Abnormality determination device, abnormality determination method, and abnormality determination program

Abstract

An action abnormality determination unit 62 classifies video data that indicates the actions of a person into action clusters and determines whether or not there is an abnormality in the actions of the person. A procedure classification unit 66 classifies the actions of the person into procedures, based on the action cluster classification results and the procedure tree. A procedure abnormality determination unit 68 determines whether or not there is an abnormality in the procedures including the actions of the person, on the basis of the procedure classification results.

Description

[Technical field]

[0001] The present disclosure relates to an abnormality determination device, an abnormality determination method, and an abnormality determination program. [Background technology]

[0002] In recent years, with the spread of high-definition cameras, there is an increasing need for technology to analyze human behavior in captured video. For example, detecting criminal behavior using surveillance cameras and dangerous behavior at construction sites. To discover these behaviors, it is necessary to observe a large amount of video. An abnormal behavior is detected by observing the behavior in the video by a person who understands the definition of abnormal behavior. However, since manual detection requires time and human costs, a method of detecting abnormal behavior automatically by constructing an algorithm is considered.

[0003] In recent years, a technology for detecting abnormal behavior using a neural network has been proposed (Non-Patent Document 1). The method in Non-Patent Document 1 detects abnormal behavior with high accuracy by clustering videos. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Zaheer MZ, Mahmood A., Astrid M., Lee SI. CLAWS: Clustering Assisted Weakly Supervised Learning with Normalcy Suppression for Anomalous Event Detection. ECCV 2020. Summary of the Invention [Problem to be solved by the invention]

[0005] In the conventional method for detecting abnormal actions shown in a video shown in Non-Patent Document 1, steps and actions are not clearly distinguished. Therefore, for example, if there are steps such as (step 1) setting up a stepladder placed on the floor, (step 2) fastening a safety belt, and (step 3) climbing the stepladder, each step has many actions, making it difficult to determine whether the steps are in the correct order. Specifically, in step 1, a person bends his / her knees to grab the stepladder, and then lifts and secures the stepladder. Similarly, the step of fastening the safety belt and climbing the stepladder in step 2 also includes a series of actions such as holding the safety belt and fastening it to the human body. Step 3 includes many actions such as walking toward the stepladder, putting a foot on the step, and climbing up the stepladder with hands. In this way, it is necessary to group actions together to some extent and check whether the steps are in the correct order, but current abnormal action detection methods are centered on detecting abnormalities in individual actions, and no consideration has been given to detecting abnormalities in a step that includes multiple actions. Therefore, it is difficult to detect dangerous actions from the video, such as fastening the safety belt after climbing the ladder. Also, since it is necessary to detect abnormalities in the actions themselves during the procedure, a method that takes into account the detection of these simultaneously is required.

[0006] The disclosed technology has been made in consideration of the above points, and aims to provide an abnormality determination device, method, and program that can accurately determine abnormalities in procedures and abnormalities in the operation itself. [Means for solving the problem]

[0007] A first aspect of the present disclosure is an anomaly judgment device including: a clustering database that stores a plurality of action clusters related to human actions based on characteristics of video data; a procedure tree database that stores a procedure tree representing relationships between a plurality of procedures including at least one action, the procedure tree storing the action cluster for each of the plurality of procedures; an action anomaly judgment unit that classifies video data representing human actions into the action clusters and judges whether the human actions are abnormal; a procedure classification unit that classifies the human actions into the procedures based on the classification results of the action clusters and the procedure tree; and a procedure anomaly judgment unit that judges whether the procedure including the human action is abnormal based on the procedure classification results.

[0008] A second aspect of the present disclosure is an anomaly detection method in an anomaly detection device including a clustering database that stores multiple action clusters related to human actions based on characteristics of video data, and a procedure tree database that stores a procedure tree representing relationships between multiple procedures including at least one action, the procedure tree storing the action cluster for each of the multiple procedures, wherein an action anomaly detection unit classifies video data representing human actions into the action clusters and determines whether the human actions are abnormal, a procedure classification unit classifies the human actions into the procedures based on the classification results of the action clusters and the procedure tree, and a procedure anomaly detection unit determines whether the procedure including the human action is abnormal based on the classification results of the procedures.

[0009] A third aspect of the present disclosure is an abnormality determination program for causing a computer to function as the abnormality determination device of the first aspect. Effect of the Invention

[0010] According to the disclosed technology, it is possible to accurately determine abnormalities in procedures and abnormalities in the operations themselves. [Brief description of the drawings]

[0011] [Figure 1]FIG. 2 is a schematic block diagram of an example of a computer that functions as a learning device and an abnormality determination device according to the first and second embodiments. [Diagram 2] FIG. 1 is a block diagram showing a configuration of a learning device according to a first embodiment and a second embodiment. [Diagram 3] 1 is a block diagram showing a configuration of an abnormality determination device according to a first embodiment and a second embodiment; [Figure 4] 5 is a flowchart showing a clustering processing routine of the learning device of the first embodiment. [Diagram 5] 5 is a flowchart showing an operation abnormality determination model learning processing routine of the learning device of the first embodiment. [Figure 6] 4 is a flowchart showing a procedure tree construction processing routine of the learning device of the first embodiment. [Figure 7] 4 is a flowchart showing an abnormality determination processing routine of the abnormality determination device of the first embodiment. [Figure 8] 5 is a flowchart showing a flow of processing by an operation abnormality determination unit of the abnormality determination device of the first embodiment. [Figure 9] 5 is a flowchart showing a process flow of a procedure classification unit of the abnormality determination device of the first embodiment. [Figure 10] 5 is a flowchart showing a flow of processing in a procedural abnormality determination unit of the abnormality determination device according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings. In each drawing, the same or equivalent components and parts are given the same reference numerals. In addition, the dimensional ratios of the drawings are exaggerated for the convenience of explanation and may differ from the actual ratios.

[0013] [First embodiment] <Outline of this embodiment> In this embodiment, the video data is divided into video segments and clustering information obtained by clustering the video features of the video segments is used as input to output an action anomaly label and an action class probability, the action class probability and procedure tree are used as input to output a procedure probability, and the procedure probability is used as input to output a procedure anomaly label.

[0014] Here, the procedure does not only mean a procedure that is manually defined, such as a procedural statement, but also includes a pseudo procedure that is not defined in advance and that is a collection of at least one action. One procedure includes at least one action.

[0015] <Configuration of the learning device according to this embodiment> FIG. 1 is a block diagram showing the hardware configuration of a learning device 10 of this embodiment.

[0016] 1, the learning device 10 has a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0017] The CPU 11 is a central processing unit, and executes various programs and controls each part. That is, the CPU 11 reads a program from the ROM 12 or the storage 14, and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various arithmetic processing according to the program stored in the ROM 12 or the storage 14. In this embodiment, a learning program is stored in the ROM 12 or the storage 14. The learning program may be one program, or may be a group of programs composed of multiple programs or modules.

[0018] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including an operating system and various data.

[0019] The input unit 15 includes a pointing device such as a mouse, and a keyboard, and is used to perform various input operations.

[0020] The input unit 15 receives learning video data as an input. Specifically, the input unit 15 receives learning video data representing at least one motion. A label indicating whether the motion itself is abnormal or normal is attached to the learning video data.

[0021] The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may function as the input unit 15 by adopting a touch panel system.

[0022] The communication interface 17 is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).

[0023] Next, a description will be given of the functional configuration of the learning device 10. FIG 2 is a block diagram showing an example of the functional configuration of the learning device 10.

[0024] Functionally, as shown in Figure 2, the learning device 10 includes a learning video database (DB) 20, a clustering unit 22, a clustering database (DB) 24, an action abnormality judgment model learning unit 26, an action class calculation unit 28, a procedure tree construction unit 30, and a procedure tree database (DB) 32.

[0025] The learning video database 20 stores a plurality of pieces of learning video data that have been input. The learning video data may be input for each video, for each divided video segment, or for each video frame. Here, a video segment is a unit obtained by dividing a video into a group of multiple frames, and for example, one segment is defined as 32 frames.

[0026] The clustering unit 22 receives as input a group of training video segments stored in the training video database 20, clusters the group of training video segments based on the features of the video data, and outputs clustering information representing a plurality of motion clusters related to human motion. The features of the video data are feature vectors extracted in advance from the video segments of the training video data. The clustering information is stored in the clustering database 24. If there are K motion clusters, the clustering information is the central vector of the feature vectors of each of the K motion clusters. Furthermore, the clustering unit 22 stores in the clustering database 24 the motion cluster to which each video segment belongs as a clustering result for the group of training video segments.

[0027] The motion abnormality judgment model learning unit 26 extracts a group of training video segments from the training video database 20, classifies video data showing human motion into motion clusters, and learns a motion abnormality judgment model for judging whether the human motion itself is abnormal or not. Here, a machine learning model such as a neural network is used as the motion abnormality judgment model. The training video segment group also holds a chronological order for each video. Each training video segment is assigned a label indicating whether it is abnormal or normal, and the motion abnormality judgment model learning unit 26 learns the motion abnormality judgment model so as to reduce the loss between the label and the clustering result for the training video segment group.

[0028] The action class calculation unit 28 receives as input the group of training video segments stored in the training video database 20 and the clustering information stored in the clustering database 24, and calculates, for each training video segment, an action class probability, which is the probability that the training video segment belongs to each of the multiple action clusters. Specifically, the action class calculation unit 28 compares, for each training video segment, the video features with the central vector of the feature vectors of the K action clusters, to calculate the action class probability, which is the probability that the training video segment belongs to each of the multiple action clusters. If there are K action clusters, the action class probability is a K-dimensional vector, and the sum of all the elements of the vector is 1.

[0029] The procedure tree construction unit 30 receives the action class probability for each training video segment and outputs a procedure tree. Here, the procedure tree is a parsed tree that represents the relationship between a plurality of procedures including at least one action, and stores action clusters for each of the plurality of procedures. Specifically, based on the action class probability for each training video segment, the procedure is divided into groups of actions represented by the training video segment to obtain the relationship between the procedures, and for each of the terminal nodes of the procedure tree that represents the obtained relationship between the procedures, the procedure tree is constructed by calculating the action class probability corresponding to the procedure represented by the terminal node.

[0030] The procedure tree database 32 stores the constructed procedure tree.

[0031] <Configuration of the abnormality determination device according to the first embodiment> FIG. 1 is a block diagram showing a hardware configuration of an abnormality determination device 50 according to the first embodiment.

[0032] As shown in FIG. 1, the abnormality determination device 50 has the same configuration as the learning device 10, and the ROM 12 or the storage 14 stores an abnormality determination program for determining abnormal operation.

[0033] The input unit 15 receives as input video data representing human movements.

[0034] Next, a description will be given of the functional configuration of the abnormality determination device 50. FIG.

[0035] Functionally, the abnormality determination device 50 includes a clustering database (DB) 60, an operation abnormality determination unit 62, a procedure tree database (DB) 64, a procedure classification unit 66, and a procedure abnormality determination unit 68, as shown in FIG.

[0036] The clustering database 60, like the clustering database 24, stores clustering information representing a plurality of action clusters relating to human actions based on the features of the video data.

[0037] The movement abnormality judgment unit 62 uses the movement abnormality judgment model learned by the learning device 10 to classify video data representing human movements into movement clusters for each time point, and judges whether the human movements are abnormal or not.

[0038] Specifically, the motion abnormality judgment unit 62 receives as input video segments obtained by dividing video data and clustering information obtained by clustering video features, and outputs a motion abnormality label and a motion class probability, which is the probability of belonging to each of a plurality of motion clusters, using a motion abnormality judgment model. The motion abnormality label indicates, by 1 or 0, whether the motion itself in the input video segment is abnormal or normal. In this embodiment, a motion abnormality label of 1 indicates that the motion itself is abnormal.

[0039] The procedure tree database 64, like the procedure tree database 32, stores procedure trees.

[0040] The procedure classification unit 66 classifies the actions represented by the video data into procedures based on the procedure tree and the classification result of the action cluster for each time. Specifically, the procedure classification unit 66 outputs a procedure probability, which is the probability that the action belongs to each of a plurality of procedures, for each time based on the action class probability and the procedure tree. For example, the procedure tree receives the action class probability up to time t as input and outputs the procedure probability at time t. Therefore, the procedure classification unit 66 holds the action class probability for each time after the input of the video data starts.

[0041] Based on the classification result of the procedure at each time, the procedure abnormality determination unit 68 determines whether or not the procedure including the action shown by the video segment is abnormal. Specifically, based on the procedure probability at each time, it outputs a procedure abnormality label indicating whether or not the procedure including the action shown by the video segment is abnormal. The procedure abnormality label takes the value 1 or 0, similar to the action abnormality label.

[0042] <Function of the learning device according to the first embodiment> Next, the operation of the learning device 10 according to the first embodiment will be described.

[0043] 4 is a flowchart showing the flow of the clustering process by the learning device 10. The CPU 11 reads out the learning program from the ROM 12 or the storage 14, expands it in the RAM 13, and executes it to perform the clustering process of constructing clusters for expressing the action class probabilities. In addition, a plurality of pieces of video data for learning are input to the learning device 10 and stored in the learning video database 20.

[0044] In step S100 , the CPU 11 functioning as the clustering unit 22 receives a group of training video segments from the training video database 20 .

[0045] In step S102, the CPU 11, functioning as the clustering unit 22, inputs each learning video segment to the motion abnormality determination model obtained by pre-learning, and obtains a feature vector.

[0046] In step S104, the CPU 11, functioning as the clustering unit 22, performs clustering to classify the group of feature vectors obtained for each learning video segment into K action clusters.

[0047] In step S106, the CPU 11 functions as the clustering unit 22 to output the center vector of the feature vector of each action cluster as clustering information, and stores it in the clustering database 24.

[0048] 5 is a flowchart showing the flow of the operation abnormality judgment model learning process by the learning device 10. The operation abnormality judgment model learning process is performed by the CPU 11 reading out a learning program from the ROM 12 or the storage 14, expanding it into the RAM 13, and executing it. The flow of the operation abnormality judgment model learning process is similar to batch learning of a general neural network.

[0049] In step S110, the CPU 11, functioning as the motion abnormality determination model learning unit 26, receives a group of learning video segments from the learning video database 20.

[0050] In step S112, the CPU 11, functioning as the motion abnormality determination model learning unit 26, samples a learning video segment batch from the learning video segment group.

[0051] In step S114, the CPU 11, functioning as the motion abnormality determination model learning unit 26, inputs each learning video segment included in the learning video segment batch to the motion abnormality determination model.

[0052] In step S116, the CPU 11, functioning as the motion abnormality detection model learning unit 26, obtains a motion abnormality score and a motion class probability for each learning video segment from the output of the motion abnormality detection model.

[0053] In step S118, the CPU 11, as the motion anomaly determination model learning unit 26, calculates a loss from the motion anomaly score and the motion class probability for each training video segment. Specifically, the CPU 11 compares the motion anomaly label and the clustering result assigned to each training video segment with the motion anomaly score and the motion class probability for each training video segment to calculate the loss.

[0054] In step S120, the CPU 11, functioning as the operation abnormality determination model learning unit 26, calculates a gradient from the obtained loss and updates the weights of the operation abnormality determination model by backpropagation.

[0055] In step S122, the CPU 11, functioning as the operation abnormality determination model learning unit 26, determines whether the loss is sufficiently small. If the loss is not sufficiently small, the CPU 11 returns to step S112. On the other hand, if the loss is sufficiently small, the CPU 11 proceeds to step S124.

[0056] In step S124, the CPU 11, functioning as the operation abnormality determination model learning unit 26, outputs the updated operation abnormality determination model, and then ends the process.

[0057] 6 is a flowchart showing the flow of a procedure tree construction process by the learning device 10. The procedure tree construction process is performed by the CPU 11 reading out a learning program from the ROM 12 or storage 14, expanding it in the RAM 13, and executing it. The procedure tree construction process is a process for constructing a procedure tree by the method described in Non-Patent Document 2.

[0058] [Non-Patent Document 2] S. Qi et al. Generalized Earley Parser: Bridging Symbolic Grammars and Sequence Data for Future Prediction. ICML2018.

[0059] Non-Patent Document 2 assumes that the class probability is obtained for each frame, but in this embodiment, a modification is applied such that the motion class probability is calculated for each segment.

[0060] In step S130, the CPU 11, functioning as the motion class calculation unit 28, receives a group of training video segments from the training video database 20.

[0061] In step S132, the CPU 11, functioning as the operation class calculation unit 28, receives the clustering information from the clustering database 24.

[0062] In step S134, the CPU 11, functioning as the motion class calculation unit 28, extracts a learning video segment from the learning video segment group.

[0063] In step S136, the CPU 11, functioning as the action class calculation unit 28, inputs the learning video segment into the action abnormality determination model to calculate the action class probability.

[0064] In step S138, the CPU 11, functioning as the motion class calculation unit 28, determines whether steps S134 and S136 have been performed for all training video segments. If there is a training video segment for which steps S134 and S136 have not been performed, the CPU 11 returns to step S134 and repeats the process for that training video segment. On the other hand, if steps S134 and S136 have been performed for all training video segments, the CPU 11 proceeds to step S140.

[0065] In step S140, the CPU 11, functioning as the motion class calculation unit 28, outputs the motion class probability of each learning video segment to the procedure tree builder 30.

[0066] In step S142, the CPU 11, functioning as the procedure tree builder 30, builds a procedure tree using the action class probabilities of each training video segment.

[0067] In step S144, the CPU 11, functioning as the procedure tree constructor 30, stores the procedure tree in the procedure tree database 32.

[0068] <Function of the Abnormality Determination Device According to the First Embodiment> Next, the operation of the abnormality determination device 50 according to the first embodiment will be described.

[0069] 7 is a flowchart showing the flow of the abnormality determination process by the abnormality determination device 50. The abnormality determination process is performed by the CPU 11 reading out an abnormality determination program from the ROM 12 or the storage 14, deploying it in the RAM 13, and executing it. Also, video data showing a person's movement is input to the abnormality determination device 50.

[0070] In step S150, the CPU 11 inputs each video segment of the video data to the operation abnormality determination section 62.

[0071] In step S152, the CPU 11 functions as the motion abnormality determination unit 62 to determine motion abnormalities for each video segment using the motion abnormality determination model, and simultaneously classifies the video segments into motion clusters.

[0072] In step S154, the CPU 11, functioning as the action abnormality judgment unit 62, outputs, via the display unit 16, the action abnormality labels for each video segment output from the action abnormality judgment model, and simultaneously outputs the action class probabilities to the procedure classification unit 66.

[0073] In step S 156 , the CPU 11 , functioning as the procedure classification unit 66 , extracts a procedure tree from the procedure tree database 64 .

[0074] In step S158 , the CPU 11 , functioning as the procedure classification unit 66 , classifies procedures for each video segment using the action class probability and the procedure tree, and outputs the procedure probability to the procedure abnormality determination unit 68 .

[0075] In step S160, the CPU 11, functioning as the procedure abnormality determination unit 68, calculates a procedure abnormality label from the procedure probability for each video segment, and ends the abnormality determination process.

[0076] Detailed operations in the processes in steps S150 to S154 above are shown in Fig. 8. The operations shown in Fig. 8 are repeatedly performed for each video segment.

[0077] In step S 170 , the CPU 11 inputs the video segment to the operation abnormality determination unit 62 .

[0078] In step S172, the CPU 11, functioning as the motion abnormality judgment unit 62, inputs the video segment to a motion abnormality judgment model. Here, the motion abnormality judgment model is a classification model such as a neural network. Specifically, a neural network configured to perform softmax output that outputs a K-dimensional vector expressing a motion class probability that is a probability of belonging to K motion clusters for classification, and sigmoid output that outputs a motion abnormality score that takes a value between 0 and 1 is used as the motion abnormality judgment model.

[0079] In step S174, the CPU 11, functioning as the motion abnormality determination unit 62, calculates a motion abnormality score and a motion class probability by using the motion abnormality determination model.

[0080] In step S176, the CPU 11, functioning as the behavior anomaly determination unit 62, determines a behavior anomaly label from the behavior anomaly score. The behavior anomaly label is obtained by determining whether the behavior anomaly score is abnormal or normal by comparing the behavior anomaly score with a specific threshold value (for example, 0.5).

[0081] In step S178, the CPU 11, functioning as the operation abnormality determination unit 62, outputs an operation abnormality label.

[0082] In step S180, the CPU 11, functioning as the action abnormality determination unit 62, outputs the action class probability to the procedure classification unit 66.

[0083] Detailed operations in the processes of steps S156 and S158 are shown in Fig. 9. The operations shown in Fig. 9 are repeatedly performed for each video segment.

[0084] In step S190, the CPU 11 inputs the action class probability to the step classification unit 66. Here, the step classification unit 66 holds the action class probability of each video segment from the start of the video (the action class probability at each time up to time t-1), and calculates the step probability using the past action class probability and the action class probability at time t. Here, assuming that there are L steps, the step probability indicates which step class of the L steps a step belongs to. Specifically, the steps and step probabilities are calculated using the method described in Non-Patent Document 2.

[0085] In step S192, the CPU 11, functioning as the procedure classification unit 66, calculates the procedure probability at time t using the action class probability and the procedure tree, and obtains a procedure class indicating which procedure the action at time t is classified into.

[0086] In step S194, the CPU 11, functioning as the procedure classification unit 66, outputs to the procedure abnormality determination unit 68 the procedure probabilities and procedure classes at time t−1 and time t, and the procedure probabilities and procedure classes at time t+1 predicted from the procedure tree.

[0087] A detailed operation in the process of step S160 above is shown in Fig. 10. The operation shown in Fig. 10 is repeatedly performed for each video segment.

[0088] In step S200, the CPU 11 inputs the procedure probability and procedure class at time t to the procedure abnormality determination unit 68.

[0089] In step S202, the CPU 11, functioning as the procedure abnormality determination unit 68, determines whether the procedure class at time t is the same as the procedure class at time t-1 and time t+1. If the procedure class at time t is the same as the procedure class at time t-1 and time t+1, the CPU 11 proceeds to step S206. On the other hand, if the procedure class at time t is not the same as the procedure class at time t-1 and time t+1, the CPU 11 proceeds to step S204. If the procedure classes at time t-1 and time t+1 are not the same, the CPU 11 proceeds to step S206.

[0090] In step S204, the CPU 11, functioning as the procedure abnormality determination unit 68, determines whether or not the procedure class differs only at time t. If the procedure classes at time t−1 and time t+1 are the same and the procedure class differs only at time t, the CPU 11 proceeds to step S208.

[0091] On the other hand, in step S206, the CPU 11, functioning as the procedure abnormality determination unit 68, determines whether the procedure probability of the procedure class at time t is equal to or less than the threshold. If the procedure probability of the procedure class at time t is equal to or less than the threshold, the CPU 11 proceeds to step S208. On the other hand, if the procedure probability of the procedure class at time t is greater than the threshold, the CPU 11 proceeds to step S210.

[0092] In step S208, the CPU 11, functioning as the procedure abnormality determination unit 68, outputs a procedure abnormality label indicating that the procedure is abnormal, and ends the process. In step S210, the CPU 11, functioning as the procedure abnormality determination unit 68, outputs a procedure abnormality label indicating that the procedure is normal, and ends the process.

[0093] As described above, the anomaly determination device according to the first embodiment classifies video data showing human actions into action clusters, determines whether the human actions are abnormal, classifies the human actions into procedures based on the action cluster classification results and a procedure tree, and determines whether the procedures including the human actions are abnormal based on the procedure classification results. This makes it possible to accurately determine anomalies in procedures and in the actions themselves.

[0094] In addition, by expressing the relationship between procedures and actions using a parse tree, it is possible to simultaneously detect procedures that represent at least one action and abnormalities in the actions. It is also possible to detect abnormal behavior in units of procedures that can be grouped together in a certain section of a video of a certain task.

[0095] [Second embodiment] Next, a second embodiment will be described. Note that parts having the same configuration as the first embodiment will be given the same reference numerals and the description will be omitted.

[0096] The second embodiment differs from the first embodiment in that the action class probability, which is a K-dimensional vector, is converted into an L-dimensional vector to classify into procedures.

[0097] <Outline of the second embodiment> Focusing on time t, the procedure tree can be changed to a converter that converts the K-dimensional vector, which is the action class probability, into an L-dimensional vector. For example, a method of converting into an L-dimensional vector using the converter in Non-Patent Document 3 can be used.

[0098] [Non-Patent Document 3] A. Vaswani et al. Attention is All you Need. NeurIPS2017.

[0099] In this case, it is necessary to prepare L procedure classes in advance so that they can be classified into the procedure classes. Specifically, a clustering method such as k-means is performed in advance so that the K-dimensional vector representing the action class probability can be classified into L procedure classes, and L central vectors are obtained as procedure classes. However, in this case, it is necessary to separately consider times t-1 and t+1. Specifically, the converter is configured to include a network that takes as input both a K-dimensional vector which is the action class probability at time t-1 and a K-dimensional vector which is the action class probability at time t, and outputs an L-dimensional vector at time t, and a network that takes as input the L-dimensional vector at time t-1 and the L-dimensional vector at time t, and outputs an L-dimensional vector at time t+1.

[0100] The procedure tree of the first embodiment can input the action class probability from time 0 to time t, but the converter of the second embodiment inputs the action class probability only for the surrounding times. Therefore, as in Non-Patent Document 4, it is also possible to input longer-term context information to the converter using LSTM (Long Short-Term Memory) or the like.

[0101] [Non-Patent Document 4] S. Hochreiter and J. Schmidhuber. Long Short-Term Memory. Neural Computation, volume 9, 1997.

[0102] <Configuration of the learning device according to the second embodiment> The learning device of the second embodiment is similar to the learning device 10 of the first embodiment, so the same reference numerals are used and the description will be omitted.

[0103] The procedure tree construction unit 30 of the learning device 10 constructs, as a procedure tree, a converter that converts the action class probability, which is a K-dimensional vector, into an L-dimensional vector, based on the action class probability for each learning video segment.

[0104] Specifically, based on the action class probability for each training video segment, a clustering technique is used to determine the central vectors of L procedure classes, and a converter that converts the action class probability, which is a K-dimensional vector, into an L-dimensional vector is constructed as a procedure tree.

[0105] <Configuration of the abnormality determination device according to the second embodiment> The abnormality determination device of the second embodiment is similar to the abnormality determination device 50 of the above-described first embodiment, and therefore the same reference numerals are used and the description thereof will be omitted.

[0106] The procedure classification unit 66 of the abnormality determination device 50 uses a converter that is a procedure tree constructed by the learning device 10 to convert the action class probability, which is a K-dimensional vector, into a procedure probability, which is an L-dimensional vector.

[0107] Other configurations and operations of the learning device 10 and the abnormality determination device 50 according to the second embodiment are similar to those of the first embodiment, and therefore will not be described.

[0108] <Modification> The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the spirit and scope of the present invention.

[0109] For example, although an example has been described in which the learning device and the abnormality determination device are configured as separate devices, the present invention is not limited to this, and the learning device and the abnormality determination device may be configured as a single device.

[0110] In addition, various processes executed by the CPU by reading the software (program) in each of the above embodiments may be executed by various processors other than the CPU. Examples of the processor in this case include a dedicated electric circuit that is a processor having a circuit configuration designed exclusively for executing a specific process, such as a GPU (Graphics Processing Unit), a PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacture, such as an FPGA (Field-Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit). The learning process and the abnormality determination process may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, a plurality of FPGAs, and a combination of a CPU and an FPGA). The hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0111] In addition, in each of the above embodiments, the learning program and the abnormality determination program are described as being stored (installed) in advance in the storage 14, but the present invention is not limited to this. The programs may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The programs may also be downloaded from an external device via a network.

[0112] The following supplementary notes are further disclosed regarding the above embodiment.

[0113] (Additional note 1) A clustering database that stores a plurality of action clusters related to human actions based on features of video data; a procedure tree representing a relationship between a plurality of procedures, each of the procedures including at least one operation; and a procedure tree database storing the procedure tree in which the operation cluster for each of the plurality of procedures is stored, Memory, at least one processor coupled to the memory; Including, The processor, classifying video data representing human movements into the movement clusters and determining whether the human movements are abnormal; classifying the human actions into procedures based on the classification results of the action clusters and the procedure tree; Based on the classification result of the procedure, it is determined whether the procedure including the human motion is abnormal. The abnormality determination device is configured as follows.

[0114] (Additional note 2) A clustering database that stores a plurality of action clusters related to human actions based on features of video data; A non-transitory storage medium storing a program executable by a computer to execute an anomaly determination process, the non-transitory storage medium including: a procedure tree representing a relationship between a plurality of procedures, each of which includes at least one operation; and a procedure tree database storing a procedure tree in which the operation cluster for each of the plurality of procedures is stored, the non-transitory storage medium storing a program executable by a computer to execute an anomaly determination process, the non-transitory storage medium including: The abnormality determination process includes: classifying video data representing human movements into the movement clusters and determining whether the human movements are abnormal; classifying the human actions into procedures based on the classification results of the action clusters and the procedure tree; Based on the classification result of the procedure, it is determined whether the procedure including the human motion is abnormal. Non-transitory storage medium. [Explanation of symbols]

[0115] 10 Learning Device 11 CPU 14. Storage 15 Input section 16 Display 20 Learning Video Database 22 Clustering Department 24, 60 Clustering Database 26. Operational anomaly detection model learning unit 28 Operation Class Calculation Unit 30 Procedure Tree Construction 32, 64 Step Tree Database 50 Abnormality determination device 62 Operation abnormality determination section 66 Procedure Classification Section 68 Procedure Abnormality Judgment Unit

Claims

1. A clustering database that stores a plurality of action clusters related to human actions based on features of video data; a procedure tree database that stores a procedure tree representing a relationship between a plurality of procedures, each of which includes at least one operation, and that stores the operation cluster for each of the plurality of procedures; a motion abnormality determination unit that classifies video data representing human motion into the motion clusters and determines whether the human motion is abnormal; a procedure classification unit that classifies the human actions into the procedures based on the classification result of the action clusters and the procedure tree; a procedure abnormality determination unit that determines whether the procedure including the human motion is abnormal based on a result of the procedure classification; An abnormality determination device comprising:

2. the motion abnormality determination unit classifies the video data into the motion clusters for each time and determines whether the motion of the person is abnormal; The procedure classification unit classifies the human actions into the procedures for each time point; 2. The abnormality determination device according to claim 1, wherein the procedure abnormality determination unit determines whether or not the procedure is abnormal based on a classification result of the procedure at each time.

3. the action abnormality determination unit classifies the action into the action clusters by calculating an action class probability, which is a probability that the action belongs to each of the plurality of action clusters; 3. The abnormality determination device according to claim 1, wherein the procedure tree stores the action class probability corresponding to each of the plurality of procedures.

4. the plurality of action clusters being K action clusters; The plurality of procedures is L procedures, The procedure tree is a converter that converts the action class probability, which is a K-dimensional vector, into an L-dimensional vector; 4. The abnormality determination device according to claim 3, wherein the procedure classification unit classifies the action class probability, which is a K-dimensional vector, into an L-dimensional vector using the procedure tree, thereby classifying the action class probability into the procedures.

5. A clustering database that stores a plurality of action clusters related to human actions based on features of video data; a procedure tree database that stores a procedure tree representing a relationship between a plurality of procedures, each of which includes at least one operation, and that stores the operation cluster for each of the plurality of procedures; An abnormality determination method for an abnormality determination device comprising: a motion abnormality determination unit classifies video data representing human motions into the motion clusters and determines whether the human motions are abnormal; a procedure classification unit classifies the human actions into the procedures based on the classification result of the action cluster and the procedure tree; A procedure abnormality determination unit determines whether the procedure including the human motion is abnormal based on the procedure classification result. Abnormality determination method.

6. An abnormality determination program for causing a computer to function as the abnormality determination device according to any one of claims 1 to 4.

Images