Method for training and method for testing deep learning-based behavior detection model that detects behavior of person through video analysis, and training device and testing device using same

Abstract

The present invention relates to a method for training a deep learning-based behavior detection model that detects the behavior of a person through video analysis, comprising steps in which a training device: (a) generates at least one first bounding box for training to at least one t-th bounding box for training, and at least one first skeleton keypoint for training to at least one t-th skeleton keypoint for training; (b) performs (i) a process of detecting a first discrete action for training to a t-th discrete action for training and (ii) a process of detecting a continuous action for training; and (c) generates at least one first loss with reference to each of the first discrete action for training to the t-th discrete action for training and each discrete action ground truth corresponding thereto, generates at least one second loss with reference to the continuous action for training and a continuous action ground truth corresponding thereto, and trains a discrete action detection network and a continuous action detection network by using the first loss and the second loss.

Description

A method for learning and testing a deep learning-based action detection model that detects human actions through video analysis, and a learning device and a testing device using the same.

[0001] The present invention relates to a method for learning and testing a deep learning-based thawing detection model that detects human behavior through video analysis, and a learning device and a testing device using the same.

[0002] Deep learning techniques have shown significant improvements in a variety of applications, including computer vision and speech recognition.

[0003] In particular, in computer vision, a specific type of deep learning model known as Convolutional Neural Networks (CNNs) has shown very good results in the field of object recognition, and CNNs have shown reliable results for object recognition and detection that are useful in real-world applications.

[0004] Recently, deep learning models are being used to recognize human behavior by analyzing video captured by cameras in various fields such as video surveillance systems, sports analysis, driving assistance systems, smart homes, and senior care systems.

[0005] In a deep learning model for human action recognition using video, spatial and temporal features are extracted from image frames of the video, and then the spatial and temporal features are implied and the action category is predicted using the implied information.

[0006] Action recognition technology that utilizes these deep learning models includes a method that combines CNN and LSTM, as in the paper 'Long-term Recurrent Convolutional Networks for visual Recognition and Description' presented at CVPR 2015; a method that processes time-varying data by increasing the dimension by one from a 2D CNN for image processing, as in the paper 'Learning Spatiotemporal Features with 3D Convolutional Networks' presented at the ICCV2015 conference; a method that processes RGB images through a Spatial stream CNN block, converts optical flow into a 3-channel image by collecting three gray images, and processes them with a Temporal stream CNN block, and then collects the processed results to output the prediction result; and a method that combines 3D CNN and 3D Convolutional Network, as in the paper 'Convolutional Two-Stream Network Fusion for Video Action Recognition' presented at the CVPR2016 conference. There are methods for predicting by processing data as a combined distribution of spatiotemporal information through pooling.

[0007] However, conventional action recognition technologies such as this one have the problem of being difficult to apply to real-time applications that perform action recognition in real time, as they not only use video data as input but also require the application of a 3D network.

[0008] Additionally, it is difficult to secure training data, which is video data related to human behavior, and accordingly, it is difficult to train a deep learning model for action recognition.

[0009] Therefore, the present applicant proposes a deep learning model that can detect human behavior, which is applicable to real-time applications and can be easily learned.

[0010] The purpose of the present invention is to solve all of the problems of the above-mentioned prior art.

[0011] In addition, another object of the present invention is to provide an action detection model applicable to real-time applications.

[0012] In addition, another object of the present invention is to provide a learnable action detection model using sufficient learning data.

[0013] In addition, another object of the invention is to provide an action detection model that detects human actions in a video using image processing technology.

[0014] A representative configuration of the present invention to achieve the above purpose is as follows.

[0015] According to one embodiment of the present invention, there is provided a method for learning a deep learning-based action detection model for detecting human action through video analysis, the method comprising: (a) when a first image frame for training to a t-th image frame for training corresponding to a video captured by a camera, wherein t is an integer greater than or equal to 2, is acquired, a learning device inputs each of the first image frame for training to the t-th image frame for training into a keypoint and bounding box extraction network, such that the keypoint and bounding box extraction network generates at least one first bounding box for training to at least one t-th bounding box for training corresponding to at least one person located in each of the first image frame for training to the t-th image frame for training, and at least one first skeleton keypoint for training to at least one t-th skeleton keypoint for training;(b) a process in which the learning device (i) inputs the first bounding box for training and the first skeleton keypoints for training or the t-bounding box for training and the t-skeleton keypoints for training into a discrete action detection network, and causes the discrete action detection network to perform a learning operation on each of the first bounding box for training and the first skeleton keypoints for training or the t-bounding box for training and the t-skeleton keypoints for training, to detect the first discrete action for training corresponding to the person located in the first image frame for training or the t-discrete action for training corresponding to the person located in the t-image frame for training, and (ii) inputs the first bounding box for training or the t-bounding box for training into a continuous action detection network, and causes the continuous action detection network to detect the first bounding box for training or the t-bounding box for training in the first image frame for training, A step of generating a learning t-crop image by cropping an area corresponding to the learning t-bounding box from a first learning crop image or a learning t-crop image frame, and performing a learning first action estimation value by detecting an individual action of the person in the learning 1st learning crop image or a learning t-action estimation value by detecting an individual action of the person in the learning t-crop image by performing a learning operation on the first learning crop image or the learning t-crop image through at least one classifier, and performing a process of detecting a learning continuous action corresponding to the person located in the learning 1st learning image frame or the learning t-crop image frame by referring to the learning 1st action estimation value or the learning t-action estimation value;And (c) a step of the learning device generating at least one first loss by referring to each of the first discrete action for learning to the t discrete action for learning and the corresponding discrete action ground truth, generating at least one second loss by referring to the continuous action for learning and the corresponding continuous action ground truth, and training the discrete action detection network and the continuous action detection network using the first loss and the second loss is provided.;

[0016] In the above embodiment, in the step (b), the learning device may cause the discrete action detection network to detect a specific discrete action by a heuristic analysis using at least one of an aspect ratio of the specific bounding box for learning and vertical distances between skeleton keypoints in the specific bounding box for learning and the specific skeleton keypoints for learning, which is one of the first bounding box for learning and the first skeleton keypoints for learning, or the t bounding box for learning and the t skeleton keypoints for learning.

[0017] In the above embodiment, in the step (b), the learning device may cause the discrete action detection network, in the process (i), to detect, by reference to the positions of each of the first bounding box for learning to the t bounding box for learning, a first sub-discrete action for learning of the person corresponding to the same discrete action for learning when the number of times the same discrete action for learning is detected among the first discrete action for learning to the t discrete action for learning is greater than or equal to a preset threshold, and to detect, by reference to the positions of each of the first bounding box for learning to the t bounding box for learning, a first sub-discrete action for learning of the person corresponding to the same discrete action for learning when the position change is less than or equal to the preset threshold position change, a second sub-discrete action for learning corresponding to the same discrete action for learning or the first sub-discrete action for learning when the position change is greater than or equal to the preset threshold position change.

[0018] In the above embodiment, in the step (b), the learning device can cause the continuous action detection network to perform a learning operation on each of the first crop image for learning to the t-crop image for learning through each of the first classifier to the t-crop image for learning, respectively, in the process (ii).

[0019] In the above embodiment, in the step (c), the learning device can further train the keypoint and bounding box extraction network using the 3_1 loss generated by referring to the first bounding box for learning to the t-th bounding box for learning and the respective bounding box ground truths corresponding thereto, and the 3_2 loss generated by referring to the first skeleton keypoints for learning to the t-th skeleton keypoints for learning and the respective skeleton keypoint ground truths corresponding thereto.

[0020] In the above embodiment, in the step (a), the learning device causes the keypoint and bounding box extraction network to perform convolution operations on a specific learning image frame among the first learning image frame to the t-th learning image frame through a feature extraction block to generate learning feature maps having different scales, and performs top-down feature fusion using a top-down pass and bottom-up feature fusion using a bottom-up pass on at least a part of the learning feature maps, the first learning feature map to the k-th learning feature map, wherein k is an integer greater than or equal to 2, through a feature fusion block to generate the first learning fusion feature map to the k-th learning fusion feature map, and detects, through a detection block, a first learning bounding box and a first learning feature map corresponding to the person in each of the first learning fusion feature map to the k-th learning fusion feature map. Skeleton keypoints or a k-th bounding box for learning and a k-th skeleton keypoints for learning may be detected, and a specific bounding box for learning and specific skeleton keypoints for learning corresponding to the person located in the specific image frame for learning may be generated using the first bounding box for learning and the first skeleton keypoints for learning or the k-th bounding box for learning and the k-th skeleton keypoints for learning.

[0021] In the above embodiment, in the step (a), the learning device may cause the keypoint and bounding box extraction network to perform a convolution operation on the learning-specific image frame at least once through a feature extraction block for any one of the first learning image frame to the t-th learning image frame to generate a learning feature map, perform a learning operation on the feature map through an object detector to generate a learning-specific bounding box that detects the person in the learning-specific image frame, and perform a learning operation on the feature map through a pose estimator to generate learning-specific skeleton keypoints corresponding to the person in the learning-specific image frame.

[0022] According to another embodiment of the present invention, a method for testing a deep learning-based action detection model for detecting human actions through video analysis comprises: (a) a process of, by a learning device, (i) acquiring a first image frame for training to a t-th image frame for training corresponding to a video captured by a camera, wherein t is an integer greater than or equal to 2, inputting each of the first image frame for training to the t-th image frame for training into a keypoint and bounding box extraction network, such that the keypoint and bounding box extraction network generates at least one first bounding box for training to at least one t-th bounding box for training corresponding to at least one person located in each of the first image frame for training to the t-th image frame for training, and at least one first skeleton keypoint for training to at least one t-th skeleton keypoint for training, (ii) (ii-1) generating the first bounding box for training and the first skeleton keypoint for training to the t-th bounding box for training and the t-th skeleton keypoint for training. A process of inputting keypoints into a discrete action detection network and causing the discrete action detection network to perform a learning operation on each of the first bounding box for training and the first skeleton keypoints for training or the t-th bounding box for training and the t-th skeleton keypoints for training, thereby detecting the first discrete action for training corresponding to the person located in the first image frame for training or the t-th discrete action for training corresponding to the person located in the t-th image frame for training, and (ii-2) inputting the first bounding box for training or the t-th bounding box for training into a continuous action detection network and causing the continuous action detection network toA process for generating a first cropped image for training by cropping an area corresponding to the first bounding box for training in the first image frame for training or a t-cropped image for training by cropping an area corresponding to the t-bounding box for training in the t-image frame for training, and performing a learning operation on the first cropped image for training or the t-cropped image for training through at least one classifier to generate a first action estimate value for training by detecting an individual action of the person in the first cropped image for training or a t-action estimate value for training by detecting an individual action of the person in the t-cropped image for training, and performing a process for detecting a continuous action for training corresponding to the person located in the first image frame for training or the t-image frame for training by referring to the first action estimate value for training or the t-action estimate value for training, and (iii) at least one discrete action for training or the t-discrete action for training and a corresponding discrete action ground truth for training, respectively. In a state where a process of generating a first loss, generating at least one second loss by referring to the learning continuous action and the corresponding continuous action ground truth, and training the discrete action detection network and the continuous action detection network using the first loss and the second loss is performed, when a first image frame for testing or a t-th image frame for testing corresponding to a video captured by a camera is acquired, the test device,A step of inputting each of the first test image frame to the t test image frame into the keypoint and bounding box extraction network, and causing the keypoint and bounding box extraction network to generate at least one first test bounding box to at least one t test bounding box corresponding to at least one person located in each of the first test image frame to the t test image frame, and at least one first test skeleton keypoint to at least one t test skeleton keypoint; And (b) the test device, (i) a process of inputting the first bounding box for testing and the first skeleton keypoints for testing or the t-bounding box for testing and the t-skeleton keypoints for testing into the discrete action detection network, and causing the discrete action detection network to perform a learning operation on each of the first bounding box for testing and the first skeleton keypoints for testing or the t-bounding box for testing and the t-skeleton keypoints for testing, to detect the first discrete action for testing or the t-discrete action for testing corresponding to the person located in the t-image frame for testing, and (ii) inputting the first bounding box for testing or the t-bounding box for testing into the continuous action detection network, and causing the continuous action detection network to detect a test image frame in which an area corresponding to the first bounding box for testing is cropped in the first image frame for testing. Generate a test t crop image by cropping an area corresponding to the test t bounding box from the first crop image or the test t image frame,A method is provided, comprising: a step of generating a test first action estimate value that detects an individual action of the person in the test first crop image through a running operation on the test first crop image to the test t-th crop image, and a test t-th action estimate value that detects an individual action of the person in the test t-th crop image; and performing a process of detecting a test continuous action corresponding to the person located in the test first image frame to the test t-th image frame by referring to the test first action estimate value to the test t-th action estimate value;

[0023] In another embodiment, in the step (b), the test device may cause the discrete action detection network to detect a specific discrete action by a heuristic analysis using at least one of an aspect ratio of the specific test bounding box and vertical distances between skeleton keypoints in the specific test skeleton keypoints, wherein the specific test bounding box and the specific test skeleton keypoints are either one of the first test bounding box and the first test skeleton keypoints or the t test bounding box and the t test skeleton keypoints.

[0024] In the above other embodiment, in the step (b), the test device may cause the discrete action detection network, in the process (i), to detect, by reference to the positions of each of the test first bounding box to the test t bounding box, a first sub-discrete action of the person corresponding to the same test discrete action when the number of times the same test discrete action is detected among the first discrete action for testing to the tth discrete action for testing is greater than or equal to a preset threshold number of times, and to detect, when the position change is less than or equal to the preset threshold position change, a second sub-discrete action for testing corresponding to the same test discrete action or the first sub-discrete action for testing when the position change is greater than or equal to the preset threshold position change.

[0025] According to another embodiment of the present invention, a learning device for learning a deep learning-based action detection model for detecting human behavior through video analysis comprises: a memory storing instructions for learning a deep learning-based action detection model for detecting human behavior through video analysis; and a processor for performing an operation for learning the deep learning-based action detection model for detecting human behavior through video analysis according to the instructions stored in the memory; , wherein the processor comprises: (I) a process for inputting each of the training first image frame to the training t-th image frame, wherein t is an integer greater than or equal to 2, into a keypoint and bounding box extraction network to cause the keypoint and bounding box extraction network to generate at least one training first bounding box to at least one training t-th bounding box corresponding to at least one person located in each of the training first image frame to the training t-th image frame, and at least one training first skeleton keypoint to at least one training t-th skeleton keypoint, (II) (i) a process for inputting each of the training first bounding box and the training first skeleton keypoints to the training t-th bounding box and the training t-th skeleton keypoints into a discrete action detection network to cause the discrete action detection network to extract the training first bounding box and the training t-th skeleton keypoints. A process of detecting a first discrete action for training corresponding to the person located in the first image frame for training or a t discrete action for training corresponding to the person located in the t image frame for training by performing a learning operation on the first bounding box and the first skeleton keypoints for training or the t bounding box for training and the t skeleton keypoints for training, respectively.And (ii) inputting the first bounding box for training to the t-th bounding box for training into a continuous action detection network, and causing the continuous action detection network to generate a first cropped image for training by cropping an area corresponding to the first bounding box for training from the first image frame for training to a t-th cropped image for training by cropping an area corresponding to the t-th bounding box for training from the t-th image frame for training, and performing a learning operation on the first cropped image for training to the t-th cropped image for training through at least one classifier to generate a first action estimation value for training by detecting an individual action of the person in the first cropped image for training to a t-th action estimation value for training by detecting an individual action of the person in the t-th cropped image for training, and referring to the first action estimation value for training to the t-th action estimation value for training, the continuous action corresponding to the person located in the first image frame for training to the t-th image frame for training A process for performing a process of detecting an action, and (III) a learning device for generating at least one first loss by referring to each of the first discrete action for learning to the tth discrete action for learning and the respective discrete action ground truths corresponding thereto, generating at least one second loss by referring to the continuous action for learning and the continuous action ground truths corresponding thereto, and performing a process of training the discrete action detection network and the continuous action detection network using the first loss and the second loss is provided.

[0026] In another embodiment, the processor may cause the discrete action detection network, in the process (i) of the process (II), to detect a specific discrete action by a heuristic analysis using at least one of an aspect ratio of the specific bounding box for training and vertical distances between skeleton keypoints in the specific bounding box for training and specific skeleton keypoints for training, wherein the specific bounding box for training and specific skeleton keypoints for training are any one of the first bounding box for training and the first skeleton keypoints for training or the t bounding box for training and the t skeleton keypoints for training.

[0027] In another embodiment, the processor may cause the discrete action detection network, in the process (i) of the process (II), to detect a first sub-discrete action of the person corresponding to the same learning discrete action by referring to the positions of each of the first bounding box for learning to the t bounding box for learning, if the number of times the same learning discrete action is detected among the first discrete action for learning to the t discrete action for learning is greater than or equal to a preset threshold number of times, and to detect a second sub-discrete action for learning corresponding to the same learning discrete action or the first sub-discrete action for learning if the position change is less than or equal to the preset threshold position change.

[0028] In another embodiment, the processor may cause the continuous action detection network, in the process (ii) of the process (II), to perform a learning operation on each of the first crop image for training to the t-crop image for training, through each of the first classifier to the t-classifier corresponding to each of the first crop image for training to the t-crop image for training.

[0029] In another embodiment, the processor may further train the keypoint and bounding box extraction network by using, in the process (III), the 3_1 loss generated by referring to the first bounding box for training to the t-th bounding box for training and the respective bounding box ground truths corresponding thereto, and the 3_2 loss generated by referring to the first skeleton keypoints for training to the t-th skeleton keypoints for training and the respective skeleton keypoint ground truths corresponding thereto.

[0030] In another embodiment, the processor, in the process (I), causes the keypoint and bounding box extraction network to perform convolution operations on a specific learning image frame among the first image frame for training to the t-th image frame for training through a feature extraction block to generate learning feature maps having different scales, and performs top-down feature fusion using a top-down pass and bottom-up feature fusion using a bottom-up pass on at least a part of the first feature map for training to the k-th feature map for training, wherein k is an integer greater than or equal to 2, among the learning feature maps through a feature fusion block to generate the first fusion feature map for training to the k-th fusion feature map for training, and detects, through a detection block, a first bounding box for training corresponding to the person and a learning feature map for training corresponding to the person in each of the first fusion feature map for training to the k-th fusion feature map for training. The first skeleton keypoints or the k-th bounding box for learning and the k-th skeleton keypoints for learning may be detected, and the first bounding box for learning and the first skeleton keypoints for learning or the k-th bounding box for learning and the k-th skeleton keypoints for learning may be used to generate the specific bounding box for learning and the specific skeleton keypoints for learning corresponding to the person located in the specific image frame for learning.

[0031] In another embodiment, the processor may cause the keypoint and bounding box extraction network, in the process (I), to perform a convolution operation on the training-specific image frame at least once through a feature extraction block for any one of the training-specific first image frame to the training-specific t-th image frame to generate a training feature map, perform a learning operation on the feature map through an object detector to generate a training-specific bounding box that detects the person in the training-specific image frame, and perform a learning operation on the feature map through a pose estimator to generate training-specific skeleton keypoints corresponding to the person in the training-specific image frame.

[0032] According to another embodiment of the present invention, a test device for testing a deep learning-based action detection model for detecting human action through video analysis comprises: a memory storing instructions for testing the deep learning-based action detection model for detecting human action through video analysis; and a processor for performing an operation for testing the deep learning-based action detection model for detecting human action through video analysis according to the instructions stored in the memory; , wherein the processor comprises: (I) a process for inputting each of the training first image frame to the training t-th image frame corresponding to a video captured by a camera, when the training first image frame to the training t-th image frame, wherein t is an integer greater than or equal to 2, into a keypoint and bounding box extraction network, and causing the keypoint and bounding box extraction network to generate at least one training first bounding box to at least one training t-th bounding box corresponding to at least one person located in each of the training first image frame to the training t-th image frame, and at least one training first skeleton keypoint to at least one training t-th skeleton keypoint;(ii) (ii-1) a process of inputting the first bounding box for training and the first skeleton keypoints for training or the t-bounding box for training and the t-skeleton keypoints for training into a discrete action detection network, and causing the discrete action detection network to perform a learning operation on each of the first bounding box for training and the first skeleton keypoints for training or the t-bounding box for training and the t-skeleton keypoints for training, to detect the first discrete action for training corresponding to the person located in the first image frame for training or the t-discrete action for training corresponding to the person located in the t-image frame for training, and (ii-2) a process of inputting the first bounding box for training or the t-bounding box for training into a continuous action detection network, and causing the continuous action detection network to detect the first discrete action for training corresponding to the person located in the first image frame for training, A process for generating a training t-crop image by cropping an area corresponding to the training t-bounding box from a training first crop image or a training t-crop image frame, and performing a learning operation on the training first crop image or the training t-crop image through at least one classifier to generate a training first action estimation value that detects an individual action of the person in the training first crop image or a training t-action estimation value that detects an individual action of the person in the training t-crop image, and performing a process of detecting a training continuous action corresponding to the person located in the training first image frame or the training t-image frame by referring to the training first action estimation value or the training t-action estimation value, and (iii) each of the training first discrete action or the training t-discrete action,A process of generating at least one first loss by referring to each corresponding discrete action ground truth, generating at least one second loss by referring to the learning continuous action and the corresponding continuous action ground truth, and training the discrete action detection network and the continuous action detection network using the first loss and the second loss is performed, and when a test first image frame to a test t-th image frame corresponding to a video captured by a camera is acquired, a process of inputting each of the test first image frame to the test t-th image frame to the keypoint and bounding box extraction network so that the keypoint and bounding box extraction network generates at least one test first bounding box to at least one test t-th bounding box corresponding to at least one person located in each of the test first image frame to the test t-th image frame, and at least one test first skeleton keypoint to at least one test t-th skeleton keypoint, and (II) (i) A process of inputting the test first bounding box and the test first skeleton keypoints or the test t bounding box and the test t skeleton keypoints into the discrete action detection network, and causing the discrete action detection network to perform a learning operation on each of the test first bounding box and the test first skeleton keypoints or the test t bounding box and the test t skeleton keypoints, to detect the test first discrete action corresponding to the person located in the test first image frame or the test t discrete action corresponding to the person located in the test t image frame.And (ii) inputting the first bounding box for test to the t-test bounding box for test into the continuous action detection network, and causing the continuous action detection network to generate a first cropped image for test by cropping an area corresponding to the first bounding box for test in the first image frame for test, or a t-cropped image for test by cropping an area corresponding to the t-test bounding box in the t-test image frame for test, and performing a running operation on the first cropped image for test to the t-test cropped image for test, respectively, through at least one classifier, to generate a first action estimate value for test by detecting an individual action of the person in the first cropped image for test, or a t-action estimate value for test by detecting an individual action of the person in the t-test cropped image for test, and referring to the first action estimate value for test to the t-test action estimate value for test, detecting a continuous action for test corresponding to the person located in the first image frame for test to the t-test image frame for test. A test device is provided that performs a process of performing a detection process.

[0033] In another embodiment, the processor may cause the discrete action detection network, in the process (i) of the process (II), to detect a specific discrete action by a heuristic analysis using at least one of an aspect ratio of the specific test bounding box and vertical distances between skeleton keypoints in the specific test skeleton keypoints, and a test specific bounding box and test specific skeleton keypoints, wherein the specific test bounding box and test specific skeleton keypoints are any one of the test first bounding box and the test first skeleton keypoints, or the test t bounding box and the test t skeleton keypoints.

[0034] In another embodiment, the processor may cause the discrete action detection network, in the process (i) of the process (II), to detect a first sub-discrete action of the person corresponding to the same test discrete action by referring to the positions of each of the first bounding box for testing to the t bounding box for testing, if the number of times the same test discrete action is detected among the first discrete action for testing to the tth discrete action for testing is greater than or equal to a preset threshold, and to detect a second sub-discrete action for testing corresponding to the same test discrete action or the first sub-discrete action for testing if the position change is less than or equal to the preset threshold position change.

[0035] In addition, a computer-readable recording medium for recording a computer program for executing the method of the present invention is further provided.

[0036] According to the present invention, it is possible to provide an action detection model applicable to real-time applications.

[0037] In addition, according to the present invention, it is possible to provide a learnable action detection model using sufficient learning data.

[0038] In addition, according to the present invention, it is possible to provide an action detection model that detects human actions in a video using image processing technology.

[0039] The drawings attached below for use in explaining embodiments of the present invention are only some of the embodiments of the present invention, and a person having ordinary knowledge in the technical field to which the present invention pertains (hereinafter “ordinary skilled in the art”) can obtain other drawings based on these drawings without performing an inventive work.

[0040] FIG. 1 schematically illustrates a learning device that learns a deep learning-based action detection model that detects human actions through video analysis according to one embodiment of the present invention.

[0041] FIG. 2 schematically illustrates a method for learning a deep learning-based action detection model for detecting human actions through video analysis according to one embodiment of the present invention.

[0042] FIG. 3 schematically illustrates one operation process of a keypoint and bounding box extraction network in a method for learning a deep learning-based action detection model for detecting human actions through video analysis according to one embodiment of the present invention.

[0043] FIG. 4 schematically illustrates another operation process of a keypoint and bounding box extraction network in a method for learning a deep learning-based action detection model for detecting human actions through video analysis according to one embodiment of the present invention.

[0044] FIG. 5a and FIG. 5b illustrate a state of detecting a discrete action in a method for learning a deep learning-based action detection model for detecting human actions through video analysis according to one embodiment of the present invention.

[0045] FIG. 6 schematically illustrates the operation process of a continuous action detection network in a method for learning a deep learning-based action detection model for detecting human actions through video analysis according to one embodiment of the present invention.

[0046] FIG. 7 schematically illustrates a test device for testing a deep learning-based action detection model that detects human actions through video analysis according to another embodiment of the present invention.

[0047] FIG. 8 schematically illustrates a method for testing a deep learning-based action detection model for detecting human actions through video analysis according to another embodiment of the present invention.

[0048] The following detailed description of the present invention refers to the accompanying drawings, which illustrate specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention, while different from each other, are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be modified and implemented from one embodiment to another without departing from the spirit and scope of the present invention. Furthermore, it should be understood that the positions or arrangements of individual components within each embodiment may be modified without departing from the spirit and scope of the present invention. Accordingly, the following detailed description is not intended to be limiting, and the scope of the present invention is to include the scope of the claims and all equivalents thereof. Like reference numerals in the drawings represent the same or similar elements throughout the several aspects.

[0049] Hereinafter, various preferred embodiments of the present invention will be described in detail with reference to the attached drawings so that a person having ordinary skill in the art to which the present invention pertains can easily practice the present invention.

[0050] FIG. 1 schematically illustrates a learning device for learning a deep learning-based action detection model for detecting human behavior through video analysis according to one embodiment of the present invention. Referring to FIG. 1, a learning device (100) may include a memory (101) in which instructions for learning a deep learning-based action detection model for detecting human behavior through video analysis are stored, and a processor (102) for performing an operation for learning a deep learning-based action detection model for detecting human behavior through video analysis according to the instructions stored in the memory (101).

[0051] Specifically, the learning device (100) may typically, but is not limited to, utilize a combination of computing devices (e.g., devices that may include computer processors, memory, storage, input devices and output devices, and other components of conventional computing devices; electronic communication devices such as routers, switches, etc.; electronic information storage systems such as network attached storage (NAS) and storage area networks (SAN)) and computer software (i.e., instructions that cause the computing device to function in a particular manner) to achieve desired system performance.

[0052] Additionally, the processor of the computing device may include hardware components such as a Micro Processing Unit (MPU) or a Central Processing Unit (CPU), cache memory, and a data bus. Furthermore, the computing device may further include software components such as an operating system and applications that perform specific purposes.

[0053] However, this does not exclude the case where the computing device includes an integrated processor in which the medium, processor, and memory for implementing the present invention are integrated.

[0054] Meanwhile, the processor (102) of the learning device (100) may, according to instructions stored in the memory (101), when a first image frame for learning to a t-th image frame for learning corresponding to a video captured by a camera is acquired, input each of the first image frame for learning to a keypoint and bounding box extraction network to cause the keypoint and bounding box extraction network to generate at least one first bounding box for learning to a t-th bounding box for learning corresponding to at least one person located in each of the first image frame for learning to a t-th image frame for learning, and at least one first skeleton keypoint for learning to a t-th skeleton keypoint for learning.And, the processor (102) of the learning device (100) is configured to (i) input the first bounding box for training and the first skeleton keypoints for training or the t-bounding box for training and the t-skeleton keypoints for training into a discrete action detection network, and cause the discrete action detection network to perform a learning operation on each of the first bounding box for training and the first skeleton keypoints for training or the t-bounding box for training and the t-skeleton keypoints for training, to detect the first discrete action for training corresponding to the person located in the first image frame for training or the t-discrete action for training corresponding to the person located in the t-image frame for training, and (ii) input the first bounding box for training or the t-bounding box for training into a continuous action detection network, and cause the continuous action detection network to detect the first discrete action for training or the t-discrete action for training corresponding to the person located in the first image frame for training, A process may be performed to generate a first cropped image for training by cropping an area corresponding to a first bounding box for training or a t-cropped image for training by cropping an area corresponding to a t-bounding box for training in the t-image frame for training, and to perform a learning operation on the first cropped image for training or the t-cropped image for training through at least one classifier to generate a first action estimate value for training by detecting an individual action of the person in the first cropped image for training or a t-action estimate value for training by detecting an individual action of the person in the t-cropped image for training, and to perform a process of detecting a continuous action for training corresponding to the person located in the first image frame for training or the t-image frame for training by referring to the first action estimate value for training or the t-action estimate value for training.Thereafter, the processor (102) of the learning device (100) may generate at least one first loss by referring to each of the first discrete action for learning to the t discrete action for learning and the corresponding discrete action ground truth, generate at least one second loss by referring to the continuous action for learning and the corresponding continuous action ground truth, and perform a process of training the discrete action detection network and the continuous action detection network using the first loss and the second loss.

[0055] A method for learning a deep learning-based behavior detection model that detects human behavior through video analysis using a learning device according to one embodiment of the present invention configured as described above is described below with reference to FIG. 2.

[0056] First, the learning device (100) can obtain a first image frame for learning or a t-th image frame for learning (11) corresponding to a video captured by a camera. The t may be an integer greater than or equal to 2. At this time, the first image frame for learning or the t-th image frame for learning (11) may be annotated with at least one ground truth among a discrete action ground truth and a continuous action ground truth. In addition, the first image frame for learning or the t-th image frame for learning (11) may be annotated with a bounding box ground truth and a skeleton keypoint ground truth corresponding to a person.

[0057] And, the learning device (100) inputs each of the first image frame for learning to the t-th image frame for learning (11) into a keypoint and bounding box extraction network (110) so that the keypoint and bounding box extraction network (110) can generate at least one first bounding box for learning to at least one t-th bounding box for learning corresponding to at least one person located in each of the first image frame for learning to the t-th image frame for learning, and at least one first skeleton keypoint for learning to at least one t-th skeleton keypoint for learning (12).

[0058] For example, referring to FIG. 3, the learning device (100) inputs each of the first image frame for training to the t-th image frame for training (11) into the keypoint and bounding box extraction network (110), and causes the keypoint and bounding box extraction network (110) to perform convolution operations on the specific image frame for training among the first image frame for training to the t-th image frame for training through the feature extraction block (111) to generate feature maps for training having different scales. At this time, the feature extraction block (111) may use the backbone network of the YOLO (You Only Look Once) model that uses the CSP (Cross Stage Partial) technique, but the present invention is not limited thereto, and various networks that generate a plurality of feature maps having different scales by using convolution blocks that perform convolution operations may be used.

[0059] And, the learning device (100) can cause the keypoint and bounding box extraction network (110) to perform feature fusion on each of the first learning feature map to the kth learning feature map, which are at least some of the learning feature maps having different scales, through the feature fusion block (112). k can be an integer greater than or equal to 2. At this time, the feature fusion block (112) can perform top-down feature fusion using a top-down pass and bottom-up feature fusion using a bottom-up pass to generate the first learning fusion feature map to the kth learning fusion feature map. That is, in the top-down pass, a learning feature map of a specific scale is up-sampled to fuse features into a learning feature map of the next scale that is larger than the specific scale, and in the bottom-up pass, a learning feature map of a specific scale is down-sampled to fuse features into a learning feature map of the next scale that is smaller than the specific scale. However, the present invention is not limited thereto, and may be performed in various ways to fuse features between the first learning feature map to the k-th learning feature map. In addition, although FIG. 3 illustrates that four learning feature maps having different scales are used, this is for the convenience of explanation, and the present invention is not limited thereto.

[0060] Thereafter, the learning device (100) can cause the keypoint and bounding box extraction network (110) to detect the first bounding box for learning and the first skeleton keypoints for learning or the k-th bounding box for learning and the k-th skeleton keypoints for learning (114) corresponding to a person in each of the first fusion feature map for learning or the k-th fusion feature map for learning through the detection block (113). At this time, the detection block (113) detects bounding boxes and skeleton keypoints through each detection head from each of the first fusion feature map for learning to the t-th fusion feature map for learning having different scales, thereby generating the first bounding box for learning and the first skeleton keypoints for learning to the k-th bounding box for learning and the k-th skeleton keypoints for learning (114) by detecting bounding boxes and skeleton keypoints corresponding to a person from each of the first fusion feature map for learning to the t-th fusion feature map for learning having different scales.

[0061] And, the learning device (100) can cause the keypoint and bounding box extraction network (110) to generate a learning-specific bounding box and learning-specific skeleton keypoints corresponding to a person located in a learning-specific image frame using the first learning-specific bounding box and the first learning-specific skeleton keypoints or the k-th learning-specific bounding box and the k-th learning-specific skeleton keypoints.

[0062] As another example, referring to FIG. 4, the learning device (100) can cause the keypoint and bounding box extraction network (110) to generate a learning feature map by performing a convolution operation on the learning specific image frame at least once through the feature extraction block (116) for any one of the learning first image frame to the learning t-th image frame.

[0063] In addition, the learning device (100) can cause the keypoint and bounding box extraction network (110) to generate a learning-specific bounding box (BB) that detects a person in a learning-specific image frame by performing a learning-specific feature map learning operation through an object detector (117), and to generate learning-specific skeleton keypoints (SK) corresponding to a person in a learning-specific image frame by performing a learning-specific feature map learning operation through a pose estimator (118). At this time, the object detector (117) may be composed of various detection networks such as an R-CNN-based object detector, a YOLO-based object detector, a SSD (Single Shot MultiBox Detector), RetinaNet, etc., and the pose estimator (118) may be composed of various pose estimation networks such as PoseNet, OpenPose, ConvNet POSE, CPM (Convolutional Pose Machines), Stacked Hourglass Networks, CPN (Cascaded Pyramid Network), BlazePose, MoveNet, etc.

[0064] Again, referring to FIG. 2, the learning device (100) may input the first bounding box for learning and the first skeleton keypoints for learning or the t-bounding box for learning and the t-skeleton keypoints for learning into a discrete action detection network (120), and cause the discrete action detection network (120) to perform a learning operation on each of the first bounding box for learning and the first skeleton keypoints for learning or the t-bounding box for learning and the t-skeleton keypoints for learning, thereby detecting the first discrete action for learning corresponding to a person located in the first image frame for learning or the t-discrete action for learning corresponding to a person located in the t-image frame for learning (13).

[0065] At this time, the learning device (100) can cause the discrete action detection network (120) to detect a specific discrete action by heuristic analysis using at least one of the aspect ratio of the specific bounding box for learning and the vertical distances between the skeleton keypoints in the specific bounding box for learning and the specific skeleton keypoints for learning, which is one of the first bounding box for learning and the first skeleton keypoints for learning, or the t bounding box for learning and the t skeleton keypoints for learning.

[0066] In addition, the learning device (100) can cause the discrete action detection network (120) to detect a first sub-discrete action for learning corresponding to a person's learning action when the number of times the same learning discrete action is detected among the first discrete action for learning to the tth discrete action for learning is greater than or equal to a preset threshold number of times, by referring to the positions of each of the first bounding box for learning to the tth bounding box for learning, when the position change is less than the preset threshold position change, and to detect a second sub-discrete action for learning corresponding to the same learning discrete action or the first sub-discrete action for learning when the position change is greater than or equal to the preset threshold position change.

[0067] As an example, let us consider the case of detecting a person's “fall” action as a discrete action as follows.

[0068] As shown in Figure 5a, the aspect ratio of a normal person in the image may be greater than that of a person falling. Therefore, if the aspect ratio of the bounding box exceeds a preset value, i.e., the vertical length is greater than the horizontal length, the person's action can be predicted as "falling."

[0069] Also, as shown in FIG. 5b, assuming that the skeleton key points of the shoulders on both sides are P1 and P4, the skeleton key points of the waists on both sides are P2 and P5, the skeleton key points of the feet on both sides are P3 and P6, and the vertical distance between P1 and P2 or between P4 and P5 is D1, and the vertical distance between P2 and P3 or between P5 and P6 is D2, the “fall” action can be predicted using the vertical distances between the skeleton key points.

[0070] For example, if the top-left coordinate in the image is (0, 0), the vertical distance between the skeleton keypoints, i.e. the y-coordinate value, can be predicted as a human action as “falling” if it satisfies the following condition.

[0071] (i) If P1(y) > P3(y) - D1, P1(y) > P2(y) - D1 / 2, and P2(y) > P3(y) - D1 / 2

[0072] (ii) If P4(y) > P6(y) - D1, P4(y) > P5(y) - D1 / 2, and P5(y) > P6(y) - D1 / 2

[0073] In addition, if the number of times predicted as “falling” in a preset time, i.e., a preset number of image frames, is greater than a preset threshold number of times, the human action can be predicted as “laying” resulting from the human “falling.”

[0074] And, when the human action is predicted as “lying down”, it is checked whether the position change of the bounding boxes corresponding to the human is less than or greater than a preset threshold position change. If the position change of the bounding boxes corresponding to the human predicted as “lying down” is less than the preset threshold position change, the human action can be predicted as “lying down”. If the position change of the bounding boxes corresponding to the human predicted as “lying down” is greater than or equal to the preset threshold position change, the human action can be predicted as “rolling” in the “lying down” state. At this time, if the position change of the human bounding boxes is a position change in the horizontal direction, the human action can be predicted as “rolling in the horizontal direction”, and if the position change of the human bounding boxes is a position change in the vertical direction, the human action can be predicted as “rolling in the vertical direction”.

[0075] Referring back to FIG. 2, the learning device (100) inputs the first bounding box for learning or the t-th bounding box for learning into the continuous action detection network (130), and causes the continuous action detection network (130) to generate a first cropped image for learning by cropping an area corresponding to the first bounding box for learning from the first image frame for learning or a t-th cropped image for learning by cropping an area corresponding to the t-th bounding box for learning from the t-th image frame for learning, and performs a learning operation on the first cropped image for learning or the t-th cropped image for learning through at least one classifier to generate a first action estimation value for learning by detecting an individual action of a person in the first cropped image for learning or a t-th action estimation value for learning by detecting an individual action of a person in the t-th cropped image for learning, and with reference to the first action estimation value for learning or the t-th action estimation value for learning, the network generates an image corresponding to a person located in the first image frame for learning or the t-th image frame for learning. A process for detecting continuous actions for learning can be performed. At this time, the classifier included in the continuous action detection network (130) can predict an action estimate value that estimates a person's action using only features extracted from a training crop image. In addition, the classifier can include various deep learning networks that extract features from a training crop image through at least one convolution operation and predict a corresponding specific action estimate value among preset action estimate value classes using the extracted features.

[0076] For example, when “pick”, “watch”, and “add to basket or cart” are set as action estimates, and the action estimates are predicted in the order of “pick” - “pick” - “put”, the person’s action can be predicted as “put”, which is the action of putting a product in a basket or card to purchase the product from a store shelf. When the action estimates are predicted in the order of “pick” - “watch” - “put”, the person’s action can be predicted as “put”, which is the action of picking up a product from a store shelf, checking it, and then putting the product in a basket or card to purchase the product. In addition, if the action estimates are predicted in the order of “put” - “look” - “pick up,” the human action can be predicted as “put back,” which is an action of putting an object into a basket or cart, checking the object, and then putting the object back on the store shelf. In addition, if the action estimates are predicted in the order of “pick up” - “pick up” - “look” - “pick up,” the human action can be predicted as “pick up,” which is an action of picking up the object from the store shelf to check the object and then purchase the object. However, the present invention is not limited thereto, and various continuous actions of a human can be predicted with reference to the action estimates predicted through image analysis.

[0077] At this time, the learning device (100) can sequentially input each of the first crop image for learning and the t-th crop image for learning into one classifier and sequentially perform learning operations on each of the first crop image for learning and the t-th crop image for learning through one classifier, thereby sequentially generating a first action estimate value for learning corresponding to the first crop image for learning and a t-th action estimate value for learning corresponding to the t-th crop image for learning.

[0078] In addition, unlike the continuous action detection network (130) that generates the first action estimate for learning or the t-th action estimate for learning through one classifier as described above, the learning device (100) may cause the continuous action detection network (130) to perform a learning operation on each of the first crop image for learning or the t-th crop image for learning through each of the first classifiers to the t-th classifiers corresponding to each of the first crop image for learning or the t-th crop image for learning.

[0079] For example, referring to FIG. 6, the continuous action detection network (130) may generate a learning (t-2) action estimate by performing a learning (t-2) crop image through a learning (t-2) classifier, generate a learning (t-1) action estimate by performing a learning (t-1) crop image through a learning (t-1) classifier, and generate a learning (t-1) action estimate by performing a learning (t-1) crop image through a learning (t-1) classifier. In addition, the continuous action detection network (130) may generate a result of predicting a learning continuous action by referring to the learning (t-2) action estimate or the learning (t-1) action estimate.

[0080] Again, referring to FIG. 2, the learning device (100) generates at least one first loss by referring to each of the first discrete action for learning to the t discrete action for learning and the corresponding discrete action ground truth, generates at least one second loss by referring to the continuous action for learning and the corresponding continuous action ground truth, and trains the discrete action detection network (120) and the continuous action detection network (130) using the first loss and the second loss.

[0081] In addition, the learning device (100) can further train the keypoint and bounding box extraction network (110) using the 3_1 loss generated by referring to the first bounding box for training or the t-th bounding box for training and the respective bounding box ground truths corresponding thereto, and the 3_2 loss generated by referring to the first skeleton keypoints for training or the t-th skeleton keypoints for training and the respective skeleton keypoint ground truths corresponding thereto. In addition, alternatively, the learning device (100) can train the discrete action detection network (120) and the continuous action detection network (130) in a state where the keypoint and bounding box extraction network (110) is trained in advance before training the discrete action detection network (120) and the continuous action detection network (130).

[0082] FIG. 7 schematically illustrates a test device for testing a deep learning-based action detection model for detecting human action through video analysis according to another embodiment of the present invention. Referring to FIG. 7, the test device (200) may include a memory (201) in which instructions for testing a deep learning-based action detection model for detecting human action through video analysis are stored, and a processor (202) for performing an operation for testing a deep learning-based action detection model for detecting human action through video analysis according to the instructions stored in the memory (201).

[0083] Specifically, the test device (200) may typically, but is not limited to, utilize a combination of computing devices (e.g., devices that may include computer processors, memory, storage, input devices and output devices, and other components of conventional computing devices; electronic communication devices such as routers, switches, and the like; electronic information storage systems such as network attached storage (NAS) and storage area networks (SAN)) and computer software (i.e., instructions that cause the computing device to function in a particular manner) to achieve desired system performance.

[0084] Additionally, the processor of the computing device may include hardware components such as a Micro Processing Unit (MPU) or a Central Processing Unit (CPU), cache memory, and a data bus. Furthermore, the computing device may further include software components such as an operating system and applications that perform specific purposes.

[0085] However, this does not exclude the case where the computing device includes an integrated processor in which the medium, processor, and memory for implementing the present invention are integrated.

[0086] Meanwhile, the processor (202) of the test device (200) executes, according to instructions stored in the memory (201), a process for: (i) inputting each of the first image frame for training to the t-th image frame for training corresponding to the video captured by the camera into a keypoint and bounding box extraction network, so that the keypoint and bounding box extraction network generates at least one first bounding box for training to at least one t-th bounding box for training corresponding to at least one person located in each of the first image frame for training to the t-th image frame for training, and at least one first skeleton keypoint for training to at least one t-th skeleton keypoint for training, (ii) inputting (ii-1) the first bounding box for training and the first skeleton keypoints for training to the t-th bounding box for training and the t-th skeleton keypoints for training into a discrete action detection network, so that the discrete A process for detecting a first discrete action for training corresponding to a person located in a first image frame for training or a t discrete action for training corresponding to a person located in a t image frame for training by performing a learning operation on the first bounding box for training and the first skeleton keypoints for training or the t bounding box for training and the t skeleton keypoints for training by an action detection network, and (ii-2) inputting the first bounding box for training or the t bounding box for training into a continuous action detection network, and causing the continuous action detection network to generate a first cropped image for training by cropping an area corresponding to the first bounding box for training in the first image frame for training or a t cropped image for training by cropping an area corresponding to the t bounding box for training in the t image frame for training,A process of generating a first action estimate value for training or a t-th action estimate value for training by detecting an individual action of a person in the first cropped image for training or a t-th cropped image for training by performing a learning operation on the first cropped image for training or the t-th cropped image for training, respectively, through at least one classifier, and detecting a continuous action for training corresponding to a person located in the first image frame for training or the t-th image frame for training by referring to the first action estimate value for training or the t-th action estimate value for training, and (iii) generating at least one first loss by referring to each of the first discrete action for training or the t-th discrete action for training and the respective corresponding discrete action ground truths, generating at least one second loss by referring to the learning continuous action and the corresponding continuous action ground truths, and training a discrete action detection network and a continuous action detection network using the first loss and the second loss, wherein a first loss for testing corresponding to a video captured by a camera is performed. When the image frame or the test t-th image frame is acquired, a process may be performed in which the first test image frame or the test t-th image frame, respectively, is input to a keypoint and bounding box extraction network, and the keypoint and bounding box extraction network generates at least one test first bounding box or at least one test t-th bounding box corresponding to at least one person located in each of the first test image frame or the test t-th image frame, and at least one test first skeleton keypoint or at least one test t-th skeleton keypoint. In addition, the processor (202) of the test device (200)A process of inputting a first bounding box for testing and first skeleton keypoints for testing or a t-bounding box for testing and a t-skeleton keypoints for testing into a discrete action detection network, and causing the discrete action detection network to perform a learning operation on each of the first bounding box for testing and the first skeleton keypoints for testing or the t-bounding box for testing and the t-skeleton keypoints for testing, to detect a first discrete action for testing corresponding to a person located in a first image frame for testing or a t-discrete action for testing corresponding to a person located in a t-image frame for testing, and inputting the first bounding box for testing or the t-bounding box for testing into a continuous action detection network, and causing the continuous action detection network to detect a first cropped image for testing, which is a cropped image for testing, which is a region corresponding to the first bounding box for testing in the first image frame for testing, or a t-cropped image for testing, which is a region corresponding to the t-bounding box for testing in the t-image frame for testing. A process may be performed to generate an image, and to detect an individual action of a person in the test first crop image through a running operation on the test first crop image to the test t-th crop image, respectively, to generate a test first action estimate value to a test t-th action estimate value to detect an individual action of a person in the test t-th crop image, and to perform a process of detecting a test continuous action corresponding to a person located in the test first image frame to the test t-th image frame by referring to the test first action estimate value to the test t-th action estimate value.

[0087] A method for testing a deep learning-based action detection model for detecting human actions through video analysis using a test device according to another embodiment of the present invention configured as described above is described below with reference to FIG. 8. In the following description, detailed descriptions of parts that can be easily understood from the description referring to FIG. 2 will be omitted.

[0088] First, with reference to FIG. 2, when the keypoint and bounding box extraction network (210), the discrete action detection network (220), and the continuous action detection network (230) are trained and the test first image frame to the test t-th image frame (21) corresponding to the video captured by the camera are acquired, the test device (200) inputs each of the test first image frame to the test t-th image frame (21) into the keypoint and bounding box extraction network (210), so that the keypoint and bounding box extraction network (210) can generate at least one test first bounding box to at least one test t-th bounding box corresponding to at least one person located in each of the test first image frame to the test t-th image frame (21), and at least one test first skeleton keypoint to at least one test t-th skeleton keypoint (22).

[0089] And, the test device (200) can input the test first bounding box and the test first skeleton keypoints or the test t bounding box and the test t skeleton keypoints (22) into the discrete action detection network (220) and cause the discrete action detection network (220) to perform a running operation on each of the test first bounding box and the test first skeleton keypoints or the test t bounding box and the test t skeleton keypoints, thereby detecting the test first discrete action corresponding to a person located in the test first image frame or the test t discrete action (23) corresponding to a person located in the test t image frame.

[0090] At this time, the test device (200) can cause the discrete action detection network (220) to detect a specific discrete action by a heuristic analysis using at least one of an aspect ratio of the specific bounding box for the test and vertical distances between skeleton keypoints in the specific skeleton keypoints for the test, with reference to a specific bounding box for the test and a specific skeleton keypoints for the test, which is one of the first bounding box for the test and the first skeleton keypoints for the test, or the t bounding box for the test and the t skeleton keypoints for the test.

[0091] In addition, the test device (200) may cause the discrete action detection network (220) to detect a first sub-discrete action of a person corresponding to the same test discrete action by referring to the positions of each of the first bounding box for testing to the t bounding box for testing when the number of times the same test discrete action is detected among the first discrete action for testing to the t discrete action for testing is greater than or equal to a preset threshold number of times, and to detect a second sub-discrete action for testing corresponding to the same test discrete action or the first sub-discrete action for testing when the position change is less than or equal to the preset threshold position change.

[0092] In addition, the test device (200) inputs the first bounding box for test to the t-test bounding box for test into the continuous action detection network (230), and causes the continuous action detection network (230) to generate a first cropped image for test by cropping an area corresponding to the first bounding box for test from the first image frame for test to a t-test cropped image by cropping an area corresponding to the t-test bounding box from the t-test image frame for test, and performs a running operation on the first cropped image for test to the t-test cropped image for test, respectively, through at least one classifier included in the continuous action detection network (230), and generates a first action estimation value for test by detecting an individual action of a person in the first cropped image for test to a t-test action estimation value for test by detecting an individual action of a person in the t-test cropped image for test, and with reference to the first action estimation value for test to the t-test action estimation value for test, located in the first image frame for test to the t-test image frame for test, A process can be performed to detect continuous actions for testing corresponding to a person.

[0093] The embodiments of the present invention described above may be implemented in the form of program commands that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, etc., either singly or in combination. The program commands recorded on the computer-readable recording medium may be those specially designed and configured for the present invention or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

[0094] Although the present invention has been described above with specific details such as specific components and limited examples and drawings, these are provided only to help a more general understanding of the present invention, and the present invention is not limited to the above examples, and those with ordinary knowledge in the technical field to which the present invention pertains can make various modifications and variations from this description.

[0095] Therefore, the idea of ​​the present invention should not be limited to the embodiments described above, and all things that are modified equally or equivalently to the following claims as well as the claims are considered to fall within the scope of the idea of ​​the present invention.

Claims

1. A method for learning a deep learning-based action detection model that detects human actions through video analysis, (a) When a first image frame for training to a t-th image frame for training corresponding to a video captured by a camera, wherein t is an integer greater than or equal to 2, is acquired, a step of a learning device inputting each of the first image frame for training to the t-th image frame for training into a keypoint and bounding box extraction network, and causing the keypoint and bounding box extraction network to generate at least one first bounding box for training to at least one t-th bounding box for training corresponding to at least one person located in each of the first image frame for training to the t-th image frame for training, and at least one first skeleton keypoint for training to at least one t-th skeleton keypoint for training; (b) a process in which the learning device (i) inputs the first bounding box for training and the first skeleton keypoints for training or the t-bounding box for training and the t-skeleton keypoints for training into a discrete action detection network, and causes the discrete action detection network to perform a learning operation on each of the first bounding box for training and the first skeleton keypoints for training or the t-bounding box for training and the t-skeleton keypoints for training, to detect the first discrete action for training corresponding to the person located in the first image frame for training or the t-discrete action for training corresponding to the person located in the t-image frame for training, and (ii) inputs the first bounding box for training or the t-bounding box for training into a continuous action detection network, and causes the continuous action detection network to detect the first bounding box for training or the t-bounding box for training in the first image frame for training, A step of generating a learning t-crop image by cropping an area corresponding to the learning t-bounding box from a first learning crop image or a learning t-crop image frame, and performing a learning first action estimation value by detecting an individual action of the person in the learning 1st crop image or a learning t-action estimation value by detecting an individual action of the person in the learning t-crop image by performing a learning operation on the first learning crop image or the learning t-crop image through at least one classifier, and performing a process of detecting a learning continuous action corresponding to the person located in the learning 1st image frame or the learning t-image frame by referring to the learning 1st action estimation value or the learning t-action estimation value; and (c) a step of the learning device generating at least one first loss by referring to each of the first discrete action for learning to the t discrete action for learning and the corresponding discrete action ground truth, generating at least one second loss by referring to the continuous action for learning and the corresponding continuous action ground truth, and training the discrete action detection network and the continuous action detection network using the first loss and the second loss; How to include.

2. In paragraph 1, In step (b) above, A method in which the learning device, in the process (i), causes the discrete action detection network to detect a specific discrete action by heuristic analysis using at least one of an aspect ratio of the specific learning bounding box and vertical distances between skeleton keypoints in the specific learning skeleton keypoints, referring to a specific learning bounding box and specific learning skeleton keypoints, which is one of the first learning bounding box and the first learning skeleton keypoints or the t learning bounding box and the t learning skeleton keypoints.

3. In paragraph 1, In step (b) above, The learning device, in the process (i), causes the discrete action detection network to detect, by referring to the positions of each of the learning first bounding box to the learning t bounding box, a first sub-discrete action for learning of the person corresponding to the same discrete action for learning if the change in position is less than the preset threshold position change, and detects a second sub-discrete action for learning corresponding to the same discrete action for learning or the first sub-discrete action for learning if the change in position is greater than the preset threshold position change, when the change in position is greater than the preset threshold position change.

4. In paragraph 1, In step (b) above, A method in which the learning device, in the process (ii), causes the continuous action detection network to perform a learning operation on each of the first crop image for learning to the t-crop image for learning through the first classifier to the t-classifier corresponding to each of the first crop image for learning to the t-crop image for learning.

5. In paragraph 1, In step (c) above, A method for further training the keypoint and bounding box extraction network using the 3_1 loss generated by referring to the first bounding box for training or the t-th bounding box for training and the respective corresponding bounding box ground truths, and the 3_2 loss generated by referring to the first skeleton keypoints for training or the t-th skeleton keypoints for training and the respective corresponding skeleton keypoint ground truths.

6. In paragraph 1, In step (a) above, The learning device causes the keypoint and bounding box extraction network to perform convolution operations on a specific learning image frame among the first learning image frame to the t-th learning image frame through a feature extraction block to generate learning feature maps having different scales, and performs top-down feature fusion using a top-down pass and bottom-up feature fusion using a bottom-up pass on at least a part of the learning feature maps, the first learning feature map to the k-th learning feature map, wherein k is an integer greater than or equal to 2, through a feature fusion block to generate the first learning fusion feature map to the k-th learning fusion feature map, and detects, through a detection block, a first learning bounding box corresponding to the person and the first learning skeleton keypoints to the k-th learning fusion feature map, respectively. A method for detecting a k-th bounding box and a k-th skeleton keypoints for training, and generating a specific bounding box and specific skeleton keypoints for training corresponding to the person located in the specific image frame for training using the first bounding box for training and the first skeleton keypoints for training or the k-th bounding box for training and the k-th skeleton keypoints for training.

7. In paragraph 1, In step (a) above, The learning device causes the keypoint and bounding box extraction network to perform a convolution operation on the learning specific image frame at least once through a feature extraction block for any one of the first learning image frame to the t-th learning image frame to generate a learning feature map, perform a learning operation on the feature map through an object detector to generate a learning specific bounding box that detects the person in the learning specific image frame, and perform a learning operation on the feature map through a pose estimator to generate learning specific skeleton keypoints corresponding to the person in the learning specific image frame.

8. A method for testing a deep learning-based action detection model that detects human actions through video analysis, (a) a process of, by a learning device, (i) acquiring a first image frame for training to a t-th image frame for training corresponding to a video captured by a camera, wherein t is an integer greater than or equal to 2, inputting each of the first image frame for training to the t-th image frame for training into a keypoint and bounding box extraction network so that the keypoint and bounding box extraction network generates at least one first bounding box for training to at least one t-th bounding box for training corresponding to at least one person located in each of the first image frame for training to the t-th image frame for training, and at least one first skeleton keypoint for training to at least one t-th skeleton keypoint for training, (ii) (ii-1) inputting the first bounding box for training and the first skeleton keypoints for training to the t-th bounding box for training and the t-th skeleton keypoints for training into a discrete action detection network so that the discrete action detection network generates the learning A process of detecting the first discrete action for training corresponding to the person located in the first image frame for training or the t discrete action for training corresponding to the person located in the t image frame for training by performing a learning operation on the first bounding box and the first skeleton keypoints for training or the t bounding box for training and the t skeleton keypoints for training, respectively, and (ii-2) inputting the first bounding box for training or the t bounding box for training into a continuous action detection network so that the continuous action detection networkA process for generating a first cropped image for training by cropping an area corresponding to the first bounding box for training in the first image frame for training or a t-cropped image for training by cropping an area corresponding to the t-bounding box for training in the t-image frame for training, and performing a learning operation on the first cropped image for training or the t-cropped image for training through at least one classifier to generate a first action estimate value for training by detecting an individual action of the person in the first cropped image for training or a t-action estimate value for training by detecting an individual action of the person in the t-cropped image for training, and performing a process for detecting a continuous action for training corresponding to the person located in the first image frame for training or the t-image frame for training by referring to the first action estimate value for training or the t-action estimate value for training, and (iii) at least one discrete action for training or the t-discrete action for training and a corresponding discrete action ground truth for training, respectively. In a state where a process of generating a first loss, generating at least one second loss by referring to the learning continuous action and the corresponding continuous action ground truth, and training the discrete action detection network and the continuous action detection network using the first loss and the second loss is performed, when a first image frame for testing or a t-th image frame for testing corresponding to a video captured by a camera is acquired, the test device,A step of inputting each of the first image frame for testing to the t-test image frame for testing into the keypoint and bounding box extraction network, and causing the keypoint and bounding box extraction network to generate at least one first bounding box for testing to at least one t-test bounding box corresponding to at least one person located in each of the first image frame for testing to the t-test image frame for testing, and at least one first skeleton keypoint for testing to at least one t-test skeleton keypoint for testing; and (b) a process in which the test device (i) inputs the first bounding box for testing and the first skeleton keypoints for testing or the t-bounding box for testing and the t-skeleton keypoints for testing into the discrete action detection network, and causes the discrete action detection network to perform a learning operation on each of the first bounding box for testing and the first skeleton keypoints for testing or the t-bounding box for testing and the t-skeleton keypoints for testing, to detect the first discrete action for testing or the t-discrete action for testing corresponding to the person located in the t-image frame for testing, and (ii) inputs the first bounding box for testing or the t-bounding box for testing into the continuous action detection network, and causes the continuous action detection network to detect a test image frame in which an area corresponding to the first bounding box for testing is cropped in the first image frame for testing. A step of generating a test t-crop image by cropping an area corresponding to the test t-bounding box from the first crop image or the test t-image frame, and performing a running operation on the test 1st crop image or the test t-crop image through at least one classifier to generate a test 1st action estimation value that detects an individual action of the person in the test 1st crop image or a test t-action estimation value that detects an individual action of the person in the test t-crop image, and performing a process of detecting a test continuous action corresponding to the person located in the test 1st image frame or the test t-image frame by referring to the test 1st action estimation value or the test t-action estimation value; How to include.

9. In paragraph 8, In step (b) above, A method for detecting a specific discrete action by a heuristic analysis using at least one of an aspect ratio of the specific bounding box for the test and vertical distances between skeleton keypoints in the specific skeleton keypoints for the test, wherein the test device causes the discrete action detection network to detect a specific discrete action by referring to a specific bounding box for the test and a specific skeleton keypoints for the test, which is one of the first bounding box for the test and the first skeleton keypoints for the test, or the t bounding box for the test and the t skeleton keypoints for the test.

10. In paragraph 8, In step (b) above, A method for detecting a test device, in the process (i), the discrete action detection network detects a first sub-discrete action of the person corresponding to the same test discrete action by referring to the positions of each of the first bounding box for the test to the t bounding box for the test when the number of times the same test discrete action is detected among the first discrete action for the test to the t discrete action for the test is greater than or equal to a preset threshold, and detects a second sub-discrete action for the test corresponding to the same test discrete action or the first sub-discrete action for the test when the position change is less than or equal to the preset threshold position change.

11. A learning device that learns a deep learning-based action detection model that detects human actions through video analysis. A memory storing instructions for training a deep learning-based action detection model that detects human actions through video analysis; and A processor that performs an operation of learning the deep learning-based action detection model that detects the human action through video analysis according to the instructions stored in the memory; Includes, The processor, (I) when a first image frame for training to a t-th image frame for training corresponding to a video captured by a camera, wherein t is an integer greater than or equal to 2, is acquired, a process for inputting each of the first image frame for training to the t-th image frame for training into a keypoint and bounding box extraction network so that the keypoint and bounding box extraction network generates at least one first bounding box for training to at least one t-th bounding box for training corresponding to at least one person located in each of the first image frame for training to the t-th image frame for training, and at least one first skeleton keypoint for training to at least one t-th skeleton keypoint for training, (II) (i) inputting the first bounding box for training and the first skeleton keypoints for training to the t-th bounding box for training and the t-th skeleton keypoints for training into a discrete action detection network so that the discrete action detection network generates the first bounding box for training and the t-th skeleton keypoints for training. A process of detecting a first discrete action for training corresponding to the person located in the first image frame for training or a t discrete action for training corresponding to the person located in the t image frame for training by performing a learning operation on each of the first skeleton keypoints for training or the t bounding box for training and the t skeleton keypoints for training, and (ii) inputting the first bounding box for training or the t bounding box for training into a continuous action detection network so that the continuous action detection network generates a first cropped image for training by cropping an area corresponding to the first bounding box for training in the first image frame for training or a t cropped image for training by cropping an area corresponding to the t bounding box for training in the t image frame for training.A process for generating a first action estimate value for training by detecting an individual action of the person in the first cropped image for training or a t-th cropped image for training by performing a learning operation on the first cropped image for training or the t-th cropped image for training, respectively, through at least one classifier, or a t-th action estimate value for training by detecting an individual action of the person in the t-th cropped image for training, and performing a process of detecting a continuous action for training corresponding to the person located in the first image frame for training or the t-th image frame for training by referring to the first action estimate value for training or the t-th action estimate value for training, and (III) generating at least one first loss by referring to each of the first discrete action for training or the t-th discrete action for training and the respective discrete action ground truths corresponding thereto, and generating at least one second loss by referring to the continuous action for training and the continuous action ground truths corresponding thereto, and training the discrete action detection network and the continuous action detection network using the first loss and the second loss. A learning device that performs a process.

12. In paragraph 11, The above processor, A learning device that detects a specific discrete action by heuristic analysis using at least one of an aspect ratio of the specific bounding box for learning and vertical distances between skeleton keypoints in the specific bounding box for learning and the specific skeleton keypoints for learning, wherein the specific bounding box for learning and the specific skeleton keypoints for learning are either one of the first bounding box for learning and the first skeleton keypoints for learning or the t bounding box for learning and the t skeleton keypoints for learning.

13. In paragraph 11, The above processor, A learning device in which, in the process (i) of the process (II), the discrete action detection network detects, when the number of times the same learning discrete action is detected among the first learning discrete action to the t learning discrete action is greater than or equal to a preset threshold number, the device detects, by referring to the positions of each of the first learning bounding box to the t learning bounding box, the first sub-discrete action for learning of the person corresponding to the same learning discrete action when the position change is less than the preset threshold position change, and detects, when the position change is greater than or equal to the preset threshold position change, the second sub-discrete action for learning corresponding to the same learning discrete action or the first learning sub-discrete action.

14. In paragraph 11, The above processor, A learning device that causes the continuous action detection network to perform a learning operation on each of the first crop image for learning to each of the t-crop images for learning through the first classifier to the t-crop image for learning, respectively, in the process (ii) of the process (II).

15. In paragraph 11, The above processor, A learning device that further trains the keypoint and bounding box extraction network in the above (III) process by using the 3_1 loss generated by referring to the first bounding box for learning to the t-th bounding box for learning and the respective bounding box ground truths corresponding thereto, and the 3_2 loss generated by referring to the first skeleton keypoints for learning to the t-th skeleton keypoints for learning and the respective skeleton keypoint ground truths corresponding thereto.

16. In paragraph 11, The above processor, In the above (I) process, the keypoint and bounding box extraction network is caused to perform convolution operations on a specific training image frame among the first training image frame to the t-th training image frame through a feature extraction block to generate training feature maps having different scales, and perform top-down feature fusion using a top-down pass and bottom-up feature fusion using a bottom-up pass on at least a part of the training feature maps, the first training feature map to the k-th training feature map, where k is an integer greater than or equal to 2, through a feature fusion block to generate the first training fusion feature map to the k-th training fusion feature map, and through a detection block, the first training bounding box and the first training skeleton keypoints corresponding to the person in each of the first training fusion feature map to the k-th training fusion feature map. A learning device that detects a k-th bounding box for learning and a k-th skeleton keypoint for learning, and generates a specific bounding box for learning and a specific skeleton keypoint for learning corresponding to the person located in the specific image frame for learning using the first bounding box for learning and the first skeleton keypoint for learning or the k-th bounding box for learning and the k-th skeleton keypoint for learning.

17. In paragraph 11, The above processor, In the above (I) process, a learning device configured to cause the keypoint and bounding box extraction network to perform a convolution operation on the learning-specific image frame at least once through a feature extraction block for any one of the learning-specific first image frame to the learning-specific t-th image frame to generate a learning feature map, perform a learning operation on the feature map through an object detector to generate a learning-specific bounding box that detects the person in the learning-specific image frame, and perform a learning operation on the feature map through a pose estimator to generate learning-specific skeleton keypoints corresponding to the person in the learning-specific image frame.

18. In a test device for testing a deep learning-based action detection model that detects human actions through video analysis, A memory storing instructions for testing a deep learning-based action detection model that detects human actions through video analysis; and A processor that performs an operation of testing the deep learning-based action detection model that detects the human action through video analysis according to the instructions stored in the memory; Includes, The processor, (I) a process for inputting, by a learning device, (i) a first image frame for training to a t-th image frame for training corresponding to a video captured by a camera, wherein t is an integer greater than or equal to 2, into a keypoint and bounding box extraction network, respectively, for inputting the first image frame for training to the t-th image frame for training, so that the keypoint and bounding box extraction network generates at least one first bounding box for training to at least one t-th bounding box for training, and at least one first skeleton keypoint for training to at least one t-th skeleton keypoint for training, corresponding to at least one person located in each of the first image frame for training to the t-th image frame for training, (ii) (ii-1) inputting the first bounding box for training and the first skeleton keypoints for training to the t-th bounding box for training and the t-th skeleton keypoints for training to a discrete action detection network, thereby generating the discrete action detection network A process of detecting the first discrete action for training corresponding to the person located in the first image frame for training or the t discrete action for training corresponding to the person located in the t image frame for training by performing a learning operation on the first bounding box for training and the first skeleton keypoints for training or the t bounding box for training and the t skeleton keypoints for training, respectively, and (ii-2) inputting the first bounding box for training or the t bounding box for training into a continuous action detection network, so that the continuous action detection networkA process for generating a first cropped image for training by cropping an area corresponding to the first bounding box for training in the first image frame for training or a t-cropped image for training by cropping an area corresponding to the t-bounding box for training in the t-image frame for training, and performing a learning operation on the first cropped image for training or the t-cropped image for training through at least one classifier to generate a first action estimate value for training by detecting an individual action of the person in the first cropped image for training or a t-action estimate value for training by detecting an individual action of the person in the t-cropped image for training, and performing a process for detecting a continuous action for training corresponding to the person located in the first image frame for training or the t-image frame for training by referring to the first action estimate value for training or the t-action estimate value for training, and (iii) at least one discrete action for training or the t-discrete action for training and a corresponding discrete action ground truth for training, respectively. In a state where a process of generating a first loss, generating at least one second loss by referring to the learning continuous action and the corresponding continuous action ground truth, and training the discrete action detection network and the continuous action detection network using the first loss and the second loss is performed, when a first image frame for testing or a t-th image frame for testing corresponding to a video captured by a camera is acquired,A process for inputting each of the test first image frame to the test t-th image frame into the keypoint and bounding box extraction network so that the keypoint and bounding box extraction network generates at least one test first bounding box to at least one test t-th bounding box corresponding to at least one person located in each of the test first image frame to the test t-th image frame, and at least one test first skeleton keypoint to at least one test t-th skeleton keypoint, and (II) (i) inputting the test first bounding box and the test first skeleton keypoints to the test t-th bounding box and the test t-th skeleton keypoints into the discrete action detection network so that the discrete action detection network performs a learning operation on each of the test first bounding box and the test first skeleton keypoints to the test t-th bounding box and the test t-th skeleton keypoints, and A process for detecting a first discrete action for testing corresponding to the person located in the first image frame or a t-th discrete action for testing corresponding to the person located in the t-th image frame for testing, and (ii) inputting the first bounding box for testing or the t-th bounding box for testing into the continuous action detection network, thereby causing the continuous action detection network to generate a first cropped image for testing by cropping an area corresponding to the first bounding box for testing in the first image frame for testing or a t-th cropped image for testing by cropping an area corresponding to the t-th bounding box for testing in the t-th image frame for testing.A test device that performs a process of detecting a test continuous action corresponding to a person located in the test first image frame or the test t-th image frame by performing a running operation on the test first crop image or the test t-th crop image through at least one of the classifiers, thereby generating a test first action estimation value or a test t-th action estimation value that detects an individual action of the person in the test first crop image or the test t-th crop image, and performing a process of detecting a test continuous action corresponding to a person located in the test first image frame or the test t-th image frame by referring to the test first action estimation value or the test t-th action estimation value.

19. In paragraph 18, The above processor, A test device for detecting a specific discrete action by heuristic analysis using at least one of an aspect ratio of the specific bounding box for the test and vertical distances between skeleton keypoints in the specific skeleton keypoints for the test, wherein the specific bounding box for the test and the specific skeleton keypoints for the test are either one of the first bounding box for the test and the first skeleton keypoints for the test, or the t bounding box for the test and the t skeleton keypoints for the test.

20. In paragraph 18, The above processor, A test device in which, in the process (i) of the process (II), the discrete action detection network detects, when the number of times the same test discrete action is detected among the first test discrete action to the t test discrete action is greater than or equal to a preset threshold number, the device detects, by referring to the positions of each of the first test bounding box to the t test bounding box, the first sub-discrete action of the person corresponding to the same test discrete action when the position change is less than the preset threshold position change, and detects, when the position change is greater than or equal to the preset threshold position change, the second sub-discrete action corresponding to the same test discrete action or the first test sub-discrete action.

Images