Device and method for classification using heterogeneous graph data regarding people and object

Abstract

The present invention relates to a classification device and method that enables more accurate and rapid situation recognition by generating graph data for each person and object from input data, combining each generated graph data to generate a graph sequence, and classifying the input data using the same. The above classification device may include a data collection module that receives input data, a graph module that extracts features from each person and predefined object included in the input data and generates a graph sequence based on the extracted features, and a classification module that generates a classification result for the graph sequence.

Description

Technology Field

[0001] The present invention relates to a classification device and method using heterogeneous graph data regarding people and objects.

[0002] Specifically, the present invention relates to a classification device and method that enables more accurate and rapid situation recognition by generating graph data for each person and object from input data, combining each generated graph data to generate a graph sequence, and classifying the input data using the same. Background Technology

[0004] This study was conducted as a result of the Food Tech Research and Technology Development Project of the Gyeongsangbuk-do Provincial Office and Pohang City Hall (GBTP2023129001).

[0005] Recently, in fields requiring risk detection, security, and monitoring, there has been a steady development of technologies utilizing Artificial Intelligence to perform monitoring.

[0006] While accurate monitoring and detection of risk factors are essential for such AI perimeter security systems, existing AI perimeter security systems still suffer from many problems, such as malfunctions and over-operations. In particular, in the defense sector where this technology is most widely used, various instances of perimeter security failures have been observed even recently, and data errors can be cited as the cause. Specifically, graph data, which is structured as a graph, cannot be processed by existing DNNs in the Euclidean domain (e.g., CNN (Convolutional Neural Network), RNN (Recurrent Neural Network)). Although CNNs are generally most commonly used for motion recognition, converting graph data (skeleton data) into images for input into CNNs is not practical due to changes in the external appearance of the motion and the inherent noise of skeleton estimation; accordingly, technologies for extracting powerful image features are being researched.

[0007] Meanwhile, in industrial sectors such as national defense, medical facilities, construction sites, and the culinary field, the need for situational awareness rather than object recognition is gradually increasing. In other words, regarding input data related to each industry, it is becoming necessary to go beyond simple object recognition and determine what situation the monitored results, including object recognition, signify. The problem to be solved

[0009] The problem that the present invention aims to solve is to provide a classification device and method that enables more accurate and rapid situation recognition, going beyond mere object recognition, by generating graph data for people and objects respectively from input data, combining each generated graph data to create a graph sequence, and classifying the input data using the same.

[0010] Another problem that the present invention aims to solve is to provide a classification device and method capable of recognizing various types of situations in security and surveillance environments (e.g., public place surveillance, border security, etc.), healthcare environments, cooking environments, sports analysis environments, environments for assisting the disabled, marketing environments, automotive safety environments, entertainment environments (e.g., virtual reality), educational environments, etc., by combining heterogeneous graph data regarding people and objects to generate a graph sequence and utilizing it.

[0011] The objects of the present invention are not limited to those mentioned above, and other unmentioned objects and advantages of the present invention may be understood from the following description and will be more clearly understood by the embodiments of the present invention. Furthermore, it will be readily apparent that the objects and advantages of the present invention can be realized by the means and combinations thereof set forth in the claims. means of solving the problem

[0013] A classification device according to some embodiments of the present invention may include a data collection module that receives input data, a graph module that extracts features from each person and a predefined object included in the input data and generates a graph sequence based on the extracted features, and a classification module that generates a classification result for the graph sequence.

[0014] In addition, the above input data includes video data, and the video data may include a plurality of frames.

[0015] Additionally, the graph module may include a person graph generation unit that generates a person graph by extracting person features from a person present in any one input frame included in the input data, an object graph generation unit that generates an object graph by extracting object features from a predefined object included in the input frame, and a combination unit that generates a combined graph by combining the person graph and the object graph, and generates the graph sequence for the input data by connecting the combined graph according to a predefined window size.

[0016] Additionally, the person feature includes a predetermined number of person key points existing at predefined body locations in the person, and the object feature includes a predetermined number of object key points existing at predefined object locations in the object, and the person graph generation unit sets the person key points as person nodes and determines the person bounding box determined based on the person nodes, person edges connecting the person nodes, and the outermost edges of a plurality of person nodes as the person graph, and the object graph generation unit sets the object key points as object nodes and determines the object bounding box determined based on the object nodes, object edges connecting the object nodes, and the outermost edges of a plurality of object nodes as the object graph.

[0017] In addition, the person graph generation unit generates the person graph by extracting the person features using a predefined graph generation algorithm, and the graph generation algorithm may include a CNN (Convolutional Neural Network)-based YOLO (You Only Look Once) algorithm.

[0018] In addition, the graph module further includes a learning unit that generates an adaptive algorithm by additionally training the graph generation algorithm, and the object graph generation unit can generate the object graph by extracting the object features using the adaptive algorithm.

[0019] In addition, the above-mentioned combining unit can generate an integrated bounding box based on the person bounding box and the object bounding box, and generate the combined graph by post-processing the person graph and the object graph based on the generated integrated bounding box.

[0020] In addition, the above-mentioned coupling part can normalize the position of the person node in the person graph and the position of the object node in the object graph based on the integrated bounding box.

[0021] In addition, the above-mentioned connecting part can set an auxiliary edge connecting any one of the person nodes included in the person graph and any one of the object nodes included in the object graph based on the above-mentioned integrated bounding box.

[0022] Additionally, the classification module generates a classification result regarding whether the input data belongs to one of a plurality of predefined classes by inputting the graph sequence into a pre-trained classification algorithm, and the classification algorithm may include an algorithm based on LSTM (Long Short Term Memory). Effects of the invention

[0024] A classification device and method according to some embodiments of the present invention generate graph data for each person and object from input data, combine each generated graph data to generate a graph sequence, and classify the input data using the same, thereby enabling more accurate and rapid situation recognition beyond just object recognition.

[0025] In addition, the classification device and method according to some embodiments of the present invention classify input data using a graph sequence generated by combining graph data for each person and object, thereby enabling accurate and rapid determination and identification of dangerous situations, etc., and thus overcoming the cost and technical limitations existing in conventional technology.

[0026] In addition, the classification device and method according to some embodiments of the present invention can be widely utilized in environments requiring various types of situational awareness (situational awareness environments), such as security and surveillance environments (e.g., public place surveillance, border security, etc.), healthcare environments, cooking environments, sports analysis environments, environments for assisting the disabled, marketing environments, automotive safety environments, entertainment (e.g., virtual reality) environments, and educational environments.

[0027] In addition to the above, the specific effects of the present invention are described together with the specific details for implementing the invention below. Brief explanation of the drawing

[0029] FIG. 1 illustrates a classification system according to some embodiments of the present invention. FIG. 2 is a block diagram of a classification device according to some embodiments of the present invention. FIG. 3 is a flowchart of a classification method according to some embodiments of the present invention. FIG. 4 is a drawing for explaining a classification device and a classification method according to some embodiments of the present invention. FIG. 5 is a drawing for explaining a figure graph according to some embodiments of the present invention. FIGS. 6a and FIGS. 6b are drawings for illustrating object graphs according to some embodiments of the present invention. FIG. 7 is a drawing for illustrating a combination graph according to some embodiments of the present invention. FIGS. 8A and FIGS. 8B are drawings for explaining the operation of a classification module according to some embodiments of the present invention. FIG. 9 is a diagram illustrating the hardware implementation of a classification device that performs a classification method according to some embodiments of the present invention. Specific details for implementing the invention

[0030] Terms and words used in this specification and claims shall not be interpreted as being limited to their general or dictionary meanings. In accordance with the principle that an inventor may define the concept of a term or word to best describe their invention, they shall be interpreted in a meaning and concept consistent with the technical spirit of the invention. Furthermore, since the embodiments described in this specification and the configurations illustrated in the drawings are merely one embodiment of the invention and do not represent the entire technical spirit of the invention, it should be understood that various equivalents, modifications, and applicable examples capable of replacing them may exist at the time of filing this application.

[0031] The terms first, second, A, B, etc., as used in this specification and claims may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0032] The terms used in this specification and claims are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" should be understood as not precluding the existence or addition of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification.

[0033] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which this invention pertains.

[0034] Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with their meanings in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.

[0035] In addition, each component, process, procedure, or method included in each embodiment of the present invention may be shared within a scope that is not technically contradictory to one another.

[0036] Hereinafter, with reference to FIGS. 1 to 9, a classification device and method according to some embodiments of the present invention and a system including the same will be described in detail.

[0038] FIG. 1 illustrates a classification system according to some embodiments of the present invention.

[0039] Referring to FIG. 1, the classification system (1) may include a data input device (100), a classification device (200), and a communication network (300).

[0040] The data input device (100) may be a device that transmits input data to a classification device (200).

[0041] For example, a data input device (100) can transmit input data to a classification device (200). The input data may be target data for which a classification operation is performed by the classification device (200). For example, the input data may include video data, but embodiments of the present invention are not limited thereto.

[0042] The data input device (100) may include a user terminal (101), an external database (102), etc. However, the embodiments of the present invention are not limited thereto, and it is obvious that the data input device (100) may include many more types of objects. The user terminal (101) may be in the form of various types of electronic devices such as smartphones, computers, laptop PCs, and wearable devices, workstations, data centers, internet data centers (IDCs), direct attached storage (DAS) systems, storage area network (SAN) systems, network attached storage (NAS) systems, and redundant array of inexpensive disks, or redundant array of independent disks (RAID) systems, but the embodiments of the present invention are not limited thereto. Additionally, the external database (102) may be in the form of a workstation, data center, internet data center (IDC), direct attached storage (DAS) system, storage area network (SAN) system, network attached storage (NAS) system, and redundant array of inexpensive disks, or redundant array of independent disks) system, but the embodiments of the present invention are not limited thereto.

[0043] The classification device (200) is a device that performs classification work on input data.

[0044] For example, the classification device (200) can classify input data transmitted from the data input device (100) into one of a plurality of predefined classes to generate a classification result. In other words, the classification device (200) can perform situational awareness on the input data to generate and output a classification result indicating what kind of situation the input data is in.

[0045] At this time, the classification device (200) can perform a classification task based on a neural network. In other words, the classification device (200) can perform a classification task on input data using a neural network model that has been pre-trained based on a neural network. At this time, the neural network model may include a graph generation algorithm, an adaptive algorithm, a classification algorithm, etc., as described below.

[0046] To explain in detail, the neural network model used by the classification device (200) can be trained based on machine learning. To explain in more detail, deep learning technology, which is a type of machine learning, learns by going down to a deep level in multiple stages based on data. That is, deep learning represents a set of machine learning algorithms that extract key data from multiple data by increasing the stages.

[0047] As a few examples, neural network models can utilize various known deep learning structures. For instance, neural network models can utilize structures such as CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), DBN (Deep Belief Network), GNN (Graph Neural Network), GAN (Generative Adversarial Network), Transformer, and Autoencoder.

[0048] Specifically, a Convolutional Neural Network (CNN) is a model that mimics the function of the human brain, based on the assumption that when humans recognize an object, they extract its basic features, perform complex calculations within the brain, and then recognize the object based on the results. CNNs may include well-known structures such as LeNet, AlexNet, VGGNet, GoogleNet, and ResNet, but are not limited to them.

[0049] Recurrent Neural Networks (RNNs) are widely used in natural language processing and are an effective structure for processing time-series data that changes over time; they can be constructed by stacking layers at every moment.

[0050] A Deep Belief Network (DBN) is a deep learning structure constructed by stacking Restricted Boltzmann Machines (RBMs), a deep learning technique, in multiple layers. When the Restricted Boltzmann Machine (RBM) training is repeated until a certain number of layers are reached, a Deep Belief Network (DBN) with that number of layers can be constructed.

[0051] A Graphic Neural Network (GNN) represents an artificial neural network structure implemented by deriving similarities and feature points between modeling data using modeling data modeled based on data mapped between specific parameters.

[0052] A Generative Adversarial Network (GAN) represents an artificial neural network structure that uses a generative neural network and a discriminative neural network to generate new data in a form similar to input data. GANs may include known DCGAN (Deep Convolutional GAN), CGAN (Conditional GAN), WGAN (Wasserstein GAN), StyleGAN (Style-Based GAN), CycleGAN, etc., but embodiments of the present invention are not limited thereto.

[0053] The Transformer is an artificial neural network with an encoder-decoder structure utilizing attention, capable of grasping the overall meaning between input and output sequences. By employing an attention mechanism, the Transformer ensures that every element of the input sequence influences the output sequence, allowing both the encoder and decoder to consider the entire sequence. The Transformer can use natural language and time-series data, as well as patched images, as input.

[0054] An autoencoder is a deep learning architecture that performs the role of extracting and reconstructing data features. Typically, an autoencoder includes an encoder that compresses input values ​​and a decoder that restores the compressed data. The encoder transforms input values ​​into low-dimensional latent representations, while the decoder restores the latent representations to the same dimension as the input values. In this process, both the encoder and decoder can be composed of Multilayer Perceptrons (MLPs). When training an autoencoder, input data is used, and weights and biases are trained to minimize the difference between the output and input values. An autoencoder trained in this way can effectively extract features from input data and restore noisy input data. Autoencoders are primarily utilized in fields such as data compression, dimensionality reduction, noise removal, and data generation; they can also be applied in areas such as image recognition, natural language processing, and speech recognition.

[0055] Meanwhile, artificial neural network training of neural network models can be achieved by adjusting the weights of the connections between nodes (and adjusting bias values ​​if necessary) to produce a desired output for a given input. Additionally, artificial neural networks can continuously update weight values ​​through learning. Furthermore, methods such as backpropagation can be used for the training of artificial neural networks.

[0056] In this case, machine learning methods for artificial neural networks, such as unsupervised learning, semi-supervised learning, and supervised learning, can be used. Additionally, the neural network model can be controlled to automatically update the artificial neural network structure to output analysis data after training, depending on the settings.

[0057] The specific process of the classification device (200) performing a classification task on input data based on this neural network model will be described later.

[0058] The communication network (300) refers to a communication means that performs data exchange between a data input device (100) and a classification device (200).

[0059] At this time, the communication network (300) may include a network based on wired internet technology, wireless internet technology, and short-range communication technology. Wired internet technology may include, for example, at least one of a local area network (LAN) and a wide area network (WAN). Wireless internet technology may include, for example, at least one of Wireless LAN (WLAN), DMNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), IEEE 802.16, Long Term Evolution (LTE), LTE-A (Long Term Evolution-Advanced), Wireless Mobile Broadband Service (WMBS), and 5G NR (New Radio) technology. However, the present embodiment is not limited thereto. Short-range communication technologies may include, for example, at least one of Bluetooth, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), Ultra-Wideband (UWB), ZigBee, Near Field Communication (NFC), Ultra Sound Communication (USC), Visible Light Communication (VLC), Wi-Fi, Wi-Fi Direct, and 5G NR (New Radio).However, the present embodiment is not limited thereto.

[0060] Hereinafter, a classification device (200) according to several embodiments of the present invention will be described in more detail with reference to FIGS. 2 to 9. Also, for convenience of explanation, the description will be made by assuming that the situational awareness environment of the classification device (200) is "gun detection." However, this is only for convenience of explanation, and the situational awareness environment of the classification device (200) can be expanded in various ways as described above, such as "other security and surveillance environments excluding gun detection (e.g., public place surveillance, etc.), healthcare environment, cooking environment, sports analysis environment, disability assistance environment, marketing environment, automotive safety environment, entertainment (e.g., virtual reality) environment, education environment, etc." and the person keypoints, object keypoints, etc. extracted according to each situational awareness environment may be modified.

[0062] FIG. 2 is a block diagram of a classification device according to some embodiment of the present invention. FIG. 3 is a flowchart of a classification method according to some embodiment of the present invention. FIG. 4 is a diagram illustrating a classification device and a classification method according to some embodiment of the present invention. Each step (S100 to S300) of FIG. 3 can be performed by the classification device (200) of FIG. 1 and FIG. 2.

[0063] Referring to FIGS. 1 to 4, a classification device (200) according to some embodiments of the present invention is a device that generates a classification result (hereinafter referred to as "CR") for input data (hereinafter referred to as "ID"), and may include a data collection module (210), a graph module (220), and a classification module (230).

[0064] To explain the operation of each component included in the classification device (200) in more detail, first, the data collection module (210) can receive input data (ID) (S100).

[0065] The input data (ID) may be target data for which a classification task is performed by a classification device (200). In some examples, the input data (ID) may include video data. In other words, the input data (ID) may include a plurality of frames composed of image data. However, embodiments of the present invention are not limited thereto.

[0066] Next, the graph module (220) can generate a graph sequence (hereinafter referred to as "GS") by extracting features from each person and object included in the input data (ID) (S200).

[0067] For some examples, the graph module (220) may include a person graph generation unit (221), an object graph generation unit (222), a learning unit (223), and a combination unit (224), and may generate a graph sequence (GS) from input data (ID) using the same.

[0068] At this time, the input data (ID) may be in the form of video data having multiple frames. The graph module (220) can generate a combination graph (hereinafter referred to as "CG") for each of the multiple frames of the input data (ID) (hereinafter referred to as "input frames (ID_F)") and then generate a graph sequence (GS) by connecting them. Below, the process of generating a combination graph (CG) for each input frame (ID_F) will be explained first, and then the process of generating a graph sequence (GS) by connecting the combination graphs (CG) will be explained.

[0069] More specifically, the people graph generation unit (221) can generate a people graph (hereinafter referred to as "G1") based on an input frame (ID_F). At this time, the people graph (G1) may have the form of predefined skeleton data used for motion capture, human body motion analysis, etc.

[0070] For example, a person graph generation unit (221) can extract person features from an input frame (ID_F) using a predefined graph generation algorithm (hereinafter referred to as "GGA") and generate a person graph (G1) based on the extracted person features. The person features may include a predetermined number of person key points existing at predefined body locations (e.g., wrist, shoulder, head, etc.) in the person. In other words, the person features may include a predetermined number of person key points extracted from predefined locations in the human body. At this time, the graph generation algorithm (GGA) may include the aforementioned CNN (Convolutional Neural Network)-based YOLO (You Only Look Once) algorithm, but the embodiments of the present invention are not limited thereto.

[0071] For example, the person graph generation unit (221) can extract person keypoints as person features from an input frame (ID_F) through a graph generation algorithm (GGA), set the extracted person keypoints as person nodes, set a person bounding box determined based on person edges connecting the person nodes and the outermost edges of a plurality of person nodes, and determine the set person nodes, person edges, and person bounding box as a person graph (G1).

[0072] Hereinafter, with reference to FIG. 5, an example of a person graph (G1) generated by a person graph generation unit (221) according to some embodiments of the present invention will be described.

[0074] FIG. 5 is a drawing for explaining a figure graph according to some embodiments of the present invention.

[0075] Referring to FIGS. 1 to 5, first, the person graph generation unit (221) can extract a person key point (Key Point of People, hereinafter referred to as "KP1") as a person feature from an input frame (ID_F) through a graph generation algorithm (GGA). In other words, the person graph generation unit (221) can identify a person included in the input frame (ID_F) using a graph generation algorithm (GGA), search for a person key point (KP1) from the person, and determine the searched person key point (KP1) as a person feature.

[0076] At this time, as shown in FIG. 5, the location of the person key point (KP1) can be predefined. That is, as shown in FIG. 5, the location of the person key point (KP1) can be predefined as the nose, left eye, right eye, left wrist, right wrist, etc. Although a total of 17 person key points (KP1) are defined in FIG. 5, it is obvious that the embodiments of the present invention are not limited thereto.

[0077] Next, the person graph generation unit (221) can set the extracted person keypoint (KP1) as a person node, set person edges connecting the person nodes, and set a person bounding box (hereinafter referred to as "BB1") determined based on the outermost edge of a plurality of person nodes. At this time, the person bounding box (BB1) can be set at the outermost edge of the person node or in an area spaced apart from the outermost edge by a predetermined distance.

[0078] Next, the person graph generation unit (221) can determine the person node (person keypoint (KP1)), the person edge, and the person bounding box (BB1) as the person graph (G1). At this time, the person graph generation unit (221) can determine the coordinates of each person node (person keypoint (KP1)) based on the person bounding box (BB1). That is, the person graph generation unit (221) can determine the relative coordinates of each person node (person keypoint (KP1)) based on an arbitrary reference point (e.g., the top-left corner) of the person bounding box (BB1). For example, the person graph generation unit (221) can determine the relative coordinates of each person node (person keypoint (KP1)) through a method similar to the method in which the object graph generation unit (222), as described later in FIG. 6b, defines the coordinates of each object node.

[0080] Referring again to FIGS. 1 to 4, the object graph generation unit (222) can generate an object graph (hereinafter referred to as "G2") based on an input frame (ID_F). At this time, the object graph (G2) may have the form of predefined skeleton data, similar to the person graph (G1) described above.

[0081] For example, the object graph generation unit (222) can extract object features from an input frame (ID_F) using an adaptive type of graph generation algorithm (hereinafter referred to as "GGA_AF") and generate an object graph (G2) based on the extracted object features. The object features may include a predetermined number of object key points existing at predefined locations (e.g., handle, center axis, opening, etc.) in the object. In other words, the object features may include a predetermined number of object key points extracted from predefined locations in the object.

[0082] For example, the object graph generation unit (222) can extract object keypoints as object features from an input frame (ID_F) through an adaptive algorithm (GGA_AF), set the extracted object keypoints as object nodes, set object bounding boxes determined based on object edges connecting object nodes and the outermost edges of a plurality of object nodes, and determine the set object nodes, object edges, and object bounding boxes as an object graph (G2).

[0083] At this time, the adaptive algorithm (GGA_AF) may be an algorithm to which the graph generation algorithm (GGA), which is the aforementioned CNN-based YOLO algorithm, has been additionally trained. That is, the learning unit (223) can generate the adaptive algorithm (GGA_AF) by additionally training the graph generation algorithm (GGA), and the object graph generation unit (222) can receive and use this adaptive algorithm (GGA_AF) from the learning unit (223). More specifically, the adaptive algorithm (GGA_AF) can be additionally trained to extract object keypoints at predefined locations based on the neural network structure of the graph generation algorithm (GGA). For example, the adaptive algorithm (GGA_AF) can be trained through a supervised learning method in which, upon receiving training input data in the "Learning Phase," it processes the training input data and outputs training object keypoints. That is, training object keypoints provided by the manager of the classification device (200) can serve as correct answer data (Labeling Data) in the supervised learning method. The adaptive algorithm (GGA_AF) that has undergone such additional learning can process the input frame (ID_F) and output object keypoints when it receives the input frame (ID_F) in the "Inference Phase".

[0084] Hereinafter, with reference to FIGS. 6a and 6b, an example of an object graph (G2) generated by an object graph generation unit (222) according to some embodiment of the present invention will be described.

[0086] FIGS. 6a and 6b are drawings for illustrating an object graph according to some embodiments of the present invention. Specifically, FIG. 6a illustrates an object key point (Key Point of Object, hereinafter referred to as "KP2") and an object graph (G2) based on the object key point (KP2), and FIG. 6b illustrates the relative coordinates of a plurality of object key points (KP2_P1, KP2_P2, KP2_P3) according to the angles ((a), (b), (c), (d)) of the object.

[0087] Referring to FIGS. 1 to 4, FIGS. 6a, and FIGS. 6b, first, the object graph generation unit (222) can extract object keypoints (KP2) as object features from an input frame (ID_F) through an adaptive algorithm (GGA_AF). In other words, the object graph generation unit (222) can identify an object included in the input frame (ID_F) using an adaptive algorithm (GGA_AF), search for object keypoints (KP2) from the object, and determine the searched object keypoints (KP2) as object features.

[0088] At this time, as illustrated in FIG. 6a, the location of the object key point (KP2) can be predefined. That is, as illustrated in FIG. 6a, the location of the object key point (KP2) can be predefined as a grip (P1), the upper part of the grip (P2), the muzzle (P3), etc. Although a total of three object key points (KP2) are defined in FIG. 6a, it is obvious that the embodiments of the present invention are not limited thereto.

[0089] Next, the object graph generation unit (222) can set the extracted object keypoint (KP2) as an object node, set object edges connecting the object nodes, and set an object bounding box (hereinafter referred to as "BB2") determined based on the outermost edge of a plurality of object nodes. At this time, the object bounding box (BB2) can be set at the outermost edge of the object node or in an area spaced apart from the outermost edge by a predetermined distance.

[0090] Next, the object graph generation unit (222) can determine the object node (object key point (KP2)), object edge, and object bounding box (BB2) as the object graph (G2).

[0091] At this time, the object graph generation unit (222) can define the coordinates of each object node in the object graph (G2). For example, the object graph generation unit (222) can determine the coordinates of each object node (object key point (KP2)) based on the object bounding box (BB2). That is, the object graph generation unit (222) can determine the relative coordinates of each object node (object key point (KP2)) based on an arbitrary reference point (e.g., the top-left corner) of the object bounding box (BB2). To explain with reference to FIG. 6b, the object graph generation unit (222) can set an arbitrary point of the object bounding box (BB2) as a reference point (hereinafter referred to as "RP") and determine the relative coordinates of each object node (object key point (KP2)) based on the set reference point (RP). That is, the object graph generation unit (222) can determine the distance between each object node (object key point (KP2_P1, KP2_P2, KP2_P3)) and the x-axis and y-axis respectively based on the reference point (RP), and determine the relative coordinates by measuring the determined distance based on the horizontal length (w) and vertical length (h) of the object bounding box (BB2).

[0093] Referring again to FIGS. 1 to 4, the coupling part (224) can generate a graph sequence (GS) based on a person graph (G1) and an object graph (G2).

[0094] In some examples, the combining unit (224) can combine a person graph (G1) and an object graph (G2) to create a combined graph (CG), and can create a graph sequence (GS) by connecting the combined graphs (CG) according to a predefined window size.

[0095] More specifically, first, the combining unit (224) can combine a person graph (G1) and an object graph (G2) generated based on any one of the input frames (ID_F) as described above to generate a combined graph (CG) for the input frame (ID_F).

[0096] Hereinafter, a combined graph (CG) according to some embodiments of the present invention will be described in more detail with reference to FIG. 7.

[0098] FIG. 7 is a drawing for illustrating a combination graph according to some embodiments of the present invention.

[0099] Referring to FIGS. 1 to 4 and FIG. 7, the combining part (224) can combine a person graph (G1) and an object graph (G2) to generate a combined graph (CG).

[0100] More specifically, first, the coupling unit (224) can generate a Union Bounding Box (hereinafter referred to as "BB_U") based on the person bounding box (BB1) included in the person graph (G1) and the object bounding box (BB2) included in the object graph (G2). As a few examples, the coupling unit (224) can set a rectangular Union Bounding Box (BB_U) along the outermost edges of the person bounding box (BB1) and the object bounding box (BB2), as shown in FIG. 7. In other words, the coupling unit (224) can determine a rectangular shape of the smallest size that includes both the person bounding box (BB1) and the object bounding box (BB2) as the Union Bounding Box (BB_U).

[0101] Next, the combination unit (224) can post-process the person graph (G1) and object graph (G2) based on the generated integrated bounding box (BB_U).

[0102] For example, the joining part (224) can normalize the position of a person node (person keypoint (KP1)) included in the person graph (G1) and the position of an object node (object keypoint (KP2)) included in the object graph (G2) based on the integrated bounding box (BB_U). That is, as described above, the coordinates of each node (person node, object node) in the person graph (G1) and the object graph (G2) are pre-set. Since the coordinates in these individual graphs are set based on the bounding box in each individual graph (person bounding box (BB1) in the case of the person graph (G1), and object bounding box (BB2) in the case of the object graph (G2)), a bias may occur in the pre-set coordinate values ​​in the integrated bounding box (BB_U). Accordingly, the combining unit (224) can perform a post-processing process of redefining the coordinates of the person node (person keypoint (KP1)) included in the person graph (G1) and the coordinates of the object node (object keypoint (KP2)) included in the object graph (G2) based on the integrated bounding box (BB_U). As a few examples, the combining unit (224) can normalize the coordinates of the object node (person keypoint (KP1)) and the object node (object keypoint (KP2)) based on an arbitrary point (e.g., the top-left vertex) of the integrated bounding box (BB_U).

[0103] As another example, the connecting part (224) can set an auxiliary edge connecting any one of the person nodes (person keypoints (KP1)) included in the person graph (G1) and any one of the object nodes (object keypoints (KP2)) included in the object graph (G2) based on the integrated bounding box (BB_U). In some examples, the connecting part (224) can set an auxiliary edge between an object node (hereinafter referred to as the "first node") corresponding to an object keypoint (KP2_P1 in FIG. 6b) for a grip (P1 in FIG. 6a) in an object, and a person node (hereinafter referred to as the "second node") corresponding to a person keypoint (KP1 with a Keypoint Index of 9 in FIG. 5) for a left wrist (Left Wrist in FIG. 5) in a person, or a person node (hereinafter referred to as the "third node") corresponding to a person keypoint (KP1 with a Keypoint Index of 10 in FIG. 5) for a right wrist (Right Wrist in FIG. 5). At this time, the connecting part (224) can calculate the distance between the second node and the third node, respectively, based on the first node, and set an auxiliary edge connecting the node with the smaller calculated distance to the first node. At this time, the separation distance may include the Euclidean distance, Manhattan distance, Minkowski distance, Chebyshev distance, etc., but the embodiments of the present invention are not limited thereto.

[0104] Next, when this post-processing is completed, the combining unit (224) can calculate the node parameters (hereinafter referred to as "NP") of each node, namely the person node (person keypoint (KP1)) and the object node (object keypoint (KP2)), and determine the combining graph (CG) based on the calculated node parameters (NP). At this time, the node parameters (NP) may represent the distance characteristics of each node expressed as a vector based on the integrated bounding box (BB_U). For example, the combining unit (224) can determine the node parameters (NP) of each node through the following <Equation 1>. At this time, <Equation 1> below assumes a case where 17 person nodes are set and 3 object nodes are set for convenience of explanation, as described above through FIGS. 5 to 7.

[0105] <Mathematical Formula 1>

[0106]

[0107] At this time, represents the node parameter (NP) of the i-th node in the set of person nodes (person keypoints (KP1)) and object nodes (object keypoints (KP2)), and represents the normalized x-axis coordinate of the corresponding i-th node, and represents the normalized y-axis coordinate of the corresponding i-th node, and represents the distance between the normalized coordinates of the i-th node and the normalized coordinates of the remaining nodes.

[0108] When the node parameters (NP) of each node are determined through this process, the combining unit (224) can generate a combined graph (CG) by combining the node parameters (NP) of each node. At this time, the combining unit (224) can generate the combined graph (CG) by inputting the node parameters (NP) of each node into a known combining algorithm, etc.

[0110] Referring again to FIGS. 1 to 4, subsequently, the coupling part (224) can generate a graph sequence (GS) by connecting the coupling graph (CG) according to a predefined window size (Window size = k).

[0111] In other words, the combined graph (CG) described above through FIGS. 5 to 7 is generated based on an input frame (ID_F), which is one frame of the input data (ID) in the form of video data having multiple frames, and the combining unit (224) can generate a graph sequence (GS) for the input data (ID) by connecting the combined graph (CG) generated for each of the multiple input frames (ID_F).

[0113] Next, the classification module (230) can generate a classification result (CR) for the graph sequence (GS) (S300).

[0114] For example, the classification module (230) can generate a classification result (CR) by inputting a graph sequence (GS) into a pre-trained classification algorithm (hereinafter referred to as "CA"). In this case, the classification result (CR) may be data regarding which of a plurality of pre-defined classes the input data (ID) belongs to.

[0115] At this time, the classification algorithm (CA) may include an algorithm based on LSTM (Long Short Term Memory). Hereinafter, the classification process using the classification algorithm (CA) of the classification module (230) will be explained in more detail with reference to FIGS. 8a and 8b.

[0117] FIGS. 8a and 8b are drawings for explaining the operation of a classification module according to some embodiments of the present invention. More specifically, FIG. 8a illustrates a plurality of classes (Class, hereinafter referred to as "CL"), and FIG. 8b illustrates a process of classifying a graph sequence (GS) into one of the plurality of classes (CL).

[0118] Referring to FIGS. 1 to 4, FIGS. 8a, and FIG. 8b, the classification algorithm (CA) can generate a classification result (CR) regarding which of the plurality of classes (CL) a graph sequence (GS) corresponds to. FIGS. 8a and FIG. 8b are illustrated as including a first class (CL1) regarding a standing situation, a second class (CL2) regarding a preparation situation, and a third class (CL3) regarding an aiming situation. However, it is obvious that embodiments of the present invention are not limited thereto.

[0119] To explain in detail, first, the classification algorithm (CA) can receive a graph sequence (GS) and perform dropout on the graph sequence (GS) using a dropout layer that randomly excludes some nodes to prevent overfitting of the model.

[0120] Next, the classification algorithm (CA) can apply the Graph Attention Mechanism (GAT) to the graph sequence (GS) through the Graph Attention Mechanism (GAT) layer. In other words, the classification algorithm (CA) can apply the attention mechanism to each node of the graph sequence (GS) to calculate the importance of each node and update features based on this.

[0121] Next, the classification algorithm (CA) can add non-linearity to the graph sequence (GS) using the Exponential Linear Unit (ELU) activation function and apply an activation function that includes negative values.

[0122] Next, the classification algorithm (CA) can apply the aforementioned Dropout and GAT again to the graph sequence (GS).

[0123] Next, the classification algorithm (CA) can extract spatial and temporal features from the graph sequence (GS) through an LSTM layer to model the temporal dependency of the time series data. As a few examples, the classification algorithm (CA) can extract spatial and temporal features by inputting the features of the graph sequence (GS) at each time step into the LSTM.

[0124] Next, the classification algorithm (CA) calculates the probability that the graph sequence (GS) belongs to each class through a fully connected layer (FC Layer) and normalizes the probability using a softmax function, thereby determining which class the graph sequence (GS) belongs to among the first class (CL1) regarding standing situations, the second class (CL2) regarding preparation situations, and the third class (CL3) regarding aiming situations, and can determine the result of the determination as the classification result (CR). Figure 8b illustrates, as an example, the result of the classification algorithm (CA) classifying the graph sequence (GS) into the second class (CL2) regarding preparation situations.

[0126] FIG. 9 is a diagram illustrating the hardware implementation of a classification device that performs a classification method according to some embodiments of the present invention.

[0127] Referring to FIGS. 1, 2 and 9, a classification device (200) according to some embodiments of the present invention may be implemented as an electronic device (1000). The electronic device (1000) may include a controller (1010), an input / output device (1020, I / O), a memory device (1030, memory device), an interface (1040, interface), and a bus (1050, bus). The controller (1010), the input / output device (1020), the memory device (1030), and / or the interface (1040) may be connected to each other through the bus (1050). In this case, the bus (1050) corresponds to a path through which data is moved.

[0128] Specifically, the controller (1010) may include at least one of a CPU (Central Processing Unit), MPU (Micro Processor Unit), MCU (Micro Controller Unit), GPU (Graphic Processing Unit), microprocessor, digital signal processor, microcontroller, application processor (AP), and logic elements capable of performing similar functions.

[0129] The input / output device (1020) may include at least one of a keypad, a keyboard, a touchscreen, and a display device.

[0130] The memory device (1030) can store data and / or programs, etc.

[0131] The interface (1040) can perform the function of transmitting data to a communication network or receiving data from a communication network. The interface (1040) may be wired or wireless. For example, the interface (1040) may include an antenna or a wired / wireless transceiver, etc. Although not illustrated, the memory device (1030) may further include high-speed DRAM and / or SRAM, etc. as an operational memory to enhance the operation of the controller (1010). The memory device (1030) may store a program or application internally.

[0132] A classification device (200) according to embodiments of the present invention may be a system formed by connecting a plurality of electronic devices (1000) to each other through a network. In such cases, each module or combination of modules may be implemented as an electronic device (1000). However, the present embodiment is not limited thereto.

[0133] Additionally, the classification device (200) may be implemented as at least one of a workstation, a data center, an internet data center (IDC), a direct attached storage (DAS) system, a storage area network (SAN) system, a network attached storage (NAS) system, a redundant array of inexpensive disks, or a redundant array of independent disks) system, and an Electronic Document Management (EDMS) system, but the present embodiment is not limited thereto.

[0134] Additionally, the classification device (200) can transmit data to the data input device (100) through a network. The network may include a network based on wired internet technology, wireless internet technology, and local area communication technology. Wired internet technology may include, for example, at least one of a local area network (LAN) and a wide area network (WAN).

[0135] Wireless internet technology may include, for example, at least one of Wireless LAN (WLAN), DMNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), IEEE 802.16, Long Term Evolution (LTE), LTE-A (Long Term Evolution-Advanced), Wireless Mobile Broadband Service (WMBS), and 5G NR (New Radio) technology. However, the present embodiment is not limited thereto.

[0136] Short-range communication technology may include, for example, at least one of Bluetooth, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), Ultra-Wideband (UWB), ZigBee, Near Field Communication (NFC), Ultra Sound Communication (USC), Visible Light Communication (VLC), Wi-Fi, Wi-Fi Direct, and 5G NR (New Radio). However, the present embodiment is not limited thereto.

[0137] A classification device (200) communicating through a network may comply with technical standards and standard communication methods for mobile communication. For example, a standard communication method may include at least one of GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), CDMA2000 (Code Division Multi Access 2000), EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), WCDMA (Wideband CDMA), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTEA (Long Term Evolution-Advanced), and 5G NR (New Radio). However, the present embodiment is not limited thereto.

[0138] The above description is merely an illustrative explanation of the technical concept of the present embodiment, and a person skilled in the art to which the present embodiment belongs would be able to make various modifications and variations within the scope of the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain, not limit, the technical concept of the present embodiment, and the scope of the technical concept of the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of the present embodiment.

Claims

Claim 1 A data collection module that receives input data; a graph module that extracts features from each person and predefined object included in the input data, and generates a graph sequence based on the extracted features; The graph module comprises a classification module that generates a classification result for the graph sequence, wherein the graph module includes: a person graph generation unit that extracts a predetermined number of person key points existing at predefined body locations as person features from a person existing in any one input frame included in the input data, sets the person key points as person nodes, and determines a person bounding box determined based on the person nodes, person edges connecting the person nodes, and the outermost edges of a plurality of person nodes as a person graph; an object graph generation unit that extracts a predetermined number of object key points existing at predefined object locations from a predefined object included in the input frame as object features, sets the object key points as object nodes, and determines an object bounding box determined based on the object nodes, object edges connecting the object nodes, and the outermost edges of a plurality of object nodes as an object graph; and a combination unit that generates a combined graph by combining the person graph and the object graph, and generates the graph sequence for the input data by connecting the combined graph according to a predefined window size. A classification device comprising, wherein the coupling part generates an integrated bounding box based on the person bounding box and the object bounding box, and generates the combined graph by post-processing the person graph and the object graph based on the generated integrated bounding box. Claim 2 A classification device according to claim 1, wherein the input data includes video data, and the video data includes a plurality of frames. Claim 3 delete Claim 4 delete Claim 5 In claim 1, the person graph generation unit generates the person graph by extracting the person features using a predefined graph generation algorithm, and the graph generation algorithm includes a CNN (Convolutional Neural Network)-based YOLO (You Only Look Once) algorithm, forming a classification device. Claim 6 In claim 5, the graph module further includes a learning unit that generates an adaptive algorithm by additionally learning the graph generation algorithm, and the object graph generation unit generates the object graph by extracting the object features using the adaptive algorithm. Claim 7 delete Claim 8 In claim 1, the coupling part is a classification device that normalizes the position of the person node of the person graph and the position of the object node of the object graph based on the integrated bounding box. Claim 9 In claim 1, the coupling part is a classification device that sets an auxiliary edge connecting any one of the person nodes included in the person graph and any one of the object nodes included in the object graph based on the integrated bounding box. Claim 10 In claim 1, the classification module generates a classification result regarding whether the input data belongs to a plurality of predefined classes by inputting the graph sequence to a pre-trained classification algorithm, and the classification algorithm includes an algorithm based on LSTM (Long Short Term Memory).

Images