Learning device, learning method, behavior recognition device, and behavior recognition method

Abstract

A technology for efficiently generating an action recognizer that recognizes relationships between people and objects is disclosed. One aspect of the present disclosure relates to a learning device including: a training data acquisition unit that acquires a training dataset including training videos, training descriptions that describe relationships between people and objects or between people, and action class labels of the people with respect to the objects in the training videos; a keypoint detection unit that detects keypoints between the people and the objects from the training videos; and a learning unit that trains an action recognizer that recognizes the actions of an agent of the person with respect to the objects or people, based on the keypoints, the training descriptions, and the action class labels.

Description

[Technical Field]

[0001] The present disclosure relates to a learning device, a learning method, an action recognition device, and an action recognition method. [Background technology]

[0002] Recent advances in deep learning technology have expanded the opportunities for using machine learning models. For example, action recognition models trained to recognize human actions from videos containing people, objects, and other objects are now being used, making it possible to recognize what people are doing in videos captured by video cameras.

[0003] Human actions often involve relationships between the subject and the objects and people around them. Human Object Interaction (HOI) recognizes the relationships between people and objects in a video, such as a person holding a shoe and putting their foot in, or a person sitting on a bed and bending over. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] “Generative Action Description Prompts for Skeleton-based Action Recognition”, ICCV 2023. [Non-patent document 2] “Multi-Modality Co-Learning for Efficient Skeleton-based Action Recognition”, https: / / arxiv.org / abs / 2407.15706. [Non-patent document 3] “Addressing missing part interaction in skeleton-text contrastive learning for action recognition”, SSRN Working Paper. https: / / ssrn.com / abstract=4868542. [Non-patent document 4] “Part-aware Unified Representation of Language and Skeleton for Zero-shot Action Recognition”, CVPR 2024. [Non-Patent Document 5] “Skeleton-based interactive graph network for human object interaction detection”, IEEE Xplore, https: / / ieeexplore.ieee.org / document / 9102755. Summary of the Invention [Problem to be solved by the invention]

[0005] Conventionally, the relationships between people and objects have not been given, and a large amount of training data is required to generate an action recognizer that can recognize action types including HOI.

[0006] In view of the above problems, one object of the present disclosure is to provide a technology for efficiently generating an action recognizer that recognizes relationships between people and objects or between people. [Means for solving the problem]

[0007] One aspect of the present disclosure is a training video and a training description that describes a relationship between a person and an object or between people, and an action class label of the person with respect to the object in the training video. Equipped with a training data acquisition unit that acquires a training data set; and a keypoint detection unit that detects keypoints between the person and the object from the training video; Using neural networks An action recognizer that recognizes the actions of an agent toward an object or person and adjusting parameters of the action recognizer according to a difference between a processing result based on the key points detected from the training video and the training description and the action class label. and a learning unit that performs the learning. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to provide a technology for efficiently generating an action recognizer that recognizes relationships between people and objects or between people. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram illustrating an activity recognition process according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a schematic diagram illustrating a learning device and an activity recognition device according to an embodiment of the present disclosure. [Figure 3] FIG. 3 is a block diagram illustrating a hardware configuration of a learning device and an action recognition device according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a block diagram illustrating a functional configuration of a learning device according to an embodiment of the present disclosure. [Figure 5] FIG. 5 illustrates an example prompt according to one embodiment of the present disclosure. [Figure 6] FIG. 6 illustrates an example prompt according to one embodiment of the present disclosure. [Figure 7] FIG. 7 illustrates an example prompt according to one embodiment of the present disclosure. [Figure 8] FIG. 8 illustrates an example prompt according to one embodiment of the present disclosure. [Figure 9] FIG. 9 is a diagram illustrating a behavior recognition result according to an embodiment of the present disclosure. [Figure 10] FIG. 10 is a schematic diagram illustrating an articulation point detector and an object detector according to one embodiment of the present disclosure. [Figure 11] FIG. 11 is a schematic diagram illustrating the training process of an activity recognizer according to one embodiment of the present disclosure. [Figure 12] FIG. 12 is a schematic diagram illustrating the training process of an activity recognizer according to one embodiment of the present disclosure. [Figure 13] FIG. 13 is a schematic diagram illustrating the training process of an activity recognizer according to one embodiment of the present disclosure. [Figure 14] FIG. 14 is a flowchart illustrating a learning process according to an embodiment of the present disclosure. [Figure 15] FIG. 15 is a block diagram illustrating a functional configuration of a behavior recognition device according to an embodiment of the present disclosure. [Figure 16] FIG. 16 is a flowchart illustrating the behavior recognition process according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0011] In the following embodiments, a learning device that learns an action recognizer to be learned and an action recognition device that uses the trained action recognizer are disclosed.

[0012] [Summary of the Disclosure] 1, an action recognition device 200 according to an embodiment of the present disclosure acquires a video captured by an imaging device 20 such as a camera, and acquires a description describing the action of a recognition target in the video from a user device 30. For example, the description may be a sentence expressing a relationship between a person and an object, such as a person holding a shoe and putting their foot in it, or a person sitting on a bed and bending over.

[0013] After acquiring the video and the explanatory text, the behavior recognition device 200 detects key points of people and / or objects captured in the video. Key points of people may be, for example, joint points useful for determining the posture of a person, such as the head, limbs, and waist, joint points necessary for determining the behavior of the recognition target, and finger positions. For example, the behavior recognition device 200 can detect key points of people using any known joint point detector. Key points of objects may be, for example, edge points of the object. For example, the behavior recognition device 200 can detect key points of an object using any known object detector.

[0014] When the keypoints are detected, the behavior recognition device 200 recognizes the behavior corresponding to the description in the video based on the detected keypoints and the description, and outputs the behavior recognition result. Specifically, the behavior recognition device 200 inputs the detected keypoints to a trained behavior recognizer and obtains a confidence level of the behavior class as the behavior recognition result from the behavior recognizer. However, the behavior to be recognized is not limited to the behavior class used during training, and may be other behavior. In other words, zero-shot behavior recognition may be performed. In the case of zero-shot behavior recognition, in addition to the keypoints, a description of the behavior to be recognized is also input to the trained behavior recognizer to determine whether the behavior to be detected is included in the video.

[0015] For example, such an action recognizer may be trained by a learning device 100 using a training dataset. Specifically, as shown in Fig. 2, the learning device 100 acquires a training dataset from a training data database (DB) 40, and trains an action recognizer 50 to be trained by supervised learning using the acquired training dataset.

[0016] Each training data is composed of a training video, a training explanation, and a corresponding behavior class label, and when the learning device 100 acquires a training video, it detects key points from the acquired training video and inputs the detected key points and the training explanation to the behavior recognizer 50 to be learned. When it acquires a processing result (e.g., a feature vector) from the behavior recognizer 50 to be learned, it adjusts the parameters of the behavior recognizer 50 to be learned according to the error between the acquired processing result and the behavior class label.

[0017] The learning target action recognizer 50 may be realized as a neural network model such as a deep learning model. In this case, the learning device 100 may calculate an error between a processing result from the learning target action recognizer 50 and an action class label, and update parameters of the learning target action recognizer 50 according to the calculated error using an error backpropagation algorithm.

[0018] When learning of the learning target action recognizer 50 is completed, the learning device 100 may make the finally acquired learning target action recognizer 50 available to the action recognition device 200 as a trained action recognizer 60. As shown in the figure, the trained action recognizer 60 may be provided within the action recognition device 200.

[0019] Alternatively, it may be provided in an external server communicatively connected to the behavior recognition device 200. In this case, when the behavior recognition device 200 acquires a video to be recognized and an explanatory text, it performs keypoint detection on the acquired video and transmits the detected keypoints or both the detected keypoints and the explanatory text to the server. The server may then input the received keypoints or both the detected keypoints and the explanatory text to the trained behavior recognizer 60 and transmit the behavior recognition result from the trained behavior recognizer 60 to the behavior recognition device 200.

[0020] In this way, the learning device 100 and the behavior recognition device 200 according to this embodiment can recognize behaviors corresponding to explanatory texts that express relationships between people and objects and / or between people from a video.

[0021] Here, the learning device 100 and the behavior recognition device 200 may be realized by a computing device such as a server, a personal computer (PC), a smartphone, or a tablet, and may have, for example, a hardware configuration as shown in Fig. 3. That is, the learning device 100 and the behavior recognition device 200 have a drive device 101, a storage device 102, a memory device 103, a processor 104, a user interface (UI) device 105, and a communication device 106, which are interconnected via a bus B.

[0022] The programs or instructions that realize the various functions and processes of the learning device 100 and the behavior recognition device 200 may be stored in a removable storage medium such as a CD-ROM (Compact Disk-Read Only Memory) or a flash memory. When the storage medium is set in the drive device 101, the programs or instructions are installed from the storage medium to the storage device 102 or the memory device 103 via the drive device 101. However, the programs or instructions do not necessarily have to be installed from the storage medium, and may be downloaded from any external device via a network or the like.

[0023] The storage device 102 is realized by a hard disk drive or the like, and stores installed programs or instructions as well as files, data, etc. used to execute the programs or instructions.

[0024] The memory device 103 is realized by a random access memory, a static memory, or the like, and when a program or instruction is activated, reads and stores the program, instruction, data, or the like from the storage device 102. The storage device 102, the memory device 103, and the removable storage medium may be collectively referred to as a non-transitory storage medium.

[0025] The processor 104 may be realized by one or more central processing units (CPUs), graphics processing units (GPUs), processing circuitry, etc., which may be composed of one or more processor cores, and executes various functions and processes of the learning device 100 and the behavior recognition device 200 in accordance with programs, instructions, data such as parameters required to execute the programs or instructions, etc., stored in the memory device 103.

[0026] The user interface (UI) device 105 may be composed of input devices such as a keyboard, mouse, camera, and microphone, output devices such as a display, speaker, headset, and printer, and input / output devices such as a touch panel, and realizes an interface between the user and the learning device 100 and the behavior recognition device 200. For example, the user operates the learning device 100 and the behavior recognition device 200 by operating a keyboard, mouse, etc. on a GUI (Graphical User Interface) displayed on a display or touch panel.

[0027] The communication device 106 is realized by various communication circuits that execute wired and / or wireless communication processing with external devices, the Internet, a LAN (Local Area Network), a cellular network, or other communication networks.

[0028] However, the above-described hardware configuration is merely an example, and the learning device 100 and the behavior recognition device 200 according to the present disclosure may be realized by any other appropriate hardware configuration.

[0029] [Learning device] Next, a learning device 100 according to an embodiment of the present disclosure will be described. Fig. 4 is a block diagram showing the functional configuration of the learning device 100 according to an embodiment of the present disclosure.

[0030] 4, the learning device 100 includes a training data acquisition unit 110, a keypoint detection unit 120, and a learning unit 130. The learning device 100 may be realized by a program that causes one or more processors 104 to operate as the training data acquisition unit 110, the keypoint detection unit 120, and / or the learning unit 130.

[0031] The training data acquisition unit 110 acquires a training dataset consisting of a training video, a training description describing a relationship between a person and an object or between people, and an action class label of the person with respect to the object in the training video. For example, the training video may be a video of a person performing some action toward an object and / or another person, and the training description may be a description of the action the person in the video is performing toward the object and / or another person. For example, if the training image is a video of a person putting on shoes, the training description may be a description of the action, such as "the person holds the shoes in their hands and puts their feet in."

[0032] Furthermore, the behavioral class labels corresponding to the training video and the training explanation are labels indicating a predetermined behavioral class. For example, if the training video is a video of a person putting on shoes, behavioral classes such as "putting on shoes" and "bending forward" may be associated as the corresponding behavioral class labels.

[0033] For example, such training explanations may be created manually or generated by a large language model (LLM). Specifically, the training explanations may be explanations focusing on the posture or state change of a person during the behavior of the recognition target. For example, explanations focusing on the posture or state change of a person during the behavior of the recognition target can be generated by providing a prompt such as that shown in FIG. 5 to the LLM. Here, "target_action_label" indicates an action class, and the LLM can generate explanations indicating the action class indicated by "target_action_label" in response to the prompt.

[0034] The training explanation may also explain the relationship between a person and each object in the behavior of the target to be recognized. For example, a prompt such as that shown in FIG. 6 may be provided to the LLM to explain the relationship between a person and each object in the behavior of the target to be recognized, thereby generating a list of objects related to the behavior of the target to be recognized. Here, "object_list_str" indicates a list of object classes that can be detected by the keypoint detection unit 120 (described later). In response to this prompt, the LLM may generate a list of objects related to the behavior of the target to be recognized, among the objects indicated in "object_list_str," as "objects_str."

[0035] Then, for each object listed in "objects_str," a description indicating the relationship between the person and the object can be generated. For example, such a description can be generated by providing a prompt such as that shown in FIG. 7 to the LLM. This prompt can cause the LLM to generate a description relating the person to the object listed in "object_str." Furthermore, when an interaction occurs between a person and another person, rather than between a person and an object, the LLM can generate a description indicating the relationship. In other words, the description generated in this way can be used as training description that explains the changes in the person's posture or state during the behavior of the recognition target and the relationship between the person and the object and / or the person and the other person.

[0036] Note that a summary of an action may be generated using the description generated based on the prompt in Figure 5 and the description generated based on the prompts in Figures 6 and / or 7. For example, such a summary of an action may be generated by providing a prompt such as that shown in Figure 8 to the LLM. Here, "person_caption" is the description generated based on the prompt in Figure 5, and "objects_caption" is the description generated based on the prompts in Figures 6 and / or 7.

[0037] In response to the prompt generated in this way, the LLM may output a generated result such as that shown in Fig. 9. In the generated result shown in the figure, the caption for the action class "cleaning floor" is explained using a caption "person_caption" that focuses on the person's posture during the action, and a caption "objects_caption" that describes objects that may interact with the person and the relationship between the person and the person performing the action. In addition, the action as a whole is explained using an "action_summary" caption.

[0038] In this way, by using LLM to generate explanatory text that indicates the relationships between people and objects and / or between people, it becomes possible to generate explanatory text efficiently.

[0039] The keypoint detection unit 120 detects keypoints between people and objects from training images. Specifically, when the training data acquisition unit 110 acquires a training dataset, the keypoint detection unit 120 performs keypoint detection on each training image in the training dataset. The keypoints here may be, for example, human keypoints such as joint points useful for determining the posture of a person's head, limbs, waist, etc., joint points necessary for determining the behavior of a recognition target, and finger positions, as well as object keypoints such as object endpoints.

[0040] 10 , such keypoint detection may be realized using any known joint point detector 70 and object detector 80. Specifically, the keypoint detection unit 120 may detect joint points of a person in the training video as keypoints using any known joint point detector 70 that detects joint points of the person captured in an image or video from the image or video. Alternatively, the keypoint detection unit 120 may detect edge points of an object in the training video as keypoints using any known object detector 80 that detects edge points of the object captured in the image or video from the image or video.

[0041] In addition, if the behavior to be recognized is related to the movement of a person's fingers, such as a person grasping an object, keypoint detection 120 may detect the position of a person's fingers in the training video using any known detector that detects the position of a person's fingers in an image or video.

[0042] When key points are detected in this manner, the key point detection unit 120 provides the detected key points to the learning unit 130. At this time, the key point detection unit 120 may generate time-series information of the position coordinates of the key points and the object type, and provide the generated time-series information to the learning unit 130. Specifically, the key point detection unit 120 may generate time-series information of the position coordinates of the positions of each joint point, each finger, and each endpoint of the object, etc., and the object type. This makes it possible to grasp the movement trajectory of each key point in the video, and to grasp the movement of a person or an object in time series.

[0043] The learning unit 130 learns the action recognizer 50 that recognizes the action of the subject person toward the object or person based on the keypoints, the training description, and the action class label. The action recognizer 50 may be, for example, a machine learning model based on a neural network, such as a deep learning model.

[0044] Specifically, as shown in Fig. 11, the learning unit 130 may train the target action recognizer 50 so that the similarity between the feature vectors of the training images output from the target action recognizer 50 and the feature vectors of the training explanation text is increased. For example, the learning unit 130 inputs training videos and / or key points detected from the training videos into the target action recognizer 50 to obtain feature vectors indicating behavior classes. On the other hand, the learning unit 130 inputs the corresponding training explanation text into any known text encoder 90 to obtain the feature vectors of the training explanation text. Then, the learning unit 130 adjusts the parameters of the target action recognizer 50 according to the similarity between the two feature vectors.

[0045] For example, when the action recognizer 50 to be learned is realized as a deep learning model, the learning unit 130 adjusts the parameters of the action recognizer 50 to be learned according to the error between the two feature vectors using the backpropagation algorithm. Then, when a predetermined termination condition is satisfied, such as when parameter adjustment is completed for all training data in the training dataset, the learning process for the action recognizer 50 to be learned is terminated. Then, the learning unit 130 provides the finally acquired action recognizer 50 to the action recognition device 200 as the trained action recognizer 60.

[0046] 12 and 13 are schematic diagrams illustrating the learning process of an action recognizer according to an embodiment of the present disclosure. First, input data is input to Backbone. For example, the input data may be the number of frames (F) × (number of people + number of objects) (I) × number of person / object keypoints (K) × number of channels (C). The channel of each keypoint includes coordinates (x, y), confidence (conf), and object type (category_id).

[0047] In Backbone, feature extraction is performed on keypoints of people and objects in each frame of input data. The resulting array has the shape (F,I,C'), with a C'-dimensional feature vector corresponding to the number of input frames and the number of input instances. The instance axis (I) holds information about each instance.

[0048] Next, behavior classification processing is performed using the features of all instances. The feature vectors of all instances are processed using Global Max Pooling (GMPool) to obtain a feature vector for each video in C' dimension. Global Max Pooling aggregates the input feature vector into a feature vector for each video by selecting the element with the largest value on the feature vector (C' axis).

[0049] The C'-dimensional feature vector is then converted into a dimension with a predetermined number of classes by a fully connected layer (FC), and converted into a probability or confidence for each behavior class for behavior classification. During learning, the parameters of the learning target behavior recognizer 50 are updated so as to minimize the classification loss (cross-entropy error) between the behavior class label and the behavior classification result.

[0050] Here, we will explain in more detail the contrastive learning between the action description shown in Figure 11 and the feature vector obtained from the target action recognizer 50. The action description is converted into a C"-dimensional text feature vector by the text encoder 90. The feature vector obtained from the backbone of the target action recognizer 50 is converted into a C'-dimensional feature vector by Global Max / Average Pooling processing. In Global Average Pooling processing, a C'-dimensional feature vector is obtained by averaging the F and I axes.

[0051] The C'-dimensional feature vector is converted into a C"-dimensional action feature vector, the same as the sentence feature vector, by a Projector consisting of a fully connected layer (FC). The cosine similarity matrix between the sentence feature vector and the action feature vector is calculated. Its shape is (B,B) (B = batch size during learning). The correct label matrix of shape (B,B) used for contrastive learning can be generated using the following procedure. The action class label given as the correct label is an array of shape (B, ), and is converted into a symmetric matrix that represents the match of labels between batch data. For example, labels are generated so that data in the batch data that have the same action class label are set to "1", and otherwise are set to "0". At this time, the diagonal elements of the generated matrix are always "1" because they are compared with themselves.

[0052] After converting the cosine similarity matrix and the ground truth label matrix into probability distributions, the Kullback-Leibler (KL) divergence is calculated, and the parameters of the target action recognizer 50 are updated to minimize this as a contrastive loss.

[0053] In the contrastive training step, three contrastive losses are calculated: 3-1: Contrastive loss using a summary of the action as a result of generating an action description and all instance feature vectors 3-2: Contrastive loss using descriptions focusing on human posture and state changes as the result of generating action descriptions and feature vectors of only the person selected from the output of Backbone 3-3: Contrastive loss using the description that explains the relationship between people and objects or between people for each object as the result of generating action descriptions, and the feature vectors of people and target instances or feature vectors of target instances (excluding people) selected from the output of Backbone In this case, if the target instance is not detected, the loss does not need to be calculated. Also, not only based on whether the target instance is detected, but also based on, for example, the distance between the person and the instance, the presence or absence of a relationship between the person and the instance is determined, and if it is determined that there is no relationship, the loss does not need to be calculated.

[0054] The above-described learning device 100 can acquire a trained action recognizer 60 capable of recognizing actions corresponding to explanatory text that expresses the relationship between a person and an object and / or between people from a video.

[0055] [Learning process] Next, a learning process according to an embodiment of the present disclosure will be described. Fig. 14 is a flowchart illustrating the learning process according to an embodiment of the present disclosure. The learning process is performed by the learning device 100, and more specifically, may be realized by one or more processors 104 of the learning device 100 executing one or more programs or instructions stored in one or more memory devices 103.

[0056] 14, in step S101, the learning device 100 acquires a training dataset including a training video, training descriptions that describe relationships between people and objects, or between people, and behavior class labels of people who are the subject of the behavior toward the objects and people in the training video. For example, the training dataset may be stored in advance in the training data DB 40 or the like, and extracted by the learning device 100 when the learning process starts.

[0057] In step S102, the learning device 100 detects key points between people and objects from the training images. For example, the learning device 100 inputs the training video to any known joint point detector 70, and the joint point detector 70 detects the joint points of people captured in the training video as key points. The learning device 100 also inputs the training video to any known object detector 80, and the object detector 80 detects the endpoints of objects captured in the training video as key points. Note that the object detector 80 may detect the type of object along with the endpoints of the object.

[0058] In step S103, the learning device 100 learns the target action recognizer 50 to be trained, which recognizes the action of a human agent toward an object or a person, based on the keypoints, the training explanation, and the action class label. Specifically, the learning device 100 inputs the keypoints and / or video to the target action recognizer 50, and inputs the training explanation to the text encoder 90. Then, the learning device 100 updates the parameters of the target action recognizer 50 to be trained so that the difference between the feature vector output from the target action recognizer 50 and the feature vector output from the text encoder 90 becomes smaller.

[0059] For example, when the above steps are performed for all the training data in the training data set, the learning device 100 terminates the learning process, and the finally obtained behavior recognizer 50 to be learned is made available to the behavior recognition device 200 as a trained behavior recognizer 60.

[0060] According to the above-described learning process, it is possible to acquire a trained behavior recognizer 60 that can recognize behaviors corresponding to explanatory texts that express relationships between people and objects and / or people from a video.

[0061] [Action recognition device] Next, a description will be given of an action recognition device 200 according to an embodiment of the present disclosure. The action recognition device 200 uses the above-described learning device 100 and / or the action recognizer 60 trained by the learning method to recognize the action of a person captured on video, from either the video of the recognition target or from both the video of the recognition target and an explanatory text that describes the relationship between the person and the object or the person captured on video.

[0062] 15 is a block diagram showing a functional configuration of an action recognition device 200 according to an embodiment of the present disclosure. As shown in FIG. 15, the action recognition device 200 includes an acquisition unit 210, a keypoint detection unit 220, and an action recognition unit 230.

[0063] The acquisition unit 210 acquires a video of the recognition target and an explanatory text that describes the relationship between a person and an object or between people. Specifically, when a user or the like of the behavior recognition device 200 provides the behavior recognition device 200 with a video of the recognition target and an explanatory text that describes the behavior of the recognition target, the acquisition unit 210 acquires the video of the recognition target and the explanatory text, and provides the acquired video to the keypoint detection unit 220 and the acquired explanatory text to the behavior recognition unit 230.

[0064] The explanatory text here may indicate an action that cannot be classified into an action class label.

[0065] The keypoint detection unit 220 detects keypoints between people and objects from the video. Specifically, the keypoint detection unit 220 detects the joint points of people and finger positions and / or endpoints of objects captured in the video, using the same or a different keypoint detection method as that used by the keypoint detection unit 120 described above, for the video of the recognition target acquired from the acquisition unit 210.

[0066] For example, the keypoint detection unit 220 may detect keypoints using a joint point detector 70 and an object detector 80. Specifically, the keypoint detection unit 220 may input a video to the joint point detector 70 and detect the positions of joint points and fingers of a person captured in the video. Alternatively, the keypoint detection unit 220 may input a video to the object detector 80 and detect the endpoints of an object captured in the video. Note that the joint point detector 70 and / or the object detector 80 may be any known detection tool or any known machine learning model.

[0067] Upon acquiring the detection results, the keypoint detection unit 220 may generate time-series information of the position coordinates of the keypoints and the object types. For example, the keypoint detection unit 220 may use any keypoint detection method to generate time-series information by associating the position coordinates of each keypoint detected in each frame of the video with the type of person or object. The keypoint detection unit 220 then provides the generated time-series information to the behavior recognition unit 230.

[0068] The behavior recognition unit 230 recognizes the behavior of a subject person toward an object or a person from the keypoints or both the keypoints and the explanatory text using the trained behavior recognizer 60. Here, the trained behavior recognizer 60 is trained using a training dataset consisting of training videos, training explanatory text that expresses the relationship between a person and an object or between a person and another person, and action class labels of people toward objects in the training videos.

[0069] Specifically, the behavior recognition unit 230 inputs only the keypoints or both the keypoints and the explanatory text acquired from the acquisition unit 210 and the keypoints detected by the keypoint detection unit 220 to the trained behavior recognizer 60, and acquires behavior recognition results of a person captured in a video from the trained behavior recognizer 60. For example, if the behavior to be recognized is a predetermined behavior class at the time of learning, the behavior recognition unit 230 inputs only the keypoints to the behavior recognizer 60 and obtains a confidence factor for the predetermined behavior class. For example, the behavior recognition result may indicate a confidence factor indicating whether the behavior described in the explanatory text was performed, in addition to the confidence factor for the predetermined behavior class. In this case, by inputting both the keypoints and the explanatory text to the trained behavior recognizer 60, it may be possible to determine whether not only the predetermined behavior class used to train the trained behavior recognizer 60 but also the behavior described in the explanatory text was recognized (zero-shot learning).

[0070] According to the above-described behavior recognition device 200, it is possible to recognize not only the predetermined behavior classes used to train the trained behavior recognizer 60, but also behaviors corresponding to explanatory text that expresses the relationships between people and objects and / or people from videos.

[0071] [Behavior recognition processing] Next, the behavior recognition process according to an embodiment of the present disclosure will be described. Fig. 16 is a flowchart showing the behavior recognition process according to an embodiment of the present disclosure. The behavior recognition process is executed by the behavior recognition device 200, and more specifically, may be realized by one or more processors 104 of the behavior recognition device 200 executing one or more programs or instructions stored in one or more memory devices 103.

[0072] 16, in step S201, the behavior recognition device 200 acquires a video of a recognition target and an explanatory sentence that describes the relationship between a person and an object or between people. For example, the video may be captured by an imaging device such as a camera and transmitted to the behavior recognition device 200 via a wired connection and / or a wireless connection. For example, if the camera is a surveillance camera, the video captured by the surveillance camera may be transmitted to the behavior recognition device 200 in real time.

[0073] In step S202, the behavior recognition device 200 detects key points between a person and an object from the video. For example, the behavior recognition device 200 inputs each frame of the video captured by a camera to the joint point detector 70 and / or the object detector 80, and acquires the joint points and finger positions of the person captured in the video and / or the corner points of the object captured in the video from the joint point detector 70 and / or the object detector 80.

[0074] In step S203, the behavior recognition device 200 recognizes the behavior of the subject person toward the object or person from the keypoints and the description using the trained behavior recognizer 60. Here, the trained behavior recognizer 60 is trained using a training dataset consisting of training videos, training description statements that represent relationships between people and objects or between people, and behavior class labels of the subject person toward the object and person in the training videos.

[0075] For example, the behavior recognition device 200 receives a video captured by a surveillance camera in real time via wired communication and / or wireless communication and detects a behavior described in a description in the received video. In this case, the behavior recognition device 200 may operate a display, a speaker, or the like to provide a visual display and / or an audio notification to a predetermined user, such as a manager of the area where the surveillance camera is installed, that the behavior described in the description has been detected. Specifically, the behavior recognition device 200 may display a predetermined message on the display or emit a predetermined electronic sound from the speaker.

[0076] According to the above-described behavior recognition process, it is possible to recognize not only the predetermined behavior classes used to train the trained behavior recognizer 60, but also behaviors corresponding to explanatory text that expresses the relationships between people and objects and / or people from videos.

[0077] In addition, the following supplementary notes are provided in relation to the above description. (Appendix 1) a training data acquisition unit that acquires a training data set including a training video, a training description that describes a relationship between a person and an object or a relationship between a person and another person, and an action class label of the person with respect to the object in the training video; a keypoint detection unit that detects keypoints between the person and the object from the training video; a learning unit that learns an action recognizer that recognizes an action of a person acting on an object or a person based on the key points, the training explanations, and the action class labels; A learning device having: (Appendix 2) The learning device according to claim 1, wherein the training explanation focuses on the posture or state changes of a person in the behavior of the recognition target. (Appendix 3) 3. The learning device according to claim 1, wherein the training explanation explains the relationship between a person and an object for each object in the behavior of the object to be recognized. (Appendix 4) 4. The learning device according to any one of appendices 1 to 3, wherein the training explanation explains the relationship between people in the behavior of the recognition target. (Appendix 5) 5. The learning device according to any one of appendices 1 to 4, wherein the training explanation explains a change in posture or state of a person in the behavior of a recognition target and a relationship between the person and an object or between people. (Appendix 6) 6. The learning device according to any one of appendices 1 to 5, wherein the training explanations are generated by a large-scale language model. (Appendix 7) 7. The learning device according to any one of claims 1 to 6, wherein the key point detection unit detects human joint points as the key points. (Appendix 8) 8. The learning device according to claim 1, wherein the keypoint detection unit detects the positions of a person's fingers as the keypoints. (Appendix 9) 9. The learning device according to any one of Supplementary notes 1 to 8, wherein the keypoint detection unit detects edge points of an object as the keypoints. (Appendix 10) 10. The learning device according to any one of Supplementary notes 1 to 9, wherein the keypoint detection unit detects the keypoints using a joint point detector and an object detector. (Appendix 11) 11. The learning device according to any one of claims 1 to 10, wherein the keypoint detection unit generates time-series information of position coordinates of the keypoints and object types. (Appendix 12) The learning device according to any one of appendices 1 to 11, wherein the learning unit trains the behavior recognizer so as to increase the similarity between the feature vectors of the training video output from the behavior recognizer and the feature vectors of the training explanation text. (Appendix 13) Obtaining a training dataset consisting of training videos, training descriptions that describe relationships between people and objects, or between people, and action class labels of people who are acting on the objects and people in the training videos; Detecting key points between the person and the object from the training video; training an action recognizer that recognizes the human action of the subject with respect to an object or person based on the keypoints, the training narrative, and the action class labels; The computer executes the learning method. (Appendix 14) an acquisition unit that acquires a video to be recognized and an explanatory sentence that represents a relationship between a person and an object or a relationship between a person and another person; a keypoint detection unit that detects keypoints between the person and the object from the video; an action recognition unit that recognizes the action of the subject person with respect to the object or person from the key points and the description using an action recognizer trained using a training dataset consisting of a training video, training description text that represents a relationship between a person and an object, or a relationship between a person and another person, and action class labels of the subject person with respect to the object and person in the training video; An activity recognition device having the above. (Appendix 15) the acquisition unit acquires an explanation indicating an action that cannot be classified into the action class label, The behavior recognition device according to claim 14, wherein the behavior recognition unit uses the behavior recognizer to recognize the behavior of the subject of the behavior toward the object or person from the key points and the unclassifiable description. (Appendix 16) Obtaining a video to be recognized and a description that describes the relationship between a person and an object or between people; Detecting key points between the person and the object from the video; Recognizing the actions of the subject person relative to the object or person from the key points and the description using an action recognizer trained using a training dataset consisting of training videos, training description sentences that represent relationships between people and objects, and between people, and action class labels of the subject person relative to the object and person in the training videos; The activity recognition method is performed by a computer.

[0078] Although the examples of the present disclosure have been described in detail above, the present disclosure is not limited to the specific embodiments described above, and various modifications and variations are possible within the scope of the gist of the present disclosure as set forth in the claims. [Explanation of symbols]

[0079] 20 Imaging device 30 User terminals 40 Training Data DB 50 Learning target action recognizer 60 Trained Action Recognizer 70 Joint Detector 80 Object Detector 90 Text Encoder 100 Learning Device 110 Training data acquisition unit 120 Keypoint detection unit 130 Learning Department 200 Behavior recognition device 210 Acquisition Department 220 Keypoint detection unit 230 Behavior Recognition Department

Claims

1. a training data acquisition unit that acquires a training dataset including a training video, a training description that represents a relationship between a person and an object, or a relationship between a person and another person, and an action class label of the person with respect to the object in the training video; a keypoint detection unit that detects keypoints between the person and the object from the training video; a learning unit that adjusts parameters of an action recognizer that recognizes an action of a subject of action toward an object or a person using a neural network, according to a difference between a processing result based on the key points detected from the training video and the training description, and the action class label; A learning device having:

2. The learning device according to claim 1 , wherein the training explanation is an explanation regarding a change in a person's posture or state during the behavior of the recognition target.

3. The learning device according to claim 1 , wherein the training explanations explain a relationship between a person and an object for each object in the behavior of a recognition target.

4. The learning device according to claim 1 , wherein the training explanations explain relationships between people in the behavior of the recognition target.

5. The learning device according to claim 1 , wherein the training explanations explain a change in a person's posture or state in the behavior of the recognition target, a relationship between the person and an object, and a relationship between people.

6. The learning device of claim 1 , wherein the training sentences are generated by a large-scale language model.

7. The learning device according to claim 1 , wherein the key point detection unit detects joint points of a person as the key points.

8. The learning device according to claim 1 , wherein the key point detection unit detects positions of fingers of a person as the key points.

9. The learning device according to claim 1 , wherein the key point detection unit detects edge points of an object as the key points.

10. The learning device according to claim 1 , wherein the keypoint detection unit detects the keypoints using a joint point detector and an object detector.

11. The learning device according to claim 1 , wherein the keypoint detection unit generates time-series information of the position coordinates of the keypoints and the object types.

12. The learning device according to claim 1 , wherein the learning unit trains the behavior recognizer so that a similarity between a feature vector of the training video output from the behavior recognizer and a feature vector of the training explanation text increases.

13. obtaining a training dataset comprising training videos and training narratives describing relationships between people and objects or people, and activity class labels of the people relative to the objects in the training videos; Detecting key points between the person and the object from the training video; adjusting parameters of an action recognizer that recognizes an action of a subject person toward an object or a person using a neural network according to a difference between a processing result based on the key points detected from the training video and the training description and the action class label; The computer executes the learning method.

14. an acquisition unit that acquires a video to be recognized and an explanatory sentence that represents a relationship between a person and an object or a relationship between a person and another person; a keypoint detection unit that detects keypoints between the person and the object from the video; an action recognition unit that uses an action recognizer whose parameters are adjusted according to a difference between a processing result based on key points between a person and an object detected from a training video and training explanatory text, and an action class label of the person with respect to the object in the training video, and recognizes the action of the person acting as the subject of the action with respect to the object or the person in the video to be recognized using a neural network based on the key points detected by the key point detection unit and the explanatory text; An activity recognition device having the above.

15. the acquisition unit acquires an explanation indicating an action that cannot be classified into the action class label, The behavior recognition device according to claim 14 , wherein the behavior recognition unit recognizes the behavior of the person with respect to the object from the key points and the unclassifiable description using the behavior recognizer.

16. Obtaining a video to be recognized and a description that describes the relationship between a person and an object or between people; Detecting key points between the person and the object from the video; Using an action recognizer whose parameters have been adjusted according to the difference between a processing result based on the key points of the person and the object detected from the training video and the training description and the action class label of the person with respect to the object in the training video, recognize the action of the subject person with respect to the object or the person in the video to be recognized using a neural network based on the key points and the description detected from the video; The activity recognition method is performed by a computer.

Images