Learning device, learning method, behavior recognition device, and behavior recognition method

Abstract

The present invention provides a learning device, a learning method, a behavior recognition device, and a behavior recognition method for efficiently generating a behavior recognition device that recognizes the relationship between people and objects. [Solution] The learning device 100 includes a training data acquisition unit that acquires a training dataset consisting of training videos, training descriptions that represent the relationship between a person and an object or between a person and another person, and class labels of human actions toward an object in the training videos; a key point detection unit that detects key points between a person and an object from the training videos; and a learning unit that learns an action recognition device that recognizes the actions of a person acting toward an object or another person based on the key points, training descriptions, and class labels of actions toward an object or another person.

Description

Technical Field

[0001] The present disclosure relates to a learning device, a learning method, an action recognition device, and an action recognition method.

Background Art

[0002] With the recent progress of deep learning technology, the opportunities for using machine learning models have been expanding. For example, an action recognition model trained to recognize human actions from videos including objects such as people and objects is used, and it has become possible to recognize what a person photographed by a video camera is doing.

[0003] Human actions often involve the relationship between the acting person and surrounding objects and people. In the relationship between humans and objects, that is, Human Object Interaction (HOI), regarding the relationship between humans and objects in a video, for example, actions of a person such as a person holding shoes in their hand and putting their foot in, or a person sitting on a bed and bending down are recognized.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

[0005] Traditionally, the relationship between people and objects has not been given in relation to actions, and a large amount of training data is required to generate an action recognizer that can recognize action types, including HoIs.

[0006] In light of the above-mentioned problems, one objective of this disclosure is to provide a technology for efficiently generating an action recognition device that recognizes the relationship between a person and an object or between people. [Means for solving the problem]

[0007] One aspect of the present disclosure relates to a learning device comprising: a training data acquisition unit that acquires a training dataset comprising training videos and training descriptions that represent the relationship between a person and an object or between a person and another person, and a class label of the person's actions toward the object in the training videos; a key point detection unit that detects key points between the person and the object from the training videos; and a learning unit that adjusts the parameters of the action recognition unit according to the difference between the processing result based on the key points detected from the training videos and the training descriptions and the action class label, using an action recognition unit that recognizes the actions of the person acting toward the object or person using a neural network. [Effects of the Invention]

[0008] According to this disclosure, it is possible to provide a technology for efficiently generating an action recognition device that recognizes the relationship between a person and an object or between a person and another person. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a schematic diagram showing an action recognition process according to one embodiment of the present disclosure. [Figure 2] Figure 2 is a schematic diagram showing a learning device and an action recognition device according to one embodiment of the present disclosure. [Figure 3] Figure 3 is a block diagram showing the hardware configuration of a learning device and an action recognition device according to one embodiment of the present disclosure. [Figure 4] Figure 4 is a block diagram showing the functional configuration of a learning device according to one embodiment of the present disclosure. [Figure 5] Figure 5 shows an example of a prompt according to one embodiment of the present disclosure. [Figure 6] Figure 6 shows an example of a prompt according to one embodiment of the present disclosure. [Figure 7] Figure 7 shows an example of a prompt according to one embodiment of the present disclosure. [Figure 8] Figure 8 shows an example of a prompt according to one embodiment of the present disclosure. [Figure 9] Figure 9 shows the behavior recognition results according to one embodiment of the present disclosure. [Figure 10] FIG. 10 is a schematic diagram showing a joint detector and an object detector according to an embodiment of the present disclosure. [Figure 11] FIG. 11 is a schematic diagram showing a learning process of an action recognizer according to an embodiment of the present disclosure. [Figure 12] FIG. 12 is a schematic diagram showing a learning process of an action recognizer according to an embodiment of the present disclosure. [Figure 13] FIG. 13 is a schematic diagram showing a learning process of an action recognizer according to an embodiment of the present disclosure. [Figure 14] FIG. 14 is a flowchart showing a learning process according to an embodiment of the present disclosure. [Figure 15] FIG. 15 is a block diagram showing a functional configuration of an action recognition device according to an embodiment of the present disclosure. [Figure 16] FIG. 16 is a flowchart showing an action recognition process according to an embodiment of the present disclosure.

MODE FOR CARRYING OUT THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0011] In the following examples, a learning device that learns an action recognizer to be learned and an action recognition device that uses the learned action recognizer are disclosed.

[0012] [SUMMARY OF THE PRESENT DISCLOSURE] Schematically, as shown in FIG. 1, an action recognition device 200 according to an embodiment of the present disclosure acquires a video captured by an imaging device 20 such as a camera, and also acquires a description text for describing the action of a recognition target in the video from a user device 30. For example, the description text may be a sentence representing the relationship between a person and an object, such as a person holding shoes in their hand and putting their foot in, or a person sitting on a bed and bending over.

[0013] Upon acquiring a video and a description, the action recognition device 200 detects key points of people and / or objects captured in the video. Key points for people may include, for example, joint points useful for determining the posture of a person, such as the head, limbs, and waist; joint points necessary for determining the action of the object being recognized; and the position of fingers. For example, the action recognition device 200 can detect key points of people using any known joint point detector. Key points for objects may include, for example, the endpoints of objects. For example, the action recognition device 200 can detect key points of objects using any known object detector.

[0014] When a keypoint is detected, the action recognition device 200 recognizes the action corresponding to the description in the video based on the detected keypoint and the description, and outputs the action recognition result. Specifically, the action recognition device 200 inputs the detected keypoint into a trained action recognizer and obtains the confidence level of the action class as the action recognition result from the action recognizer. However, the action to be recognized is not limited to the action class used during training, but may be any other action. In other words, zero-shot action recognition may be performed. In the case of zero-shot action recognition, in order to determine whether the action to be detected is included in the video, the description of the action to be recognized is also input into the trained action recognizer in addition to the keypoint.

[0015] For example, such an action recognition system may be trained by a learning device 100 using a training dataset. Specifically, as shown in Figure 2, the learning device 100 obtains a training dataset from a training data database (DB) 40 and trains the action recognition system 50 to be trained by supervised learning using the obtained training dataset.

[0016] Each training data set consists of a training video, a training description, and a corresponding behavior class label. When the learning device 100 acquires a training video, it detects key points from the acquired training video and inputs the detected key points and the training description to the behavior recognition device 50 to be learned. When the learning device 100 acquires processing results (e.g., feature vectors) from the behavior recognition device 50 to be learned, it adjusts the parameters of the behavior recognition device 50 to be learned according to the error between the acquired processing results and the behavior class label.

[0017] The behavior recognition unit 50 to be trained may be implemented as a neural network model, such as a deep learning model. In this case, the learning device 100 may calculate the error between the processing result from the behavior recognition unit 50 to be trained and the behavior class label, and update the parameters of the behavior recognition unit 50 according to the backpropagation method based on the calculated error.

[0018] Once the learning of the target behavior recognition unit 50 is complete, the learning device 100 may make the finally acquired target behavior recognition unit 50 available to the behavior recognition device 200 as a trained behavior recognition unit 60. As shown in the figure, the trained behavior recognition unit 60 may be provided within the behavior recognition device 200.

[0019] Alternatively, it may be provided on an external server connected to the action recognition device 200 via communication. In this case, when the action recognition device 200 acquires the video and explanatory text to be recognized, it performs keypoint detection on the acquired video and sends the detected keypoints, or both the detected keypoints and the explanatory text, to the server. The server then inputs the received keypoints, or both the detected keypoints and the explanatory text, into the trained action recognition unit 60 and sends the action recognition results from the trained action recognition unit 60 to the action recognition device 200.

[0020] In this way, the learning device 100 and the action recognition device 200 according to this embodiment can recognize actions from a video that correspond to descriptive text that describes the relationship between a person and an object and / or between people.

[0021] Here, the learning device 100 and the behavior recognition device 200 may be implemented by computing devices such as servers, personal computers (PCs), smartphones, and tablets, and may have a hardware configuration such as that shown in Figure 3. That is, the learning device 100 and the behavior recognition device 200 have a drive device 101, a storage device 102, a memory device 103, a processor 104, a user interface (UI) device 105, and a communication device 106 that are interconnected via bus B.

[0022] The programs or instructions that implement the various functions and processes in the learning device 100 and the behavior recognition device 200 may be stored in a removable storage medium such as a CD-ROM (Compact Disk-Read Only Memory) or flash memory. When the storage medium is set in the drive device 101, the programs or instructions are installed from the storage medium to the storage device 102 or memory device 103 via the drive device 101. However, the programs or instructions do not necessarily have to be installed from the storage medium; they may also be downloaded from any external device via a network or the like.

[0023] The storage device 102 is implemented by a hard disk drive or the like, and stores files, data, etc., used to execute the installed program or instruction, along with the program or instruction itself.

[0024] The memory device 103 is implemented using random access memory, static memory, etc., and when a program or instruction is activated, it reads the program or instruction, data, etc. from the storage device 102 and stores it. The storage device 102, the memory device 103, and the removable storage medium may be collectively referred to as a non-transitory storage medium.

[0025] The processor 104 may be implemented by one or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), processing circuits, etc., which may consist of one or more processor cores, and executes various functions and processes of the learning device 100 and the action recognition device 200 according to data such as programs, instructions, and parameters necessary to execute said programs or instructions stored in the memory device 103.

[0026] The user interface (UI) device 105 may consist of input devices such as a keyboard, mouse, camera, and microphone, output devices such as a display, speaker, headset, and printer, and input / output devices such as a touch panel, and realizes an interface between the user and the learning device 100 and the behavior recognition device 200. For example, the user operates the learning device 100 and the behavior recognition device 200 by operating a GUI (Graphical User Interface) displayed on a display or touch panel with a keyboard, mouse, etc.

[0027] The communication device 106 is implemented by various communication circuits that perform wired and / or wireless communication processing with external devices, the Internet, LAN (Local Area Network), cellular networks, and other communication networks.

[0028] However, the hardware configuration described above is merely an example, and the learning device 100 and the behavior recognition device 200 according to this disclosure may be implemented by any other suitable hardware configuration.

[0029] [Learning device] Next, a learning device 100 according to one embodiment of the present disclosure will be described. Figure 4 is a block diagram showing the functional configuration of the learning device 100 according to one embodiment of the present disclosure.

[0030] As shown in Figure 4, the learning device 100 includes a training data acquisition unit 110, a keypoint detection unit 120, and a learning unit 130. The learning device 100 may be implemented by a program that operates one or more processors 104 as the training data acquisition unit 110, the keypoint detection unit 120, and / or the learning unit 130.

[0031] The training data acquisition unit 110 acquires a training dataset consisting of training videos, training descriptions that represent the relationship between a person and an object or between people, and class labels for the person's actions toward the object in the training videos. For example, the training video may be a video in which a person is performing some action toward an object and / or another person, and the training description may be a sentence that describes the action that the person in the video is performing toward the object and / or another person. For example, if the training image is a video in which a person is putting on shoes, the training description could be a sentence that describes the action, such as "the person is holding the shoe in their hand and putting their foot in."

[0032] Furthermore, the behavior class labels corresponding to the training videos and training descriptions are labels that indicate a predetermined behavior class. For example, if the training video is footage of a person putting on shoes, behavior classes such as "putting on shoes" and "bending forward" may be associated with it as the corresponding behavior class labels.

[0033] For example, such training descriptions may be created manually or generated by a Large Language Model (LLM). Specifically, the training descriptions may focus on the posture or state changes of a person in the action to be recognized. For example, descriptions that focus on the posture or state changes of a person in the action to be recognized can be generated by giving the LLM a prompt as shown in Figure 5. Here, “target_action_label” indicates the action class, and this prompt can cause the LLM to generate a description that indicates the action class indicated by “target_action_label”.

[0034] Furthermore, the training description may also describe the relationship between a person and an object for each object in the action to be recognized. For example, in order to describe the relationship between a person and an object for each object in the action to be recognized, a list of objects related to the action to be recognized can be generated by giving the LLM a prompt as shown in Figure 6. Here, “object_list_str” represents a list of object classes that can be detected by the keypoint detection unit 120 described later, and in response to this prompt, the LLM may generate a list of objects related to the action to be recognized from among the objects shown in “object_list_str” as “objects_str”.

[0035] Then, for each object indicated in “objects_str”, a descriptive text describing the relationship between a person and that object can be generated. For example, such a descriptive text can be generated by giving the LLM a prompt as shown in Figure 7. In response to this prompt, the LLM can generate a descriptive text describing the relationship between a person and the object indicated in “object_str”. Furthermore, if it is a relationship between people rather than between a person and an object, the LLM can provide a descriptive text describing that relationship. In other words, the descriptive texts generated in this way can be used as training descriptive texts that explain the posture or state changes of a person in the behavior of the recognized object, and the relationship between the person and the object and / or between people.

[0036] Furthermore, a summary of the action may be generated using the caption generated based on the prompt in Figure 5 and the caption generated based on the prompts in Figure 6 and / or Figure 7. For example, such a summary of the action can be generated by giving the LLM a prompt as shown in Figure 8. Here, “person_caption” is the caption generated based on the prompt in Figure 5, and “objects_caption” is the caption generated based on the prompts in Figure 6 and / or Figure 7.

[0037] In response to prompts generated in this manner, LLM may output a generated result such as that shown in Figure 9. In the generated result shown, the caption for the action class "cleaning floor" is explained by a "person_caption" that focuses on the person's posture during the action, and a "objects_caption" that describes objects that may interact with the person and the relationship between the person and the person performing the action. In addition, the "action_summary" caption explains the entire action.

[0038] In this way, by using LLM to generate descriptive texts that show the relationship between people and objects and / or between people, it becomes possible to generate descriptive texts efficiently.

[0039] The keypoint detection unit 120 detects keypoints between people and objects from training images. Specifically, when the training data acquisition unit 110 acquires a training dataset, the keypoint detection unit 120 performs keypoint detection on each training image in the training dataset. Here, keypoints may include, for example, joint points useful for determining the posture of a person, such as the head, limbs, and waist, joint points necessary for determining the actions of the object to be recognized, and the position of fingers, as well as keypoints of objects, such as the endpoints of objects.

[0040] As shown in Figure 10, such keypoint detection may be implemented using either a known joint point detector 70 or an object detector 80. Specifically, the keypoint detection unit 120 may use either a known joint point detector 70, which detects the joint points of a person captured in an image or video, to detect the joint points of a person in the training video as keypoints. Alternatively, the keypoint detection unit 120 may use either a known object detector 80, which detects the endpoints of an object captured in an image or video, to detect the endpoints of an object in the training video as keypoints.

[0041] Furthermore, if the action to be recognized is related to the movement of a person's fingers, such as grasping an object, the keypoint detection 120 may use any known detector that detects the position of a person's fingers in an image or video to detect the position of a person's fingers in the training video.

[0042] When keypoints are detected in this manner, the keypoint detection unit 120 provides the detected keypoints to the learning unit 130. At this time, the keypoint detection unit 120 may generate time-series information of the position coordinates of the keypoints and the object type, and provide the generated time-series information to the learning unit 130. Specifically, the keypoint detection unit 120 may generate time-series information of the position coordinates of each joint point, each finger, and each endpoint of an object, and the object type. This makes it possible to understand the movement trajectory of each keypoint in the video, and to understand the movement of people and objects in a time-series manner.

[0043] The learning unit 130 learns an action recognizer 50 that recognizes the actions of an agent of an object or person based on key points, training descriptions, and action class labels. The action recognizer 50 may be a machine learning model based on a neural network, such as a deep learning model.

[0044] Specifically, as shown in Figure 11, the learning unit 130 may train the target action recognition unit 50 so that the similarity between the feature vector of the training image output from the target action recognition unit 50 and the feature vector of the training description is high. For example, the learning unit 130 inputs a training video and / or keypoints detected from the training video into the target action recognition unit 50 and obtains a feature vector indicating the action class. On the other hand, the learning unit 130 inputs the corresponding training description into any known text encoder 90 and obtains a feature vector of the training description. Then, the learning unit 130 adjusts the parameters of the target action recognition unit 50 according to the similarity between the two feature vectors.

[0045] For example, if the action recognition system 50 to be trained is implemented as a deep learning model, the learning unit 130 adjusts the parameters of the action recognition system 50 according to the error between two feature vectors using backpropagation. When predetermined termination conditions are met, such as when parameter adjustment is completed for all training data in the training dataset, the training process for the action recognition system 50 is terminated. The learning unit 130 then provides the finally acquired action recognition system 50 as a trained action recognition system 60 to the action recognition device 200.

[0046] Figures 12 and 13 are schematic diagrams showing the learning process of an action recognition system according to one embodiment of the present disclosure. First, input data is input to the Backbone. For example, the input data may be the number of frames (F) × (number of people + number of objects) (I) × number of keypoints for people / objects (K) × number of channels (C). The channels for each keypoint are the coordinates (x,y), confidence level (conf), and object type (category_id).

[0047] Backbone performs feature extraction on keypoint inputs of people and objects in each frame of the input data. The resulting array has the shape of (F,I,C') and contains a C'-dimensional feature vector corresponding to the number of input frames and the number of input instances. The instance axis (I) holds information for each instance.

[0048] Next, a behavior classification process is performed using the features of all instances. The feature vectors of all instances are processed using Global Max Pooling (GMPool) to obtain a C'-dimensional feature vector for each video. Global Max Pooling aggregates the input video-specific feature vector by selecting the element with the maximum value of the feature vector (C' axis).

[0049] The C'-dimensional feature vector is then transformed into a dimension of a predetermined number of classes by a fully connected layer (FC), and converted into the probability or confidence level of each behavior class to perform behavior classification. During training, the parameters of the behavior class recognition unit 50 being trained are updated to minimize the classification loss (cross-entropy error) between the behavior class label and the behavior classification result.

[0050] Here, we will explain in more detail the contrastive learning process between the action description text shown in Figure 11 and the feature vectors obtained from the action recognition unit 50 being trained. The action description text is converted into a C″-dimensional text feature vector by the text encoder 90. The feature vectors obtained from the backbone of the action recognition unit 50 being trained are converted into a C'-dimensional feature vector by the Global Max / Average Pooling process. In the Global Average Pooling process, a C'-dimensional feature vector is obtained by averaging the F and I axes.

[0051] The C'-dimensional feature vector is transformed into a C″-dimensional behavior feature vector, the same as the text feature vector, by a Projector consisting of fully connected layers (FC). A cosine similarity matrix between the text feature vector and the behavior feature vector is calculated. Its shape is (B,B) (B = batch size during training). The (B,B)-shaped ground truth label matrix used for comparative learning can be generated using the following procedure. The behavior class labels given as ground truth labels are an array of (B,) shape, which is transformed into a symmetric matrix representing the matching of labels between batch data. For example, labels are generated such that data with the same behavior class label in the batch data are assigned "1", and data without the same label are assigned "0". In this case, the diagonal elements of the generated matrix are always "1" because they are comparisons with themselves.

[0052] After converting the cosine similarity matrix and the ground truth label matrix into probability distributions, the Kullback-Leibler (KL) divergence is calculated, and the parameters of the training action recognition model 50 are updated to minimize this KL divergence as the control loss.

[0053] In the comparative learning step, the following three types of comparative losses are calculated. 3-1: Controlled loss using the descriptive text showing the summary of the action as a result of generating the action description text and the total instance feature vector 3-2: Controlled loss using explanatory texts that focus on changes in a person's posture and state as a result of generating behavioral explanatory texts, and feature vectors of only the people selected from the output of the backbone. 3-3: Control loss using the descriptive text that explains the relationship between a person and an object or a person and another person as a result of generating the action description text, and the feature vectors of the person and the target instance or the target instance (excluding the person) selected from the Backbone output. In this case, if the target instance is not detected, the loss does not need to be calculated. Furthermore, in addition to detection, the existence of a relationship between a person and an instance can be determined based on, for example, the distance between the person and the instance. If it is determined that there is no relationship, the loss does not need to be calculated.

[0054] According to the learning device 100 described above, a trained action recognizer 60 can be obtained that is capable of recognizing actions corresponding to descriptive texts that represent the relationship between people and objects and / or people and people from a video.

[0055] [Learning Process] Next, a learning process according to one embodiment of the present disclosure will be described. Figure 14 is a flowchart of the learning process according to one embodiment of the present disclosure. The learning process is performed by a learning device 100, and more specifically, it may be realized by one or more processors 104 of the learning device 100 executing one or more programs or instructions stored in one or more memory devices 103.

[0056] As shown in Figure 14, in step S101, the learning device 100 acquires a training dataset consisting of training videos, training descriptions representing the relationship between people and objects or people and people, and action class labels of the people acting on the objects and people in the training videos. For example, the training dataset may be stored in advance in a training data DB 40 or the like, and extracted by the learning device 100 when the learning process starts.

[0057] In step S102, the learning device 100 detects key points between people and objects from the training images. For example, the learning device 100 inputs the training video to any known joint point detector 70 and detects the joint points of people captured in the training video as key points from the joint point detector 70. Alternatively, the learning device 100 inputs the training video to any known object detector 80 and detects the endpoints of objects captured in the training video as key points from the object detector 80. The object detector 80 may also detect the type of object along with its endpoints.

[0058] In step S103, the learning device 100 learns a target action recognizer 50 that recognizes the actions of an agent acting on an object or person, based on key points, training descriptions, and action class labels. Specifically, the learning device 100 inputs key points and / or videos into the target action recognizer 50 and the training descriptions into the text encoder 90. The learning device 100 then updates the parameters of the target action recognizer 50 so that the difference between the feature vector output from the target action recognizer 50 and the feature vector output from the text encoder 90 becomes smaller.

[0059] For example, once the above steps have been performed on all the training data in the training dataset, the learning device 100 terminates the learning process and makes the resulting training target behavior recognition device 50 available to the behavior recognition device 200 as a trained behavior recognition device 60.

[0060] According to the learning process described above, a trained action recognizer 60 can be obtained that is capable of recognizing actions corresponding to descriptive texts that represent the relationship between people and objects and / or people and people from a video.

[0061] [Action recognition device] Next, an action recognition device 200 according to one embodiment of the present disclosure will be described. The action recognition device 200 uses an action recognition unit 60 trained by the learning device 100 and / or learning method described above to recognize the actions of a person being filmed from both the video to be recognized and the video to be recognized and the descriptive text that shows the relationship between the person and the object or the person and the person being filmed.

[0062] Figure 15 is a block diagram showing the functional configuration of an action recognition device 200 according to one embodiment of the present disclosure. As shown in Figure 15, the action recognition device 200 has an acquisition unit 210, a key point detection unit 220, and an action recognition unit 230.

[0063] The acquisition unit 210 acquires the video to be recognized and a descriptive text that describes the relationship between a person and an object or between a person and another person. Specifically, when a user of the action recognition device 200 provides the action recognition device 200 with the video to be recognized and a descriptive text describing the actions of the person being recognized, the acquisition unit 210 acquires the video and the descriptive text, provides the acquired video to the keypoint detection unit 220, and provides the acquired descriptive text to the action recognition unit 230.

[0064] The descriptions provided here may include behaviors that cannot be classified into behavioral class labels.

[0065] The keypoint detection unit 220 detects keypoints of people and objects from the video. Specifically, the keypoint detection unit 220 detects the joint points and finger positions of people and / or endpoints of objects captured in the video acquired from the acquisition unit 210 using the same or different keypoint detection method as the keypoint detection unit 120 described above.

[0066] For example, the keypoint detection unit 220 may detect keypoints using the joint point detector 70 and the object detector 80. Specifically, the keypoint detection unit 220 may input video to the joint point detector 70 and detect the joint points and finger positions of a person captured in the video. Alternatively, the keypoint detection unit 220 may input video to the object detector 80 and detect the endpoints of objects captured in the video. The joint point detector 70 and / or the object detector 80 may be either a known detection tool or either a known machine learning model.

[0067] Upon obtaining the detection results, the keypoint detection unit 220 may generate time-series information relating the position coordinates of the keypoints and the type of object. For example, the keypoint detection unit 220 may use any keypoint detection method to associate the position coordinates of each keypoint detected in each frame of the video with the type of person or object, and generate this as time-series information. The keypoint detection unit 220 then provides the generated time-series information to the action recognition unit 230.

[0068] The action recognition unit 230 uses the trained action recognition unit 60 to recognize the actions of a person acting on an object or person from key points, or from both key points and explanatory text. Here, the trained action recognition unit 60 is trained using a training dataset consisting of training videos, training explanatory texts that describe the relationship between a person and an object or between a person and another person, and class labels of the person's actions toward an object in the training videos.

[0069] Specifically, the action recognition unit 230 inputs either only the key points, or both the key points and the description, from the descriptive text acquired from the acquisition unit 210 and the key points detected by the key point detection unit 220, into the trained action recognition unit 60, and obtains the action recognition result of the person captured in the video from the trained action recognition unit 60. For example, if the action to be recognized is a predetermined action class used during training, the action recognition unit 230 inputs only the key points into the action recognition unit 60 and obtains the confidence level for the predetermined action class. For example, the action recognition result may show the confidence level for the predetermined action class, as well as a confidence level indicating whether the action described in the description was performed. In this case, both the key points and the description are input into the trained action recognition unit 60, and it is possible to determine not only whether the predetermined action class used to train the trained action recognition unit 60 was recognized, but also whether the action described in the description was recognized (zero-shot learning).

[0070] According to the behavior recognition device 200 described above, it is possible to recognize not only predetermined behavior classes used to train the trained behavior recognition device 60, but also behaviors corresponding to descriptive texts that represent the relationship between people and objects and / or people and people, from the video.

[0071] [Action Recognition Processing] Next, an action recognition process according to one embodiment of the present disclosure will be described. Figure 16 is a flowchart of the action recognition process according to one embodiment of the present disclosure. The action recognition process is performed by an action recognition device 200, and more specifically, it may be realized by one or more processors 104 of the action recognition device 200 executing one or more programs or instructions stored in one or more memory devices 103.

[0072] As shown in Figure 16, in step S201, the behavior recognition device 200 acquires a video of the object to be recognized and a descriptive text that represents the relationship between a person and an object or between a person and another person. For example, the video may be captured by an imaging device such as a camera and transmitted to the behavior recognition device 200 via a wired connection and / or a wireless connection. For example, if the camera is a surveillance camera, the video captured by the surveillance camera may be transmitted to the behavior recognition device 200 in real time.

[0073] In step S202, the action recognition device 200 detects key points of people and objects from the video. For example, the action recognition device 200 inputs each frame of the video captured by the camera to the joint point detector 70 and / or object detector 80, and obtains the joint points and finger positions of people and / or endpoints of objects captured in the video from the joint point detector 70 and / or object detector 80.

[0074] In step S203, the action recognition device 200 uses the trained action recognition unit 60 to recognize the actions of a person acting on an object or person from the key points and explanatory text. Here, the trained action recognition unit 60 is trained using a training dataset consisting of training videos, training explanatory texts that describe the relationship between a person and an object or between a person and another person, and the action class labels of the person acting on the object and person in the training videos.

[0075] For example, suppose the behavior recognition device 200 receives video footage captured by a surveillance camera in real time via wired and / or wireless communication, and detects an action described in a descriptive text in the received video. In this case, the behavior recognition device 200 may activate a display, speaker, etc., to provide a visual display and / or audio notification to a designated user, such as the administrator of the area where the surveillance camera is installed, indicating that the action described in the descriptive text has been detected. Specifically, the behavior recognition device 200 may display a predetermined message on the display or emit a predetermined electronic sound from the speaker.

[0076] According to the action recognition process described above, it is possible to recognize not only predetermined action classes used to train the trained action recognizer 60, but also actions corresponding to descriptive texts that represent the relationship between people and objects and / or people and people, from the video.

[0077] Furthermore, the following additional information is disclosed regarding the above explanation. (Note 1) A training data acquisition unit acquires a training dataset consisting of training videos, training descriptions representing the relationship between a person and an object or between people, and class labels of the person's actions toward the object in the training videos. A key point detection unit that detects key points between the person and the object from the training video, A learning unit that learns an action recognition device to recognize the actions of a person acting on an object or person based on the aforementioned key points, the training description, and the action class label, A learning device having the following features. (Note 2) The training instructions described above are based on the learning device described in Appendix 1, which focuses on changes in a person's posture or state during the actions of the object being recognized. (Note 3) The training description is a learning device as described in Appendix 1 or 2, which explains the relationship between a person and an object for each object in the behavior of the object to be recognized. (Note 4) The aforementioned training description is a learning device described in any of the appendices 1 to 3, which explains the relationships between people in the behavior of the object to be recognized. (Note 5) The training description is a learning device described in any of the appendices 1 to 4, which explains the relationship between a person's posture or state changes in the behavior of the object being recognized and the relationship between a person and an object or between people. (Note 6) The aforementioned training description is generated by a large-scale language model using a learning device described in any of the appendices 1 to 5. (Note 7) The learning device according to any one of the appendices 1 to 6, wherein the key point detection unit detects the joint points of a person as the key points. (Note 8) The learning device according to any one of the appendices 1 to 7, wherein the key point detection unit detects the position of a person's fingers as the key point. (Note 9) The learning device according to any one of the appendices 1 to 8, wherein the key point detection unit detects the endpoints of an object as the key points. (Note 10) The learning device according to any one of the appendices 1 to 9, wherein the key point detection unit detects the key point using an articulation point detector and an object detector. (Note 11) The keypoint detection unit is a learning device according to any one of the appendices 1 to 10, which generates time-series information of the position coordinates and object type of the keypoint. (Note 12) The learning unit trains the behavior recognition device to increase the similarity between the feature vector of the training video output from the behavior recognition device and the feature vector of the training description text, as described in any of the appendices 1 to 11. (Note 13) Obtain a training dataset consisting of training videos, training descriptions describing the relationship between people and objects or people and people, and behavior class labels of the agents acting on the objects and people in the training videos. To detect key points between the person and the object from the aforementioned training video, Based on the aforementioned key points, the training description, and the behavior class label, the behavior recognition device learns to recognize the actions of the subject towards an object or person. A learning method that a computer uses. (Note 14) An acquisition unit that acquires a video to be recognized and a descriptive text that represents the relationship between a person and an object or between a person and another person, A key point detection unit that detects key points between the person and the object from the aforementioned video, An action recognition unit that recognizes the actions of the person acting on the object or person from the key points and the description, using an action recognition unit trained with a training dataset consisting of a training video, a training description describing the relationship between a person and an object or between people, and the action class labels of the person acting on the object and person in the training video. A behavior recognition device having the following features. (Note 15) The acquisition unit acquires a descriptive text indicating an action that cannot be classified into the action class label. The behavior recognition device described in Appendix 14, wherein the behavior recognition unit uses the behavior recognition device to recognize the behavior of the person acting on the object or person from the key points and the unclassifiable explanatory text. (Note 16) The process involves obtaining the video to be recognized and a descriptive text that represents the relationship between people and objects or people and people, To detect key points between the person and the object from the aforementioned video, Using an action recognizer trained with a training dataset consisting of training videos, training descriptions describing relationships between people and objects, and people and people, and action class labels of the actors acting on the objects and people in the training videos, the system recognizes the actions of the actors acting on the objects or people from the key points and the descriptions. A method of action recognition performed by a computer.

[0078] Although embodiments of this disclosure have been described in detail above, this disclosure is not limited to the specific embodiments described above, and various modifications and changes are possible within the scope of the gist of this disclosure as described in the claims. [Explanation of Symbols]

[0079] 20 Imaging device 30 User terminals 40 Training Data Database 50. Action recognition system for learning 60 pre-trained behavior recognition systems 70 Joint detectors 80 Object detectors 90 Text Encoders 100 Learning Devices 110 Training data acquisition unit 120 Keypoint detection unit 130 Learning Department 200 Behavior recognition device 210 Acquisition Department 220 Keypoint detection unit 230 Behavior Recognition Department

Claims

1. A training data acquisition unit acquires a training dataset comprising a training video, a training description that represents the relationship between a person and an object or between people, and a class label of the person's behavior towards the object in the training video. A key point detection unit that detects key points between the person and the object from the training video, A learning unit adjusts the parameters of an action recognition device, which uses a neural network to recognize the actions of an agent acting on an object or person, based on the difference between the processing result, which is based on the key points detected from the training video and the training description text, and the action class label. A learning device having the following features.

2. The learning device according to claim 1, wherein the training description is a description of a change in a person's posture or state in the behavior of the object to be recognized.

3. The learning device according to claim 1, wherein the training description explains the relationship between a person and an object for each object in the behavior of the object to be recognized.

4. The learning device according to claim 1, wherein the training description explains the relationship between people in the behavior of the object to be recognized.

5. The learning device according to claim 1, wherein the training description explains the relationship between a person and an object and the relationship between people in the actions of the object to be recognized, as well as the changes in a person's posture or state in the actions of the object to be recognized.

6. The learning device according to claim 1, wherein the training description is generated by a large-scale language model.

7. The learning device according to claim 1, wherein the key point detection unit detects human joint points as the key points.

8. The learning device according to claim 1, wherein the key point detection unit detects the position of a person's fingers as the key point.

9. The learning device according to claim 1, wherein the key point detection unit detects the endpoints of an object as the key points.

10. The learning device according to claim 1, wherein the key point detection unit detects the key point using an articulation point detector and an object detector.

11. The learning device according to claim 1, wherein the key point detection unit generates time-series information of the position coordinates of the key point and the type of object.

12. The learning device according to claim 1, wherein the learning unit trains the behavior recognition device so that the similarity between the feature vector of the training video output from the behavior recognition device and the feature vector of the training description text is increased.

13. Obtain a training dataset comprising training videos, training descriptions representing the relationship between a person and an object or between people, and class labels for the person's actions toward the object in the training videos. To detect key points between the person and the object from the aforementioned training video, The behavior recognition system, which uses a neural network to recognize the actions of an agent acting on an object or person, adjusts the parameters of the behavior recognition system according to the difference between the processing result based on the key points detected from the training video and the training description text, and the behavior class label. A learning method that a computer uses.

14. An acquisition unit that acquires a video to be recognized and a descriptive text that represents the relationship between a person and an object or between a person and another person, A key point detection unit that detects key points between the person and the object from the aforementioned video, An action recognition unit uses an action recognition device whose parameters are adjusted according to the difference between the processing result based on key points between people and objects detected from the training video and the training description text, and the action class label of the person's action toward the object in the training video, to recognize the action of the person acting toward the object or person in the video to be recognized, using a neural network based on the key points detected by the key point detection unit and the description text. A behavior recognition device having the following features.

15. The acquisition unit acquires a descriptive text indicating an action that cannot be classified into the action class label. The behavior recognition device according to claim 14, wherein the behavior recognition unit uses the behavior recognition device to recognize the person's behavior toward the object from the key points and the unclassifiable description.

16. The process involves obtaining the video to be recognized and a descriptive text that represents the relationship between people and objects or people and people, To detect key points between the person and the object from the aforementioned video, Based on the processing results derived from key points between people and objects detected from training videos and training descriptions, and the difference between these results and the behavior class labels of the people's actions toward the objects in the training videos, a behavior recognition system with adjusted parameters is used to recognize the actions of the person acting toward the object or person in the target video, using a neural network based on the key points detected from the video and the descriptions. A method of action recognition performed by a computer.

Images