Behavior recognition method, behavior recognition device, and computer-readable recording medium
By using the weights that change according to the behavior type in behavior recognition technology to calculate the characteristic value and identify local behaviors, the problem of low accuracy in recognition of part of the body's behavior is solved, and the accurate identification and accuracy of unpreset behaviors are achieved.
Patent Information
- Application Number
- CN202110205353.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-02-28
- Filing Date
- 2021-02-24
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-02-24
AI Technical Summary
Existing behavior recognition techniques are difficult to accurately identify moving behaviors occurring in parts of the body, especially due to reduced recognition accuracy due to motor interference in other parts of the body.
By calculating the position and time series data of the corresponding joints, and calculating characteristic values based on the weights changed according to the behavior type, local behaviors are identified, and motor interference in other body parts are suppressed, so as to realize the identification of unpreset behaviors.
Improves the accuracy of recognition of moving behaviors occurring in parts of the body, reduces the time and energy to reset the time series pattern, and enhances the ability to recognize behaviors of unpreset elements.
Smart Images

Figure CN113326726B_ABST
Abstract
Description
Technical Field
[0001] The embodiments discussed herein relate to a behavior recognition method, a behavior recognition program, and a behavior recognition device. Background Art
[0002] As a technology for recognizing behavior from a video, a technology for recognizing behavior based on basic features of 15 body joints estimated from each frame of a 3D video, ie, features such as posture, speed, and movement is known (for example, refer to Japanese Laid-Open Patent Publication No. 2015-130151).
[0003] However, the above-mentioned technology sometimes has difficulty in recognizing that an action of movement occurs in a part of the body.
[0004] By this technology, behavior is recognized based on basic features extracted from 15 body joints, i.e., features such as posture, speed, and movement. As described above, in the aforementioned technology, behavior is recognized using features such as posture, speed, and movement associated with joints of the whole body. For this reason, in this technology, when the behavior as an object to be recognized is a movement in which movement occurs only in a part of the body, the movement occurring in another part hinders recognition, and thus reduces the accuracy of recognition.
[0005] Therefore, an object in one aspect of an embodiment of the present invention is to provide a behavior recognition method, a behavior recognition program, and a behavior recognition device that make it possible to recognize a behavior whose movement occurs in a part of the body. Summary of the invention
[0006] According to one aspect of the embodiment, a behavior recognition method executed by a computer includes: obtaining the positions of multiple corresponding joints; and identifying the local behavior as an object to be identified based on a feature value, which is calculated according to the position of the corresponding joint, using a weight to be applied to the corresponding joint and changed according to the type of local behavior. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] Figure 1 is a diagram showing an example of a behavior recognition service according to the first embodiment;
[0008] Figure 2 is a graph representing an example of a rule;
[0009] Figure 3 is a schematic diagram showing an example of an application behavior recognition service according to a first embodiment;
[0010] Figure 4A is a schematic diagram showing weights used for joints in an application of the related art;
[0011] Figure 4B is a schematic diagram showing weights for joints in the application of the element behavior recognition function according to the first embodiment;
[0012] Figure 5 is a block diagram showing an example of a functional configuration of a behavior recognition device according to the first embodiment;
[0013] Figure 6 is a diagram showing an example of first weight data;
[0014] Figure 7 is a diagram showing an example of second weight data;
[0015] Figure 8 is a flowchart showing the steps of a model learning process according to a first embodiment;
[0016] Fig. 9 is a flowchart showing the procedure of the behavior recognition processing according to the first embodiment;
[0017] Fig.10 is a diagram showing an example of application to GCNN;
[0018] Fig.11 is a schematic diagram showing an image of parameters; and
[0019] Fig.12 is a diagram showing an example of the hardware configuration of a computer. DETAILED DESCRIPTION
[0020] The preferred embodiments will be described with reference to the accompanying drawings. The embodiments do not limit the disclosed technology. The embodiments can be appropriately combined within the range in which no inconsistency is caused in terms of processing contents.
[0021] [a] First Embodiment
[0022] Example of behavior recognition service
[0023] Figure 1 is a diagram showing an example of a behavior recognition service according to the first embodiment. Figure 1 The behavior recognition service 1 shown in FIG. 1 provides a service for recognizing the behavior of an object such as a person from a video of a camera 2. In the behavior recognition service 1, as an example only, an upper-level behavior that is complicated by combining multiple actions such as suspicious behavior or purchase behavior is an object to be recognized.
[0024] In addition to the above-mentioned behavior recognition service 1, in order to recognize the "upper-level behavior", a large amount of videos containing the behavior must be prepared as training data. Preparing such a large amount of training data may be difficult in practice, or even when a large amount of training data can be prepared, it may take time and effort, and therefore there is an aspect that it is difficult to introduce upper-level behavior recognition into the field.
[0025] To eliminate the need to prepare such a large amount of training data, the behavior recognition service 1 recognizes upper-level behaviors based on rules and elements of upper-level behaviors, where the rules define combinations of "element behaviors" that can be included as "basic movements" taken in daily life.
[0026] like Figure 1 As shown, the behavior recognition service 1 may include a skeleton detection function 3 , a basic motion recognition function 4 and an upper-level behavior recognition function 5 .
[0027] The skeleton detection function 3 is a function of detecting skeleton information including the position of each joint of the object in each frame of the video captured by the camera 2. As an example only, the skeleton detection function 3 may be implemented by a two-dimensional or three-dimensional motion capture technology.
[0028] The basic motion recognition function 4 is a function of recognizing basic motions based on the skeleton information of each frame. As an example only, the basic motion recognition function 4 can be implemented using a model that has learned basic motions based on a machine learning algorithm such as deep learning.
[0029] For example, "basic motion" may include "whole body behavior" whose movement occurs in the whole human body and "local behavior" whose movement occurs in a part of the human body. Behaviors such as "walking", "running" and "stopping" may be taken as examples of "whole body behavior". Behaviors such as "raising right hand", "looking down" and "looking forward" may be taken as examples of "local behavior".
[0030] "Whole body behavior" and "local behavior" are "basic movements" that are commonly taken, and are therefore simple actions compared to "upper-level behavior". For this reason, it is easy to collect videos of "basic movements", and the trained model can be used to achieve accurate recognition of "basic movements".
[0031] The upper layer behavior recognition function 5 is a function for recognizing upper layer behaviors. Figure 1 As shown, the upper-layer behavior recognition function 5 further includes a time series analysis function 5A and a rule checking function 5B.
[0032] The time series analysis function 5A is a function for analyzing whether a basic movement corresponds to a given time series pattern. For example, the continuity of the basic movement or the transition of the state of the basic movement is taken as an example of a time series pattern. The basic movement corresponding to such a time series pattern is identified as an "element behavior".
[0033] The rule checking function 5B is a function of checking the recognition results of the basic behavior and the element behavior with the rule 6. When the rule 6 is satisfied, the upper-level behavior as the object to be recognized is recognized.
[0034] On the one hand, rule 6 defines the upper-level behavior of the object to be identified according to the combination of basic behavior and element behavior. For example, the combination can include basic behavior or element behavior, or both basic behavior and element behavior. When rule 6 defines multiple combinations, the order and logical relationship between the combinations can be defined.
[0035] In the above behavior recognition service 1, the rules 6 can be set not only according to system definition but also according to user definition. For example, the user U, such as a system engineer of the service provider providing the above behavior recognition service 1 or a customer of the behavior recognition service 1, is allowed to edit the rules 6 so that they can be customized by the user U.
[0036] Figure 1 Time series analysis is selectively illustrated. In parallel with the time series analysis, a spatial analysis can be performed on whether the place where the basic behavior occurs satisfies a given spatial condition, such as whether the place is within a region of interest (ROI). For example, when the suspicious behavior is an object to be identified or monitored, such as in the case of picking, the door or the vicinity of the door can be set as the ROI. When the purchase behavior is an object to be identified, the shelf of the product can be set as the ROI.
[0037] Example of application behavior recognition service
[0038] use Figure 2 and Figure 3 , an example of the application behavior identification service 1 according to the first embodiment will be described. Figure 2 is a diagram showing an example of Rule 6. Figure 2 This rule represents an example of limiting "using a smart phone while walking" as an upper-level behavior of an object to be identified. Figure 2As shown, Rule 6 defines the combination in each step in the order from step S1 to step S2. For example, in step S1, the whole body behavior C1 "walking", the element behavior C2 "holding a smart phone", the element behavior C3 "looking down", and the element behavior C5 "start operating a smart phone" are set. In step S2, the whole body behavior C1 "walking", the element behavior C2 "holding a smart phone", the element behavior C4 "looking forward", and the element behavior C6 "ending operating a smart phone" are set.
[0039] Figure 3 is a schematic diagram showing an example of the application behavior identification service 1 according to the first embodiment. Figure 3 Schematically represents the Figure 2 In the case of Rule 6 represented in , the result of identifying the overall behavior and the local behavior in each of the frames t1 to t6 of the video input from the camera 2.
[0040] like Figure 3 As indicated, the whole body behavior C1 "walking" at steps S1 and S2 is continuously identified from frame t1 to frame t6. The local behavior of the right arm "forward" and the local behavior of the right hand "holding an object" continue from frame t1 to frame t6. The continuity of the local behavior of the right arm "forward" and the local behavior of the right hand "holding an object" corresponds to the time series pattern of the element behavior "holding a smart phone". For this reason, the element behavior C2 "holding a smart phone" at steps S1 and S2 is continuously identified from frame t1 to frame t6.
[0041] In addition, in frames t1 and t2, the local behavior of the face changes from "forward" to "downward". Such a chronological change in the local behavior of the face corresponds to the time series pattern of the element behavior C3 "look down". Therefore, the element behavior C3 at step S1 is identified at the moment of frame t2. In addition, in frames t2 and t3, the local behavior of the left arm changes from "downward" to "forward". Such a chronological change in the local behavior of the left arm corresponds to the time series pattern of the element behavior C5 "start operating the smart phone". Therefore, the element behavior C5 at step S1 is identified at the moment of frame t3.
[0042] As described above, at the moment of frame t3, under the AND condition, the whole body behavior C1 "walking", the element behavior C2 "holding a smart phone", the element behavior C3 "looking down", and the element behavior C5 "starting to operate a smart phone" are recognized. Therefore, at the moment of frame t3, step S1 is completed.
[0043] Afterwards, in frames t5 and t6, the local behavior of the face changes from "downward" to "forward". Such a chronological change in the local behavior of the face corresponds to the time series pattern of the element behavior C4 "looking forward". Therefore, the element behavior C4 at step S2 is identified at the moment of frame t6. In addition, in frames t5 and t6, the local behavior of the left arm changes from "forward" to "downward". Such a chronological change in the local behavior of the left arm corresponds to the time series pattern of the element behavior C6 "end operating the smart phone". Therefore, the element behavior C6 at step S2 is identified at the moment of frame t6.
[0044] As described above, at the moment of frame t6, under the AND condition, the whole body behavior C1 "walking", the element behavior C2 "holding a smart phone", the element behavior C4 "looking forward", and the element behavior C6 "ending operating the smart phone" are recognized. Therefore, at the moment of frame t6, step S2 is completed.
[0045] At the time of frame t6 , both step S1 and step S2 are completed in this order, and thus the upper-level behavior of “using a smartphone while walking” satisfying rule 6 is identified.
[0046] As described above, according to the behavior recognition service 1 according to the first embodiment, the upper-level behavior is recognized according to the rule 6 that defines the combination of basic movements and element behaviors, which makes it possible to eliminate the necessity of preparing a large amount of training data. In addition, according to the behavior recognition service 1 according to the first embodiment, not only the rule 6 preset by the system can be used, but also the user U can be allowed to customize the rule 6, which makes it possible to realize the recognition of the upper-level behavior corresponding to the scene.
[0047] One side of the problem
[0048] As described above, according to the behavior recognition service 1 according to the first embodiment, the rule 6 can be customized using basic motions and element behaviors preset by system definition.
[0049] However, the required element behavior varies according to the site, and therefore it is not necessarily possible to define the upper-level behavior corresponding to the site using only the preset element behavior. In order to identify the unpreset element behavior, the time series mode for time series analysis must be reset.
[0050] However, setting the time series pattern has the following aspects: since the expertise of both the behavior recognition service 1 and the site is required, and time and effort are spent, it is difficult to recognize unpreset element behaviors.
[0051] Even using the technology described in the background technology section above, it is difficult to identify such element behaviors. In other words, as described in the background technology section above, in the above technology, behaviors are identified based on basic features extracted from 15 joints, namely, features such as posture, speed, and movement. As described above, in the above technology, behaviors are identified using features such as posture, speed, and movement associated with joints throughout the body. For this reason, in the above technology, when the behavior to be identified is a movement that occurs only in a part of the body, the movement that occurs in other parts hinders the identification, and thus reduces the accuracy of the identification.
[0052] One aspect of a method used to solve a problem
[0053] In this case, in the first embodiment, an element behavior recognition function 7 is added to the behavior recognition service 1, and the element behavior recognition function 7 recognizes the element behavior as the object to be recognized based on the feature value calculated using the weight to be applied to the corresponding joint and changed according to the type of behavior. The element behavior recognition function 7 adopts a method for solving the problem to suppress the movement that hinders the recognition in other parts except the body part corresponding to the behavior element as the object to be recognized.
[0054] Here, only element behaviors are illustrated as examples of behaviors as objects to be identified. As an example only, behaviors as objects to be identified may cover general behaviors whose movement occurs in a part of the body, for example, may be local behaviors. For example, when local behaviors are objects to be identified, local behaviors as objects to be identified may be identified based on feature values calculated according to time series data of the positions of corresponding joints, using weights to be applied to the corresponding joints and changed according to the type of local behaviors.
[0055] Figure 4A is a schematic diagram showing weights for joints in a case where the related art is applied. Figure 4B : is a diagram schematically showing weights of joints in the case where the element behavior recognition function 7 according to the first embodiment is applied. Figure 4A and Figure 4B An example in which “raise the right hand” is an object to be recognized is shown as an example of element behavior, and a skeleton model including positions of joints in frame tk and frame t is shown.
[0056] As an example only, an example is illustrated in which the difference between the position of each joint in frame t and the position of each joint in frame tk, that is, the relative position, is calculated as a velocity feature and the element behavior as the object to be identified is identified based on the velocity features of the corresponding joints.
[0057] When calculating the velocity features of the corresponding joints, the relative positions are calculated for each of the joints, such as the right and left wrists, elbows, shoulders, waists, knees, and ankles. For example, the velocity feature of the right wrist is calculated based on the difference between the position of the right wrist joint in frame t and the position of the right wrist joint in frame tk, the difference between the position of the right wrist joint in frame t and the position of the right elbow joint in frame tk, ..., and the difference between the position of the right wrist joint in frame t and the position of the left ankle joint in frame tk.
[0058] Figure 4A The thickness of the dotted line indicates the size of the weight applied to each joint relative to the right wrist when calculating the velocity feature of the right wrist in the related art. Figure 4A In other words, the velocity feature of the right wrist is calculated in a state where the same weight is applied between the relative positions calculated based on the positions of the right wrist, right elbow, right shoulder, etc. related to the element behavior "raise right hand" in the frame tk and the relative positions calculated based on the positions of the left arm, lower body, etc. that are not related to the element behavior "raise right hand" in the frame tk.
[0059] exist Figure 4B In the figure, only as an example, the thickness of the dotted line indicates the size of the weight applied to the corresponding joint relative to the right wrist when calculating the velocity characteristics of the right wrist in the element behavior recognition function 7. Different weights are applied between the relative positions of the joints related to the element behavior "raise the right hand" in frame tk and the relative positions of the joints not related to the element behavior "raise the right hand" in frame tk. Figure 4B As shown, weights are applied according to weight data, in which weights to be applied to corresponding nodes are used according to the type of element behavior to be identified to calculate the relative position of the joint relative to the velocity feature to be calculated. For example, a weight of "1" is applied to the relative position calculated between the position of the joint of the right wrist in frame t and the position of the joint of the right wrist in frame tk. A weight of "1 / 2" is applied to the relative position calculated between the position of the joint of the right wrist in frame t and the position of the joint of the right elbow in frame tk. A weight of "1 / 4" is applied to the relative position calculated between the position of the joint of the right wrist in frame t and the position of the joint of the right shoulder in frame tk. On the other hand, a weight of "0" is applied to the relative position calculated according to the position of the joint such as the left arm or the lower body in frame tk. In this way, the velocity feature of the right wrist is calculated using a weight to be applied to the relative position calculated according to the joint position related to the element behavior "raising the right hand" in frame tk and greater than the weight to be applied to the relative position calculated according to the joint position unrelated to the element behavior "raising the right hand".
[0060] The relative position calculated from the position of the part irrelevant to the element behavior of "raising the right hand" generates an obstacle, i.e., noise, in identifying the element behavior of "raising the right hand". As an example only, a case where a captured video of "raising the right hand while walking" and a captured video of "walking" are input is illustrated. Advantageously, it is assumed that the movement of joints other than the right wrist, right elbow, and right shoulder is the same between the two videos.
[0061] For example, when a skeleton model including the positions of 12 joints excluding the head is obtained, the number of joints with the same positions except for the right wrist, right elbow, and right shoulder in each frame between the two videos is 9. Even when the positions of the remaining three joints are different in each frame between the two videos, 75% (=9÷12×100) of the relative positions have the same value in calculating the velocity feature of the right wrist.
[0062] For this reason, according to the related art, the same weight is applied to each joint, and in the calculation of the velocity feature of the right wrist, 75% (=9÷12×100) of the relative positions generate noise. This increases the possibility that the velocity features of the right wrist calculated between the two videos are similar to each other. Therefore, even a captured video of "walking" is recognized as an element behavior "raising the right hand" as an object to be recognized, and therefore, the accuracy of recognition is reduced.
[0063] On the other hand, according to the element behavior recognition function 7 according to the first embodiment, since a larger weight is applied to the joint related to the element behavior "raising the right hand" than the weight for the joint unrelated to the element behavior "raising the right hand", the speed feature of the right wrist can be calculated in a state where noise is excluded. Therefore, even the recognition of the captured "walking" video as the element behavior "raising the right hand" as the object to be recognized can be suppressed.
[0064] Therefore, according to the element behavior recognition function 7, it is possible to recognize an element behavior that is not pre-set as an example of a behavior in which movement occurs in a part of the body. In addition, according to the element behavior recognition function 7, when an unpreset element behavior is added to a rule, the time and effort for resetting a time series pattern for time series analysis can be reduced.
[0065] Functional configuration of behavior recognition equipment
[0066] Figure 5 : is a block diagram showing an example of the functional configuration of the behavior recognition device 10 according to the first embodiment. Figure 5 The illustrated behavior recognition device 10 corresponds to a computer that provides the above-mentioned behavior recognition service 1 .
[0067] In one embodiment, the behavior recognition device 10 can be implemented by installing a behavior recognition program that implements the above-mentioned behavior recognition device 10 in a given computer as packaged software or online software. For example, the behavior recognition device 10 can be installed as a SaaS (Software as a Service) application, thereby providing the above-mentioned behavior recognition service 1 as a cloud service. The behavior recognition device 10 is not limited to this, and the behavior recognition device 10 can be installed as a server that locally provides functions corresponding to the behavior recognition service 1.
[0068] like Figure 5 As shown, the camera 2 corresponding to the example of the imaging device can be connected to the behavior recognition device 10. The behavior recognition device 10 and the camera 2 can be connected to each other via a given network so that the behavior recognition device 10 and the camera 2 can communicate with each other. For example, the network can be any type of communication network, such as the Internet or a LAN (Local Area Network), regardless of whether the network is wired or wireless. Figure 1 and Figure 5 An example in which a single camera 2 is connected to a single behavior recognition device 10 is conveniently illustrated; however, this does not prevent a plurality of cameras 2 from being connected to a single behavior recognition device 10 .
[0069] like Figure 5 As shown, the behavior recognition device 10 includes a video acquisition unit 11, a skeleton information acquisition unit 12, a basic motion recognition unit 13, an upper-layer behavior recognition unit 14, a rule storage device 15, an element behavior recognition unit 17, a first weight storage device 18, and a second weight storage device 19. In addition to the Figure 5 In addition to the functional units shown, various functional units of a known computer may also be included, such as functions corresponding to an input / output interface and a communication interface.
[0070] Functional units such as the video acquisition unit 11, the skeleton information acquisition unit 12, the basic motion recognition unit 13, the upper-level behavior recognition unit 14, and the element behavior recognition unit 17 are actually implemented by a hardware processor such as a central processing unit (CPU) or a microprocessing unit (MPU). In other words, in addition to reading an operating system (OS) from a storage device (not shown in the drawings) such as a hard disk drive (HDD), an optical disk, or a solid-state drive (SSD), the processor also reads a program such as a behavior recognition program obtained by packaging the above-mentioned behavior recognition service 1. Then, the processor executes the above-mentioned behavior recognition program, thereby loading the processing corresponding to the above-mentioned functional unit into a memory such as a random access memory (RAM). As a result of executing the above-mentioned behavior recognition program as described above, the above-mentioned functional unit is actually implemented as a process. Here, a CPU or an MPU is taken as an example of a processor, and the above-mentioned functional unit can be implemented by a given processor, regardless of whether the processor is a general-purpose processor or a special-purpose processor. The above-mentioned functional unit or a part of the functional unit can be implemented using wired logic by hardware such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA).
[0071] The above-mentioned storage means, such as the rule storage means 15, the first weight storage means 18 and the second weight storage means 19 can be set as an auxiliary storage device, such as an HDD, an optical disk or an SSD, or can be set by allocating part of the storage area of an auxiliary device.
[0072] The video acquisition unit 11 is a processor that acquires a video.
[0073] In an embodiment, the video acquisition unit 11 can acquire the video sent from the camera 2 frame by frame. The information source from which the video acquisition unit 11 acquires the video can be any information source and is not limited to the camera 2. For example, the video acquisition unit 11 can also acquire the video from an auxiliary storage device such as a hard disk or an optical disk storing the video, or acquire the video from a removable medium such as a memory card or a universal serial bus (USB). The video acquisition unit 11 can also acquire the video from an external device other than the camera 2 via the network NW.
[0074] The skeleton information acquisition unit 12 is a processor that detects skeleton information including the positions of the corresponding joints of the object in each frame of the video. As an example only, the skeleton information acquisition unit 12 inputs the video into the motion capture engine for each frame in which the video is captured by the video acquisition unit 11, so that the skeleton information including the positions of the corresponding joints can be acquired. For example, when performing two-dimensional motion capture, the coordinate set (x t t ) of the joint n = 1, ..., N in frame t can be acquired. t 1, y t 1) (x t 2, y t2) ...(x t N,y t N). The index n identifies the right wrist, right elbow, right shoulder, left wrist, ... and left ankle in the order of n=1, ..., N. The motion capture engine does not necessarily have to be executed by the processor on the behavior recognition device 10, and an external engine may be used.
[0075] The basic motion recognition unit 13 is a processor that recognizes basic motions based on the skeleton information on each frame. As an example only, the basic motion recognition unit 13 inputs the positions of the corresponding joints acquired in each frame by the skeleton information acquisition unit 12 into a model that has learned the basic motions based on a machine learning algorithm such as deep learning. Therefore, the result of recognizing the basic motions can be obtained on a frame-by-frame basis.
[0076] The upper layer behavior recognition unit 14 is a processor for recognizing upper layer behaviors. Figure 5 As shown, the upper-level behavior recognition unit 14 further includes a time series analyzer 14A and a rule checker 14B.
[0077] The time series analyzer 14A is a processor that analyzes whether a basic motion corresponds to a given time series pattern. For example, the continuity of the basic motion and the transition of the state of the basic motion can be taken as examples of the time series pattern. Such basic motions corresponding to the time series pattern are identified as "element behaviors".
[0078] The rule checker 14B is a processor that checks the recognition results of the basic behavior and the element behavior with the rules stored in the rule storage device 15. For example, the rule checker 14B uses not only the element behavior recognized in the time series analysis performed by the time series analyzer 14A but also the element behavior recognized by the element behavior recognition unit 17, which will be described below, to check with the rules. When the rule is satisfied, the upper-level behavior as the object to be recognized is recognized.
[0079] The rule storage device 15 stores rules defining the upper-level behaviors of objects to be identified by the combination of basic behaviors and element behaviors. For example, a combination may include basic behaviors or element behaviors, or both basic behaviors and element behaviors. When a rule defines multiple combinations, the order and logical relationship between the combinations may be defined.
[0080] As an example only, the rule storage device 15 may store Figure 2 The rule 6, etc. Figure 2As shown, Rule 6 defines the combination in each step in the order from step S1 to step S2. For example, in step S1, the basic behavior (whole body behavior) C1 "walking", the element behavior C2 "holding a smart phone", the element behavior C3 "looking down" and the element behavior C5 "start operating a smart phone" are set. In step S2, the basic behavior (whole body behavior) C1 "walking", the element behavior C2 "holding a smart phone", the element behavior C4 "looking forward" and the element behavior C6 "end operating a smart phone" are set.
[0081] As described above, the rules 6 set not only according to the system definition but also according to the user definition can be stored in the rule storage device 15. For example, allowing a user U, such as a system engineer of a business operator providing the above-mentioned behavior recognition service 1 or a customer of the behavior recognition service 1, to edit the rules 6 enables customization by the user U.
[0082] The element behavior recognition unit 17 is a processor that recognizes the element behavior of the object to be recognized. Figure 5 As shown, the element behavior identification unit 17 further includes a calculator 17A and a determination unit 17B.
[0083] The calculator 17A is a processor that calculates the characteristic value of the element behavior.
[0084] On the one hand, the calculator 17A changes the weight to be applied to each joint according to the type of element behavior as the object to be recognized. The weight of each joint changed according to the type of element behavior as the object to be recognized may be referred to as a "first weight" hereinafter.
[0085] The example of calculating the velocity feature represented by the following equation (1) will be used only as an example of the feature value. In the velocity feature represented by the following equation (1), the velocity feature f of the joint n in the frame t is calculated according to the following equation (2): s (X t n ). Note that "k" expressed in the following equation (2) represents a constant set in advance, and, for example, any natural number can be set.
[0086]
[0087]
[0088] Just as an example, an example will be described in which a speed feature is calculated in a case where an element behavior as an object to be recognized is “raising the right hand”.
[0089] First, an example is given in which the velocity feature f of the joint of the right wrist in frame t is calculated from the velocity features of the N joints in frame t. s (x t右手腕 In other words, the calculator 17A sets the first weight w in the above equation (2) according to the first weight data stored in the first weight storage device 18. i .
[0090] Figure 6 is a diagram showing an example of first weight data. Figure 6 The weights for joints that have been learned according to a machine learning algorithm such as SVM (Support Vector Machine) and model parameters are merely taken as examples of weight data. Figure 6 As an example of the first weight data, the first weight w for each joint i=1, ..., N to be used for calculating the relative position with respect to the joint whose velocity characteristic is to be calculated is represented as follows: i Data associated with each element behavior as an object to be identified may be used as an example.
[0091] For example, when calculating the relative position between the position of the joint of the right wrist in frame t and the position of the joint of the right wrist in frame tk (when i=1), the first weight w 1 set up Figure 6 The weight of the right wrist in the first weight data represented is "1". When calculating the relative position between the position of the right wrist joint in frame t and the position of the right elbow joint in frame tk (when i=2), the first weight w 2 set up Figure 6 The weight of the right elbow in the first weight data represented is "1 / 2". In addition, when calculating the relative position between the position of the right wrist joint in frame t and the position of the right shoulder joint in frame tk (when i=3), the first weight w 3 set up Figure 6 The weight of the right shoulder in the first weight data is represented as "1 / 4". In this way, the same process is repeated until the index i reaches N.
[0092] On the other hand, the calculator 17A changes the weight to be applied to each joint according to the cooperativeness between the joints. The weight for each joint changed according to the cooperativeness between the joints may be referred to as a "second weight" hereinafter.
[0093] For example, the calculator 17A sets the second weight b in the above equation (2) according to the second weight data stored in the second weight storage device 19. n .
[0094] Figure 7 is a diagram showing an example of the second weight data. Figure 7, where the second weight b to be applied to each joint i in frame tk is used to calculate the relative position of the joint n whose velocity characteristic is to be calculated. n Data associated with each joint n can be used as an example. For example, in the case where a high value is set for the second weight as the cooperation between joint n and joint i increases, a low value is set for the second weight as the cooperation between joint n and joint i decreases.
[0095] For example, when calculating the velocity feature f of the right wrist s (x t 右手腕 ) (when n=1), refer to Figure 7 First, when calculating the relative position between the position of the joint of the right wrist in frame t and the position of the joint of the right wrist in frame tk (when i=1), the second weight b1 is set to include Figure 7 The weight of the right wrist identified by index i=1 in the second weights of the joints i=1, ..., N in the record in the first row identified is "1". In addition, when the joints in frame tk are the right elbow and the right shoulder, the second weight b1 is set to "1". When calculating the relative position between the position of the joint of the right wrist in frame t and the position of the joint of the left wrist in frame tk (when i=4), the second weight b1 is set to the value contained in Figure 7 The weight of the left wrist identified by index i=4 in the second weights of joints i=1, ..., N in the record of the first row shown is "0." In this way, the same process is repeated until index i reaches N.
[0096] After setting the first weight w as described above i and the second weight b n Then, the calculator 17A assigns the position of each joint in frame t and the position of each joint in frame tk to the above equation (2), thereby calculating the velocity feature f of the right wrist joint in frame t. s (x t 右手腕 ). In the same way, the calculator 17A calculates the velocity feature f of the right elbow joint in frame t. s (x t 右肘 ),…, velocity feature f of the left ankle joint in frame t s (x t 左脚踝 ) and the like. Thereafter, the calculator 17A calculates the N speed features f in frame t. s (x t 1) to fs(x tN) is assigned to equation (1), thereby calculating the velocity feature of the element behavior "raising the right hand" as the object to be recognized in frame t.
[0097] Figure 6 and Figure 7 The case where each data is stored in a table form is illustrated; however, this is only an example, and the data structure is not limited to a relational database. For example, the following data can be used: data described in a tag form using a markup language such as XML (Extensible Markup Language), or data described using commas or line breaks such as CSV (Comma Separated Values).
[0098] Figure 7 The case where "1" or "0" is set as the second weight is illustrated; however, the second weight is not necessarily limited to the binary digit "1" or "0", and any value from "0" to "1" may be set.
[0099] The determination unit 17B is a processor that determines whether or not there is an element behavior as an object to be recognized based on the feature value calculated by the calculator 17A.
[0100] As an example only, the determination unit 17B inputs the speed feature calculated by the calculator 17A into the trained SVM model. For example, when the trained model is a linear recognition model, the trained model is based on Figure 6 The identification boundary parameters of the model represented, namely the slope a and the intercept b, are used to determine the identification boundary to output a label "1" with element behavior or a label "0" without element behavior. In this way, it is possible to identify whether there is an element behavior as an object to be identified based on the label output from the trained model.
[0101] As an example only, such model parameters such as slope a and intercept b can be learned as follows. For example, a video corresponding to a positive example or a negative example of an element behavior of an object to be identified and training data containing correct labels for the positive and negative examples are used. Initial values are set for the "first weight" in the following equation (3) and the "recognition boundary parameter" of the model, and then the element behavior of the object to be identified is identified based on the speed features calculated based on the training data and the recognition boundary parameters of the model. Then, the process of updating the "first weight" and the "recognition boundary parameter" is repeated within a given number of epochs so that the loss function L in the following equation (4) is minimized. In equation (4), "q(x)" represents an activation function, and for example, a tanh function, a sigmoid function, or a RELU function can be used. In equation (4), "f s ” represents the first weight w to be applied to each joint i iFor example, the “first weight” and the “identification boundary parameter” are optimized by optimizing the log-likelihood that maximizes the parameters (t, q(fs)) on the loss function L, the value of which decreases as the number of correct solutions increases, and the value increases as the number of correct solutions decreases.
[0102]
[0103] L(t,q(f s ))=log(1+exp(+tq(f s ))) (4)
[0104] Processing Flow
[0105] The flow of processing performed by the behavior recognition device 10 according to the first embodiment will be described. A (1) model learning process will be described, and then (2) behavior recognition processing will be described.
[0106] (1) Model learning process
[0107] Figure 8 is a flowchart showing the steps of the model learning process according to the first embodiment. The model learning process can be performed by any learning device including the behavior recognition device 10. For example, when a request for model learning is received, the model learning process starts. Note that initial values are set for the "first weight" and "recognition boundary parameter" in the above equation (3).
[0108] like Figure 8 As shown, the learning device obtains training data, which includes videos corresponding to positive or negative examples of element behaviors of the object to be identified and correct labels of the positive or negative examples (step S101).
[0109] Then, the learning device acquires time series data on the positions of joints from the videos included in each training data set acquired at step S101 (step S102 ).
[0110] Thereafter, the learning device repeats the process from the following step S103 to the following step S108 until a given number of epochs is reached, and then the process is terminated. In addition, in one epoch, the learning device repeats the process from the following step S103 to the following step S107 a number of times corresponding to the number of training data sets. In addition, for one training data set, the learning device repeats the process from the following step S103 to the following step S106 a number of times corresponding to the number of element behaviors as objects to be recognized.
[0111] In other words, the learning device refers to the first weight data stored in the first weight storage device 18, and sets the first weight w to be applied to each joint i=1, ..., N for calculating the relative position relative to the joint n for each element behavior as the object to be recognized. i (Step S103). Then, the learning device refers to the second weight data stored in the second weight storage device 19, and sets the second weight b to be applied to each joint n=1, ..., N for calculating its velocity characteristic and each joint i=1, ..., N for calculating the relative position relative to the joint n in the frame tk. n , (step S104).
[0112] Then, the learning device also calculates the speed features f in N frames s (x t 1) to f s (x t N ) is assigned to the above equation (1), these speed features are obtained by setting the first weight w i and the second weight b n Then, the position of each joint in frame t and the position of each joint at frame tk are assigned to equation (2), thereby calculating the velocity feature of the element behavior in frame t (step S105).
[0113] The learning device inputs the speed characteristics of the element behavior in frame t calculated at step S105 into the recognition boundary parameters, such as a model with slope a and intercept b, thereby determining whether there is an element behavior as an object to be recognized based on the label output from the model (step S106).
[0114] Thereafter, the learning device compares the recognition result of the element behavior as the object to be recognized with the correct label, and thus counts the number of correct solutions and the number of incorrect solutions (step S107 ).
[0115] Then, the learning device updates parameters including the “first weight” and the “identification boundary parameter” based on the number of correct solutions and the number of incorrect solutions counted at step S107 (step S108 ).
[0116] (2) Behavior recognition processing
[0117] Fig. 9 : is a flowchart showing the procedure of the action recognition process according to the first embodiment. As an example, this process may be started when the video acquisition unit 11 acquires a video of a new frame.
[0118] like Fig. 9As shown, the video acquisition unit 11 acquires a frame of video from the camera 2 or the like (step S301). The skeleton information acquisition unit 12 inputs the video acquired at step S301 into the motion capture engine, thereby acquiring skeleton information including the position of each joint (step S302).
[0119] After step S302, the A process of steps SA303 to SA307 and the B process of steps SB303 and SB304 regarding the element behavior are performed in parallel. The A process and the B process do not necessarily need to be performed in parallel, but may be performed in series. In this case, the A process and the B process may be performed in a random order.
[0120] For example, at step SB303, the basic motion recognition unit 13 inputs the position of each joint acquired at step S302 into the model that has learned the basic motion, thereby obtaining the result of recognizing the basic motion output from the model. At step SB304, the time series analyzer 14A analyzes whether the basic motion recognized at step SB303 corresponds to a given time series pattern.
[0121] As described above, when a basic motion is identified at step SB303, the basic motion is saved in a working area of a memory not shown in the drawings. When a basic motion corresponds to a time series pattern at step SB304, an element behavior corresponding to the time series pattern of the basic motion is saved in a working area of a memory not shown in the drawings.
[0122] On the other hand, the processing from step SA303 to step SA307 is repeated on each frame a number of times corresponding to the number of element behaviors as objects to be recognized.
[0123] In other words, the calculator 17A refers to the first weight data stored in the first weight storage device 18, and sets the first weight w to be applied to each joint i=1, ..., N for calculating the relative position with respect to the joint n for calculating the velocity feature for each element behavior as the object to be recognized. i (Step SA303).
[0124] Subsequently, the calculator 17A refers to the second weight data stored in the second weight storage device 19 and sets the second weight b to be applied to each joint n=1, ..., N for calculating the velocity feature and each joint i=1, ..., N in the frame tk for calculating the relative position with respect to the joint n. n (Step SA304).
[0125] Then, the calculator 17A also calculates the velocity features f in the N frames. s (x t 1) to f s (xt N ) is assigned to the above equation (1), these speed features are in setting the first weight w i and the second weight b n Then, the velocity feature of the element behavior in frame t is calculated by assigning the position of each joint in frame t and the position of each joint in joint tk to equation (2) (step SA305).
[0126] When the element behavior is recognized at step SA306 (Yes at step SA306), the element behavior recognized at step SA306 is saved in a work area of a memory not shown in the drawing (step SA307). On the other hand, when the element behavior is not recognized at step SA306 (No at step SA306), the above-mentioned processing at step SA307 is skipped.
[0127] After steps SB304 and SA307 are completed, the rule checker 14B checks the basic behavior identified at step SB303, the element behavior identified at step SB304, and the element behavior identified at step SA307 with the rules stored in the rule storage device 15 (step S308).
[0128] The rule checker 14B outputs the recognition result of the upper-level behavior to a given output destination (step S309). The "output destination" herein can be any hardware or software. For example, the recognition result of the upper-level behavior can be output to a service or function that performs any back-end processing based on the recognition result of the upper-level behavior. Alternatively, the recognition result of the upper-level behavior can be notified to a display device, an audio output device, an information processing device, or a mobile terminal device set by the user U of the above-mentioned behavior recognition service 1. Any form of notification such as web (webpage) or email can be used.
[0129] One aspect of the effect
[0130] As described above, when identifying element behaviors based on time series data of the positions of corresponding joints, the behavior recognition device 10 according to the first embodiment recognizes the element behaviors as the objects to be recognized based on feature values calculated using weights to be applied to the corresponding joints and changed according to the types of element behaviors. This suppresses the recognition of motion obstructions that occur in other body parts other than the body part corresponding to the element behaviors as the objects to be recognized. Therefore, according to the behavior recognition device 10 according to the first embodiment, it is possible to realize the recognition of element behaviors that are not preset as examples of behaviors in which movement occurs in a part of the body. In addition, according to the behavior recognition device 10 according to the first embodiment, when non-preset element behaviors are added to the rules, it is also possible to save time and effort for resetting the time series pattern used in the time series analysis.
[0131] [b] Second Embodiment
[0132] The embodiments of the disclosed device have been described, and the present disclosure may be performed in various different modes in addition to the above-described embodiments. Another embodiment covered by the present disclosure will be described below.
[0133] Application scope of element behavior recognition
[0134] In the first embodiment described above, only an example in which the recognition of element behavior is achieved using the first weight and model parameters learned according to the SVM algorithm is illustrated as an example; however, the recognition is not limited to this, and the recognition of element behavior can be achieved by any method. As an example thereof, an example in which the recognition of element behavior is achieved by GCNN (Graph Convolutional Neural Network) that performs a convolution operation on a graph is illustrated.
[0135] Fig.10 is a diagram showing an example of applying GCNN. Fig.11 is a diagram schematically showing an image of parameters. Fig.10 A GCNN is shown where joints are represented by nodes and the human body structure is represented using a graph structure. Fig.10 and Fig.11 This example shows that when calculating the velocity feature f of the right wrist joint in frame t among the velocity features of N joints in frame t, s (x t 右手腕 ) (when n=1) performs a convolution operation.
[0136] According to equation (2), when calculating the velocity feature f of the right wrist joint in frame t s (x t 右手腕), calculate the position (x) of joint n=1 in frame t t 1, y t 1) and the position (x) of each joint i=1, ..., N in frame tk t-k i ,y t-k i ). For these N relative positions (x t n -x t-k i ), applying the second weight b 右手腕 × first weight w i .
[0137] On the other hand, Fig.10 In the GCNN model shown, the second weight b 右手腕 × first weight w i Applied to the position (x t-k i ,y t-k i ), the second weight b is applied as described above in frame tk 右手腕 × first weight w i The position of joint i=1, ..., N (x t-k i ,y t-k i ) convolution at the position (x) of the right wrist joint n=1 in frame t t 1, y t 1) In.
[0138] Its schematic representation is generated in Fig.11 For example, Fig.11 As shown, when i=1, the second weight b 右手腕 (=1)×first weight w 1 (=1) is applied to the position (x) of the right wrist joint with i=1 in frame tk t-k 右手腕 ,y t-k 右手腕 ). When i=2, the second weight b 右手腕 (=1)×first weight w 2 (=1 / 2) is applied to the position of the right elbow joint ((x t-k 右肘 ,y t-k 右肘 ). When i=3, the second weight b 右手腕 (=1)×first weight w 3(=1 / 4) is applied to the position (x) of the right shoulder joint with i=3 in frame tk t-k 右肩 ,y t-k 右肩 ). The first weight and the second weight are similarly applied to i=4 and below. For example, when i=N, the second weight b 右手腕 (=0)×first weight w N (=1 / 128) is applied to the position (x) of the left ankle joint with i=N in frame tk. t-k 左脚踝 ,y t-k 左脚踝 ). By executing Figure 8 The model training process shown in the figure is similar to the process of learning the second weight b 右手腕 × first weight w i As the weights of the synapses of the GCNN model.
[0139] As mentioned above, in the GCNN model, according to the above equation (2), the second weight b 右手腕 × first weight w i Applied to the position (x t-k i ,y t-k i ) convolution at the position (x) of the right wrist joint n=1 in frame t t 1. Y t 1). Similarly, for the velocity feature f of the joint at or after n=2 in frame t s (x t n ), perform similar convolution operations. As a result of performing such convolution operations until n=N, the output layer of the GCNN model outputs the result (correct or wrong) of the element behavior of the object to be recognized or the certainty factor of the element behavior.
[0140] As described above, it is apparent that the above-mentioned recognition of element behavior may be achieved using a GCNN model or by any other method other than the example shown in the above-mentioned first embodiment.
[0141] Application scope of the object to be identified
[0142] In the above-mentioned first embodiment, only element behaviors are illustrated as examples of behaviors of objects to be identified; however, behaviors are not limited to this. As an example only, the behavior of an object to be identified may be a general behavior whose movement occurs in a part of a body, such as a local behavior. For example, when a local behavior is an object to be identified, the local behavior can be identified based on feature values calculated based on time series data about the positions of corresponding joints, using weights to be applied to the corresponding joints and changed according to the type of local behavior. In other words, reading "element behaviors" in the description of the element behavior identification unit 17 as "local behaviors" enables the identification of local behaviors, just like identifying element behaviors.
[0143] Distribution and Integration
[0144] Each component of each device shown in the drawings is not necessarily physically configured as shown in the drawings. In other words, the specific form of distribution and integration of the devices is not limited to the form shown in the drawings, and all or part of the components can be configured by functionally or physically distributing or integrating in any unit according to various types of loads and uses. For example, the video acquisition unit 11, the skeleton information acquisition unit 12, the basic motion recognition unit 13, the upper-level behavior recognition unit 14, or the element behavior recognition unit 17 can be connected via a network as an external device of the behavior recognition device 10. Different devices can include a video acquisition unit 11, a skeleton information acquisition unit 12, a basic motion recognition unit 13, an upper-level behavior recognition unit 14, and an element behavior recognition unit 17, respectively, and can be connected and cooperate with each other via a network, thereby realizing the functions of the above-mentioned behavior recognition device 10.
[0145] Behavior recognition program
[0146] The various processes described in the above embodiments can be realized by executing a program prepared in advance using a computer such as a personal computer or a workstation. Fig.12 , an example of a computer that executes a behavior recognition program having the same functions as those of the above-described first and second embodiments will be described.
[0147] Fig.12 is a diagram showing an example of the hardware configuration of a computer. Fig.12 As shown, the computer 100 includes an operation unit 110a, a speaker 110b, a camera 110c, a display 120, and a communication unit 130. The computer 100 also includes a CPU 150, a ROM 160, a HDD 170, and a RAM 180. The units 110 to 180 are connected to each other via a bus 140.
[0148] like Fig.12As shown, a behavior recognition program 170a that realizes the same functions as those of the video acquisition unit 11, the skeleton information acquisition unit 12, the basic motion recognition unit 13, the upper-layer behavior recognition unit 14, and the element behavior recognition unit 17 is stored in the HDD 170. The behavior recognition program 170a may be integrated or separated as components of the video acquisition unit 11, the skeleton information acquisition unit 12, the basic motion recognition unit 13, the upper-layer behavior recognition unit 14, and the element behavior recognition unit 17. In other words, all the data represented in the above-mentioned embodiment need not necessarily be stored in the HDD 170, and data used for processing may be stored in the HDD 170.
[0149] In this case, the CPU 150 reads the behavior recognition program 170a from the HDD 170, and then loads the behavior recognition program 170a into the RAM 180. Fig.12 As shown, the behavior recognition program 170a is used as the behavior recognition process 180a. The behavior recognition process 180a loads various types of data read from the HDD 170 into an area allocated to the behavior recognition process 180a in the storage area of the RAM 180, and performs various processes using the loaded various types of data. For example, examples of the processes performed by the behavior recognition process 180a include Figure 8 and Fig. 9 In the CPU 150, all the processors shown in the above-mentioned first embodiment do not necessarily need to be operated, and a processor corresponding to the processing to be executed can be actually realized.
[0150] The behavior recognition program 170a does not necessarily need to be stored in the HDD 170 or the ROM 160 from the beginning. For example, the behavior recognition program 170a may be stored in a "portable physical medium" such as a floppy disk (i.e., FD), a CD-ROM, a DVD disk, a magneto-optical disk, or an IC card inserted into the computer 100. The computer 100 may obtain the behavior recognition program 170a from the portable physical medium and execute the behavior recognition program 170a. The behavior recognition program 170a may be stored in another computer or a server device connected to the computer 100 via a public line, the Internet, a LAN, or a WAN, and the computer 100 may obtain the behavior recognition program 170a from the computer or the server device and execute the behavior recognition program 170a.
[0151] It is possible to recognize the behavior of a part of the body in which movement occurs.
Claims
1. A behavior recognition method executed by a computer, the behavior recognition method comprising: Obtaining positions of a plurality of corresponding joints in a human body; as well as identifying a type of a local behavior among a plurality of local behaviors based on a feature value calculated according to the position of the corresponding joint using a weight to be applied to the corresponding joint and stored in a storage unit according to the types of the plurality of local behaviors, wherein Among the plurality of local behaviors, the local behaviors are objects to be recognized and movements of the plurality of local behaviors occur in a part of the body of the person, and and also changing the weights to be applied to the corresponding joints according to the cooperativity between the joints, the cooperativity being stored in a storage unit for the corresponding joints, wherein the identifying comprises: referring to weight data in which a weight to be applied to the corresponding joint has been learned according to a given machine learning algorithm, and calculating a feature value based on a relative position between a position of the joint at a first time point and a position of the corresponding joint at a second time point before the first time point according to time series data on the position of the corresponding joint, and Based on the calculated feature values and the model whose parameters have been learned according to the machine learning algorithm, it is determined whether the type of local behavior exists.
2. The behavior recognition method according to claim 1, wherein: The identification includes: identifying the type of element behavior corresponding to the time series pattern of the local behavior based on feature values, wherein the feature values are calculated based on time series data about the position of the corresponding joint, using weights to be applied to the corresponding joint and stored in the storage unit according to the type of the element behavior.
3. A computer-readable recording medium having a behavior recognition program stored therein, wherein the behavior recognition program causes a computer to execute a process including: obtaining positions of a plurality of corresponding joints in a human body; and identifying a type of a local behavior among a plurality of local behaviors based on a feature value calculated according to the position of the corresponding joint using a weight to be applied to the corresponding joint and stored in a storage unit according to the types of the plurality of local behaviors, wherein Among the plurality of local behaviors, the local behaviors are objects to be recognized and movements of the plurality of local behaviors occur in a part of the body of the person, and The weights to be applied to the corresponding joints are also changed according to the cooperation between the joints, the cooperation being stored in the storage unit for the corresponding joints, wherein, The identification includes: referring to weight data in which a weight to be applied to the corresponding joint has been learned according to a given machine learning algorithm, and calculating a feature value based on a relative position between a position of the joint at a first time point and a position of the corresponding joint at a second time point before the first time point according to time series data on the position of the corresponding joint, and Based on the calculated feature values and the model whose parameters have been learned according to the machine learning algorithm, it is determined whether the type of local behavior exists.
4. The computer-readable recording medium according to claim 3, wherein: The identification includes: identifying the type of element behavior corresponding to the time series pattern of the local behavior based on feature values, wherein the feature values are calculated based on time series data about the position of the corresponding joint, using weights to be applied to the corresponding joint and stored in the storage unit according to the type of the element behavior.
5. A behavior recognition device, comprising: an acquisition unit configured to acquire positions of a plurality of corresponding joints in a human body; as well as an identification unit configured to identify a type of a local behavior among a plurality of local behaviors based on a feature value calculated according to the position of the corresponding joint using a weight to be applied to the corresponding joint and stored in a storage unit according to the types of the plurality of local behaviors, wherein Among the plurality of local behaviors, the local behaviors are objects to be recognized and movements of the plurality of local behaviors occur in a part of the body of the person, and and also changing the weights to be applied to the corresponding joints according to the cooperation between the joints, the cooperation being stored in a storage unit for the corresponding joints, wherein the recognition unit comprises: a calculator configured to: refer to weight data in which a weight to be applied to the corresponding joint has been learned according to a given machine learning algorithm, and calculate a feature value based on a relative position between a position of the joint at a first time point and a position of the corresponding joint at a second time point before the first time point according to time series data on the position of the corresponding joint, and A determination unit is configured to determine whether the type of local behavior exists based on the calculated feature value and a model whose parameters have been learned according to the machine learning algorithm.
6. The behavior recognition device according to claim 5, wherein: The identification unit is configured to identify the type of element behavior corresponding to the time series pattern of the local behavior based on a feature value, wherein the feature value is calculated based on the time series data about the position of the corresponding joint, using a weight to be applied to the corresponding joint and stored in the storage unit according to the type of the element behavior.
