Behavior recognition device, behavior recognition method, and program

Abstract

To provide a behavior recognition device, a behavior recognition method and a program for accurately recognizing the behavior of an object.SOLUTION: A behavior recognition device 10 includes: an estimation part 11 for estimating the motion of a target object for each target object image by using a target object image corresponding to the target object included in an image acquired in time series; and a behavior determination part 12 for determining the motion of the target object to be the estimated motion when the same estimated motion continues for the preset number of times, and also determines the behavior whose order matches as the behavior of the target object when the order in which the target object's motions are determined matches the order in which the motions preset for each behavior representing a series of motions of the target object are transited.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technical field relates to a behavior recognition device, a behavior recognition method, and a program for recognizing behavior. [Background technology]

[0002] A device that uses the results of behavior recognition to execute appropriate processing can execute appropriate processing as the accuracy of human behavior recognition improves. However, because human behavior differs from person to person, it is difficult to recognize behavior through simple matching alone.

[0003] As a related technology, a system capable of detecting human behavior is proposed in Patent Document 1. The system in Patent Document 1 includes a plurality of RFID (Radio Frequency Identification) tags attached to a plurality of objects present in a space, a portable device carried by a user moving within the space, and a processing unit that receives and processes information from the RFID tags from the portable device.

[0004] The processing unit first stores in advance a behavior pattern corresponding to a specific behavior of the user, and counts the number of times an ordered pair (feature) included in the behavior pattern is included in the user's behavior log.The processing unit then calculates feature points based on the number of times, and detects the specific behavior corresponding to the behavior pattern by comparing the feature points with a threshold. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-097472 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in Patent Document 1, an RFID reader carried by a user moving through a space is used to receive information from RFID tags attached to objects such as switches for electrical appliances or lights, bags, mobile phones, and wallets present in spaces such as offices, schools, stores, and homes, but images acquired in a time series are not used.

[0007] Furthermore, Patent Document 1 uses an ordered pair that represents a combination of two or more objects touched by a user when performing a behavior and the order in which they were touched. Patent Document 1 then counts the number of times an ordered pair included in a behavior pattern appears in a behavior log, calculates feature points based on the number of times the ordered pair appears in the behavior log, and compares the feature points with a predetermined threshold to determine a behavior corresponding to the behavior pattern.

[0008] Therefore, when two objects are not available, for example, the behavior of a person standing up from a chair or a person standing still to talk on a phone cannot be recognized even with the technology of Patent Document 1.

[0009] One aspect of the present invention is to provide a behavior recognition device, a behavior recognition method, and a program that accurately recognize the behavior of an object. [Means for solving the problem]

[0010] In order to achieve the above object, a behavior recognition device in one aspect includes: an estimation unit that estimates a motion of a target object for each target object image by using target object images corresponding to the target object included in images acquired in time series; a behavior determination unit that, when the same estimated behavior occurs consecutively a preset number of times, determines the behavior of the target object to be the estimated behavior, and, when the order in which the target object's behaviors are determined matches the order in which predetermined behaviors transition for each behavior representing a series of behaviors of the target object, determines the behavior with the matching order as the behavior of the target object; The present invention is characterized by having the following.

[0011] In order to achieve the above object, a behavior recognition method in one aspect includes: an estimation step of estimating a motion of a target object for each target object image using target object images corresponding to the target object included in images acquired in time series; a behavior determination step of determining the behavior of the target object as the estimated behavior when the same estimated behavior occurs consecutively a predetermined number of times, and determining the behavior with the matching order as the behavior of the target object when the order in which the target object's behaviors are determined matches the order in which predetermined behaviors transition for each behavior representing a series of behaviors of the target object; The present invention is characterized by having the following.

[0012] Furthermore, in order to achieve the above object, the program in one aspect comprises: an estimation step of estimating a motion of a target object for each target object image using target object images corresponding to the target object included in images acquired in time series; a behavior determination step of determining the behavior of the target object as the estimated behavior when the same estimated behavior occurs consecutively a predetermined number of times, and determining the behavior with the matching order as the behavior of the target object when the order in which the target object's behaviors are determined matches the order in which predetermined behaviors transition for each behavior representing a series of behaviors of the target object; The present invention is characterized in that the following is executed. [Effects of the Invention]

[0013] One aspect is that it can accurately recognize the behavior of an object. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating an example of a behavior recognition device. [Figure 2A] FIG. 2A is a diagram for explaining the behavior of a person standing up from a chair. [Figure 2B] FIG. 2B is a diagram for explaining the behavior of stopping and talking on the phone. [Figure 3] FIG. 3 is a diagram illustrating an example of a system having a behavior recognition device. [Figure 4] FIG. 4 is a diagram illustrating an example of the operation information. [Figure 5] FIG. 5 is a diagram illustrating an example of behavior determination information. [Figure 6] FIG. 6 is a diagram for explaining the behavior determination unit in the first modification. [Figure 7] FIG. 7 is a diagram for explaining the transition of the action determination of the behavior of the first modification. [Figure 8] FIG. 8 is a diagram for explaining the reduction of the influence of noise in the first modification. [Figure 9] FIG. 9 is a diagram for explaining the transition of the action determination of the behavior of the second modification. [Figure 10] FIG. 10 is a diagram for explaining the reduction of the influence of noise in the second modification. [Figure 11] FIG. 11 is a diagram for explaining the transition of the action determination of the behavior of the third modification. [Figure 12] FIG. 12 is a diagram for explaining a configuration in which processes for determining a plurality of behaviors are executed in parallel. [Figure 13] FIG. 13 is a diagram illustrating an example of the operation of the behavior recognition device. [Figure 14] FIG. 14 is a block diagram illustrating an example of a computer that realizes the behavior recognition device according to the embodiment and the first to fourth modifications. DETAILED DESCRIPTION OF THE INVENTION

[0015] First, an overview will be provided to facilitate understanding of the embodiments described below. A device that executes appropriate processing using the results of recognizing human behavior can execute appropriate processing more accurately the more accurately it recognizes human behavior. In other words, being able to recognize human behavior with high accuracy enables a variety of applications.

[0016] Specifically, it can be applied to devices that recognize and provide assistance to people, devices that recognize people's gestures and allow them to perform command operations, and devices that recognize and point out the form of people playing sports (for example, pitching form in baseball, swing form in golf, etc.).

[0017] Furthermore, it can also be applied to devices that recognize the behavior of objects other than humans, such as animals and robots, and execute processing using the recognition results.

[0018] Behavior refers to a series of actions performed by an object such as a person, animal, or robot. Examples of human behavior include standing up from a chair, stopping to talk on the phone, picking up a cup to drink, putting down a bag and walking away, and getting up from a bed.

[0019] As described above, a behavior is a series of movements of an object, and can be regarded as one or more movements performed in a predetermined order. For example, when standing up from a chair, a person can be regarded as performing the movements of sitting, squatting, and standing, in that order.

[0020] In the case of the behavior of standing still and talking on the phone, the person can be considered to be performing the following actions in sequence: walking, operating a communication device, talking on the communication device, and operating another communication device. The communication device could be, for example, a smartphone.

[0021] However, the time required for each action that constitutes a behavior varies depending on the person, the surrounding environment, etc. Therefore, recognition cannot be achieved through simple matching processing alone.

[0022] Through this process, the inventors have identified the challenges involved in accurately recognizing the behavior of objects such as people, animals, and robots, and have also derived means for solving these challenges.

[0023] In other words, the inventors have derived a means for recognizing behavior regardless of the time required for each action by considering the actions and their order as a sequence of predetermined actions, thereby enabling the behavior of objects such as people, animals, and robots to be recognized with high accuracy.

[0024] Hereinafter, embodiments will be described with reference to the drawings. In the drawings described below, elements having the same or corresponding functions are denoted by the same reference numerals, and repeated description thereof may be omitted.

[0025] (Embodiment) The configuration of a behavior recognition device 10 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram illustrating an example of a behavior recognition device.

[0026] [Device configuration] 1 is a device that recognizes the behavior of an object with high accuracy. As shown in FIG. 1, the behavior recognition device 10 includes an estimation unit 11 and a behavior determination unit 12.

[0027] The estimation unit 11 estimates the motion of the target object for each target object image using the target object image corresponding to the target object included in the images acquired in time series. The target object images may be all frames or may be selected and used at predetermined intervals.

[0028] When the same estimated motion occurs a predetermined number of times in succession, the motion of the target object is determined to be the estimated motion. Furthermore, when the order in which the motions of the target object are determined matches the order in which motions preset for each motion representing a series of motions of the target object transition, the behavior determination unit 12 determines the behavior with the matching order as the motion of the target object.

[0029] The number of times (motion confirmation threshold) is determined based on the rate of motion error estimation. The number of times is determined, for example, through experiments, simulations, etc. For example, even when a person makes a fast motion, the same motion continues for several frames, so setting a motion confirmation threshold is effective.

[0030] As described above, according to the embodiment, even if the time required for the behavior varies, the behavior can be recognized with high accuracy. Specific explanations will be given using Figures 2A and 2B. Figure 2A is a diagram for explaining the behavior of a person standing up from a chair. Figure 2B is a diagram for explaining the behavior of a person standing and talking on the phone.

[0031] In the example of Fig. 2A, first, the estimation unit 11 acquires images in time series and estimates the person's motion in the order of acquisition, i.e., the motion of sitting, the motion of crouching, and the motion of standing.

[0032] Next, if the same estimated motion is repeated a preset number of times (motion confirmation threshold), for example, twice in a row, the behavior confirmation unit 12 confirms the estimated motion. In the example of Fig. 2A, the sitting motion is estimated three times, the squatting motion is estimated two times, and the standing motion is estimated three times, so the motions are confirmed as sitting, squatting, and standing.

[0033] Next, the behavior determination unit 12 determines that the behavior is that of a person standing up from a chair because the determined behavior and the order of the determined behaviors match the order of transition of preset actions as the behavior of a person standing up from a chair, i.e., the sitting action, the half-squatting action, and the standing action.

[0034] 2B, to recognize the behavior of a person standing still while talking on the phone, the behavior determination unit 12 first acquires a plurality of images in time series and estimates the person's actions in the order of acquisition, i.e., the actions of walking, operating a communication device, talking on a communication device, and operating a communication device are estimated.

[0035] Next, behavior determination unit 12 determines the estimated behavior when the same estimated behavior is detected a preset number of times (behavior determination threshold), for example, twice consecutively in the example of Fig. 2B. In the example of Fig. 2B, the behavior of walking is estimated twice, the behavior of operating a communication device is estimated twice, and the behavior of talking on a communication device is estimated three times, so the behaviors are determined to be the behavior of walking, the behavior of operating a communication device, and the behavior of talking on a communication device.

[0036] Next, the behavior determination unit 12 determines that the behavior is that of a person stopping and talking because the determined actions and the order of the determined actions match the order of transition of pre-set actions for the behavior of a person stopping and talking, i.e., the action of walking, the action of operating a communication device, and the action of talking on the communication device.

[0037] As can be seen from the above, even if the time required for an action differs from person to person, the action can be determined regardless of the length of time, and behavior can be recognized based on the determined action and the order of the determined actions, so behavior can be recognized with high accuracy.

[0038] [System Configuration] The configuration of the behavior recognition device 10 in the embodiment will be described more specifically with reference to Fig. 3. Fig. 3 is a diagram for explaining an example of a system including a behavior recognition device.

[0039] As shown in FIG. 3, the system 100 according to the embodiment includes a behavior recognition device 10, an imaging device 20, and an information processing device 30.

[0040] The behavior recognition device 10 is, for example, an information processing device such as a CPU (Central Processing Unit), a programmable device such as an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), or a circuit equipped with one or more of these, a server computer, a personal computer, or a mobile terminal.

[0041] The imaging device 20 outputs two-dimensional or three-dimensional images captured in time series to the behavior recognition device 10. The imaging device 20 is, for example, a camera, or a device equipped with a camera and LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging). Note that the camera may be, for example, a monocular camera (for example, a wide-angle camera, a fisheye camera, a spherical camera, etc.), a compound eye camera (for example, a stereo camera, a multi-camera, etc.), an RGB-D camera (for example, a depth camera, a ToF camera, etc.), etc.

[0042] The information processing device 30 acquires the behavior recognition results (behavior information) from the behavior recognition device 10 and executes predetermined processing using the acquired behavior information. The information processing device 30 is, for example, a device that recognizes human behavior and provides support, a device that recognizes human gestures and allows commands to be operated, or a device that recognizes and indicates the form of a person playing a sport (for example, a pitching form in baseball or a swing form in golf).

[0043] The information processing device 30 is, for example, a CPU, a programmable device such as an FPGA, a GPU, or a circuit equipped with any one or more of them, a server computer, a personal computer, a mobile terminal, or other information processing device.

[0044] 3, the behavior recognition device 10, the imaging device 20, and the information processing device 30 are shown separately, but the behavior recognition device 10, the imaging device 20, and the information processing device 30 may be integrated into one device. Also, the behavior recognition device 10 and the imaging device 20 may be integrated into one device.

[0045] The behavior recognition device 10 includes a preprocessing unit 13, an estimation unit 11, a behavior determination unit 12, and a notification unit 16.

[0046] The pre-processing unit 13 processes the images captured by the imaging device 20 so that actions and behaviors can be efficiently recognized in subsequent processing. Specifically, the pre-processing unit 13 first acquires images captured in real time by the imaging device 20 in chronological order, or images captured in the past in chronological order from a storage device (not shown).

[0047] Next, the preprocessing unit 13 converts the acquired image into an image that is easy to recognize behavior using existing techniques, such as contrast adjustment and noise removal.

[0048] The estimation unit 11 estimates the motion of a target object for each target object image using a target object image corresponding to the target object whose behavior is that of a person, animal, robot, etc. The estimation unit 11 includes an object information extraction unit 14 and a motion estimation unit 15.

[0049] The object information extraction unit 14 extracts information about the target object from the image acquired from the pre-processing unit 13. Specifically, first, the object information extraction unit 14 acquires the image from the pre-processing unit 13. Then, the object information extraction unit 14 extracts information about the target object and information about things other than the target object from the target object image using existing technology. The existing technology is, for example, object recognition using deep learning, pattern matching, or other processing.

[0050] For example, if the target object is a person, the information about the person is the person's skeletal coordinates, face information, etc. The information about a non-person is information indicating the name, position, etc. of an object other than a person captured in the target object image.

[0051] The process of extracting information about people (information about the target object) and the process of extracting information about things other than people (information other than the target object) may be executed in parallel, or one of the processes may be executed first.

[0052] The motion estimation unit 15 estimates the motion of the target object for each target object image using information about the target object, or information about the target object and information about things other than the target object. Specifically, the motion estimation unit 15 first acquires information about the target object, or information about the target object and information about things other than the target object, from the object information extraction unit 14.

[0053] Next, the motion estimation unit 15 estimates the motion of the target object for each target object image using information about the target object, or information about the target object and information about other than the target object. Thereafter, the motion estimation unit 15 outputs motion information representing the estimated motion of the target object to the behavior determination unit 12.

[0054] For example, when the target object is a person, the motion is estimated for each target object image (frame) using existing technologies such as matching processing using human skeletal information, motion classification by time series learning, etc. When estimating the target object's motion by motion classification by time series learning, the results of person recognition and object recognition for the past few frames may be used.

[0055] The matching process using human skeletal information uses multiple pieces of skeletal information (for example, combinations of angles of three skeletal coordinate points) that have been registered in advance. For example, the angles of the three points of the right wrist, right elbow, and right shoulder, and the angles of the three points of the right wrist, neck, and right elbow are used to prepare whether the arm is outstretched or bent, and the similarity is calculated to perform matching.

[0056] The process of motion classification using time series learning extracts features that represent the characteristics of the motion from skeletal information, and then uses an RNN (Recurrent Neural Network) machine learning model to classify very short duration motions (short duration motions of a few frames, rather than motions that span seconds like behavior).

[0057] Action information is information that identifies actions that make up a behavior, such as sitting, squatting, walking, standing, operating a smartphone, talking on the phone, holding a cup, drinking from a cup, walking with a bag, putting the bag down, sleeping in bed, and sitting on a bed.

[0058] Fig. 4 is a diagram illustrating an example of the motion information. In the example of Fig. 4, motion information "1" to "12" is associated with each motion.

[0059] The behavior determination unit 12 determines the motion of the target object as the estimated motion when the estimated motion continues a preset number of times (motion determination threshold).

[0060] Specifically, first, the behavior determination unit 12 acquires motion information from the estimation unit 11. Next, if the same motion information is received a predetermined number of times (motion determination threshold), the behavior determination unit 12 determines the motion of the target object to be the estimated motion.

[0061] Thereafter, if the order in which the actions are determined matches the order in which pre-set actions transition for each action representing a series of actions of the target object, the behavior determination unit 12 determines the behavior with the matching order as the behavior of the target object.

[0062] Specifically, first, the behavior determination unit 12 acquires the determined motion information from the motion estimation unit 15. Next, the behavior determination unit 12 refers to the behavior determination information using the determined motion information and the order in which the determined motion information was acquired, and detects a matching behavior.

[0063] Fig. 5 is a diagram illustrating an example of behavior determination information. In the example of Fig. 5, the behavior determination information associates a behavior, behavior information for identifying the behavior, and information indicating the order of actions (action order). In the example of Fig. 5, the behavior information associates "A" to "F" with each behavior. The action order associates each behavior with information indicating the order of actions used to recognize the behavior (the order in which the action information was determined).

[0064] The notification unit 16 outputs information representing the detected behavior to the information processing device 30. Specifically, the notification unit 16 first acquires, from the behavior determination unit 12, information representing the detected behavior.

[0065] Next, the notification unit 16 determines whether to notify the user of the behavior according to a predetermined priority associated with each behavior. For example, the notification unit 16 outputs only behaviors with high priority to the information processing device 30. The priority is set to an arbitrary priority determined in advance. The priority is determined, for example, by experiment, simulation, or the like.

[0066] (Variation 1) Fig. 6 is a diagram for explaining the behavior determination unit in Modification 1. In the example of Fig. 6, the behavior determination unit 12 has action determination units 61 (61A, 61B, 61C). The number of action determination units 61 is not limited to three. It is determined by the number of actions for which the behavior is determined.

[0067] Each of the operation determination units 61 (61A, 61B, 61C) has a state management unit 62 (62A, 62B, 62C) and a counter 63 (63A, 63B, 63C).

[0068] The state management unit 62 (62A, 62B, 62C) adds the count value of the counter when it acquires predetermined motion information for each motion, subtracts the count value according to the noise tolerance value when it acquires motion information other than the predetermined motion information, and when the count value reaches a predetermined motion confirmation threshold, it confirms the motion of the target object as the estimated motion.

[0069] In the first modification, a motion determination threshold and a noise tolerance value are set. In the modification, the motion determination threshold is a value for determining the motion of the target object. The noise tolerance value is a value indicating how many times noise acquisition is permitted.

[0070] Fig. 7 is a diagram for explaining the transition of the action determination of the behavior of Modification 1. The example of Fig. 7 shows the initial state and action states 1, 2, and 4 corresponding to action information "1," "2," and "4." Note that the count value of each counter is 0 in the initial state.

[0071] The operating state 1 has a state 1T_1 (a state in which the count value is 1) and a state 1C (a state in which the count value is 2). In the state management unit 62A (operating state 1), the operation determination threshold (corresponding to the number of states (number of stages)) is set to 2. Also, the noise tolerance is set to 1.

[0072] The operating state 2 has a state 2T_1 (a state in which the count value is 1) and a state 2C (a state in which the count value is 2). In the state management unit 62B (operating state 2), the operation determination threshold is set to 2. Also, the noise tolerance is set to 1.

[0073] The operating state 4 has a state 4T_1 (a state in which the count value is 1) and a state 4C (a state in which the count value is 2). In the state management unit 62C (operating state 4), the operation determination threshold is set to 4. Also, the noise tolerance is set to 1.

[0074] In the example of Figure 7, first, when operation information "1" is acquired once, the state transitions from the initial state to state 1T_1. Next, when operation information "1" is acquired twice, the state transitions to state 1C, and operation 1 is confirmed. In other words, when the count value corresponding to operation state 1 becomes 2, operation 1 is confirmed.

[0075] Next, after action 1 is confirmed, if action information "2" is acquired once, a transition occurs from action state 1 to state 2T_1. Next, if action information "2" is acquired twice, a transition occurs to state 2C, and action 2 is confirmed. In other words, when the count value corresponding to action state 2 becomes 2, action 2 is confirmed.

[0076] Next, after actions 1 and 2 are determined, if action information "4" is acquired once, the state transitions from action state 2 to state 4T_1. Next, if action information "4" is acquired twice, the state transitions to state 4C, and action 4 is determined. In other words, when the count value corresponding to action state 4 becomes 2, action 4 is determined.

[0077] Thereafter, if the order in which the actions are determined matches the order in which the actions transition, which is set in advance for each of the actions representing a series of actions of the target object, the behavior determination unit 12 determines the behavior with the matching order as the behavior of the target object. Note that the count value of each counter is initially set to 0.

[0078] Fig. 8 is a diagram for explaining the reduction of the influence of noise in Modification 1. Assume that the motion information is acquired in the order shown in Fig. 8. In this case, in the example of Fig. 8, after motion 1 is determined, when motion information "2" is acquired once, a transition occurs from motion state 1 to state 2T_1.

[0079] Next, if the motion information "2" is acquired twice, the state transitions to state 2C and the motion 2 is confirmed. However, in the example of FIG. 8, the motion information "6", which is noise, is acquired next. In this case, since the motion information is not "2" or "4", the state transitions to state 2T_1.

[0080] 8, the operation information "2" is acquired again, so the state transitions to state 4C. After that, the operation information "4" is acquired, so the state transitions from operation state 2 to state 4T_1.

[0081] According to the first modification, even when erroneously estimated motion information is acquired (when noise is acquired), the behavior can be recognized with high accuracy. Therefore, according to the first modification, even when erroneously estimated motion information is acquired (when noise is acquired), the behavior can be recognized with high accuracy.

[0082] However, in the example of FIG. 8, since the noise tolerance value is 1, if noise is detected twice in succession, the state transitions to the initial state and the count value is set to 0.

[0083] (Variation 2) Fig. 9 is a diagram for explaining the transition of the action determination of the behavior in Modification 2. In the example of Fig. 9, an initial state and action states 1, 2, and 4 corresponding to action information "1", "2", and "4" are shown.

[0084] The operating state 1 has a state 1T_1 (a state in which the count value is 1), a state 1T_2 (a state in which the count value is 2), and a state 1C (a state in which the count value is 3). In the state management unit 62A (operating state 1), the operation determination threshold (corresponding to the number of states (number of stages)) is set to 3. Also, the noise tolerance is set to 2.

[0085] The operating state 2 has a state 2T_1 (a state in which the count value is 1), a state 2T_2 (a state in which the count value is 2), and a state 2C (a state in which the count value is 3). In the state management unit 62B (operating state 2), the operation determination threshold is set to 3. Also, the noise tolerance is set to 2.

[0086] The operating state 4 has a state 4T_1 (a state in which the count value is 1), a state 4T_2 (a state in which the count value is 2), and a state 4C (a state in which the count value is 3). In the state management unit 62C (operating state 4), the operation determination threshold is set to 3. Also, the noise tolerance is set to 2.

[0087] In the example of Figure 9, first, when the action information "1" is acquired once, the state transitions from the initial state to state 1T_1. Next, when the action information "1" is acquired twice, the state transitions from state 1T_1 to state 1T_2. Next, when the action information "1" is acquired three times, the state transitions to state 1C, and action 1 is confirmed.

[0088] Next, after action 1 is confirmed, if action information "2" is acquired once, the state transitions from action state 1 to state 2T_1. Next, if action information "2" is acquired twice, the state transitions from state 2T_1 to state 2T_2. Next, if action information "2" is acquired three times, the state transitions to state 2C, and action 2 is confirmed.

[0089] Next, after actions 1 and 2 are confirmed, if action information "4" is acquired once, the state changes from action state 2 to state 4T_1. Next, if action information "4" is acquired twice, the state transitions from state 4T_1 to state 4T_2. Next, if action information "4" is acquired three times, the state transitions to state 4C, and action 4 is confirmed.

[0090] Fig. 10 is a diagram for explaining the reduction of the influence of noise in Modification 2. Assume that the motion information is acquired in the order shown in Fig. 10. In this case, once motion 1 is determined and motion information "2" is acquired once, a transition occurs from motion state 1 to state 2T_1.

[0091] Next, if the motion information "2" is acquired twice, the system transitions to state 2T_1, and if the motion information "2" is acquired twice, the system transitions to state 2C and confirms motion 2. However, in the example of FIG. 10, the motion information "5", which is noise, is acquired twice in a row. In this case, since the motion information is not "2" or "4", the system transitions to state 2T_2 and then to state 2T_1.

[0092] 10, after acquiring the motion information "2" twice in succession, the motion information "4" is acquired, so that the motion state transitions from the motion state 2 to the state 4T_1. Therefore, according to the second modification, even when erroneously estimated motion information is acquired (when noise is acquired), the behavior can be recognized with high accuracy.

[0093] However, in the example of FIG. 10, since the noise tolerance value is 2, if noise is detected three times in succession, the state transitions to the initial state and the count value is set to 0.

[0094] (Variation 3) Fig. 11 is a diagram for explaining the transition of the action determination of the behavior in Modification 3. In the example of Fig. 11, an initial state and action states 1, 2, and 4 corresponding to action information "1", "2", and "4" are shown.

[0095] The operating state 1 has state 1T_1 (state where count value = 1), state 1T_2 (state where count value = 2), and state 1C (state where count value = 3). In the state management unit 62A (operating state 1), the operation determination threshold (corresponding to the number of states (number of stages)) is set to 3. Also, the noise tolerance is set to 1. Therefore, no matter what state the counter is in, it transitions to state 1T_1 (state where count value = 1).

[0096] The operating state 2 has state 2T_1 (state where count value = 1), state 2T_2 (state where count value = 2), and state 2C (state where count value = 3). In the state management unit 62B (operating state 2), the operation determination threshold is set to 3. Also, the noise tolerance value is set to 1. Therefore, no matter what state the counter is in, the state transitions to state 2T_1 (state where count value = 1).

[0097] The operating state 4 has a state 4T_1 (a state in which the count value is 1), a state 4T_2 (a state in which the count value is 2), and a state 4C (a state in which the count value is 3). In the state management unit 62C (operating state 4), the operation determination threshold is set to 3. Also, the noise tolerance value is set to 1. Therefore, no matter what state the counter is in, the state transitions to a state 4T_1 (a state in which the count value is 1).

[0098] In the example of Figure 11, first, when the action information "1" is acquired once, the state transitions from the initial state to state 1T_1. Next, when the action information "1" is acquired twice, the state transitions from state 1T_1 to state 1T_2. Next, when the action information "1" is acquired three times, the state transitions to state 1C, and action 1 is confirmed.

[0099] Next, after action 1 is confirmed, if action information "2" is acquired once, the state transitions from action state 1 to state 2T_1. Next, if action information "2" is acquired twice, the state transitions from state 2T_1 to state 2T_2. Next, if action information "2" is acquired three times, the state transitions to state 2C, and action 2 is confirmed.

[0100] Next, after actions 1 and 2 are confirmed, if action information "4" is acquired once, the state changes from action state 2 to state 4T_1. Next, if action information "4" is acquired twice, the state transitions from state 4T_1 to state 4T_2. Next, if action information "4" is acquired three times, the state transitions to state 4C, and action 4 is confirmed.

[0101] Therefore, according to the third modification, even when erroneously estimated motion information is acquired (when noise is acquired), the behavior can be recognized with high accuracy.

[0102] (Variation 4) Fig. 12 is a diagram illustrating a configuration for executing processes for determining multiple behaviors in parallel. In the example of Fig. 12, behaviors A, B, and C are executed in parallel to reduce the time required to determine behaviors A, B, and C.

[0103] 12, the behavior determination unit 12A for behavior A has operation determination units 61A, 61B, and 61C as described above. The behavior determination unit 12B for behavior B has operation determination units 61D, 61E, and 61F. The behavior determination unit 12C for behavior C has operation determination units 61D, 61E, 61F, and 61D.

[0104] The movement determining unit 61A determines the sitting movement (movement 1). The movement determining unit 61B determines the crouching movement (movement 2). The movement determining unit 61C determines the standing movement (movement 4).

[0105] The action determination unit 61D determines the action of walking (action 3). The action determination unit 61E determines the action of operating a smartphone (action 5). The action determination unit 61F determines the action of talking on the phone (action 6).

[0106] Therefore, according to Modification 4, authentication can be completed in a short time by performing multiple behavior recognitions in parallel. However, the number of behavior determination units 12 to be performed in parallel and the types of behaviors are not limited to the example in FIG.

[0107] [Device operation] Next, the operation of the behavior recognition device in the embodiment and modifications 1 to 4 will be described with reference to FIG. 13. FIG. 13 is a diagram for explaining an example of the operation of the behavior recognition device. In the following description, the diagram will be referred to as appropriate. Furthermore, in the embodiment and modifications 1 to 4, the behavior recognition method is implemented by operating the behavior recognition device. Therefore, the description of the behavior recognition method in the embodiment will be replaced by the following description of the operation of the behavior recognition device.

[0108] As shown in FIG. 13, first, the preprocessing unit 13 acquires images captured in real time by the imaging device 20, or images captured in past time series from a storage device (not shown) (step A1).

[0109] Next, the pre-processing unit 13 processes the image captured by the imaging device 20 so that the actions and behaviors can be efficiently recognized in subsequent processing (step A2).

[0110] Next, the object information extraction unit 14 extracts information about the target object from the image acquired from the preprocessing unit 13 (step A3). Specifically, in step A3, the object information extraction unit 14 extracts information about the target object and information about things other than the target object from the target object image using existing technology.

[0111] Next, the motion estimation unit 15 estimates the motion of the target object for each target object image using information about the target object, or information about the target object and information about things other than the target object (step A4).

[0112] Specifically, in step A4, the motion estimation unit 15 estimates the motion of the target object for each target object image using information about the target object, or information about the target object and information about things other than the target object.

[0113] Next, if the estimated movement continues a preset number of times (movement confirmation threshold), the behavior determination unit 12 determines the movement of the target object as the estimated movement (step A5).

[0114] Specifically, in step A5, the behavior determination unit 12 uses the determined motion information and the order in which the determined motion information was acquired to refer to the motion determination information and detects a matching motion. Note that in step A5, the behavior may be determined by executing the processes shown in the above-mentioned modifications 1 to 4.

[0115] Next, the notification unit 16 outputs information representing the detected behavior to the information processing device 30 (step A6). Specifically, in step A6, the notification unit 16 determines whether to notify the user of the behavior according to a preset priority associated with each behavior. The notification unit 16 outputs only behaviors with high priority to the information processing device 30, for example.

[0116] [Effects of the embodiment and modifications 1 to 4] According to the embodiment, a behavior can be considered as a plurality of different predetermined actions performed in a predetermined order, and by using the actions and their order, the behavior can be recognized regardless of the time required for the actions.

[0117] Furthermore, according to the first to fourth modifications, even when erroneously estimated motion information is acquired (when noise is acquired), the behavior can be recognized with high accuracy.

[0118] [program] The program in the embodiment and modifications 1 to 4 may be a program that causes a computer to execute steps A1 to A6 shown in Fig. 13. By installing and executing this program in a computer, the behavior recognition device and behavior recognition method in the embodiment and modifications 1 to 4 can be realized. In this case, the processor of the computer functions as a preprocessing unit 13, an estimation unit 11 (object information extraction unit 14, motion estimation unit 15), a behavior determination unit 12, and a notification unit 16, and performs processing.

[0119] The programs in the embodiment and modifications 1 to 4 may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as one of the preprocessing unit 13, the estimation unit 11 (object information extraction unit 14, motion estimation unit 15), the behavior determination unit 12, and the notification unit 16.

[0120] [Physical configuration] Here, a computer that realizes the behavior recognition device by executing the program in the embodiment and modifications 1 to 4 will be described with reference to Fig. 14. Fig. 14 is a block diagram showing an example of a computer that realizes the behavior recognition device in the embodiment and modifications 1 to 4.

[0121] 14, the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other. Note that the computer 110 may include a GPU or an FPGA in addition to or instead of the CPU 111.

[0122] The CPU 111 loads the programs (codes) in the embodiments and modifications 1 to 4 stored in the storage device 113 into the main memory 112 and executes them in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory). The programs in the embodiments and modifications 1 to 4 are provided in a state stored in a computer-readable recording medium 120. The programs in the embodiments and modifications 1 to 4 may be distributed over the Internet connected via the communication interface 117. The recording medium 120 is a non-volatile recording medium.

[0123] Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to a display device 119 and controls the display on the display device 119.

[0124] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0125] Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as flexible disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0126] The behavior recognition device in the embodiment and modifications 1 to 4 can be realized by using hardware corresponding to each part, instead of a computer on which a program is installed. Furthermore, the behavior recognition device may be realized in part by a program and in the remaining part by hardware.

[0127] [Note] The following supplementary notes are further provided with respect to the above-described embodiments. The above-described embodiments and modifications 1 to 4 can be partly or entirely expressed by (Supplementary Note 1) to (Supplementary Note 9) described below, but are not limited to the following descriptions.

[0128] (Appendix 1) an estimation unit that estimates a motion of a target object for each target object image by using target object images corresponding to the target object included in images acquired in time series; a behavior determination unit that, when the same estimated behavior occurs consecutively a preset number of times, determines the behavior of the target object to be the estimated behavior, and, when the order in which the target object's behaviors are determined matches the order in which predetermined behaviors transition for each behavior representing a series of behaviors of the target object, determines the behavior with the matching order as the behavior of the target object; A behavior recognition device having the above.

[0129] (Appendix 2) 2. The behavior recognition device according to claim 1, The behavior determination unit increments a count value of a counter when acquiring predetermined motion information for each motion, decrements the count value according to a noise tolerance value when acquiring motion information other than the predetermined motion information, and determines the motion of the target object as an estimated motion when the count value reaches a predetermined motion determination threshold. Behavior recognizer.

[0130] (Appendix 3) 3. The behavior recognition device according to claim 1, a notification unit that determines whether to notify the user of the behavior according to a preset priority; A behavior recognition device having the above.

[0131] (Appendix 4) The computer an estimation step of estimating a motion of a target object for each target object image using target object images corresponding to the target object included in images acquired in time series; a behavior determination step of determining the behavior of the target object as the estimated behavior when the same estimated behavior occurs consecutively a predetermined number of times, and determining the behavior with the matching order as the behavior of the target object when the order in which the target object's behaviors are determined matches the order in which predetermined behaviors transition for each behavior representing a series of behaviors of the target object; A behavior recognition method that performs the above.

[0132] (Appendix 5) 5. The behavior recognition method according to claim 4, further comprising: The behavior determination step increments a count value of a counter when predetermined motion information is acquired for each motion, decrements the count value according to a noise tolerance when motion information other than the predetermined motion information is acquired, and determines the motion of the target object as an estimated motion when the count value reaches a predetermined motion determination threshold. Behavioral recognition methods.

[0133] (Appendix 6) 6. The behavior recognition method according to claim 4 or 5, a notification unit that determines whether to notify the user of the behavior according to a preset priority; A behavior recognition method having the following.

[0134] (Appendix 7) On the computer, an estimation step of estimating a motion of a target object for each target object image using target object images corresponding to the target object included in images acquired in time series; a behavior determination step of determining the behavior of the target object as the estimated behavior when the same estimated behavior occurs consecutively a predetermined number of times, and determining the behavior with the matching order as the behavior of the target object when the order in which the target object's behaviors are determined matches the order in which predetermined behaviors transition for each behavior representing a series of behaviors of the target object; A program that executes the following.

[0135] (Appendix 8) 8. The program of claim 7, The behavior determination step increments a count value of a counter when predetermined motion information is acquired for each motion, decrements the count value according to a noise tolerance when motion information other than the predetermined motion information is acquired, and determines the motion of the target object as an estimated motion when the count value reaches a predetermined motion determination threshold. program.

[0136] (Appendix 9) 9. The program according to claim 7 or 8, a notification step for determining whether or not to notify the user of the behavior according to a preset priority; A program that executes the following.

[0137] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Industrial Applicability]

[0138] According to the above description, the behavior of an object can be recognized with high accuracy, and is useful in fields where the behavior of an object needs to be recognized. [Explanation of symbols]

[0139] 10 Behavior Recognition Device 11 Estimation part 12, 12A, 12B, 12C Behavior determination section 13 Pretreatment section 14 Object information extraction section 15 Motion estimation section 16 Notification Department 20 Imaging device 30 Information processing equipment 61, 61A, 61B, 61C, 61D, 61E, 61F Operation confirmation section 62, 62A, 62B, 62C Status management unit 63, 63A, 63B, 63C Counters 110 Computer 111 CPU 112 main memory 113 Storage device 114 Input Interface 115 Display Controller 116 Data Reader / Writer 117 Communication Interface 118 Input Devices 119 Display Device 120 Recording Media 121 Bus

Claims

1. an estimation means for estimating a motion of a target object for each target object image using target object images corresponding to the target object included in images acquired in time series; a behavior determination means for determining the behavior of the target object as the estimated behavior when the same estimated behavior occurs a predetermined number of times in succession, and determining the behavior with the matching order as the behavior of the target object when the order in which the target object's behaviors are determined matches the order in which predetermined behaviors transition for each behavior representing a series of behaviors of the target object; The behavior determination means increments a count value of a counter when it acquires predetermined motion information for each motion, decrements the count value according to a noise tolerance value when it acquires motion information other than the predetermined motion information, and determines the motion of the target object as an estimated motion when the count value reaches a predetermined motion determination threshold. Behavior recognizer.

2. The behavior recognition device according to claim 1, a notification means for determining whether or not to notify the user of the behavior according to a preset priority; A behavior recognition device having the above.

3. using target object images corresponding to the target object included in the images acquired in time series, estimating a movement of the target object for each of the target object images; When the same estimated action is performed a predetermined number of times in succession, the action of the target object is determined to be the estimated action, and when the order in which the actions of the target object are determined matches the order in which actions that are preset for each behavior representing a series of actions of the target object transition, the behavior with the matching order is determined to be the behavior of the target object; Furthermore, when predetermined motion information for each motion is acquired, the count value of the counter is incremented, and when motion information other than the predetermined motion information is acquired, the count value is decremented according to a noise tolerance value, and when the count value reaches a predetermined motion determination threshold, the motion of the target object is determined to be the estimated motion. Behavioral recognition methods.

4. The behavior recognition method according to claim 3, Decide whether to notify the user of the behavior according to a preset priority. Behavioral recognition methods.

5. On the computer, using target object images corresponding to the target object included in the images acquired in time series, estimating the motion of the target object for each of the target object images; When the same estimated action is performed a predetermined number of times in succession, the action of the target object is determined to be the estimated action; when the order in which the actions of the target object are determined matches the order in which actions preset for each behavior representing a series of actions of the target object transition, the behavior with the matching order is determined to be the behavior of the target object; Furthermore, when predetermined motion information for each motion is acquired, the count value of the counter is incremented, and when motion information other than the predetermined motion information is acquired, the count value is decremented according to a noise tolerance value. When the count value reaches a predetermined motion determination threshold, the motion of the target object is determined to be the estimated motion. program.

6. 6. The program according to claim 5, Whether or not to notify the user of the behavior is determined according to a preset priority. program.

Images