Action recognition device, action recognition method, and action recognition program

Abstract

The action recognition device estimates skeleton point coordinates of the user from the image, and calculates, on the basis of the skeleton point coordinates, a time-series feature vector representing a feature vector connecting the body of the user and the head of the user in a time-series manner. An input time-series feature vector, which is the calculated time-series feature vector, is compared with a reference time-series feature vector to determine an action of the user and a change in the action, and information indicating the determined action and the change in the action is output.

Description

Technical Field

[0001] The present disclosure relates to a technology for recognizing a user's action based on an image. Background Art

[0002] As a technology for recognizing the actions of a user using images such as static images and dynamic images, for example, Patent Document 1 and Patent Document 2 are known. Patent Document 1 discloses a technology that extracts time series data of a person's skeleton based on a dynamic image obtained by photographing a target user for the purpose of determining the fatigue level of a person, and determines the fatigue level based on the user's arm swing and step movements. Patent Document 2 discloses a technology that detects time series data of a specific part of a person based on a dynamic image obtained by photographing a target user for the purpose of determining the walking state and age of a pedestrian, and estimates the walking state and walking speed classification of the person.

[0003] However, the above-mentioned conventional action recognition technology has a problem that it is not possible to recognize the action of the user with high accuracy in a scene where the entire body of the user to be action recognized is not captured due to obstacles or the like.

[0004] Prior Art Literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent No. 6873344

[0007] Patent Document 2: Japanese Patent No. 3655618 Summary of the invention

[0008] The present disclosure has been made to solve such a problem, and an object of the present disclosure is to provide a technology capable of recognizing a user's behavior with high accuracy even in a scene where the whole body of the user to be the target of behavior recognition is not captured.

[0009] An action recognition device in one mode of the present disclosure is an action recognition device for recognizing the action of a user, and the action recognition device comprises: an acquisition unit for acquiring an image; an estimation unit for estimating the coordinates of the skeleton points of the user based on the image acquired by the acquisition unit; a calculation unit for calculating a time series feature vector representing the feature vector connecting the body of the user and the head of the user in a time series based on the skeleton point coordinates; a storage unit for storing a reference time series feature vector, which is the time series feature vector serving as a reference; and a determination unit for determining the action of the user and the change in the action by comparing the input time series feature vector serving as the time series feature vector calculated by the calculation unit with the reference time series feature vector; and the action determined by the determination unit.

[0010] According to the present disclosure, even if the image does not capture the whole body, the user's behavior can be recognized with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a block diagram showing an example of the structure of the action recognition system in the embodiment of the present disclosure.

[0012] Figure 2 FIG. 1 is a diagram showing an example of skeleton information including the coordinates of skeleton points estimated by the estimating unit.

[0013] Figure 3 This is a flowchart showing an example of the processing of the action recognition device in the embodiment of the present disclosure.

[0014] Figure 4 FIG. 1 is a diagram showing an example of a feature vector calculated from an image captured of a user's sitting action.

[0015] Figure 5 It is shown Figure 3 Flowchart of details of the comparison process in step S7.

[0016] Figure 6 This is a graph showing the temporal transition of the vector length of the feature vector when two different sitting conditions are imposed on the subject for the action label “sitting down”. DETAILED DESCRIPTION

[0017] The following is a description of an embodiment of the present invention with reference to the drawings. The following embodiment is an example of the present invention, and is not intended to limit the technical scope of the present invention.

[0018] (Knowledge that forms the basis of this disclosure)

[0019] In recent years, a method of estimating a person's skeleton points from an image and recognizing the user's actions based on the estimated skeleton points has become known. In such a recognition method, an image obtained by shooting the user's entire body at a camera angle that is advantageous for sensing and an image obtained by shooting the user's walking state for a long time are used. In addition, in such a recognition method, a deep neural network including a convolution layer and a pooling layer is used to estimate the skeleton points, thereby achieving high accuracy.

[0020] However, in recent years, in order to sense the decline of the body, etc., the technology of detecting the movement displacement of the user in daily life has been highly anticipated. However, in scenes such as daily life, it is common to see parts of the body of the user who is the object of action recognition hidden by some obstacles or parts of the body sticking out of the image. Therefore, in the previous recognition methods, there is a problem that the user's action cannot be recognized with high accuracy when shooting actions under poor shooting conditions. This problem is more likely to occur in residential areas where there are restrictions on the location of the camera.

[0021] The present disclosure has been made in view of the above-mentioned problems, and provides a technology capable of recognizing the actions of a user with high accuracy even in a scene where the whole body of the user to be the target of action recognition is not captured.

[0022] (1) An action recognition device in one embodiment of the present invention is an action recognition device for recognizing the action of a user, the action recognition device comprising: an acquisition unit for acquiring an image; an estimation unit for estimating the coordinates of the skeletal points of the user based on the image acquired by the acquisition unit; a calculation unit for calculating, based on the skeletal point coordinates, a time series feature vector representing a feature vector connecting the torso of the user and the head of the user in a time series; a storage unit for storing a reference time series feature vector, the reference time series feature vector being the time series feature vector serving as a reference; and a determination unit for determining the action of the user and changes in the action by comparing an input time series feature vector serving as the time series feature vector calculated by the calculation unit with the reference time series feature vector; and the action determined by the determination unit.

[0023] According to this structure, the user's action and the change of the action are determined by comparing the input time series feature vector and the reference time series feature vector. Here, the input time series feature vector and the reference time series feature vector are respectively composed of feature vectors connecting the user's body and head. This feature vector is a feature vector that can be generated as long as there is information about the user's upper body, and will not change significantly depending on the camera angle. Therefore, according to this structure, even in a scene where the whole body of the user who is the object of action recognition is not captured, the user's action can be recognized with high accuracy.

[0024] (2) In the action recognition device described in (1) above, the comparison performed by the determination unit may be: comparing the feature vector constituting the input time series feature vector with the feature vector constituting the reference time series feature vector in terms of length, angle, time variation of the length, and time variation of the angle.

[0025] According to this structure, since the input time series feature vector and the benchmark time series feature vector are compared using the length, angle, time change of length, and time change of angle of their respective feature vectors, the user's actions can be identified with high accuracy.

[0026] (3) In the action recognition device described in (1) or (2) above, the time series feature vector may be normalized so that time becomes a reference value.

[0027] According to this configuration, since the input time series feature vector and the reference time series feature vector are each normalized so that time becomes a reference value, it is easy to compare the input time series feature vector and the reference time series feature vector.

[0028] (4) In the action recognition device described in any one of (1) to (3) above, the bone point coordinates estimated by the estimating unit may include a reliability indicating the estimation accuracy, and the calculation unit may calculate the coordinates of the body and the coordinates of the head by weighting the bone point coordinates using the reliability, and calculate the feature vector based on the calculated coordinates of the body and the coordinates of the head.

[0029] According to this configuration, the feature vector can be calculated taking into account the reliability of the skeleton point coordinates.

[0030] (5) In the action recognition device described in any one of (1) to (4) above, it may be that the baseline time series feature vector includes a plurality of first baseline time series feature vectors corresponding to action labels representing the types of the actions, and the determination unit determines the action label representing the action of the user by comparing the input time series feature vector with the plurality of first baseline time series feature vectors, calculates an average time series feature vector by averaging the first baseline time series feature vectors corresponding to the determined action label, and determines the change in the action of the user by comparing the average time series feature vector with the input time series feature vector.

[0031] According to this structure, the action label is determined by comparing the first reference time series feature vector corresponding to the action label with the input time series feature vector, so the user's action can be determined with high accuracy. Furthermore, the change in the user's action is determined by comparing the average time series feature vector obtained by averaging the first reference time series feature vector corresponding to the determined action label with the input time series feature vector, so the input time series feature vector and the first reference time series feature vector can be compared one-to-one.

[0032] (6) In the action recognition device described in any one of (1) to (5) above, the feature vector calculated by the calculation unit may use the coordinates of the image to represent the coordinates of the body and the coordinates of the head respectively, and the determination unit may translate the feature vector so that the coordinates of the body of the feature vector calculated by the calculation unit are located at the origin of the coordinate system of the image, and use the feature vector after the translation to represent the time series feature vector.

[0033] According to this configuration, since the feature vector is translated so that the starting point of the feature vector calculated by the calculation unit is located at the origin of the coordinate system of the image, the feature vectors can be compared while ignoring the positions on the image where the feature vector appears.

[0034] (7) In the action recognition device described in any one of (1) to (6) above, the determination unit may normalize the feature vector so that the vertical length and horizontal length of the image are 1 respectively, and use the normalized feature vector to represent the time series feature vector.

[0035] According to this configuration, feature vectors obtained from images of different sizes can be compared with each other with high accuracy.

[0036] (8) In the action recognition device described in any one of (1) to (7) above, the calculation unit may determine the feature vector included in the interval as the input time series feature vector when the interval of the feature vector that can be continuously calculated is longer than a given time.

[0037] According to this configuration, a time-series feature vector composed of a series of feature vectors representing one action can be determined as an input time-series feature vector, and thus the action of the user can be recognized with high accuracy.

[0038] (9) In the action recognition device described in any one of (1) to (8) above, the reference time-series feature vector may be the input time-series feature vector calculated in the past by the calculation unit.

[0039] According to this structure, since the input time series feature vector calculated in the past is used as the benchmark time series feature vector, it is easy to collect the benchmark time series feature vector, and since the past input time series feature vector and the current input time series feature vector are compared, the change relative to the past behavior can be determined with high accuracy.

[0040] (10) In the action recognition device described in any one of (1) to (9) above, the reference time series feature vector and the input time series feature vector may be time series feature vectors of the same user.

[0041] According to this configuration, it is possible to detect changes in the current behavior by comparing it with the user's past behavior.

[0042] (11) In the action recognition device described in any one of (1) to (10) above, the decision unit may determine that there is a change in the action of the user when a statistical value representing the correlation between the feature vectors corresponding to the time series in the input time series feature vector and the benchmark time series feature vector exceeds a threshold value.

[0043] According to this configuration, when the statistical value of the difference between the input time series feature vector and the reference time series feature vector exceeds the threshold, it is determined that there is a change in the user's behavior. Therefore, it is possible to accurately determine that the user's behavior has changed.

[0044] (12) In the action recognition device described in any one of (1) to (11) above, the action recognition device may further include an output unit that outputs the action and the change in the action determined by the determination unit.

[0045] According to this configuration, it is possible to output the determined action and the change in the action.

[0046] (13) Another embodiment of the present invention provides an action recognition method in a motion recognition device for recognizing the action of a user, the action recognition method comprising: acquiring an image; estimating the coordinates of the skeletal points of the user based on the acquired image; calculating a time series feature vector representing the feature vector connecting the torso of the user and the head of the user in a time series based on the skeletal point coordinates; and determining the action of the user and the change in the action by comparing an input time series feature vector serving as the calculated time series feature vector with a reference time series feature vector, wherein the reference time series feature vector is the time series feature vector serving as a reference.

[0047] According to this configuration, it is possible to provide a behavior recognition method that can achieve the same operational effects as those of the above-mentioned behavior recognition device.

[0048] (14) In another embodiment of the present invention, the action recognition program enables a computer to function as an action recognition device for recognizing the action of a user, and the action recognition program enables the computer to perform the following processing: acquiring an image; estimating the coordinates of the skeletal points of the user based on the acquired image; calculating a time series feature vector representing the feature vector connecting the torso of the user and the head of the user in a time series based on the skeletal point coordinates; and determining the action of the user and the change in the action by comparing an input time series feature vector as the calculated time series feature vector with a reference time series feature vector, wherein the reference time series feature vector is the time series feature vector that serves as a reference.

[0049] According to this structure, it is possible to provide a behavior recognition program that can obtain the same effects as the above-mentioned behavior recognition device.

[0050] (15) A non-transitory computer-readable recording medium recording an action recognition program in another embodiment of the present invention is a recording medium recording an action recognition program that enables a computer to function as an action recognition device for recognizing a user's action, wherein the action recognition program enables the computer to perform the following processing: acquiring an image; inferring the coordinates of the user's skeletal points based on the acquired image, and calculating a time series feature vector that represents the feature vector connecting the user's torso and the user's head in a time series based on the skeletal point coordinates; determining the user's action and changes in the action by comparing an input time series feature vector that is the calculated time series feature vector with a reference time series feature vector, wherein the reference time series feature vector is the reference time series feature vector; and outputting information representing the determined action and changes in the action.

[0051] The present disclosure can also be implemented as a behavior recognition system that operates according to such a behavior recognition program. In addition, it is needless to say that such a computer program can be circulated via a computer-readable non-transitory recording medium such as a CD-ROM or a communication network such as the Internet.

[0052] In addition, the embodiments described below all represent a specific example of the present disclosure. The numerical values, shapes, structural elements, steps, the order of steps, etc. shown in the following embodiments are examples and are not intended to limit the present disclosure. In addition, among the structural elements in the following embodiments, the structural elements that are not recorded in the independent claims representing the highest concept are described as arbitrary structural elements. In addition, in all embodiments, the various contents can also be combined.

[0053] (Implementation Method)

[0054] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Figure 1 : is a block diagram showing an example of the structure of the action recognition system in the embodiment of the present disclosure. The action recognition system includes an action recognition device 1 and a camera 4. The camera 4 is a photographing device for photographing the action of a user (hereinafter, the target user) who is the object of action recognition. For example, the camera 4 is equivalent to a fixed camera installed in a house where the user lives, but the method is not particularly limited. The camera 4 photographs the user at a given frame rate and inputs the obtained image to the action recognition device 1 at a given frame rate. The action recognition device 1 is a device for determining changes in the action of the target user using the image captured by the camera. The action recognition device 1 is composed of a computer including a processor 2, a memory 3 and an interface circuit (not shown).

[0055] The processor 2 is hardware for determining a change in the behavior of the target user based on the image, such as a central processing unit. The processor 2 includes an acquisition unit 21, an estimation unit 22, a calculation unit 23, a determination unit 24, and an output unit 25. The acquisition unit 21 to the output unit 25 can be realized by the central processing unit executing a behavior recognition program, or can be constituted by a dedicated hardware circuit such as an ASIC.

[0056] The memory 3 is a storage device for recording the target user's behavior obtained from the image, and is a non-volatile rewritable storage device such as a flash memory, a hard disk drive, or a solid state drive. The interface circuit is a data input and output mechanism, and is, for example, a communication circuit.

[0057] The memory 3 includes a frame memory 31 and an action database 32. The frame memory 31 stores the image acquired by the acquisition unit 21 from the camera 4. The action database 32 is a database that stores the input time series feature vector calculated in the past based on the image of the target user as a reference time series feature vector. Specifically, the action database 32 stores a plurality of action labels representing the types of actions that become decision candidates in correspondence with a plurality of reference time series feature vectors, and the plurality of reference time series feature vectors correspond to any one of the action labels. In addition, the so-called reference time series feature vector represents a time series feature vector that serves as a reference (comparison object) when comparing the input time series feature vector along the time series in order to calculate the user's action changes. The actions that become decision candidates are various actions that the user is more likely to perform indoors, such as walking, sitting, standing up, and bending over.

[0058] The acquisition unit 21 acquires the image captured by the camera 4 and inputs the acquired image to the frame memory 31. The frame memory 31 stores the image input from the acquisition unit 21.

[0059] The estimation unit 22 reads an image from the frame memory 31 and estimates a plurality of skeleton point coordinates of the target user and the reliability of each skeleton point coordinate based on the read image. The skeleton point coordinates refer to the coordinates of the skeleton points of the target user detected from the image in the image coordinate system.

[0060] The so-called reliability is a value indicating the certainty of the estimation of each bone point coordinate, and is a value indicating the certainty of the estimated bone point coordinate by probability. As the value of the reliability of the bone point coordinate increases, the certainty increases. For example, the reliability takes a value greater than 0 and less than 1.

[0061] The estimation unit 22 inputs an image into a learned model obtained by machine learning the relationship between the image and the bone point coordinates, thereby estimating multiple bone point coordinates and reliability. An example of a learned model is a deep neural network. An example of a deep neural network is a convolutional neural network including a convolutional layer and a pooling layer. In addition, the estimation unit 22 can also be composed of a learning model other than a deep neural network.

[0062] Figure 2 2 is a diagram showing an example of skeleton information 201 including the skeleton point coordinates P estimated by the estimation unit 22. Figure 2 In the figure, the dotted lines are auxiliary lines indicating the contour of the face and the position of the neck.

[0063] The skeleton information 201 is information indicating the skeleton point coordinates P of a single target user. The skeleton information 201 includes, for example, 17 skeleton point coordinates P consisting of the left eye, right eye, left ear, right ear, nose, left shoulder, right shoulder, left waist, right waist, left elbow, right elbow, left wrist, right wrist, left knee, right knee, left ankle, and right ankle. The learned model is constructed to estimate the 17 skeleton point coordinates P. Figure 2 In the example, the skeleton information 201 is composed of 17 skeleton point coordinates P, but this is only an example, and the number of skeleton point coordinates P can be less than 16 or more than 18. In this case, the learned model is constructed to estimate the given number of skeleton point coordinates P less than 16 or more than 18. In addition, the skeleton information 201 may also include Figure 2 Skeletal points other than the skeletal point coordinates P shown (for example, skeletal points such as fingers and mouth).

[0064] Furthermore, the skeleton information 201 includes a link K representing the connection between the skeleton point coordinates P. The skeleton point coordinates P are represented by the coordinate component in the horizontal direction of the image, i.e., the X coordinate, and the coordinate component in the vertical direction of the image, i.e., the Y coordinate. The skeleton information 201 is represented by the part key points that uniquely determine the skeleton point coordinates P, the skeleton point coordinates P, and the reliability of the skeleton point coordinates P. For example, the skeleton information 201 is represented in a dictionary form such as {part key point "right eye": [X coordinate, Y coordinate, reliability], part key point "left eye": [X coordinate, Y coordinate, reliability], ..., part key point "left ankle": [X coordinate, Y coordinate, reliability]}.

[0065] The calculation unit 23 calculates the time series feature vector based on the skeletal point coordinates P estimated by the estimation unit 22. Hereinafter, the time series feature vector is referred to as the input time series feature vector. The input time series feature vector includes a plurality of feature vectors arranged in a time series. A feature vector is calculated based on one image, and the feature vector is a two-dimensional vector having the body of the object user as the starting point and the head of the object user as the end point. The feature vector is represented by a vector obtained by subtracting the coordinates of the end point from the coordinates of the starting point. However, this is an example, and the feature vector can also be represented by a vector obtained by subtracting the coordinates of the starting point from the coordinates of the end point.

[0066] The decision unit 24 determines the action and change of the action of the target user by comparing the input time series feature vector calculated by the calculation unit 23 and the reference time series feature vector stored in the action database 32. The details of this process will be described later. The decision unit 24 can also use a true or false value to represent the change of the action. However, this is an example, and the decision unit 24 can also calculate a numerical value representing the size of the difference between the input time series feature vector and the reference time series feature vector as the change of the action of the target user. In addition, the true or false value is "true" when there is a change in the action, and "false" when there is no change in the action.

[0067] The reference time series feature vector to be compared with the input time series feature vector is, for example, the input time series feature vector of the target user calculated in the past by the calculation unit 23 as in the past six months. However, this is just an example, and the reference time series feature vector to be compared with the input time series feature vector may be a time series feature vector of another user of the same age as the target user, or a pre-saved time series feature vector.

[0068] The output unit 25 outputs information indicating the action and the change in action determined by the determination unit 24. Examples of cases where the action of the target user is different include a case where the action of sitting down six months ago is different from the current action of sitting down, or a case where the action of the user is different due to physical decline or injury caused by aging. The output unit 25 may also send the above information to a computer held by the target user via an input / output interface and store the above information in a storage device.

[0069] The action recognition device 1 may be composed of an edge server installed in a house, a smart speaker installed in a house, or a cloud server. When the action recognition device 1 is composed of an edge server, the camera 4 and the action recognition device 1 are connected via a local area network. When the action recognition device 1 is composed of a cloud server, the camera 4 and the action recognition device 1 are connected via a wide area communication network such as the Internet. In addition, the action recognition device 1 may be configured in a distributed manner such that some of the components are installed on the edge side and the remaining components are installed on the cloud side.

[0070] In addition, the action recognition device 1 does not necessarily need to be implemented by a separate computer device, and can also be implemented by a distributed processing system (not shown) including a terminal device and a server. For example, the acquisition unit 21, the frame memory 31, and the estimation unit 22 can be set in the terminal device, and the action database 32, the calculation unit 23, the determination unit 24, and the output unit 25 can be set in the server. In this case, the data exchange between the components is performed via a wide area communication network.

[0071] The above is the configuration of the action recognition device 1. Next, the processing of the action recognition device 1 will be described. Figure 3 This is a flowchart showing an example of the processing of the behavior recognition device 1 in the embodiment of the present disclosure.

[0072] (Step S1)

[0073] The acquisition unit 21 acquires an image from the camera 4, and stores the acquired image in the frame memory 31. The acquisition unit 21 may acquire an image from the camera 4 at all times, or may acquire an image from the camera 4 periodically.

[0074] (Step S2)

[0075] The estimation unit 22 estimates the multiple skeleton point coordinates P in each image and the reliability of each skeleton point coordinate P by acquiring multiple images from the frame memory 31 in time series and inputting the acquired images into the learned model in sequence. Thus, time series data of the skeleton point coordinates P and the reliability can be obtained. The estimation unit 22 can also perform a process of tracking the target user in the time series data. For example, the estimation unit 22 calculates the center of gravity of the circumscribed rectangle of the skeleton point coordinates P between images that are continuous in time series, and establishes a correspondence between the circumscribed rectangles whose calculated centers of gravity are located on the nearest side as the circumscribed rectangles of the same user, thereby tracking the target user. Thus, even if multiple target users are included in the image, each target user can be tracked separately. In addition, the estimation unit 22 can also use the Hungarian algorithm to establish the correspondence of the circumscribed rectangles between adjacent images. By using the Hungarian algorithm, when there are multiple centers of gravity in each image between adjacent images, a combination of centers of gravity with the smallest sum of distances between the centers of gravity can be determined.

[0076] In addition, in the present embodiment, the case where the target user included in the image is one is described, but this is an example, and the target user may be multiple. In this case, the action recognition device 1 estimates the skeleton point coordinates P and the reliability of the skeleton point coordinates P for each of the multiple target users, and uses the estimation results to determine the change of the action of each target user.

[0077] (Step S3)

[0078] The calculation unit 23 calculates a feature vector having the body of the target user as a starting point and the head of the target user as an end point, based on the skeleton point coordinates P acquired in step S2 .

[0079] Figure 4 is a diagram showing an example of a feature vector L calculated from an image obtained by photographing a user's sitting action. Figure 4 As shown, the feature vector L is a two-dimensional vector representing the posture of the upper body of the target user. The starting point 401 of the feature vector L is the center of gravity of the body, and the end point 402 of the feature vector L is the center of gravity of the head. The starting point 401 is, for example, the center of gravity coordinates of the skeletal point coordinates P of a total of four points, including two points of the two shoulders and two points of the two waists included in the body. The end point 402 is the center of gravity coordinates of the skeletal point coordinates P of a total of five points, including two points of the two ears, two points of the two eyes, and one point of the nose.

[0080] In order to improve the accuracy of the feature vector L, the calculation unit 23 may exclude the bone point coordinates P that are not detected due to occlusion and the orientation of the body from the calculation of the feature vector L. In this case, the calculation unit 23 only uses the detected bone point coordinates P to calculate the starting point 401 and the end point 402. When the calculation unit 23 does not detect any point of the bone point coordinates P required for the calculation of the starting point 401 or the end point 402 in the acquired image, the feature vector L may not be calculated. In this case, the calculation unit 23 replaces the feature vector L and records the missing value indicating the absence of the feature vector L. Thus, the calculation unit 23 can calculate a set of feature vectors L contained in the interval after excluding the missing values ​​as the input time series feature vector.

[0081] The calculation unit 23 may also calculate the center of gravity of the trunk or head by weighted averaging the skeleton point coordinates P included in the trunk or head using the reliability of each skeleton point coordinate P. The calculation unit 23 may also calculate the center of gravity of the circumscribed rectangle of the skeleton point coordinates P of the trunk or head instead of the center of gravity as the starting point 401 or the end point 402. The skeleton point coordinates P used in calculating the center of gravity of the trunk or head are not limited to the above-mentioned skeleton point coordinates P, and may also include other skeleton point coordinates P. For example, the calculation unit 23 may further use the skeleton point coordinates P of both knees to calculate the starting point 401.

[0082] (Step S4)

[0083] The calculation unit 23 calculates the input time series feature vector by cutting out the feature vector L of a fixed period from the current time to a certain time point in the past from the feature vector L calculated by step S3. An example of a fixed period is 10 seconds, but it is not particularly limited. There may be multiple actions in the fixed period. In this case, the calculation unit 23 can cut out the feature vector L for each action. Specifically, in a fixed period, when the length of the interval in which the feature vector L is continuously calculated is greater than a threshold value (for example, 4 seconds), the calculation unit 23 regards the interval as a valid interval representing one action, and calculates the time series data of the feature vector L contained in the valid interval as the input time series feature vector. The calculation unit 23 regards the missing interval in which the missing value is recorded in step S3 as the boundary of the valid interval. Therefore, in a fixed period, when the user takes multiple actions, the valid interval corresponding to each action is cut out, and the input time series feature vector corresponding to each valid interval is calculated. As a result, the input time series feature vector having an interval greater than the threshold value can be calculated, and the user's action can be determined with high accuracy. For example, it is possible to distinguish between the action of squatting down to put on shoes at the entrance and the action of bending down to pick up a shoe brush on the floor, and determine the change of action, and analyze the user's action in detail. That is, in this embodiment, when multiple input time series feature vectors are calculated based on a fixed period, the change of action is determined for each input time series feature vector. In addition, the operator who visually views the image can also specify the fixed period and the cutting of the effective interval according to the operation manual.

[0084] (Step S5)

[0085] In step S4, when the input time series feature vector is calculated within the fixed period (yes in step S5), the calculation unit 23 advances the processing to step S6. When the input time series feature vector is not calculated (no in step S5), it is assumed that the action of the target user is not included within the fixed period, and the processing is returned to step S1.

[0086] (Step S6)

[0087] The calculation unit 23 standardizes the input time series feature vector so that the time of the input time series feature vector calculated by step S4 becomes a reference value (for example, 1). Even for the same action, the time required for the action sometimes changes, but through such standardization, the time scale of the input time series feature vector and the reference time series feature vector can be made consistent, making it easy to compare the two vectors. For example, when it is assumed that there is a peak of the action in the center of the effective interval, the peak of the action in the 4-second input time series feature vector is located at the 2-second position, and the peak of the action in the 6-second reference time series feature vector is located at the 3-second position, so the positions of the peaks of the two vectors are inconsistent. For this reason, by performing such standardization, the positions of the peaks of the two vectors are both 0.5, making it easy to compare the two vectors.

[0088] The method of performing the standardization is not limited to the above. The calculation unit 23 may also translate the feature vector L so that the starting point 401 of the feature vector L constituting the input time series feature vector is located at the origin of the image coordinate system. By standardizing the feature vector L in this way, the feature vector L can be calculated without the influence of the position after the target user has taken action. For example, the position of the feature vector L that appears in the image is different for the case of squatting in the middle of the corridor and the case of squatting at the right end of the corridor. However, as described above, by performing the standardization of translating the feature vector L so that the starting point 401 of the feature vector L is located at the origin, the feature vector L can be compared while ignoring the difference between the case of the feature vector L appearing in the middle of the corridor and the case of the feature vector L appearing at the right end of the corridor. Thus, it is easy to compare the feature vectors L with each other. In addition, the calculation unit 23 may also standardize the feature vector L so that the vertical length and horizontal length of the image are respectively reference values ​​(for example, 1). In this case, the feature vectors L that appear in the images captured by cameras of different resolutions can be compared with each other.

[0089] (Step S7)

[0090] The decision unit 24 performs a process of comparing the input time series feature vector normalized in step S6 with the normalized reference time series feature vector stored in the behavior database 32. The process of step S7 will be described in detail below.

[0091] Figure 5 It is shown Figure 3 Flowchart of details of the comparison process in step S7.

[0092] (Step S71)

[0093] The decision unit 24 retrieves the reference time series feature vector that is most similar to the input time series feature vector standardized by step S6 from among all the reference time series feature vectors stored in the action database 32. Specifically, the decision unit 24 calculates the inter-feature vector distance between the input time series feature vector and the reference time series feature vector, and retrieves the reference time series feature vector with the smallest calculated inter-feature vector distance as the most similar reference time series feature vector.

[0094] The distance between feature vectors is calculated as follows. First, the decision unit 24 extracts feature vectors L of a given number of frames from the input time series feature vector and the reference time series feature vector, respectively. The given number of frames may be, for example, all frames or an appropriate number of frames such as 10 frames. Here, the feature vector L extracted from the input time series feature vector is set as the first feature vector, and the feature vector L extracted from the reference time series feature vector is set as the second feature vector. Next, the decision unit 24 calculates the average distance between the distances between the starting points and the distances between the end points for the first feature vector and the second feature vector of the same frame as the first feature vector. Furthermore, the decision unit 24 calculates the value obtained by averaging the average distance calculated for each frame over all frames as the distance between feature vectors.

[0095] (Step S72)

[0096] The decision unit 24 assigns the action label assigned to the reference time-series feature vector determined to be most similar in step S71 to the input time-series feature vector normalized in step S6 .

[0097] When the operation of the service provided by the action recognition device 1 to the target user starts, a default reference time series feature vector is stored in the action database 32. The default reference time series feature vector is generated by initialization processing.

[0098] The initialization process is as follows. First, the action recognition device 1 causes the target user to perform multiple action tests that are candidates for decision. In this case, the action recognition device 1 may also prompt the target user's portable terminal to guide actions such as please walk, please sit down, please stand up, and please bend down. The test performed by the target user may also be performed multiple times for one action. Next, the action recognition device 1 obtains an image of the test performed by the target user for a certain action from the camera 4, and calculates a time series feature vector based on the acquired image. The process of calculating the time series feature vector is performed by the estimation unit 22, the calculation unit 23, and the decision unit 24 using the above method. Next, the decision unit 24 of the action recognition device 1 establishes a correspondence between the calculated time series feature vector and the action label representing the action corresponding to the test performed by the target user. In addition, the decision unit 24 stores the time series feature vector corresponding to the action label as a reference time series feature vector in the action database 32. As a result, the default time series feature vector is stored in the action database 32.

[0099] In addition, in the initialization process or step S72, the process of assigning the action label is performed by the decision unit 24, but the present disclosure is not limited thereto, and the process may also be performed by an operator. For example, the operator may determine the action of the target user based on the image of the camera, and assign the action label to the reference time series feature vector by the operator inputting an operation of assigning the action label representing the determined action to the time series feature vector calculated based on the image.

[0100] There are various actions that the target user performs indoors, and a general action recognizer constructed using a deep neural network cannot recognize all of these actions. For example, a general action recognizer cannot recognize an action such as bending over. Therefore, in this embodiment, by performing such initialization processing, various actions performed by the target user indoors can be recognized.

[0101] In the above step S72, the input time series feature vector is compared with all the reference time series feature vectors stored in the action database 32, but this is an example, and it can also be compared with the reference time series feature vector representing each action. In this case, the decision unit 24 assigns the action label assigned to the reference time series feature vector that is most similar to the input time series feature vector among the reference time series feature vectors representing each action to the input time series feature vector. The reference time series feature vector representing each action is, for example, a reference time series feature vector obtained by averaging the reference time series feature vectors assigned the same action label.

[0102] From the perspective of identifying action changes, it is preferred to assign action labels to a series of time series feature vectors in which the target user takes the same action. On the other hand, it is not preferred to assign the same action label to a series of time series feature vectors containing discontinuous intervals. For example, since a posture change such as "squatting" does not contain an interval in which the action is discontinuous, there is no problem even if the action label of "squatting" is assigned to a series of time series feature vectors representing "squatting". However, since an action such as "opening the door to go out and returning home 1 hour later" contains a discontinuous interval, it is not preferred to assign the same action label to a series of time series feature vectors representing such an action. For this reason, in the present embodiment, for an action such as "opening the door to go out and returning home 1 hour later", the action labels are divided into "opening the door to go out" action and "opening the door to go in and out" action. Thereby, it is possible to prevent a time series feature vector containing an interval in which the action is discontinuous from being assigned one action label.

[0103] (Step S73)

[0104] The decision unit 24 stores the input time-series feature vector to which the action label is assigned in step S72 in the action database 32 .

[0105] (Step S74)

[0106] The decision unit 24 obtains a plurality of reference time series feature vectors (an example of the first reference time series feature vector) to which the same action label as the action label assigned to the input time series feature vector in step S72 is assigned from the action database 32. Here, the decision unit 24 may also obtain, from the action database 32, reference time series feature vectors from two years ago or more among the reference time series feature vectors to which the same action label as the action label assigned to the input time series feature vector is assigned. Thus, for the target user, the change in the current action based on the past action can be clearly captured.

[0107] (Step S75)

[0108] The decision unit 24 calculates the average time series feature vector by averaging the multiple benchmark time series feature vectors obtained by step S74. As a result, it is possible to compare the input time series feature vector with the multiple benchmark time series feature vectors one-to-one. In the case where the average time series feature vector is not calculated, the comparison of the input time series feature vector with the multiple benchmark time series feature vectors becomes a one-to-many comparison, which is more complicated to handle. In addition, from the perspective of reducing the noise contained in the multiple benchmark time series feature vectors, the averaging of the multiple benchmark time series feature vectors is useful. Specifically, the averaging has the effect of reducing the estimation error of the skeleton estimation that is the basis of the multiple benchmark time series feature vectors.

[0109] The decision unit 24 calculates the average value of the starting point 401 and the average value of the end point 402 of the feature vector L for each identical frame for a plurality of reference time series feature vectors, thereby calculating the average time series feature vector. In this case, when the number of frames (amount of data in the time direction) of the plurality of reference time series feature vectors is different, the decision unit 24 interpolates the missing feature vector L using linear interpolation, etc., thereby making the number of frames of the plurality of reference time series feature vectors consistent. For example, when the average time series feature vector is calculated based on the reference time series feature vector consisting of 10 frames and the reference time series feature vector consisting of 30 frames, if interpolation is not performed, only the average time series feature vector of 10 frames can be generated. In this way, 20 frames of the latter reference time series feature vector are not used for calculating the average time series feature vector. On the other hand, when interpolation is performed, the average time series feature vector of 30 frames can be calculated, and the average time series feature vector can be calculated more accurately.

[0110] (Step S76)

[0111] The decision unit 24 generates a comparison result by comparing the input time series feature vector normalized in step S6 with the average time series feature vector obtained in step S75. Specifically, the decision unit 24 can compare the input time series feature vector and the average time series feature vector by calculating the feature vector distance between the input time series feature vector and the average time series feature vector as described in step S71.

[0112] That is, the decision unit 24 calculates the average distances between the starting points and the end points for each frame for the input time series feature vector and the average time series feature vector, and further averages the average distances over all frames to calculate the distance between feature vectors.

[0113] Then, the decision unit 24 may generate a comparison result of “no change in behavior” when the distance between feature vectors is less than a threshold value, and may generate a comparison result of “change in behavior” when the distance between feature vectors exceeds the threshold value.

[0114] Furthermore, the decision unit 24 may compare the input time series feature vector with the average time series feature vector by calculating a statistic indicating the correlation between the feature vectors L corresponding to the time series in the input time series feature vector and the average time series feature vector. The statistic a is represented by the following mathematical formula.

[0115] a=R(Ci)

[0116] Here, i is the frame index of the input time series feature vector and the average time series feature vector. Ci is the correlation value between the input time series feature vector and the average time series feature vector. R is a contraction operator for contracting the correlation value Ci. The contraction operator R is an operator for contracting the correlation value Ci as time series data to calculate a scalar statistical value a. When the contraction operation is an average, the contraction operator R calculates the average value of the correlation value Ci.

[0117] When considering the temporal changes related to the actions of the target user, assuming that the skeleton of the target user is a rigid body, the actions of the upper body of the target user are set to be close to the rotational motion around the navel. Therefore, the statistical value a calculated based on the above mathematical formula can effectively express the user's actions with fewer dimensions.

[0118] As the relevant value Ci, Figure 4 As shown, the difference in vector length of the feature vector L, the difference in the differential value of the vector length of the feature vector L, the difference in the vector angle θ of the feature vector L, or the difference in the vector angular velocity can be used. The vector length is the distance from the starting point 401 to the end point 402 of the feature vector L. The differential value of the vector length is the time differential of the vector length, and is the change in the vector length between adjacent frames. In addition, the differential value of the vector length may be replaced by the N (N is an integer greater than or equal to 2) times differential value. The vector angle θ is the angle of the feature vector L relative to the horizontal direction of the image. The vector angular velocity is the time differential of the vector angle θ, and is the change in the vector angle θ between adjacent frames.

[0119] The abbreviation operator R can calculate the average value, absolute value average value, square mean value, standard deviation, and variance of the correlation value Ci. Therefore, the statistical value a is the average value, absolute value average value, square mean value, standard deviation, or variance of the correlation value Ci. Thus, the time series data is abbreviated to a scalar value.

[0120] The decision unit 24 may determine the change in behavior by comparing the statistic a with the threshold. Specifically, the decision unit 24 may generate a comparison result of "no change in behavior" when the statistic a is below the threshold, and may generate a comparison result of "change in behavior" when the statistic a exceeds the threshold.

[0121] Here, when the number of frames of the input time series feature vector and the average time series feature vector differ, the decision unit 24 may make the number of frames of both equal by using linear interpolation or the like.

[0122] Figure 6 This is a graph showing the temporal transition of the vector length of the feature vector L when two different sitting conditions are imposed on the subject for the action label “sitting down”. Figure 6 The left figure is a graph when the subject sits down without any load being applied (normal sitting). Figure 6 The right graph is a graph of a subject wearing a knee brace to simulate physical decline while sitting down. In both graphs, one broken line corresponds to one sitting down.

[0123] Comparing the two graphs, it can be seen that in the case of sitting with knee pads, a peak occurs in the vector length when the normalized time is around 0.2. This peak is not observed in normal sitting. Therefore, it is confirmed that there is a difference between normal sitting and sitting with knee pads. Therefore, by comparing the vector length of the feature vector L connecting the center of gravity of the trunk and the center of gravity of the head, for example, it is possible to confirm that there is a difference in the time series feature vector before and after the user's body declines. As a result, even in a scene where the whole body of the user who is the object of action recognition is not captured, changes in the user's actions can be recognized with high accuracy.

[0124] (Step S8)

[0125] The output unit 25 outputs the comparison result and action label generated in step S76. The comparison result and action label are output to, for example, a portable terminal of the target user. Thus, for example, when it is determined that there is a change in walking, the portable terminal displays a message indicating that there is a change in walking behavior on a display, or outputs the message from a speaker.

[0126] Thus, according to this embodiment, even when the camera angle of the camera 4 installed indoors is not favorable for sensing, it is possible to recognize the change of the action with high accuracy for each action of the user. Therefore, the action recognition device 1 of this embodiment is useful for the action recognition of the user in a house where the installation position of the camera 4 is often limited.

[0127] (Variation)

[0128] The above describes the action recognition device 1 involved in one or more embodiments of the present disclosure based on the implementation, but the present disclosure is not limited to the implementation. As long as it does not deviate from the main purpose of the present disclosure, various modifications that can be thought of by those skilled in the art are applied to the present embodiment, and the method of combining the structural elements in different embodiments can also be included in the scope of one or more embodiments of the present disclosure. For example, the following modification examples can be adopted.

[0129] (1) In the above description, the feature vector L starts from the center of gravity of the body and ends at the center of gravity of the head. However, the present disclosure is not limited to this. The center of gravity of the body may be the end point and the center of gravity of the head may be the start point.

[0130] (2) The output unit 25 outputs a message indicating that there is a change in the behavior. However, this is an example, and a message indicating that a decline is observed in the behavior may be output. In addition, the output unit 25 may output an index indicating the change in the behavior. As the index, for example, the above-mentioned distance between feature vectors or a statistical value may be used.

[0131] Industrial Applicability

[0132] The behavior recognition device disclosed herein is useful for recognizing the status of a user in a residence.

Claims

1. A motion recognition device for recognizing a user's motion, comprising: An acquisition unit, for acquiring an image; an estimating unit, configured to estimate the coordinates of the skeleton points of the user based on the image acquired by the acquiring unit; a calculation unit that calculates a time series feature vector that represents a feature vector connecting the body of the user and the head of the user in a time series based on the skeleton point coordinates; a storage unit storing a reference time series feature vector, the reference time series feature vector being the time series feature vector serving as a reference; and The determination unit determines the action of the user and the change in the action by comparing an input time-series feature vector, which is the time-series feature vector calculated by the calculation unit, with the reference time-series feature vector.

2. The action recognition device according to claim 1, wherein: The comparison performed by the determination unit is to compare any one of the length, angle, time change of the length, and time change of the angle of the feature vector constituting the input time series feature vector and the feature vector constituting the reference time series feature vector.

3. The action recognition device according to claim 1 or 2, wherein: The time series feature vector is standardized so that time becomes a reference value.

4. The action recognition device according to claim 1 or 2, wherein: The skeleton point coordinates estimated by the estimating unit include reliability indicating estimation accuracy, The calculation unit calculates the coordinates of the trunk and the coordinates of the head by weighting the coordinates of the skeleton points using the reliability, and calculates the feature vector based on the calculated coordinates of the trunk and the coordinates of the head.

5. The action recognition device according to claim 1 or 2, wherein: The reference time series feature vector includes a plurality of first reference time series feature vectors associated with action labels indicating types of the actions, The determination unit determines the action label representing the action of the user by comparing the input time series feature vector and the multiple first benchmark time series feature vectors, calculates an average time series feature vector by averaging the first benchmark time series feature vectors corresponding to the determined action label, and determines the change of the action of the user by comparing the average time series feature vector and the input time series feature vector.

6. The action recognition device according to claim 1 or 2, wherein: The feature vector calculated by the calculation unit represents the coordinates of the body and the coordinates of the head respectively using the coordinates of the image, The determination unit translates the feature vector so that the coordinates of the body of the feature vector calculated by the calculation unit are located at the origin of the coordinate system of the image, and uses the translated feature vector to represent the input time series feature vector.

7. The action recognition device according to claim 1 or 2, wherein: The determination unit normalizes the feature vector so that the vertical length and the horizontal length of the image are respectively 1, and uses the normalized feature vector to represent the time series feature vector.

8. The action recognition device according to claim 1 or 2, wherein: When a section in which the feature vectors can be continuously calculated is equal to or longer than a predetermined time, the calculation unit determines the feature vector included in the section as the input time-series feature vector.

9. The action recognition device according to claim 1 or 2, wherein: The reference time series feature vector is the input time series feature vector calculated in the past by the calculation unit.

10. The action recognition device according to claim 1 or 2, wherein: The reference time series feature vector and the input time series feature vector are time series feature vectors of the same user.

11. The action recognition device according to claim 1 or 2, wherein: The determination unit determines that there is a change in the behavior of the user when a statistic indicating a correlation between feature vectors corresponding to a time series in the input time series feature vector and the reference time series feature vector exceeds a threshold value.

12. The action recognition device according to claim 1 or 2, wherein: The action recognition device further includes an output unit that outputs the action and the change in the action determined by the determination unit.

13. A method for identifying an action, which is a method for identifying an action of a user in an action identification device, The action recognition method comprises: Get the image; Inferring the coordinates of the skeleton points of the user according to the acquired image; Based on the skeletal point coordinates, calculating a time series feature vector representing a feature vector connecting the user's body and the user's head in a time series; The action of the user and the change in the action are determined by comparing an input time series feature vector, which is the calculated time series feature vector, with a reference time series feature vector, which is the time series feature vector serving as a reference.

14. A motion recognition program that causes a computer to function as a motion recognition device that recognizes a user's motion, the motion recognition program causing the computer to execute the following processing: Get the image; Inferring the coordinates of the skeleton points of the user according to the acquired image; Based on the skeletal point coordinates, calculating a time series feature vector representing a feature vector connecting the user's body and the user's head in a time series; The action of the user and the change in the action are determined by comparing an input time series feature vector, which is the calculated time series feature vector, with a reference time series feature vector, which is the time series feature vector serving as a reference.

Images