Action recognition device, action recognition method, and action recognition program

Abstract

The action recognition device is provided with a determination unit that estimates skeleton point coordinates of a user from an image, calculates time-series feature points representing feature points of a body of the user in a time series on the basis of the estimated skeleton point coordinates, and recognizes the action of the user on the basis of the calculated time-series feature points. The input time-series feature point, which is the calculated time-series feature point, is compared with a reference time-series feature point, which is a reference time-series feature point, to determine a change in the walking of the user.

Description

Technical Field

[0001] The present disclosure relates to a technology for recognizing a user's action based on an image. Background Art

[0002] As a technology for recognizing the actions of a user using images such as static images and dynamic images, for example, Patent Document 1 and Patent Document 2 are known. Patent Document 1 discloses a technology that extracts time series data of a person's skeleton based on a dynamic image obtained by photographing a target user for the purpose of determining the fatigue level of a person, and determines the fatigue level based on the user's arm swing and step movements. Patent Document 2 discloses a technology that detects time series data of a specific part of a person based on a dynamic image obtained by photographing a target user for the purpose of determining the walking state and age of a pedestrian, and estimates the walking state and walking speed classification of the person.

[0003] There is an example where a camera installed indoors cannot be installed at a position that is advantageous for sensing due to limitations on the layout and area of ​​the room, and cannot perform frontal photography of walking at a distance of about 5 to 10 meters, which is usually used in analyzing walking motions.

[0004] However, the above-mentioned conventional behavior recognition technology does not take into account the case where a walking user cannot be photographed from the front, and therefore further improvement is required in order to determine changes in the user's walking.

[0005] Prior Art Literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent No. 6873344

[0008] Patent Document 2: Japanese Patent No. 3655618 Summary of the invention

[0009] The present disclosure has been made to solve the above-mentioned problems, and an object of the present disclosure is to provide a technology capable of determining changes in a user's walking with high accuracy even in a case where a walking user cannot be photographed from the front.

[0010] An action recognition device in one embodiment of the present invention is a recognition device for recognizing a user's walking, and the action recognition device comprises: an acquisition unit for acquiring an image; an estimation unit for estimating the coordinates of the user's skeletal points based on the image acquired by the acquisition unit; a calculation unit for calculating time series feature points representing feature points of the user's body in a time series based on the skeletal point coordinates estimated by the estimation unit; a storage unit for storing the time series feature points serving as a benchmark, namely, benchmark time series feature points; and a determination unit for determining changes in the user's walking by comparing the time series feature points calculated by the calculation unit, namely, the input time series feature points, with the benchmark time series feature points.

[0011] According to the present disclosure, even in a case where a walking user cannot be photographed from the front, changes in the user's walking can be determined with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a block diagram showing an example of the structure of the action recognition system in Embodiment 1 of the present disclosure.

[0013] Figure 2 FIG. 1 is a diagram showing an example of skeleton information including the coordinates of skeleton points estimated by the estimating unit.

[0014] Figure 3 This is a flowchart showing an example of the processing of the action recognition device according to the first embodiment of the present disclosure.

[0015] Figure 4 is a diagram showing an example of feature points.

[0016] Figure 5 is a diagram showing an example of a polynomial approximation curve.

[0017] Figure 6 It is shown Figure 3 A flowchart showing details of the process of step S7.

[0018] Figure 7 This is a graph for explaining a method of determining changes in walking using the standard deviation of the error calculated in step S8.

[0019] Figure 8 This is a block diagram showing an example of the structure of the action recognition device system in the second embodiment of the present disclosure.

[0020] Fig. 9 This is a flowchart showing an example of processing of the action recognition device according to the second embodiment of the present disclosure.

[0021] Fig.10This is a block diagram showing an example of the structure of a behavior recognition device system according to Embodiment 3 of the present disclosure.

[0022] Fig.11 This is a flowchart showing an example of processing of the action recognition device according to the third embodiment of the present disclosure.

[0023] Fig.12 It is shown Fig.11 A flowchart showing the details of the processing of step S303. DETAILED DESCRIPTION

[0024] (Insights that form the basis of this disclosure)

[0025] In recent years, a method of estimating a person's skeleton points from an image and recognizing the user's actions based on the estimated skeleton points has become known. In such a recognition method, an image obtained by shooting the user's entire body at a camera angle that is advantageous for sensing and an image obtained by shooting the user's walking state for a long time are used. In addition, in such a recognition method, a deep neural network including a convolution layer and a pooling layer is used to estimate the skeleton points, thereby achieving high accuracy.

[0026] However, in recent years, there has been a great demand for technology that detects changes in a user's walking in daily life in order to sense physical weakness, etc. However, in scenes such as daily life, it is common to see parts of the user's body that are the subject of action recognition hidden by some obstacles or parts of the body that are out of the image. Therefore, in conventional recognition methods, there is a problem that changes in the user's walking cannot be determined with high accuracy when an action is captured under poor shooting conditions. In particular, such a problem is more likely to occur in a residence where there are restrictions on the location of the camera.

[0027] The present disclosure has been made in view of the above-mentioned problems, and provides a technology capable of determining changes in the user's walking with high accuracy even in a scene where the entire body of the user to be recognized is not captured.

[0028] (1) An action recognition device in one embodiment of the present invention is an action recognition device for recognizing a user's walking, and the action recognition device comprises: an acquisition unit for acquiring an image; an estimation unit for estimating the coordinates of the user's skeleton points based on the image acquired by the acquisition unit; a calculation unit for calculating time series feature points representing feature points of the user's body in a time series based on the skeleton point coordinates estimated by the estimation unit; a storage unit for storing the time series feature points serving as a benchmark, namely, the benchmark time series feature points; and a determination unit for determining changes in the user's walking by comparing the time series feature points calculated by the calculation unit, namely, the input time series feature points, with the benchmark time series feature points.

[0029] According to this structure, the change of the user's walking is determined by comparing the input time series feature points and the reference time series feature points. Here, the input time series feature points and the reference time series feature points are respectively composed of feature points of the user's body. The feature points are feature points that can be generated if there is information about the user's upper body, and do not change significantly depending on the camera angle. Therefore, according to this structure, even in a scene where the whole body of the user to be recognized is not captured, the change of the user's walking can be determined with high accuracy.

[0030] (2) In the action recognition device described in (1), the time series feature points may be normalized so that the time required for the walking becomes a reference value.

[0031] According to this configuration, since the input time series feature points and the reference time series feature points are normalized so that the time required for walking becomes the reference value, the input time series feature points and the reference time series feature points can be easily compared.

[0032] (3) In the action recognition device described in (1) or (2), the action recognition device may also include: a judgment unit that judges whether the user is walking straight based on the time series feature points calculated by the calculation unit, and when it is judged that the user is walking straight, the decision unit performs processing to determine changes in the walking of the user.

[0033] According to this configuration, when the user is walking straight, the process of determining the change in the user's walking is executed, and when the user is not walking straight, the process of determining the change in the user's walking is not executed, so the change in the user's walking can be determined with high accuracy.

[0034] (4) In the action recognition device described in any one of (1) to (3), it is also possible that the action recognition device further includes: a depth correction unit that calculates a correction coefficient corresponding to the distance between the user contained in the image and the camera photographing the user in a time series, and the determination unit normalizes the time series feature points by multiplying each feature point constituting the time series feature points by the correction coefficient corresponding to each feature point, and determines the change in the walking of the user by comparing the normalized time series feature points with the normalized baseline time series feature points.

[0035] According to this configuration, the input time-series feature points and the reference time-series feature points can be compared after removing the size change in the image of the user corresponding to the distance of the user from the camera.

[0036] (5) In the action recognition device described in any one of (1) to (4), the calculation unit may calculate the polynomial approximation curves of the X coordinates and Y coordinates of the time series feature points based on the calculated image coordinates of the time series feature points, and use the polynomial approximation curves of the calculated X coordinates and Y coordinates to correct the values ​​of the X coordinates and Y coordinates of each feature point constituting the time series feature points.

[0037] According to this structure, the estimated coordinates of the skeletal points of the time series may contain errors, but the X-coordinate and Y-coordinate values ​​of the feature points of the time series are corrected using the polynomial approximation curves of the X-coordinate and the Y-coordinate, so that the feature points of the time series become straight-line trajectories, which can determine the changes in the user's walking with high precision.

[0038] (6) In the action recognition device described in any one of (1) to (5), it may be that the baseline time series feature point includes multiple first baseline time series feature points, and the determination unit calculates an average time series feature point by averaging the multiple first baseline time series feature points, and determines the change in the user's walking by comparing the average time series feature point with the input time series feature point.

[0039] According to this configuration, since the average time series feature point obtained by averaging a plurality of first reference time series feature points is compared with the input time series feature point, the input time series feature point and the first reference time series feature point can be compared in a one-to-one correspondence.

[0040] (7) In the action recognition device described in any one of (1) to (6), the calculation unit may determine the feature points included in a section in which the time series feature points can be continuously calculated as the input time series feature points when the section is longer than a given time.

[0041] According to this configuration, since time-series feature points composed of a series of feature points representing one walk can be determined as input time-series feature points, changes in the user's walk can be recognized with high accuracy.

[0042] (8) In the action recognition device described in any one of (1) to (7), the reference time-series feature points may be the input time-series feature points calculated in the past by the calculation unit.

[0043] According to this structure, since the input time series feature points calculated in the past are used as the benchmark time series feature points, it is easy to collect the benchmark time series feature points, and the past input time series feature points and the current input time series feature points are compared, so the changes relative to the past walking can be determined with good accuracy.

[0044] (9) In the action recognition device described in any one of (1) to (8), the reference time-series feature points and the input time-series feature points may be time-series feature points of the same user.

[0045] According to this configuration, it is possible to determine a change in the current walking by comparing with the user's past walking.

[0046] (10) In the action recognition device described in any one of (1) to (9), the action recognition device may further include an output unit that outputs information indicating the change in the walking of the user determined by the determination unit.

[0047] According to this configuration, it is possible to output information indicating the determined change in walking.

[0048] (11) Another embodiment of the present invention is an action recognition method in an identification device for identifying a user's walking. In the action recognition method, an image is acquired, and the coordinates of the user's skeletal points are estimated based on the acquired image. Based on the estimated skeletal point coordinates, time series feature points representing feature points of the user's body in a time series are calculated. By comparing the calculated time series feature points, i.e., the input time series feature points, with the time series feature points that serve as a benchmark, i.e., the benchmark time series feature points, changes in the user's walking are determined.

[0049] According to this configuration, it is possible to provide a behavior estimation method that can achieve the same effects as those of the above-mentioned behavior recognition device.

[0050] (12) In another embodiment of the present invention, an action recognition program is an action recognition program that enables a computer to execute an action recognition method for recognizing a user's walking, wherein the action recognition program enables the computer to execute the following processing: acquiring an image, estimating the coordinates of the user's skeletal points based on the acquired image, calculating time series feature points representing feature points of the user's body in a time series based on the estimated skeletal point coordinates, and determining changes in the user's walking by comparing the calculated time series feature points, i.e., the input time series feature points, with the time series feature points that serve as a benchmark, i.e., the benchmark time series feature points.

[0051] According to this configuration, it is possible to provide a behavior estimation method that can achieve the same effects as those of the above-mentioned behavior recognition device.

[0052] (13) In another embodiment of the present invention, the recording medium is a non-temporary computer-readable recording medium that records an action recognition program that enables a computer to execute an action recognition method for recognizing a user's walking, wherein the program enables the computer to execute the following processing: acquiring an image, estimating the coordinates of the user's skeletal points based on the acquired image, calculating time series feature points that represent feature points of the user's body in a time series based on the estimated skeletal point coordinates, and determining changes in the user's walking by comparing the calculated time series feature points, i.e., the input time series feature points, with the time series feature points that serve as a benchmark, i.e., the benchmark time series feature points.

[0053] The present disclosure can also be implemented as a behavior recognition system that operates according to such a behavior recognition program. In addition, such a computer program can of course be circulated via a computer-readable non-transitory recording medium such as a CD-ROM or a communication network such as the Internet.

[0054] In addition, the embodiments described below all represent a specific example of the present disclosure. The numerical values, shapes, structural elements, steps, the order of steps, etc. shown in the following embodiments are examples and are not intended to limit the present disclosure. In addition, among the structural elements in the following embodiments, the structural elements that are not recorded in the independent claims representing the highest concept are described as arbitrary structural elements. In addition, in all embodiments, the various contents can also be combined.

[0055] (Implementation method 1)

[0056] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Figure 1 : is a block diagram showing an example of the structure of the action recognition system in embodiment 1 of the present disclosure. The action recognition system includes an action recognition device 1 and a camera 4. The camera 4 is a photographing device for photographing the action of a user (hereinafter, the target user) who is the object of action recognition. For example, it is equivalent to a fixed camera installed in a house where the user lives, but this method is not particularly limited. The camera 4 photographs the user at a given frame rate and inputs the obtained image to the action recognition device 1 at a given frame rate. The action recognition device 1 is a device for determining changes in the walking of the target user using the image captured by the camera. The action recognition device 1 is composed of a computer including a processor 2, a memory 3 and an interface circuit (not shown).

[0057] The processor 2 is hardware for determining changes in the walking of the target user based on the image, such as a central processing unit. The processor 2 includes an acquisition unit 21, an estimation unit 22, a calculation unit 23, a determination unit 24, and an output unit 25. The acquisition unit 21 to the output unit 25 can be implemented by the central processing unit executing a behavior recognition program, or can be configured by a dedicated hardware circuit such as an ASIC.

[0058] The memory 3 is a storage device for recording the walking of the target user obtained from the image, and is a non-volatile rewritable storage device such as a flash memory, a hard disk drive, a solid state drive, etc. The interface circuit is a data input and output mechanism, such as a communication circuit.

[0059] The memory 3 includes a frame memory 31 and a walking database 32. The frame memory 31 stores images acquired by the acquisition unit 21 from the camera 4. The walking database 32 is a database that stores input time-series feature points calculated in the past from images of the target user as reference time-series feature points.

[0060] The acquisition unit 21 acquires the image captured by the camera 4 and stores the acquired image in the frame memory 31. The frame memory 31 stores the image input from the acquisition unit 21.

[0061] The estimation unit 22 reads an image from the frame memory 31 and estimates a plurality of skeleton point coordinates of the target user and the reliability of each skeleton point coordinate based on the read image. The skeleton point coordinates refer to the coordinates of the skeleton points of the target user detected from the image in the image coordinate system.

[0062] The so-called reliability is a value indicating the certainty of the estimation of each bone point coordinate, and is a value indicating the certainty of the estimated bone point coordinate by probability. The larger the reliability of the bone point coordinate, the higher the certainty. For example, the reliability takes a value greater than 0 and less than 1.

[0063] The estimation unit 22 inputs an image into a learned model obtained by machine learning the relationship between the image and the bone point coordinates, thereby estimating multiple bone point coordinates and reliability. An example of a learned model is a deep neural network. An example of a deep neural network is a convolutional neural network including a convolutional layer and a pooling layer. In addition, the estimation unit 22 can also be composed of a learning model other than a deep neural network.

[0064] Figure 2 2 is a diagram showing an example of skeleton information 201 including the skeleton point coordinates P estimated by the estimation unit 22. Figure 2 In the figure, the dotted lines are auxiliary lines indicating the contour of the face and the position of the neck.

[0065] The skeleton information 201 is information indicating the skeleton point coordinates P of a single target user. The skeleton information 201 includes, for example, 17 skeleton point coordinates P consisting of the left eye, right eye, left ear, right ear, nose, left shoulder, right shoulder, left waist, right waist, left elbow, right elbow, left wrist, right wrist, left knee, right knee, left ankle, and right ankle. The estimation unit 22 is configured to estimate these skeleton point coordinates P. Figure 2 In the example, the skeleton information 201 is composed of 17 skeleton point coordinates P, but this is only an example, and the number of skeleton point coordinates P can be less than 16 or more than 18. In this case, the learned model is constructed to estimate the given number of skeleton point coordinates P less than 16 or more than 18. In addition, the skeleton information 201 may also include Figure 2 Skeletal points other than the skeletal point coordinates P shown (for example, skeletal points of fingers and mouth, etc.).

[0066] Furthermore, the skeleton information 201 includes a link L representing the connection between the skeleton point coordinates P. The skeleton point coordinates P are represented by the coordinate component in the horizontal direction of the image, i.e., the X coordinate, and the coordinate component in the vertical direction of the image, i.e., the Y coordinate. The skeleton information 201 is represented by the part key points that uniquely determine the skeleton point coordinates P, the skeleton point coordinates P, and the reliability of the skeleton point coordinates P. For example, the skeleton information 201 is represented in a dictionary form such as {part key point "right eye": [X coordinate, Y coordinate, reliability], part key point "left eye": [X coordinate, Y coordinate, reliability], ..., part key point "left ankle": [X coordinate, Y coordinate, reliability]}.

[0067] The calculation unit 23 calculates time series feature points representing feature points of the body of the target user in time series based on the skeleton point coordinates P estimated by the estimation unit 22. Hereinafter, the time series feature points are referred to as input time series feature points. Feature points are points of the image coordinate system calculated from one image.

[0068] The decision unit 24 determines the change in the walking of the target user by comparing the input time series feature points calculated by the calculation unit 23 with the reference time series feature points stored in the walking database 32. The details of this process will be described later. The decision unit 24 can also use true and false values ​​to represent the change in behavior. However, this is an example, and the decision unit 24 can also calculate a numerical value representing the size of the difference between the input time series feature points and the reference time series feature points as the change in the walking of the target user. In addition, the true and false value is "true" when there is a change in the behavior, and "false" when there is no change in the behavior.

[0069] The reference time series feature points to be compared with the input time series feature points are, for example, the input time series feature points of the target user calculated in the past by the calculation unit 23 as in the past six months. However, this is just an example, and the reference time series feature points to be compared with the input time series feature points may be time series feature points of another user of the same age as the target user, or may be time series feature points representing reference walking stored in advance.

[0070] The output unit 25 outputs information indicating changes in the walking of the target user determined by the determination unit 24. Examples of changes in the walking of the target user include a case where the walking of the target user photographed six months ago is different from the walking of the target user photographed currently, or a case where a difference occurs in the walking of the user due to physical weakness or injury caused by aging. The output unit 25 may also send the above information to a computer held by the target user via an input / output interface and cause a storage device to store the above information.

[0071] The action recognition device 1 may be composed of an edge server installed in a house, a smart speaker installed in a house, or a cloud server. When the action recognition device 1 is composed of an edge server, the camera 4 and the action recognition device 1 are connected via a local area network. When the action recognition device 1 is composed of a cloud server, the camera 4 and the action recognition device 1 are connected via a wide area communication network such as the Internet. In addition, the action recognition device 1 may also be configured such that some of the structural elements are arranged on the edge side, and the remaining structural elements are arranged on the cloud side.

[0072] In addition, the behavior recognition device 1 does not necessarily need to be implemented by a separate computer device, and can also be implemented by a distributed processing system (not shown) including a terminal device and a server. In this case, the acquisition unit 21, the frame memory 31, and the estimation unit 22 can also be set in the terminal device, and the walking database 32, the calculation unit 23, the determination unit 24, and the output unit 25 can also be set in the server. In this case, the data transmission between the structural elements is performed via a wide area communication network.

[0073] The above is the configuration of the action recognition device 1. Next, the processing of the action recognition device 1 will be described. Figure 3 This is a flowchart showing an example of the processing of the action recognition device 1 in the first embodiment of the present disclosure.

[0074] (Step S1)

[0075] The acquisition unit 21 acquires images from the camera 4, and stores the acquired images in the frame memory 31. The acquisition unit 21 may always acquire images from the camera 4, or may acquire images periodically.

[0076] (Step S2)

[0077] The estimation unit 22 estimates the multiple skeleton point coordinates P and the reliability of each skeleton point coordinate P in each image by acquiring multiple images from the frame memory 31 in time series and inputting the acquired images into the learned model in sequence. Thus, time series data of the skeleton point coordinates P and the reliability can be obtained. The estimation unit 22 can also perform a process of tracking the target user in the time series data. For example, the estimation unit 22 calculates the center of gravity of the circumscribed rectangle of the skeleton point coordinates P between images that are continuous in time series, and establishes a correspondence between the circumscribed rectangles whose calculated centers of gravity are located on the nearest side as the circumscribed rectangles of the same user, thereby tracking the target user. Thus, even if multiple target users are included in the image, each target user can be tracked separately. In addition, the estimation unit 22 can also use the Hungarian algorithm to perform the correspondence relationship between the circumscribed rectangles of adjacent images. By using the Hungarian algorithm, when there are multiple centers of gravity in each image between adjacent images, a combination of centers of gravity with the smallest sum of distances between the centers of gravity can be determined.

[0078] In the first embodiment, the case where the target user included in the image is one is described, but this is an example, and the target user may be multiple. In this case, the action recognition device 1 estimates the skeleton point coordinates P and the reliability of the skeleton point coordinates P for each of the multiple target users, and uses the estimation results to determine the change of the action of each target user.

[0079] (Step S3)

[0080] The calculation unit 23 calculates the characteristic points of the body of the target user based on the skeleton point coordinates P obtained in step S2. In this embodiment, the characteristic points calculated from the skeleton point coordinates P based on the skeleton point coordinates P along the time series and the reliability are called time series characteristic points.

[0081] Figure 4 4 is a diagram showing an example of a feature point 401 calculated from an image captured of a walking user. Feature point 401 represents a reference point of the upper body in image coordinates and is the centroid of the skeleton point coordinates P of the body. Feature point 401 is the centroid of the skeleton point coordinates P of a total of four points, namely, the skeleton point coordinates P1 and P2 of the shoulders and the skeleton point coordinates P3 and P4 of the waist of the target user.

[0082] In order to improve the estimation accuracy of the feature point 401, the calculation unit 23 may exclude the bone point coordinates P that are not detected due to occlusion and the orientation of the body from the calculation of the feature point 401. In this case, the calculation unit 23 only uses the detected bone point coordinates P to calculate the feature point 401. The calculation unit 23 may not calculate the feature point 401 when the bone point coordinates P required for calculating the feature point 401 are not detected at any point in the acquired image. In this case, the calculation unit 23 may replace the feature point 401 and record the missing value indicating that there is no feature point 401. In this way, the calculation unit 23 can calculate a group of feature points 401 included in the interval excluding the missing value as the input time series feature point.

[0083] The calculation unit 23 may also calculate the feature point 401 by weighted averaging the skeletal point coordinates P1 to P4 included in the body using the reliability of the skeletal point coordinates P1 to P4. The calculation unit 23 may also calculate the center of gravity of the circumscribed rectangle including the skeletal point coordinates P1 to P4 instead of the center of gravity of the skeletal point coordinates P1 to P4 as the feature point 401. The skeletal point coordinates P used in calculating the feature point 401 are not limited to the skeletal point coordinates P1 to P4, and may include other skeletal point coordinates P. For example, the calculation unit 23 may further use the skeletal point coordinates P of both knees or both elbows to calculate the skeletal point coordinates.

[0084] (Step S4)

[0085] The calculation unit 23 calculates the time series feature point by cutting out the feature points 401 of a fixed period from the present to a certain time point in the past from the feature points 401 calculated in step S3. An example of the fixed period is 10 seconds, but it is not particularly limited. The fixed period may include multiple walks. In this case, the calculation unit 23 can cut out the feature point 401 for each of the multiple walks. Specifically, in the fixed period, when the length of the interval in which the feature points 401 are continuously calculated is greater than a threshold value (for example, 2 seconds), the calculation unit 23 regards the interval as a walking interval representing one walk, and calculates the time series data of the feature points 401 included in the walking interval as the input time series feature point. The calculation unit 23 can regard the missing interval in which the missing value is recorded in step S3 as the boundary of the walking interval. Therefore, when there are multiple walking intervals in the fixed period, the input time series feature points corresponding to each walking interval are calculated. As a result, the input time series feature points of the walking intervals having more than the threshold value are calculated, so that the user's walking can be determined with good accuracy. That is, in this embodiment, when a plurality of input time series feature points are calculated from a fixed period, the change in walking is determined for each input time series feature point. In addition, the operator who visually views the image can also specify the fixed period and the cutting of the walking section through the operation manual.

[0086] (Step S5)

[0087] In step S4 , when the fixed period includes a walking section (Yes in step S5 ), the calculation unit 23 advances the process to step S6 , and when the fixed period does not include a walking section (No in step S5 ), the calculation unit 23 returns the process to step S1 .

[0088] (Step S6)

[0089] The calculation unit 23 normalizes the input time series feature points so that the time of the input time series feature points calculated in step S4 is a reference value (e.g., 1). Even for the same walking, the time required for the action may sometimes change, but by performing such normalization, the time scale of the input time series feature points and the reference time series feature points can be made consistent, making it easy to compare the two time series feature points. For example, when it is assumed that there is a walking peak in the center of the valid interval, the walking peak in the 4-second input time series feature point is located at the 2-second position, and the walking peak in the 6-second reference time series feature point is located at the 3-second position, so the positions of the peaks of the two time series feature points are inconsistent. For this reason, by performing such normalization, the positions of the peaks of the two time series feature points are both 0.5, making it easy to compare the two time series feature points.

[0090] The method of normalization is not limited to the above. For example, the calculation unit 23 may normalize the X coordinate (horizontal coordinate of the image) and the Y coordinate (vertical coordinate of the image) of the feature point so that the horizontal length and the vertical length of the image of each frame of the input time series feature point become a reference value (for example, 1). In this case, the feature points 401 appearing in images captured by cameras of different resolutions can be compared with each other.

[0091] (Step S7)

[0092] The calculation unit 23 corrects each feature point included in the input time series feature point. The skeleton point coordinates P estimated by the estimation unit 22 include estimation errors, and the trajectory on the image of each feature point 401 constituting the input time series feature point is not smooth. Therefore, the calculation unit 23 approximates the feature point of the time series with a polynomial so that the trajectory on the image of each feature point 401 constituting the input time series feature point moves smoothly.

[0093] Figure 5 is a diagram showing an example of a polynomial approximation curve for correcting input time series feature points. Figure 5 In the figure, the horizontal axis represents the frame, and the vertical axis represents the X coordinate of the feature point.

[0094] First, the calculation unit 23 calculates a polynomial approximation curve of the X coordinate of the input time series feature point based on the calculated image coordinate of the input time series feature point. The calculation unit 23 fits the polynomial g set to v=g(u) to the input time series feature point when depicting the value (v) of the X coordinate of the input time series feature point relative to the value (u) of the frame (time) of the input time series feature point, and calculates the polynomial approximation curve. The degree N of the polynomial g is, for example, 4.

[0095] The calculation unit 23 corrects the X coordinate of the input time series feature point using the calculated polynomial approximation curve of the X coordinate. The calculation unit 23 calculates the corrected value of the X coordinate of the feature point 401 by substituting the value u of the calculated value of the feature point before correction into the polynomial g.

[0096] Similarly, the calculation unit 23 calculates the correction value of the Y coordinate of the feature point. That is, the calculation unit 23 calculates the polynomial approximation curve of the Y coordinate of the input time series feature point based on the calculated image coordinates of the input time series feature point. And, the calculation unit 23 corrects the value of the Y coordinate of the time series feature point 401 using the calculated polynomial approximation curve of the Y coordinate.

[0097] (Step S8)

[0098] The calculation unit 23 calculates the error between the polynomial g and the X coordinate of each feature point 401 constituting the feature point of the input time series, and calculates the error between the polynomial g and the Y coordinate of each feature point 401 constituting the feature point of the input time series. As the error, an absolute value error or a square error can be used. Furthermore, the Hubor error, which is stronger than the outlier, can be used to replace these errors. The Hubor error is an error function that becomes a square error when the difference between the polynomial g and each feature point 401 is less than 1, and becomes an absolute value error when the difference is greater than 1. When the error between the polynomial g and the feature point 401 is used as the error, even when the shooting direction of the camera 4 is inconsistent with the walking direction of the user, the walking can be analyzed without being affected by the deviation of the feature point caused by the angle of the camera 4.

[0099] (Step S9)

[0100] The decision unit 24 performs a process of comparing the input time-series feature points normalized in step S6 and corrected in step S7 with the normalized reference time-series feature points recorded in the travel database 32 .

[0101] Figure 6 It is shown Figure 3 A flowchart showing details of the processing of step S9.

[0102] (Step S91)

[0103] The decision unit 24 obtains the normalized and corrected input time series feature points from the calculation unit 23. At this time, the decision unit 24 stores the obtained input time series feature points as reference time series feature points in the walking database 32. Thus, the normalized and corrected input time series feature points are stored in the walking database 32 as reference time series feature points.

[0104] (Step S92)

[0105] The decision unit 24 calculates the average time series feature point as a separate time series feature point by averaging the multiple benchmark time series feature points (the first benchmark time series feature point) stored in the walking database 32. As a result, it is possible to compare the input time series feature point with the multiple benchmark time series feature points on a one-to-one basis. In the case where the average time series feature point is not calculated, the comparison of the input time series feature point with the multiple benchmark time series feature points becomes a one-to-many comparison, which is more complicated to handle. In addition, from the perspective of reducing the noise contained in the multiple benchmark time series feature points, averaging the multiple benchmark time series feature points is useful. Specifically, the averaging has the effect of reducing the estimation error of the skeleton estimation that is the basis of the multiple benchmark time series feature points.

[0106] The decision unit 24 calculates the average value of the feature point 401 for each identical frame for a plurality of reference time series feature points, thereby calculating the average time series feature point. In this case, when the number of frames (amount of data in the time direction) of the time series feature points is different, the decision unit 24 uses linear interpolation or the like to interpolate the missing feature points, thereby making the number of frames of the plurality of reference time series feature points consistent. For example, when the average time series feature point is calculated based on the reference time series feature points consisting of 10 frames and the reference time series feature points consisting of 30 frames, if interpolation is not performed, only the average time series feature points of 10 frames can be generated. In this way, the latter's 20 frames of reference time series feature points are not used for the calculation of the average time series feature points. On the other hand, when interpolation is performed, the average time series feature points of 30 frames can be calculated, and the average time series feature points can be calculated more accurately.

[0107] (Step S93)

[0108] The decision unit 24 generates a comparison result by comparing the input time series feature points normalized in step S6 and corrected in step S7 with the average time series feature points obtained in step S92. Specifically, the decision unit 24 calculates the distance between the input time series feature points and the average time series feature points to compare the two.

[0109] The distance between feature points is calculated as follows. First, the decision unit 24 extracts a given number of feature points 401 from the input time series feature points and the reference time series feature points. The given number of frames may be, for example, all frames or an appropriate number of frames such as 10 frames. Furthermore, the decision unit 24 calculates the distance between the feature points 401 for each frame for the input time series feature points and the average time series feature points, and calculates the average value of the calculated distances over all frames as the distance between feature points.

[0110] Furthermore, the decision unit 24 compares the distance between the feature points with a threshold value (for example, 20 pixels), and generates a comparison result of "no change during walking" when the distance between the feature points is below the threshold value, and generates a comparison result of "change during walking" when the distance between the feature points exceeds the threshold value. Alternatively, the decision unit 24 may also generate a comparison result by comparing the time series feature amount calculated based on the feature point 401 with the threshold value. For example, the decision unit 24 may calculate the distance between the feature points 401 for each frame for the input time series feature points and the average time series feature points, calculate the difference between the calculated distances between adjacent frames, and calculate the value obtained by averaging the calculated differences over all frames as the time series feature amount. Alternatively, the decision unit 24 may calculate the distance between the feature points 401 for each frame for the input time series feature points and the average time series feature points, and calculate the statistical value of the calculated distance as the time series feature amount. The statistical value is, for example, an absolute value average, a square mean, a standard deviation, a dispersion, or a skewness. In addition, the determination unit 24 generates the presence or absence of walking changes as the comparison result, but may generate the distance between feature points or the time series feature quantity as the comparison result instead of or in addition to the presence or absence of walking changes. Alternatively, the target user or operator may visually compare the distance between feature points or the time series feature quantity with the threshold value, and input the comparison result into the terminal device held by the target user or operator, thereby generating the comparison result.

[0111] Alternatively, the decision unit 24 may calculate the standard deviation of the error calculated in step S8 as the time series feature value, and compare the calculated time series feature value with the standard deviation of the error calculated in step S8. Figure 7 The change in walking is determined by comparing it with the threshold straight line 710 shown by the dotted line.

[0112] Figure 7 : is a graph for explaining a method of determining a change in walking using the standard deviation of the error calculated in step S8. Figure 7In the figure, the vertical axis represents the standard deviation of the Y coordinate, and the horizontal axis represents the standard deviation of the X coordinate. The standard deviation of the X coordinate refers to the standard deviation of the error between the calculated value of the X coordinate before correction calculated in step S8 and the polynomial g of the X coordinate for one time series feature point. The standard deviation of the Y coordinate refers to the standard deviation of the error between the calculated value of the Y coordinate before correction calculated in step S8 and the polynomial g of the Y coordinate for one time series feature point.

[0113] exist Figure 7 In the figure, point 701 shows the test result for normal straight walking at a constant speed (straight), and points 702 to 705 show the test results for each of the four types of abnormal walking simulations. Point 702 is a gradually increasing walking speed (boost), point 703 is a slow walking (slow), point 704 is a stumbling walking (stumble), and point 705 is a personal habitual walking with a tendency of right shoulder drooping (habit).

[0114] exist Figure 7 In the graph of , five tests were conducted for each of the five types of walking, and the standard deviation of the error of the X coordinate and the standard deviation of the Y coordinate were obtained for each test. Points 701 to 705 are points where the average value of the standard deviation of the X coordinate and the average value of the standard deviation of the Y coordinate obtained in each test are plotted. In each of the points 701 to 705, the vertical line segment is the error bar of the standard deviation of the Y coordinate, and the horizontal line segment is the error bar of the standard deviation of the X coordinate.

[0115] The threshold line 710 is used to compare Figure 7 The coordinates of the points 701 and 704 shown are respectively used to distinguish between normal walking and abnormal walking. The area 720 to the left of the threshold straight line 710 contains only the point 701 indicating normal walking and is determined as a normal walking area. The area 730 to the right of the threshold straight line 710 contains the points 702 to 705 indicating abnormal walking and is determined as an abnormal walking area.

[0116] For example, if the standard deviation of the error obtained in step S8 for the input time series feature points determined as the walking section in step S5 is within the region 720, the decision unit 24 determines that it is normal walking. On the other hand, if the standard deviation of the error obtained in step S8 for the input time series feature points determined as the walking section in step S5 is within the region 730, the decision unit 24 determines that it is abnormal walking.

[0117] In addition, when using Figure 7In the case of the method, a threshold straight line 710 is pre-stored in the memory 3, and the decision unit 24 reads the threshold straight line 710 from the memory 3 as a reference time series feature point, and determines the change in walking by comparing the threshold straight line 710 with the standard deviation of the error of the input time series feature point.

[0118] (Step S10)

[0119] The output unit 25 outputs the comparison result generated in step S93. The comparison result is output to the terminal device of the target user, for example. Thus, when it is determined that there is a change in walking, the terminal device displays a message indicating that there is a change in walking behavior on the display or outputs the message from the speaker.

[0120] Thus, according to Embodiment 1, the centroid of the skeletal point coordinates P included in the trunk is used as the feature point 401, but since the estimation accuracy of the skeletal point coordinates P is higher than that of the parts of the trunk that move more, such as the fingertips, the skeletal point coordinates P of the trunk can be stably estimated even in an action with movement such as walking. Therefore, according to Embodiment 1, even in a case where the camera angle of the camera 4 installed indoors is not favorable for sensing, the change of walking can be accurately determined for the target user.

[0121] (Implementation method 2)

[0122] In the first embodiment, whether the photographed user is walking straight is not considered. If the photographed user changes the direction while walking, the accuracy of the determination of the change in walking may be reduced. Therefore, in the second embodiment, it is determined whether the photographed user is walking straight.

[0123] In the following description, only the differences from the first embodiment are described.

[0124] Figure 8 This is a block diagram showing an example of the structure of the action recognition device system in the second embodiment of the present disclosure.

[0125] In Embodiment 2, the same reference numerals are given to the same structures as those in Embodiment 1, and the description thereof will be omitted.

[0126] The behavior recognition device 1A further includes a determination unit 26. The determination unit 26 determines whether the target user is walking straight based on the input time series feature points calculated by the calculation unit 23. When the determination unit 26 determines that the target user is walking straight, the decision unit 24 performs processing to determine a change in the walking of the target user.

[0127] Fig. 9 is a flowchart showing an example of processing of the action recognition device 1A according to the second embodiment of the present disclosure. Fig. 9 In Figure 4 The same processes are given the same symbols, and their description is omitted.

[0128] (Step S201)

[0129] The determination unit 26 determines whether the target user is walking straight toward the camera 4. The determination unit 26 calculates a body index for determining whether the front of the target user is facing the direction of the camera 4 for each frame based on the following equation (1).

[0130] Body index = (left shoulder's X coordinate + left waist's X coordinate) - (right shoulder's X coordinate + right waist's X coordinate)... (1)

[0131] When the user walks straight toward the camera 4 and the front of the target user is photographed, the body index is always a positive value. On the other hand, when the target user bends or turns back while walking and the back of the target user is photographed, the body index is a negative value. Therefore, by determining whether the sign of the body index is positive or negative, it is possible to determine whether the front of the target user is facing the direction of the camera 4.

[0132] The determination unit 26 determines whether the ratio of the number of frames in which the body index is positive to the total number of frames in the walking interval is greater than a threshold value. The threshold value is, for example, 0.7. When the ratio of the number of frames in which the body index is positive to the total number of frames in the walking interval is greater than the threshold value, the determination unit 26 determines that the target user is walking straight toward the camera 4. On the other hand, when the ratio of the number of frames in which the body index is positive to the total number of frames in the walking interval is less than the threshold value, the determination unit 26 determines that the target user is not walking straight toward the camera 4.

[0133] When it is determined that the target user is not walking straight toward the camera 4 (NO in step S201 ), the process returns to step S1 .

[0134] On the other hand, when it is determined that the target user is walking straight toward the camera 4 (YES in step S201 ), the process proceeds to step S6 .

[0135] In the second embodiment, the behavior of the target user can be recognized with high accuracy by excluding feature points calculated based on walking in directions other than straight walking that have an adverse effect on recognition of changes in walking motion.

[0136] (Implementation method 3)

[0137] Embodiment 3 corrects feature points 401 according to the distance of the target user from the camera 4. The size of the target user in the image depends on the depth (horizontal distance) of the position of the target user relative to the camera 4. When it is assumed that the walking direction of the target user is roughly toward the camera, the size of the target user in the image is proportional to the inverse of the depth. That is, the closer the target user is to the camera 4, the larger the image appears. For this reason, in Embodiment 3, in order to remove the change in the size of the target user in the image based on the distance from the camera 4, the following depth correction is performed.

[0138] Fig.10 This is a block diagram showing an example of the structure of a behavior recognition device system according to Embodiment 3 of the present disclosure.

[0139] In Embodiment 3, the same reference numerals are given to the same structures as those in Embodiment 1, and the description thereof will be omitted.

[0140] The action recognition device 1B further includes a depth correction unit 27. The depth correction unit 27 calculates a correction parameter in time series, and the correction parameter corresponds to the distance between the target user included in the image acquired by the acquisition unit 21 and the camera that photographs the target user. Here, the correction coefficient has a plurality of correction coefficients corresponding to each frame of the input time series feature point. The correction coefficient has a larger value as the distance between the target user included in the image and the camera 4 increases.

[0141] The determination unit 24 normalizes the time series feature points by multiplying each feature point constituting the time series feature points by a correction coefficient corresponding to each feature point, and determines a change in the walking of the target user by comparing the normalized time series feature points with the normalized reference time series feature points.

[0142] Fig.11 is a flowchart showing an example of processing of the action recognition device 1B according to the third embodiment of the present disclosure. Fig.10 In Figure 4 The same processes are given the same symbols, and their description is omitted.

[0143] (Step S301)

[0144] The depth correction unit 27 calculates the depth corresponding to each feature point constituting the input time series data. Specifically, the depth correction unit 27 obtains a predetermined value as the walking distance of the target user in the space around the camera 4 from the memory 3. The walking distance is the distance between the walking start position where the target user enters the camera 4 and the walking end position where the target user exits the camera 4 in the space around the camera 4. The value of the walking distance can be a value input by the user via the terminal device, or a general walking distance of a hallway (for example, 1.9 m). In addition, assuming that the target user performs uniform linear motion between the walking start position and the walking end position, the depth correction unit 27 calculates the depth of the user from the camera 4 in each frame of the input time series feature point.

[0145] For example, the depth correction unit 27 calculates the walking distance per frame by dividing the walking distance by the total number of frames required for walking, starting from the position of the camera 4. Furthermore, the depth correction unit 27 calculates the depth of each frame by multiplying the walking distance per frame by the number of frames from the camera entrance to the frame.

[0146] (Step S302)

[0147] The depth correction unit 27 calculates a correction coefficient for each frame based on the depth of each frame calculated in step S301 .

[0148] Specifically, the depth correction unit 27 may calculate the correction coefficient using the following equation (2).

[0149] wi=di / W, where W=Σ i＝1 M w i … (2)

[0150] wi is the correction coefficient of the i-th frame, di is the depth of the i-th frame. M is the number of frames included in the input time series feature points, and W is the sum of the correction coefficients wi. That is, the correction coefficients wi are normalized so that the sum is 1.

[0151] (Step S303)

[0152] The processing of step S303 is shown in Fig.12 Flowchart of the process. Fig.12 It is shown Fig.11 Detailed description of the process of step S303. Fig.12 In Figure 6 The same processing is assigned the same processing number, and the description thereof is omitted.

[0153] (Step S1201)

[0154] The determination unit 24 corrects each feature point 401 by multiplying each feature point 401 included in the input time-series feature points corrected in step S7 by the correction coefficient wi calculated in step S302 .

[0155] (Step S1202)

[0156] The determination unit 24 stores the input time-series feature points obtained by correcting each feature point 401 using the correction coefficient wi in step S1201 as reference time-series feature points in the travel database 32 .

[0157] Next, the processing of steps S92 to S93 is performed to determine the change of the user's walking. In addition, since the input time series feature points obtained by correcting each feature point 401 with the correction coefficient wi in step S1202 are stored in the walking database 32 as the reference time series feature points, the average time series feature points calculated in step S92 become the average time series feature points obtained by correcting each feature point 401 with the correction coefficient wi. Therefore, in step S93, the comparison result is generated by comparing the input time series feature points obtained by correcting each feature point 401 with the correction coefficient wi and the average time series feature points obtained by correcting each feature point 401 with the correction coefficient wi.

[0158] Thus, according to Embodiment 3, since the changes in the user's walking can be determined on the basis of eliminating the changes in the size of the object user in the image caused by the depth from the camera 4 which has an adverse effect on the determination of the changes in the walking action, the changes in the user's walking can be determined with high accuracy.

[0159] (Variation)

[0160] The above describes the action recognition device involved in one or more embodiments of the present disclosure based on the implementation method, but the present disclosure is not limited to the above implementation methods. As long as it does not deviate from the main purpose of the present disclosure, the method of applying various deformations that can be thought of by those skilled in the art to the present embodiment and the method of combining the structural elements in different implementation methods can also be included in the scope of one or more embodiments of the present invention.

[0161] (1) Embodiment 2 and Embodiment 3 may be combined. That is, when the determination unit 26 determines that the vehicle is traveling in a straight line, the depth correction unit 27 may calculate the correction coefficient wi.

[0162] (2) The output unit 25 outputs a message indicating that there is a change in walking, but this is just an example, and a message indicating that weakness is observed during walking may be output. In addition, the output unit 25 may output an index indicating a change in walking. As the index, for example, the above-mentioned distance between feature points or a time series feature amount may be used.

[0163] Industrial Applicability

[0164] The action recognition device disclosed in the present invention is useful in recognizing the status of a user in a house.

Claims

1. A motion recognition device for recognizing a user's walking, comprising: An acquisition unit, for acquiring an image; an estimating unit, which estimates the coordinates of the skeleton points of the user based on the image acquired by the acquiring unit; a calculation unit that calculates time series feature points representing feature points of the user's body in time series based on the skeleton point coordinates estimated by the estimation unit; a storage unit storing the time series feature points serving as a reference, namely, reference time series feature points; and The determination unit determines a change in the walking of the user by comparing the input time-series feature points, which are the time-series feature points calculated by the calculation unit, with the reference time-series feature points.

2. The action recognition device according to claim 1, wherein: The time series feature points are normalized so that the time required for the walking becomes a reference value.

3. The action recognition device according to claim 1 or 2, wherein: The action recognition device further includes: a determination unit that determines whether the user is walking straight based on the time series feature points calculated by the calculation unit, When it is determined that the user is walking straight, the determination unit executes a process of determining a change in the walking of the user.

4. The action recognition device according to claim 1 or 2, wherein: The action recognition device further comprises: a depth correction unit that calculates, in time series, a correction coefficient corresponding to the distance between the user included in the image and a camera that photographs the user; The determination unit normalizes the time series feature points by multiplying each feature point constituting the time series feature points by the correction coefficient corresponding to each feature point, and determines the change in the user's walking by comparing the normalized time series feature points with the normalized baseline time series feature points.

5. The action recognition device according to claim 1 or 2, wherein: The calculation unit calculates the polynomial approximation curves of the X coordinates and Y coordinates of the time series feature points based on the calculated image coordinates of the time series feature points, and uses the calculated polynomial approximation curves of the X coordinates and Y coordinates to correct the values ​​of the X coordinates and Y coordinates of each feature point constituting the time series feature points.

6. The action recognition device according to claim 1 or 2, wherein: The reference time series feature points include a plurality of first reference time series feature points. The determination unit calculates an average time-series feature point by averaging the plurality of first reference time-series feature points, and determines a change in the walking of the user by comparing the average time-series feature point with the input time-series feature point.

7. The action recognition device according to claim 1 or 2, wherein: The calculation unit determines the feature points included in a section in which the time-series feature points can be continuously calculated as the input time-series feature points, when the section in which the time-series feature points can be continuously calculated is equal to or longer than a predetermined time.

8. The action recognition device according to claim 1 or 2, wherein: The reference time-series feature points are the input time-series feature points calculated in the past by the calculation unit.

9. The action recognition device according to claim 1 or 2, wherein: The reference time series feature points and the input time series feature points are time series feature points of the same user.

10. The action recognition device according to claim 1 or 2, wherein: The behavior recognition device further includes an output unit that outputs information indicating the change in the walking of the user determined by the determination unit.

11. A method for identifying a movement, comprising: In the action recognition method, Get the image, According to the obtained image, the coordinates of the skeleton points of the user are estimated, Based on the estimated coordinates of the skeleton points, time series feature points representing feature points of the user's body in time series are calculated, By comparing the calculated time-series feature points, namely input time-series feature points, with the time-series feature points, namely reference time-series feature points, which serve as references, changes in the walking of the user are determined.

12. A motion recognition program, which causes a computer to execute a motion recognition method for recognizing a user's walking, wherein the motion recognition program causes the computer to execute the following processing: Get the image, According to the obtained image, the coordinates of the skeleton points of the user are estimated, Based on the estimated coordinates of the skeleton points, time series feature points representing feature points of the user's body in time series are calculated, By comparing the calculated time-series feature points, namely input time-series feature points, with the time-series feature points, namely reference time-series feature points, which serve as references, changes in the walking of the user are determined.

Images