Method and apparatus for recognizing user motion based on deep learning

Abstract

The present invention relates to a deep learning-based method and device for recognizing user movements, wherein user movement data and user location data are collected in real time to generate a dataset consisting of movement data and location data for each period, and the user's movements are classified into one of a plurality of primary movement types by inputting the movement data for each period of the dataset into a deep learning model, and the movements represented by one of a plurality of primary movement types are reclassified into one of a plurality of secondary movement types by inputting the location data for each period of the dataset into a deep learning model, thereby enabling the recognition of the user's current movements by distinguishing between daily movements such as standing, sitting, lying down, and walking, and dangerous movements such as movement transitions and direction changes.

Description

Technology Field

[0001] It is about human motion recognition. Background Technology

[0002] Human motion recognition is mostly achieved using body-worn sensors, such as Inertial Measurement Units (IMUs). Recently, research aimed at improving the accuracy of human motion recognition has been actively underway. To enhance accuracy, research primarily focuses on increasing the number of sensors attached to the human body or utilizing the fusion of IMU sensor outputs with video data. However, increasing the number of sensors attached to the human body exacerbates user inconvenience, and video data raises privacy concerns.

[0003] For example, Korean Patent Publication No. 10-2013-0110565, "Device and Method for Real-time User Behavior Recognition," presents a technology to improve the accuracy of human motion recognition. Movement transitions, such as sitting down and standing up, or changes in direction are movements that pose a high risk of falls for the elderly or patients. However, most existing technologies in the field of human motion recognition focus on improving the accuracy of recognition for dynamic movements such as walking, and static movements such as sitting, lying down, and standing; technology capable of distinguishing and recognizing movement transitions or changes in direction has not yet been presented. The problem to be solved

[0004] The invention provides a user motion recognition method and device capable of distinguishing between everyday movements such as standing, sitting, lying down, and walking, and dangerous movements such as motion switching and direction switching. The invention is not limited to the technical problems described above, and other technical problems may be derived from the following description. means of solving the problem

[0005] A user motion recognition method according to one aspect of the present invention comprises: a step of collecting motion data representing the user’s motion and location data representing the user’s location in real time using at least one sensor attached to the user; a step of generating a dataset consisting of motion data for each period and location data for each period by periodically sampling the motion data and location data collected in real time; a step of classifying the user’s motion into one of a plurality of primary motion types by inputting the motion data for each period of the generated dataset into a deep learning model; a step of reclassifying the motion represented by one of the primary motion types into one of a plurality of secondary motion types by inputting the location data for each period of the generated dataset into the deep learning model; and a step of outputting one of the primary motion type and one of the secondary motion types as a result of user motion recognition.

[0006] The above user action recognition method further includes a step of determining whether the action represented by any one primary action type is accompanied by a change in the user's location, and the reclassification step may input location data for each period of the generated dataset into the deep learning model when it is determined that the action represented by any one primary action type is accompanied by a change in the user's location.

[0007] The above outputting step may output either of the above primary action type and the above secondary action type as the result of recognizing the user's action, depending on the result of determining whether the action represented by any one primary action type is accompanied by a change in the user's position.

[0008] The above plurality of primary motion types include standing motion type, sitting motion type, lying motion type, and dynamic motion type, and the determining step may determine that when any one of the primary motion types is the walking motion type, the motion represented by any one of the primary motion types is accompanied by a change in the user's position.

[0009] The plurality of secondary operation types may include an operation switching type representing a dynamic operation in which the operation type is being switched between the plurality of primary operation types, and a direction switching type representing a dynamic operation in which the direction of the operation corresponding to any one of the primary operation types is being switched.

[0010] The deep learning model is a combined model of a Convolutional Neural Network (CNN) model and a Long Short Term Memory (LSTM) model, and the classification step classifies the user's action into one of the plurality of primary action types based on the data output from the LSTM model by inputting the action data for each period of the generated dataset into the CNN model, and as the action data for each period of the generated dataset is input into the CNN model, the data output from the CNN model can be input into the LSTM model.

[0011] The above reclassification step can reclassify the action represented by any one primary action type into one of a plurality of secondary action types based on the data output from the LSTM model by inputting position data for each period of the generated dataset into the CNN model.

[0012] The above user action recognition method further includes a step of determining whether the action represented by any one primary action type is accompanied by a change in the user's location, and the reclassification step may input location data for each period of the generated dataset into the CNN model when it is determined that the action represented by any one primary action type is accompanied by a change in the user's location.

[0013] According to another aspect of the present invention, a computer-readable recording medium is provided that records a program for executing the user action recognition method on a computer.

[0014] A user motion recognition device according to another aspect of the present invention comprises: a data collection unit that collects motion data representing the user's motion and location data representing the user's location in real time using at least one sensor attached to the user; a dataset generation unit that generates a dataset consisting of motion data for each period and location data for each period by periodically sampling the motion data and location data collected in real time; a first motion classification unit that classifies the user's motion into one of a plurality of primary motion types by inputting the motion data for each period of the generated dataset into a deep learning model; a second motion classification unit that reclassifies the motion represented by one of the primary motion types into one of a plurality of secondary motion types by inputting the location data for each period of the generated dataset into the deep learning model; and a user interface that outputs one of the primary motion type and one of the secondary motion types as a result of the user's motion recognition. Effects of the invention

[0015] By collecting user motion data and user location data in real time to generate a dataset consisting of motion data and location data for each period, and by inputting the motion data for each period of the dataset into a deep learning model to classify the user's motion into one of a plurality of primary motion types, and by inputting the location data for each period of the dataset into a deep learning model to reclassify the motion represented by one of a plurality of primary motion types into one of a plurality of secondary motion types, it is possible to distinguish and recognize daily motions such as standing, sitting, lying down, and walking, and dangerous motions such as motion transition and direction change regarding the user's current motion.

[0016] In this way, as it is possible to distinguish and recognize daily movements such as standing, sitting, lying down, and walking, as well as dangerous movements such as movement transitions and direction changes, regarding the user's current movements, it is possible to conduct real-time safety assessments of movements of elderly patients with age-related diseases or elderly people living alone who are receiving treatment in nursing hospitals or general hospitals. For example, the risk of falls regarding the movements of elderly patients with age-related diseases or elderly people living alone can be monitored in real time. Furthermore, information regarding areas where dangerous movements such as movement transitions and direction changes frequently occur can be utilized to improve the living environment of elderly patients with age-related diseases or elderly people living alone. The results of the user's movements according to the present invention can be utilized for predicting risks such as falls without the need for a separate TUG (Timed Up and Go) test.

[0017] The effects are not limited to those described above, and other effects may be derived from the following description. Brief explanation of the drawing

[0018] FIG. 1 is a configuration diagram of a user motion recognition device according to one embodiment of the present invention. FIG. 2 is a flowchart of a user action recognition method according to an embodiment of the present invention. Figure 3 is an example diagram of the classification of action types for user actions. Figure 4 is a conceptual diagram of user action classification in steps 24 and 26 shown in Figure 2. Specific details for implementing the invention

[0019] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The embodiments of the present invention described below relate to a user motion recognition method and apparatus capable of distinguishing and recognizing daily movements such as standing, sitting, lying down, and walking, and dangerous movements such as motion switching and direction switching. Hereinafter, such a method and apparatus will be briefly referred to as the "user motion recognition method" and the "user motion recognition apparatus."

[0020] FIG. 1 is a configuration diagram of a user motion recognition device according to an embodiment of the present invention. Referring to FIG. 1, the user motion recognition device according to the present embodiment is composed of a data collection unit (10), a dataset generation unit (20), a data division unit (30), a first motion classification unit (40), a second motion classification unit (50), a control unit (60), and a user interface (70). The data collection unit (10), the dataset generation unit (20), the data division unit (30), the first motion classification unit (40), the second motion classification unit (50), and the control unit (60) may be implemented by a combination of at least one computer processor, at least one storage, and a computer program, or may be implemented by at least one Field Programmable Gate Array (FPGA). The user interface (70) receives commands or information from the user or displays images and messages to the user. The user interface (70) may be implemented as a display panel, a touchscreen, etc.

[0021] FIG. 2 is a flowchart of a user motion recognition method according to an embodiment of the present invention. Referring to FIG. 2, the user motion recognition method according to the present embodiment consists of the following steps performed by the user motion recognition device shown in FIG. 1. Hereinafter, the user motion recognition device shown in FIG. 1 will be described in detail with reference to FIG. 1 and 2. The control unit (60) plays the role of controlling the operation of each of the data collection unit (10), dataset generation unit (20), data division unit (30), first motion classification unit (40), and second motion classification unit (50) so that the user motion recognition device shown in FIG. 1 can perform the steps shown in FIG. 2.

[0022] In step 21, the data collection unit (10) collects motion data representing the user's movements and location data representing the user's location in real time using at least one sensor attached to the user. The motion data representing the user's movements may consist of acc(x, y, z), gyro(x, y, z), grav(x, y, z), and quat(x, y, z, w) of at least one point on the user's body. acc(x, y, z) represents the acceleration value in the x-axis, y-axis, and z-axis, i.e., the 3-axis direction; gyro(x, y, z) represents the angular velocity value in the 3-axis direction; grav(x, y, z) represents the gravitational acceleration value in the 3-axis direction; and quat(x, y, z, w) represents four quaternion values.

[0023] For example, the data collection unit (10) can collect acc(x, y, z), gyro(x, y, z), grav(x, y, z), and quat(x, y, z, w) of a point using a 3-axis IMU sensor attached to a point on the user's body. Although acc(x, y, z), gyro(x, y, z), grav(x, y, z), and quat(x, y, z, w) from multiple points can be collected using multiple 3-axis IMU sensors attached to multiple points on the user's body, quat(x, y, z, w) is collected in addition to acc(x, y, z), gyro(x, y, z), and grav(x, y, z), so the user's movement can be accurately recognized using only acc(x, y, z), gyro(x, y, z), grav(x, y, z), and quat(x, y, z, w) from a single point on the user's body.

[0024] The motion acceleration value in the x-axis direction at any point on the user's body is the value obtained by subtracting the gravitational acceleration value in the x-axis direction from the acceleration value in the x-axis direction at that point, the motion acceleration value in the y-axis direction is the value obtained by subtracting the gravitational acceleration value in the y-axis direction from the acceleration value in the y-axis direction at that point, and the motion acceleration value in the z-axis direction is the value obtained by subtracting the gravitational acceleration value in the z-axis direction from the acceleration value in the z-axis direction at that point. Since the gravitational acceleration value always exists even in a static state, static motion and dynamic motion can be distinguished using the acceleration value and the gravitational acceleration value.

[0025] Location data representing the location of a user may consist of two-dimensional coordinate values ​​user(x, y) of at least one point on the user's body. For example, a data collection unit (10) may collect two-dimensional coordinate values ​​user(x, y) of at least one point on the user's body using Ultra-Wideband (UWB) signals transmitted from at least three beacons installed in a space where the user resides, such as a hospital or a house. At least one receiver is attached to at least one point on the user's body. A receiver attached to a point on the user's body receives UWB signals transmitted from at least three beacons. The data collection unit (10) may measure the two-dimensional coordinate values ​​user(x, y) of that point by measuring the distance between each beacon and a point on the user's body using the arrival time of the UWB signals received by the receiver attached to a point on the user's body.

[0026] In step 22, the dataset generation unit (20) generates a dataset consisting of motion data and location data for each period by periodically sampling motion data and location data collected in real time by the data collection unit (10) in step 21. Motion data and location data for each period refer to motion data and location data extracted from motion data and location data collected in real time by the data collection unit (10) whenever motion data and location data collected in real time by the data collection unit (10) are periodically sampled. The dataset generation unit (20) can periodically sample motion data and location data collected in real time by the data collection unit (10) in step 21 by extracting motion data and location data corresponding to a predetermined number of samples at a predetermined period from motion data and location data collected in real time by the data collection unit (10).

[0027] For example, the dataset generation unit (20) can periodically sample the motion data and location data collected in real time by the data collection unit (10) in step 21 by extracting motion data and location data corresponding to 1,000 samples at a 1-minute interval from the motion data and location data collected in real time by the data collection unit (10). In this example, the motion data for each period of the dataset generated by the dataset generation unit (20) consists of 1,000 sets of acc(x, y, z), gyro(x, y, z), grav(x, y, z), and quat(x, y, z, w), and the location data for each period consists of 1,000 sets of 2D coordinate values ​​user(x, y).

[0028] In step 23, the data partitioning unit (30) divides the motion data for each period and the position data for each period of the dataset generated by the dataset generation unit (20) in step 22 into each window using a sliding window technique. The data partitioning unit (30) divides the motion data for each period and the position data for each period of the dataset generated by the dataset generation unit (20) into each window according to the window size and sliding interval pre-set by the user. For example, the window size can be set to a data length corresponding to 100 samples, and the sliding interval can be set to a window movement interval corresponding to 50 samples. In this example, the motion data divided into each window consists of 100 sets of acc(x, y, z), gyro(x, y, z), grav(x, y, z), and quat(x, y, z, w), and the position data divided into each window consists of 100 sets of 2D coordinate values ​​user(x, y).

[0029] In step 24, the first motion classification unit (40) classifies the user's motion into one of a plurality of primary motion types for each sampling period in step 22 by inputting motion data for each period of the dataset generated by the dataset generation unit (20) in step 22 into the deep learning model (100). FIG. 3 is an example diagram of motion type classification for user motion. For example, the plurality of primary motion types may consist of a standing motion type, a sitting motion type, a lying motion type, and a walking motion type. The plurality of primary motion types may include other motion types in addition to the motion types listed above. Here, the standing motion type represents a static motion in which the user is standing, the sitting motion type represents a static motion in which the user is sitting, the lying motion type represents a static motion in which the user is lying down, and the walking motion type represents a dynamic motion in which the user is walking.

[0030] As illustrated in FIG. 1, the deep learning model (100) may be built in the user motion recognition device according to the present embodiment or may be built on an external server. The deep learning model (100) of the present embodiment may be implemented as a combined model of a CNN (Convolutional Neural Network) model and an LSTM (Long Short Term Memory) model. The first motion classification unit (40) classifies the user's motion into one of a plurality of primary motion types based on the data output from the LSTM model by inputting motion data for each period of the dataset generated by the dataset generation unit (20) in step 22 into the CNN model. In the combined model of the CNN model and the LSTM model, as motion data for each period of the dataset generated by the dataset generation unit (20) is input into the CNN model, the data output from the CNN model is input into the LSTM model.

[0031] The first motion classification unit (40) inputs motion data for each period of the dataset generated by the dataset generation unit (20) into the CNN model in a manner that sequentially inputs motion data divided by each window by the data division unit (30) in step 23 into the CNN model. When motion data divided by each window is sequentially input into the CNN model, the CNN model extracts feature vectors for each window from the data divided by each window. The CNN model sequentially outputs the feature vectors for each window extracted in this manner, and the feature vectors for each window sequentially output from the CNN model are input into the LSTM model. In this embodiment, the data output from the CNN model is the feature vectors for each window that are sequentially output in this manner.

[0032] When feature vectors for each window sequentially output from a CNN model are input into an LSTM model, the LSTM model infers the temporal change characteristics between the sequentially input feature vectors for each window and generates probability values ​​for each of the multiple primary motion types based on the temporal change characteristics between the feature vectors for each window. In this embodiment, the data output from the LSTM model is the probability value for each of the multiple primary motion types. The first motion classification unit (40) classifies the user's motion into one of the primary motion types having the highest probability value among the data output from the LSTM model, that is, the probability values ​​for each of the multiple primary motion types. For example, if one of the primary motion types having the highest probability value among the probability values ​​for each of the multiple primary motion types is a walking type, the first motion classification unit (40) classifies the user's motion as a walking type.

[0033] In step 25, the control unit (60) determines whether the action represented by any one primary action type corresponding to the classification result of the first action classification unit (40) in step 24 is accompanied by a change in the user's position. If, in step 25, it is determined that the action represented by any one primary action type corresponding to the classification result of the first action classification unit (40) in step 24 is accompanied by a change in the user's position, the process proceeds to step 26. If, in step 25, it is determined that the action represented by any one primary action type corresponding to the classification result of the first action classification unit (40) in step 24 is not accompanied by a change in the user's position, the process proceeds to step 27. In the present embodiment, the control unit (60) determines that the action represented by any one primary action type is accompanied by a change in the user's position when the primary action type corresponding to the classification result of the first action classification unit (40) in step 24 is a walking action type. Among the multiple primary motion types, the standing motion type, sitting motion type, and lying motion type correspond to static motions and do not involve a change in the user's position.

[0034] In step 26, the second motion classification unit (50) inputs position data for each period of the dataset generated by the dataset generation unit (20) in step 22 into the deep learning model (100), thereby reclassifying the motion represented by any one primary motion type corresponding to the classification result of the first motion classification unit (40) in step 24 into any one of a plurality of secondary motion types. According to the present embodiment, when any one primary motion type corresponding to the classification result of the first motion classification unit (40) is a walking motion type, the second motion classification unit (50) inputs position data for each period of the dataset generated by the dataset generation unit (20) in step 22 into the deep learning model (100), thereby reclassifying the motion represented by the walking motion type into any one of a plurality of secondary motion types.

[0035] The second motion classification unit (50) inputs position data for each period of the dataset generated by the dataset generation unit (20) in step 22 into the CNN model, and based on the data output from the LSTM model, reclassifies the motion represented by any one primary motion type corresponding to the classification result of the first motion classification unit (40) in step 24 into one of a plurality of secondary motion types. In the combined model of the CNN model and the LSTM model, as position data for each period of the dataset generated by the dataset generation unit (20) is input into the CNN model, the data output from the CNN model is input into the LSTM model.

[0036] The second motion classification unit (50) inputs position data for each period of the dataset generated by the dataset generation unit (20) into the CNN model in a manner that sequentially inputs position data divided by each window by the data division unit (30) in step 23 into the CNN model. When position data divided by each window is sequentially input into the CNN model, the CNN model extracts feature vectors for each window from the data divided by each window. The CNN model sequentially outputs the feature vectors for each window extracted in this manner, and the feature vectors for each window sequentially output from the CNN model are input into the LSTM model. In this embodiment, the data output from the CNN model is the feature vectors for each window that are sequentially output in this manner.

[0037] When feature vectors for each window sequentially output from a CNN model are input into an LSTM model, the LSTM model infers the temporal change characteristics between the sequentially input feature vectors for each window and generates probability values ​​for each of the multiple secondary action types based on the temporal change characteristics between the feature vectors for each window. In this embodiment, the data output from the LSTM model is the probability value for each of the multiple secondary action types. The first action classification unit (40) classifies the user's action into one of the secondary action types having the highest probability value among the data output from the LSTM model, that is, the probability values ​​for each of the multiple secondary action types. For example, if one of the secondary action types having the highest probability value among the probability values ​​for each of the multiple secondary action types is an action switching type, the first action classification unit (40) classifies the user's action into an action switching type.

[0038] Multiple secondary motion types may consist of motion maintenance types, motion switching types, and direction switching types. Here, the motion maintenance type represents a motion in which a motion corresponding to one primary motion type corresponding to the classification result of the first motion classification unit (40) is maintained, and an example of the motion maintenance type is a motion of walking continuously in one direction. The motion switching type represents a dynamic motion in which the motion type is switching between multiple primary motion types, and examples of the motion switching type include a motion of walking and then standing, or a motion of sitting from a standing position. In this embodiment, all dynamic motions, such as a motion of sitting from a standing position, are classified as walking motion types in step 23. The direction switching type represents a dynamic motion in which the direction of a motion corresponding to one primary motion type corresponding to the classification result of the first motion classification unit (40) is switching, and examples of the direction switching type include a motion of walking in one direction and then turning around to walk in the opposite direction, a motion of walking in one direction and then walking to the left, or a motion of walking in one direction and then walking to the right.

[0039] The second motion classification unit (50) inputs position data for each period of the dataset generated by the dataset generation unit (20) in step 22 into the CNN model, and based on the data output from the LSTM model, reclassifies the motion represented by any one primary motion type corresponding to the classification result of the first motion classification unit (40) in step 24 into one of a plurality of secondary motion types. In the combined model of the CNN model and the LSTM model, as position data for each period of the dataset generated by the dataset generation unit (20) is input into the CNN model, the data output from the CNN model is input into the LSTM model.

[0040] The second motion classification unit (50) inputs position data for each period of the dataset generated by the dataset generation unit (20) into the CNN model in a manner that sequentially inputs position data divided by each window by the data division unit (30) in step 23 into the CNN model. When position data divided by each window is sequentially input into the CNN model, the CNN model extracts feature vectors for each window from the data divided by each window. The CNN model sequentially outputs the feature vectors for each window extracted in this manner, and the feature vectors for each window sequentially output from the CNN model are input into the LSTM model. In this embodiment, the data output from the CNN model is the feature vectors for each window that are sequentially output in this manner.

[0041] When feature vectors for each window sequentially output from a CNN model are input into an LSTM model, the LSTM model infers the temporal change characteristics between the sequentially input feature vectors for each window and generates probability values ​​for each of the multiple secondary action types based on the temporal change characteristics between the feature vectors for each window. In this embodiment, the data output from the LSTM model is the probability value for each of the multiple secondary action types. The first action classification unit (40) classifies the user's action into one of the secondary action types having the highest probability value among the data output from the LSTM model, that is, the probability values ​​for each of the multiple secondary action types. For example, if one of the secondary action types having the highest probability value among the probability values ​​for each of the multiple secondary action types is an action switching type, the first action classification unit (40) classifies the user's action into an action switching type.

[0042] Figure 4 is a conceptual diagram of user action classification in steps 24 and 26 illustrated in Figure 2. As illustrated in Figure 4, when action data for each period of the dataset is input into the CNN model, probability values ​​for each of the multiple primary action types are output from the LSTM model. Only when the primary action type with the highest probability value among the multiple primary action types is the walking type is position data for each period of the dataset input into the CNN model, and probability values ​​for each of the multiple secondary action types are output from the LSTM model. In Figure 4, solid arrows indicate the data processing path based on the input of action data for each period, and dotted arrows indicate the data processing path based on the input of position data for each period. Since position data for each period is used only when the primary action type is the walking type, daily actions such as standing, sitting, lying down, and walking, and dangerous actions such as action transitions and direction changes can be rapidly distinguished and recognized in real time.

[0043] The deep learning model (100) used by the first motion classification unit (40) and the second motion classification unit (50) is trained in the following manner. A large number of training datasets are prepared in advance before the execution of steps 21 to 27 in the same manner as described in steps 21 and 22. Labeling is performed by the user for each training dataset. For example, if a training dataset represents a standing motion type, a label corresponding to the standing motion type is assigned to that training dataset; if a training dataset represents a walking motion type, a label corresponding to the walking motion type is assigned to that training dataset; and if a training dataset represents a motion transition type, a label corresponding to the motion transition type is assigned to that training dataset.

[0044] In the same manner as described above, the training of the deep learning model (100) is performed by updating various training parameters, such as the weights of the CNN model and the LSTM model, so that the label assigned to each training dataset is output from the LSTM model while each training dataset is input into the CNN model. For example, while a training dataset representing a walking motion type is input into the CNN model, various training parameters, such as the weights of the CNN model and the LSTM model, are updated so that the label corresponding to the walking motion type is output from the LSTM model. When the training of the deep learning model (100) is completed for all of the large training datasets, the training of the deep learning model (100) is completed.

[0045] In step 27, the user interface (70), under the control of the control unit (60), outputs either one of the primary motion type corresponding to the classification result of the first motion classification unit (40) in step 24 and either one of the secondary motion type corresponding to the classification result of the second motion classification unit (50) in step 26 as the result of the user's motion recognition. The user interface (70) can output either one of the primary motion type corresponding to the classification result of the first motion classification unit (40) and either one of the secondary motion type corresponding to the classification result of the second motion classification unit (50) in various forms such as text, image, and voice. When the control unit (60) determines that any one primary motion type corresponding to the classification result of the first motion classification unit (40) in step 25 does not accompany a change in the user's position, the control unit (60) controls the user interface (70) so that the user interface (70) outputs any one primary motion type corresponding to the classification result of the first motion classification unit (40) in step 24 as the user's motion recognition result.

[0046] When the control unit (60) determines that the action represented by any one primary action type corresponding to the classification result of the first action classification unit (40) in step 25 is accompanied by a change in the user's position, the control unit (60) controls the user interface (70) to output any one secondary action type corresponding to the classification result of the second action classification unit (50) in step 26 as the user's action recognition result. In this way, the user interface (70) outputs either one of the primary action type corresponding to the classification result of the first action classification unit (40) in step 24 and any one secondary action type corresponding to the classification result of the second action classification unit (50) in step 26 as the user's action recognition result, depending on the determination result of whether the action represented by any one primary action type corresponding to the classification result of the first action classification unit (40) in step 24 is accompanied by a change in the user's position.

[0047] As steps 21 to 27 described above are repeatedly performed, user motion data and location data are collected in real time by the data collection unit (10), and then a new dataset is generated by periodically sampling the user motion data and location data by the dataset generation unit (20). Whenever a new dataset is generated, user motion recognition based on that dataset is performed in real time. According to one embodiment of the present invention described above, regarding the user's current motion, daily motions such as standing, sitting, lying down, and walking, and dangerous motions such as motion switching and direction switching can be distinguished and recognized. There is a high risk of accidents such as falls during motion switching, such as walking and then standing or sitting from a standing position, or during direction switching, such as walking in one direction and then turning around to walk in the opposite direction.

[0048] In this way, as it is possible to distinguish and recognize daily movements such as standing, sitting, lying down, and walking, as well as dangerous movements such as movement transitions and direction changes, regarding the user's current movements, it is possible to conduct real-time safety assessments of movements of elderly patients with age-related diseases or elderly people living alone who are receiving treatment in nursing hospitals or general hospitals. For example, the risk of falls regarding the movements of elderly patients with age-related diseases or elderly people living alone can be monitored in real time. Furthermore, information regarding areas where dangerous movements such as movement transitions and direction changes frequently occur can be utilized to improve the living environment of elderly patients with age-related diseases or elderly people living alone. The results of the user's movements according to the present invention can be utilized for predicting risks such as falls without the need for a separate TUG (Timed Up and Go) test.

[0049] Meanwhile, a user action recognition method according to one embodiment of the present invention as described above can be written as a program executable on a computer processor and can be implemented in a computer that executes the program by recording it on a computer-readable recording medium. The computer includes all types of computers capable of executing the program, such as desktop computers, notebook computers, smartphones, and embedded type computers. A computer-readable recording medium includes storage media such as RAM, ROM, SSD (Solid State Drive), magnetic storage media (e.g., floppy disk, hard disk, etc.), and optical reading media (e.g., CD-ROM, DVD, etc.).

[0050] The present invention has been described above with reference to its preferred embodiments. Those skilled in the art will understand that the present invention may be embodied in modified forms without departing from the essential characteristics of the invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the invention is defined by the claims, not by the foregoing description, and all variations within the scope of the claims should be interpreted as being included in the invention. Explanation of the symbols

[0051] 10 ... Data Collection Department 20 ... dataset creation section 30 ... data partitioning section 40 ... 1st Action Classification Unit 50 ... 2nd Action Classification Unit 60 ... control unit 70 ... user interface 100 ... deep learning model

Claims

Claim 1 A user motion recognition method characterized by comprising: a step of collecting motion data representing the user's motion and location data representing the user's location in real time using at least one sensor attached to the user; a step of generating a dataset consisting of motion data for each period and location data for each period by periodically sampling the motion data and location data collected in real time; a step of classifying the user's motion into one of a plurality of primary motion types by inputting the motion data for each period of the generated dataset into a deep learning model; a step of reclassifying the motion represented by one of the primary motion types into one of a plurality of secondary motion types by inputting the location data for each period of the generated dataset into the deep learning model; and a step of outputting one of the primary motion type and one of the secondary motion types as a result of user motion recognition. Claim 2 A user action recognition method according to claim 1, further comprising a step of determining whether the action represented by any one primary action type is accompanied by a change in the user's location, wherein the reclassification step is characterized by inputting location data for each period of the generated dataset into the deep learning model when it is determined that the action represented by any one primary action type is accompanied by a change in the user's location. Claim 3 A user motion recognition method according to claim 2, wherein the outputting step outputs either one of the primary motion type and one of the secondary motion type as a result of user motion recognition, depending on the result of determining whether the motion represented by any one primary motion type is accompanied by a change in the user's position. Claim 4 A user motion recognition method according to claim 2, wherein the plurality of primary motion types include a standing motion type, a sitting motion type, a lying motion type, and a dynamic motion type, and the determining step is characterized by determining that when any one of the primary motion types is the walking motion type, the motion represented by any one of the primary motion types is accompanied by a change in the user's position. Claim 5 A user motion recognition method according to claim 4, wherein the plurality of secondary motion types include a motion switching type representing a dynamic motion in which the motion type is in a state of switching between the plurality of primary motion types, and a direction switching type representing a dynamic motion in which the direction of the motion corresponding to any one of the primary motion types is in a state of switching. Claim 6 A user action recognition method according to claim 1, wherein the deep learning model is a combined model of a CNN (Convolutional Neural Network) model and an LSTM (Long Short Term Memory) model, and the classification step classifies the user's action into one of the plurality of primary action types based on the data output from the LSTM model by inputting the action data for each period of the generated dataset into the CNN model, and as the action data for each period of the generated dataset is input into the CNN model, the data output from the CNN model is input into the LSTM model. Claim 7 A user action recognition method according to claim 6, wherein the reclassification step is characterized by reclassifying the action represented by any one primary action type into one of a plurality of secondary action types based on data output from the LSTM model by inputting position data for each period of the generated dataset into the CNN model. Claim 8 A user action recognition method according to claim 7, further comprising a step of determining whether the action represented by any one primary action type is accompanied by a change in the user's location, and wherein the reclassification step is characterized by inputting location data for each period of the generated dataset into the CNN model when it is determined that the action represented by any one primary action type is accompanied by a change in the user's location. Claim 9 A computer-readable recording medium storing a program for executing the method of claim 1 on a computer. Claim 10 A user motion recognition device characterized by comprising: a data collection unit that collects motion data representing the user's motion and location data representing the user's location in real time using at least one sensor attached to the user; a dataset generation unit that generates a dataset consisting of motion data for each period and location data for each period by periodically sampling the motion data and location data collected in real time; a first motion classification unit that classifies the user's motion into one of a plurality of primary motion types by inputting the motion data for each period of the generated dataset into a deep learning model; a second motion classification unit that reclassifies the motion represented by one of the primary motion types into one of a plurality of secondary motion types by inputting the location data for each period of the generated dataset into the deep learning model; and a user interface that outputs one of the primary motion type and one of the secondary motion types as a result of recognizing the user's motion.

Images