Method and apparatus for monitoring user behavior

Abstract

The present invention relates to a method and device for monitoring user behavior, wherein user motion data and user location data are collected in real time to generate a dataset consisting of motion data and location data for each period, the space where the user is currently located is identified based on the location data for each period of the dataset, the user's motion is classified into one of a plurality of motion types based on the motion data for each period of the dataset, and by monitoring whether the space motion combination for each period consisting of the identified space information and one of the motion types satisfies at least one of a plurality of preset abnormal behavior conditions, it is possible to monitor whether the user is engaging in abnormal behavior by considering the characteristics of the space where the user is currently located.

Description

Technology Field

[0001] This relates to a method and device for monitoring user behavior. Background Technology

[0002] Recently, as the aging society accelerates, particularly in developed countries, and infectious diseases such as COVID-19 spread globally, the number of patients requiring isolation in specific areas—such as psychiatric hospitals or nursing homes—is increasing, as is the number of people whose entry into spaces frequented by the general public must be prevented. Consequently, active research and development is being conducted on technologies to monitor the behavior of individuals, including dementia patients, those with geriatric diseases, and the elderly living alone. Conventional behavioral monitoring technology has been limited to monitoring entry into or exit from specific areas based on the user's location.

[0003] For example, Korean Registered Patent Publication No. 10-2250424, "Monitoring Device for Self-Quarantined Persons and Method Using the Same," presents a technology that automatically reports to medical authorities when a self-quarantined person leaves their designated residence. However, this prior art determines whether the self-quarantined person has left their designated residence by using an app installed on the self-quarantined person's mobile phone and utilizing information on Bluetooth and Wi-Fi connection strength between the mobile phone and a GPS origin, or distance information between the GPS origin and the mobile phone. The problem to be solved

[0004] The purpose is to provide a user behavior monitoring method and device capable of monitoring in real time whether a user is engaging in abnormal behavior by considering the characteristics of the space where the user is currently located. The invention is not limited to the technical challenges described above, and other technical challenges may be derived from the following description. means of solving the problem

[0005] A user behavior monitoring method according to one aspect of the present invention comprises: a step of collecting motion data representing the user's actions and location data representing the user's location in real time using at least one sensor attached to the user; a step of generating a dataset consisting of motion data for each period and location data for each period by periodically sampling the motion data and location data collected in real time; a step of identifying the space where the user is currently located based on the location data for each period of the generated dataset; a step of classifying the user's actions into one of a plurality of motion types based on the motion data for each period of the generated dataset; a step of generating a spatial motion combination for each period consisting of the information of the identified space and the one of the motion types; and a step of monitoring whether the generated spatial motion combination for each period satisfies at least one of a plurality of abnormal behavior conditions.

[0006] The above-mentioned identifying step determines one of a plurality of virtual spaces set in a specific area of ​​the real world where the user is currently located based on location data for each period of the generated dataset, and identifies the space where the user is currently located by obtaining information of the determined virtual space, and the step of generating spatial operation combinations for each period can generate spatial operation combinations for each period consisting of the obtained virtual space information and one of the operation types.

[0007] The monitoring step described above can monitor whether each of the spatial operation combinations for each period satisfies each of the plurality of abnormal behavior conditions by identifying at least one abnormal behavior condition among the plurality of abnormal behavior conditions that includes a virtual space type identical to that of each spatial operation combination for each period, and by confirming whether there exists an abnormal behavior condition among the identified at least one abnormal behavior condition that includes an operation type matching that of each spatial operation combination for each period.

[0008] The above monitoring step can monitor whether each of the multiple abnormal behavior conditions is satisfied by detecting a change in the user's location in a virtual space corresponding to the space where the user is currently located based on the location data of each period of the generated dataset when there is an abnormal behavior condition that includes a motion type matching the motion type of each periodic spatial motion combination, and confirming whether the detected change in the user's location satisfies the condition for a change in location according to the virtual space type of each periodic spatial motion combination.

[0009] The above-described user behavior monitoring method further includes a step of counting the number of times a user is continuously classified into any one of the above-described behavior types whenever a user is classified into any one of the above-described behavior types, and the monitoring step can monitor whether each of the spatial operation combinations for each period satisfies each of the plurality of abnormal behavior conditions by verifying whether at least one of the detected user location change and the counted number satisfies the location change condition according to the virtual space type of each periodic spatial operation combination.

[0010] The above user behavior monitoring method may further include the step of setting a plurality of virtual beacons in a specific area based on a two-dimensional coordinate system of a plurality of actual beacons installed in the specific area; and the step of setting a plurality of virtual spaces for the specific area using at least one of the set plurality of virtual beacons.

[0011] The step of setting the plurality of virtual spaces above can be performed by partitioning each virtual space using at least one virtual beacon among the set plurality of virtual beacons, and by assigning one of the plurality of virtual space types to each partitioned virtual space, thereby setting the plurality of virtual spaces in the specific area.

[0012] The step of setting the plurality of virtual spaces above can define a virtual boundary line, which is a straight-line virtual space with the two virtual beacons as ends, by using two of the plurality of virtual beacons set in the specific area.

[0013] The step of setting the plurality of virtual spaces above may partition a circular virtual space with one of the virtual beacons as the center point by using one of the virtual beacons among the plurality of virtual beacons set in the specific area.

[0014] The step of setting the plurality of virtual spaces above may partition a polygonal virtual space with each of the at least three virtual beacons as corners using at least three virtual beacons among the plurality of virtual beacons set in the specific area.

[0015] According to another aspect of the present invention, a computer-readable recording medium is provided that records a program for executing the user behavior monitoring method on a computer.

[0016] A user behavior monitoring device according to another aspect of the present invention comprises: a data collection unit that collects motion data representing the user's actions and location data representing the user's location in real time using at least one sensor attached to the user; a dataset generation unit that generates a dataset consisting of motion data for each period and location data for each period by periodically sampling the motion data and location data collected in real time; a space identification unit that identifies the space where the user is currently located based on the location data for each period of the generated dataset; a motion classification unit that classifies the user's actions into one of a plurality of motion types based on the motion data for each period of the generated dataset; a combination generation unit that generates a spatial motion combination for each period consisting of the information of the identified space and one of the motion types; and a behavior monitoring unit that monitors whether the generated spatial motion combination for each period satisfies at least one of a plurality of abnormal behavior conditions. Effects of the invention

[0017] By collecting user motion data and user location data in real time to generate a dataset consisting of motion data and location data for each period, identifying the space where the user is currently located based on the location data for each period of the dataset, classifying the user's motion into one of a plurality of motion types based on the motion data for each period of the dataset, and monitoring whether the spatial motion combination for each period consisting of the identified space information and one of the motion types satisfies at least one of a plurality of preset abnormal behavior conditions, it is possible to monitor whether the user is exhibiting abnormal behavior by considering the characteristics of the space where the user is currently located.

[0018] For example, it is possible to monitor whether a patient who must be isolated in a specific area, such as a psychiatric hospital or nursing home, is exhibiting abnormal behavior by leaving that area. It is also possible to monitor whether a person vulnerable to a certain infectious disease is exhibiting abnormal behavior by entering a specific area frequented by an unspecified number of people. Furthermore, it is possible to monitor whether individuals at high risk of falls, such as patients with geriatric diseases or elderly people living alone, are exhibiting abnormal behavior by staying for extended periods in specific areas with minimal other access, such as private bathrooms. In this way, based on the characteristics of the space where the user is currently located, the user's routine behavior can be considered abnormal, and the occurrence of an abnormal behavior event can be notified to the user's manager, such as a guardian or attending physician.

[0019] The effects are not limited to those described above, and other effects may be derived from the following description. Brief explanation of the drawing

[0020] FIG. 1 is a configuration diagram of a user behavior monitoring device according to one embodiment of the present invention. FIG. 2 is a flowchart of a user behavior monitoring method according to one embodiment of the present invention. Figure 3 is a detailed flowchart of the 26 steps shown in Figure 2. Figure 4 is an example diagram of the classification of action types for user actions. Figure 5 is a conceptual diagram of user action classification in steps 262 and 264 shown in Figure 3. Figure 6 is an example of a case where the virtual space type of the spatial operation combination for each cycle is a door type. Figure 7 is an example of a case where the virtual space type of the spatial operation combination for each cycle is a lobby type. Figure 8 is an example of a case where the virtual space type of the spatial motion combination for each cycle is a toilet type. Specific details for implementing the invention

[0021] Embodiments of the present invention will be described in detail below with reference to the drawings. The embodiments of the present invention described below relate to a user behavior monitoring method and apparatus capable of monitoring in real time whether a user is engaging in abnormal behavior by considering the characteristics of the space where the user is currently located. Hereinafter, such a method and apparatus will be briefly referred to as the "user behavior monitoring method" and the "user behavior monitoring apparatus." The term "user" appearing below refers to a subject being monitored by the embodiments of the present invention, and may be, for example, a dementia patient, a person with geriatric diseases, or an elderly person living alone. The user behavior monitoring apparatus may primarily be used by the user's manager, such as a guardian or attending physician. Hereinafter, the user's manager, such as a guardian or attending physician, will be briefly referred to as the "manager."

[0022] FIG. 1 is a configuration diagram of a user behavior monitoring device according to an embodiment of the present invention. Referring to FIG. 1, the user behavior monitoring device according to the present embodiment is composed of a virtual beacon setting unit (10), a virtual space setting unit (20), a data collection unit (30), a dataset generation unit (40), a space identification unit (50), an action classification unit (60), a combination generation unit (70), a behavior monitoring unit (80), a counter (90), a control unit (100), and a user interface (110). The virtual beacon setting unit (10), virtual space setting unit (20), data collection unit (30), dataset generation unit (40), space identification unit (50), action classification unit (60), combination generation unit (70), behavior monitoring unit (80), counter (90), and control unit (100) may be implemented by a combination of at least one computer processor, at least one storage, and a computer program, or by at least one FPGA (Field Programmable Gate Array). The user interface (110) receives commands or information from the user's manager or displays images and messages to the user's manager. The user interface (110) may be implemented as a display panel, a touchscreen, etc.

[0023] FIG. 2 is a flowchart of a user behavior monitoring method according to an embodiment of the present invention. Referring to FIG. 2, the user behavior monitoring method according to the present embodiment consists of the following steps performed by the user behavior monitoring device shown in FIG. 1. Hereinafter, the user behavior monitoring device shown in FIG. 1 will be described in detail with reference to FIG. 1 and 2. The control unit (100) plays the role of controlling the operation of each of the virtual beacon setting unit (10), virtual space setting unit (20), data collection unit (30), dataset generation unit (40), space identification unit (50), action classification unit (60), combination generation unit (70), behavior monitoring unit (80), and counter (90) so that the user behavior monitoring device shown in FIG. 1 can perform the steps shown in FIG. 2.

[0024] In step 21, the virtual beacon setting unit (10) sets multiple virtual beacons in a specific area of ​​the real world based on the 2D coordinate system of multiple actual beacons installed in that specific area of ​​the real world, according to the virtual beacon setting information of the administrator entered through the user interface (110). Examples of specific areas of the real world include spaces where users reside, such as hospitals or houses. In the case of a hospital, it consists of various spaces such as entrances, lobbies, examination rooms, inpatient rooms, operating rooms, and restrooms. The boundaries of each of these spaces are difficult to determine by actual beacons due to architectural structures such as walls. Such specific areas of the real world can be represented by a 2D coordinate system consisting of an x-axis and a y-axis, and the location of each of the multiple actual beacons installed in the specific area of ​​the real world is the 2D coordinate value RB n (x n , y n It can be expressed as ). For example, the location of each of the three real beacons installed in a specific area of ​​the real world can be expressed as RB1(x1, y1), RB2(x2, y2), and RB3(x3, y3).

[0025] In this embodiment, the two-dimensional coordinate system of a plurality of actual beacons installed in a specific area of ​​the real world refers to a two-dimensional coordinate system that expresses the location of each of the plurality of actual beacons installed in a specific area of ​​the real world as two-dimensional coordinate values, and the administrator's virtual beacon setting information may be location designation information for each of the plurality of virtual beacons in that two-dimensional coordinate system. The virtual space setting unit (20) [represents] the coordinate values ​​VB of each of the plurality of virtual beacons according to the administrator's virtual beacon setting information, for example, the cursor position of the user interface (110) adjusted by the administrator, in a two-dimensional coordinate system that expresses the location of each of the plurality of actual beacons installed in a specific area of ​​the real world as two-dimensional coordinate values. n (x n , y n By setting ) based on the 2D coordinate system of multiple actual beacons installed in a specific area of ​​the real world, multiple virtual beacons can be set up in that specific area.

[0026] In step 22, the virtual space setting unit (20) sets up multiple virtual spaces for a specific area of ​​the real world according to the virtual space setting information of the administrator input through the user interface (110) using at least one virtual beacon among the multiple virtual beacons set up in a specific area of ​​the real world by the virtual beacon setting unit (10) in step 21. In this embodiment, the virtual space setting information of the administrator may be partition information and type information for each of the multiple virtual spaces using at least one virtual beacon among the multiple virtual beacons set up in a specific area of ​​the real world. In step 21, the virtual space setting unit (20) partitions each virtual space according to the virtual space setting information of the administrator input through the user interface (110) using at least one virtual beacon among the multiple virtual beacons set up in a specific area of ​​the real world by the virtual beacon setting unit (10), and sets up multiple virtual spaces in a specific area of ​​the real world by assigning one of the multiple virtual space types to each virtual space partitioned in this way.

[0027] For example, multiple virtual space types may consist of a door type, a lobby type, a medical examination room type, an inpatient room type, an operating room type, and a restroom type. The virtual space setting unit (20) may use two virtual beacons among multiple virtual beacons set in a specific area of ​​the real world to define a virtual boundary line, which is a straight virtual space with the two virtual beacons at both ends, and may assign a door type among multiple virtual space types to the virtual boundary line thus defined. In this case, the virtual space setting information of the administrator may be demarcation information indicating the selection of two virtual beacons among the multiple virtual beacons and type information indicating the door type among the multiple virtual space types. The virtual space setting unit (20) may define a virtual boundary line using the two-dimensional coordinate values ​​of the two virtual beacons.

[0028] The virtual space setting unit (20) can partition a circular virtual space with one of the virtual beacons as the center point using one of the virtual beacons among the multiple virtual beacons set in a specific area of ​​the real world, and can assign a lobby type among the multiple virtual space types to the circular virtual space partitioned in this way. In this case, the virtual space setting information of the administrator can be partition information indicating the selection and radius value of one of the virtual beacons among the multiple virtual beacons, and type information indicating the lobby type among the multiple virtual space types. The virtual space setting unit (20) can partition a circular virtual space using the 2D coordinate value and radius value of one of the virtual beacons among the multiple virtual beacons.

[0029] The virtual space setting unit (20) can partition a polygonal virtual space using at least three virtual beacons among a plurality of virtual beacons set in a specific area of ​​the real world, with each of the at least three virtual beacons serving as a corner, and can assign a toilet type among a plurality of virtual space types to the polygonal virtual space partitioned in this way. In this case, the virtual space setting information of the administrator can be partition information indicating the selection of at least three virtual beacons among the plurality of virtual beacons, and type information indicating the toilet type among the plurality of virtual space types. The virtual space setting unit (20) can partition a triangle-shaped virtual space using the 2D coordinate values ​​of three virtual beacons.

[0030] In step 23, the data collection unit (30) collects motion data representing the user's movements and location data representing the user's location in real time using at least one sensor attached to the user. The motion data representing the user's movements may consist of acc(x, y, z), gyro(x, y, z), grav(x, y, z), and quat(x, y, z, w) of at least one point on the user's body. acc(x, y, z) represents the acceleration value in the x-axis, y-axis, and z-axis, i.e., the 3-axis direction; gyro(x, y, z) represents the angular velocity value in the 3-axis direction; grav(x, y, z) represents the gravitational acceleration value in the 3-axis direction; and quat(x, y, z, w) represents the 4-quaternion value.

[0031] For example, the data collection unit (30) can collect acc(x, y, z), gyro(x, y, z), grav(x, y, z), and quat(x, y, z, w) of a point using a 3-axis IMU sensor attached to a point on the user's body. Although acc(x, y, z), gyro(x, y, z), grav(x, y, z), and quat(x, y, z, w) from multiple points can be collected using multiple 3-axis IMU sensors attached to multiple points on the user's body, quat(x, y, z, w) is collected in addition to acc(x, y, z), gyro(x, y, z), and grav(x, y, z), so the user's movement can be accurately recognized using only acc(x, y, z), gyro(x, y, z), grav(x, y, z), and quat(x, y, z, w) from a single point on the user's body.

[0032] The motion acceleration value in the x-axis direction at any point on the user's body is the value obtained by subtracting the gravitational acceleration value in the x-axis direction from the acceleration value in the x-axis direction at that point, the motion acceleration value in the y-axis direction is the value obtained by subtracting the gravitational acceleration value in the y-axis direction from the acceleration value in the y-axis direction at that point, and the motion acceleration value in the z-axis direction is the value obtained by subtracting the gravitational acceleration value in the z-axis direction from the acceleration value in the z-axis direction at that point. Since the gravitational acceleration value always exists even in a static state, static motion and dynamic motion can be distinguished using the acceleration value and the gravitational acceleration value.

[0033] Location data representing the location of a user may consist of two-dimensional coordinate values ​​user(x, y) of at least one point on the user's body. For example, the data collection unit (30) may collect two-dimensional coordinate values ​​user(x, y) of at least one point on the user's body using Ultra-Wideband (UWB) signals transmitted from at least three actual beacons installed in a space where the user resides, such as a hospital or a house. At least one receiver is attached to at least one point on the user's body. The receiver attached to any point on the user's body receives UWB signals transmitted from at least three actual beacons. The data collection unit (30) may measure the two-dimensional coordinate values ​​user(x, y) of that point by measuring the distance between that point on the user's body and each beacon using the arrival time of the UWB signals received by the receiver attached to that point on the user's body according to triangulation.

[0034] In step 24, the dataset generation unit (40) generates a dataset consisting of motion data and location data for each period by periodically sampling motion data and location data collected in real time by the data collection unit (30) in step 23. Motion data and location data for each period refer to motion data and location data extracted from motion data and location data collected in real time by the data collection unit (30) whenever motion data and location data collected in real time by the data collection unit (30) are periodically sampled. The dataset generation unit (40) can periodically sample motion data and location data collected in real time by the data collection unit (30) in step 21 by extracting motion data and location data corresponding to a predetermined number of samples at a predetermined period from motion data and location data collected in real time by the data collection unit (30).

[0035] For example, the dataset generation unit (40) can periodically sample the motion data and location data collected in real time by the data collection unit (30) in step 21 by extracting motion data and location data corresponding to 1,000 samples at a 1-minute interval from the motion data and location data collected in real time by the data collection unit (30). In this example, the motion data for each period of the dataset generated by the dataset generation unit (40) consists of 1,000 sets of acc(x, y, z), gyro(x, y, z), grav(x, y, z), and quat(x, y, z, w), and the location data for each period consists of 1,000 sets of user(x, y).

[0036] In step 25, the spatial identification unit (50) identifies the space where the user is currently located for each sampling period in step 24 based on the location data for each period of the dataset generated by the dataset generation unit (40) in step 24. Based on the location data for each period of the dataset generated by the dataset generation unit (40) in step 24, the spatial identification unit (50) determines one of the multiple virtual spaces where the user is currently located among the virtual spaces set in a specific area of ​​the real world by the virtual space setting unit (20) in step 22, and can identify the space where the user is currently located by acquiring information on the virtual space thus determined. The information on the virtual space acquired by the spatial identification unit (50) in step 25 consists of the partition information of the virtual space in a two-dimensional coordinate system representing a specific area of ​​the real world and the virtual space type.

[0037] In step 26, the action classification unit (60) classifies the user's action into one of a plurality of action types based on the action data for each period of the dataset generated by the dataset generation unit (40) in step 24. Referring to FIG. 1, the action classification unit (60) is composed of a data division unit (61), a first action classification unit (62), and a second action classification unit (63). FIG. 3 is a detailed flowchart of step 26 shown in FIG. 2. Referring to FIG. 3, step 26 consists of the following steps performed by the action classification unit (60) shown in FIG. 1. Below, the action classification unit (60) will be described in detail with reference to FIG. 1 and 2.

[0038] In step 261, the data partitioning unit (61) divides the motion data for each period and the position data for each period of the dataset generated by the dataset generation unit (40) into each window using a sliding window technique. The data partitioning unit (61) divides the motion data for each period and the position data for each period of the dataset generated by the dataset generation unit (40) into each window according to the window size and sliding interval pre-set by the user. For example, the window size can be set to a data length corresponding to 100 samples, and the sliding interval can be set to a window movement interval corresponding to 50 samples. In this example, the motion data divided into each window consists of 100 sets of acc(x, y, z), gyro(x, y, z), grav(x, y, z), and quat(x, y, z, w), and the position data divided into each window consists of 100 sets of 2D coordinate values ​​user(x, y).

[0039] In step 262, the first motion classification unit (62) classifies the user's motion into one of a plurality of primary motion types by inputting motion data for each period of the dataset generated by the dataset generation unit (40) in step 24 into the deep learning model (200). FIG. 4 is an example diagram of motion type classification for user motion. For example, the plurality of primary motion types may consist of a standing motion type, a sitting motion type, a lying motion type, and a walking motion type. The plurality of primary motion types may include other motion types in addition to the motion types listed above. Here, the standing motion type represents a static motion in which the user is standing, the sitting motion type represents a static motion in which the user is sitting, the lying motion type represents a static motion in which the user is lying down, and the walking motion type represents a dynamic motion in which the user is walking.

[0040] As illustrated in FIG. 1, the deep learning model (200) may be built in the user motion recognition device according to the present embodiment or may be built on an external server. The deep learning model (200) of the present embodiment may be implemented as a combined model of a CNN (Convolutional Neural Network) model and an LSTM (Long Short Term Memory) model. The first motion classification unit (62) classifies the user's motion into one of a plurality of primary motion types based on the data output from the LSTM model by inputting motion data for each period of the dataset generated by the dataset generation unit (40) in step 24 into the CNN model. In the combined model of the CNN model and the LSTM model, as motion data for each period of the dataset generated by the dataset generation unit (40) is input into the CNN model, the data output from the CNN model is input into the LSTM model.

[0041] The first motion classification unit (62) inputs motion data for each period of the dataset generated by the dataset generation unit (40) into the CNN model in a manner that sequentially inputs motion data divided by each window by the data division unit (61) in step 261 into the CNN model. When motion data divided by each window is sequentially input into the CNN model, the CNN model extracts feature vectors for each window from the data divided by each window. The CNN model sequentially outputs the feature vectors for each window extracted in this manner, and the feature vectors for each window sequentially output from the CNN model are input into the LSTM model. In this embodiment, the data output from the CNN model is the feature vectors for each window that are sequentially output in this manner.

[0042] When feature vectors for each window sequentially output from a CNN model are input into an LSTM model, the LSTM model infers the temporal change characteristics between the sequentially input feature vectors for each window and generates probability values ​​for each of the multiple primary motion types based on the temporal change characteristics between the feature vectors for each window. In this embodiment, the data output from the LSTM model is the probability value for each of the multiple primary motion types. The first motion classification unit (62) classifies the user's motion into one of the primary motion types having the highest probability value among the data output from the LSTM model, that is, the probability values ​​for each of the multiple primary motion types. For example, if one of the primary motion types having the highest probability value among the probability values ​​for each of the multiple primary motion types is a walking type, the first motion classification unit (62) classifies the user's motion as a walking type.

[0043] In step 263, the control unit (100) determines whether the action represented by any one primary action type corresponding to the classification result of the first action classification unit (62) in step 262 is accompanied by a change in the user's location. If, in step 263, it is determined that the action represented by any one primary action type corresponding to the classification result of the first action classification unit (62) in step 262 is accompanied by a change in the user's location, the process proceeds to step 264. If, in step 263, it is determined that the action represented by any one primary action type corresponding to the classification result of the first action classification unit (62) in step 262 is not accompanied by a change in the user's location, the process proceeds to step 265. In the present embodiment, the control unit (100) determines that the action represented by any one primary action type is accompanied by a change in the user's location when the primary action type corresponding to the classification result of the first action classification unit (62) in step 262 is a walking action type. Among the multiple primary motion types, the standing motion type, sitting motion type, and lying motion type correspond to static motions and do not involve a change in the user's position.

[0044] In step 264, the second motion classification unit (63) inputs position data for each period of the dataset generated by the dataset generation unit (40) in step 24 into the deep learning model (200), thereby reclassifying the motion represented by any one primary motion type corresponding to the classification result of the first motion classification unit (62) in step 262 into any one of a plurality of secondary motion types. According to the present embodiment, when any one primary motion type corresponding to the classification result of the first motion classification unit (62) is a walking motion type, the second motion classification unit (63) inputs position data for each period of the dataset generated by the dataset generation unit (40) in step 24 into the deep learning model (200), thereby reclassifying the motion represented by the walking motion type into any one of a plurality of secondary motion types.

[0045] The second motion classification unit (63) inputs position data for each period of the dataset generated by the dataset generation unit (40) in step 24 into the CNN model, and based on the data output from the LSTM model, reclassifies the motion represented by any one primary motion type corresponding to the classification result of the first motion classification unit (62) in step 262 into one of a plurality of secondary motion types. In the combined model of the CNN model and the LSTM model, as the position data for each period of the dataset generated by the dataset generation unit (40) is input into the CNN model, the data output from the CNN model is input into the LSTM model.

[0046] The second motion classification unit (63) inputs position data for each period of the dataset generated by the dataset generation unit (40) into the CNN model in a manner that sequentially inputs position data divided by each window by the data division unit (61) in step 261 into the CNN model. When position data divided by each window is sequentially input into the CNN model, the CNN model extracts feature vectors for each window from the data divided by each window. The CNN model sequentially outputs the feature vectors for each window extracted in this manner, and the feature vectors for each window sequentially output from the CNN model are input into the LSTM model. In this embodiment, the data output from the CNN model is the feature vectors for each window that are sequentially output in this manner.

[0047] When feature vectors for each window sequentially output from a CNN model are input into an LSTM model, the LSTM model infers the temporal change characteristics between the sequentially input feature vectors for each window and generates probability values ​​for each of the multiple secondary action types based on the temporal change characteristics between the feature vectors for each window. In this embodiment, the data output from the LSTM model is the probability value for each of the multiple secondary action types. The first action classification unit (62) classifies the user's action into one of the secondary action types having the highest probability value among the data output from the LSTM model, that is, the probability values ​​for each of the multiple secondary action types. For example, if one of the secondary action types having the highest probability value among the probability values ​​for each of the multiple secondary action types is an action switching type, the first action classification unit (62) classifies the user's action into an action switching type.

[0048] Multiple secondary motion types may consist of motion maintenance types, motion switching types, and direction switching types. Here, the motion maintenance type represents a motion in which a motion corresponding to one primary motion type corresponding to the classification result of the first motion classification unit (62) is maintained, and an example of the motion maintenance type is a motion of walking continuously in one direction. The motion switching type represents a dynamic motion in which the motion type is switching between multiple primary motion types, and examples of the motion switching type include a motion of walking and then standing, or a motion of sitting from a standing position. In this embodiment, all dynamic motions, such as a motion of sitting from a standing position, are classified as walking motion types in step 261. The direction switching type represents a dynamic motion in which the direction of a motion corresponding to one primary motion type corresponding to the classification result of the first motion classification unit (62) is switching, and examples of the direction switching type include a motion of walking in one direction and then turning around to walk in the opposite direction, a motion of walking in one direction and then walking to the left, or a motion of walking in one direction and then walking to the right.

[0049] The second motion classification unit (63) inputs position data for each period of the dataset generated by the dataset generation unit (40) in step 24 into the CNN model, and based on the data output from the LSTM model, reclassifies the motion represented by any one primary motion type corresponding to the classification result of the first motion classification unit (62) in step 262 into one of a plurality of secondary motion types. In the combined model of the CNN model and the LSTM model, as the position data for each period of the dataset generated by the dataset generation unit (40) is input into the CNN model, the data output from the CNN model is input into the LSTM model.

[0050] The second motion classification unit (63) inputs position data for each period of the dataset generated by the dataset generation unit (40) into the CNN model in a manner that sequentially inputs position data divided by each window by the data division unit (61) in step 261 into the CNN model. When position data divided by each window is sequentially input into the CNN model, the CNN model extracts feature vectors for each window from the data divided by each window. The CNN model sequentially outputs the feature vectors for each window extracted in this manner, and the feature vectors for each window sequentially output from the CNN model are input into the LSTM model. In this embodiment, the data output from the CNN model is the feature vectors for each window that are sequentially output in this manner.

[0051] When feature vectors for each window sequentially output from a CNN model are input into an LSTM model, the LSTM model infers the temporal change characteristics between the sequentially input feature vectors for each window and generates probability values ​​for each of the multiple secondary action types based on the temporal change characteristics between the feature vectors for each window. In this embodiment, the data output from the LSTM model is the probability value for each of the multiple secondary action types. The first action classification unit (62) classifies the user's action into one of the secondary action types having the highest probability value among the data output from the LSTM model, that is, the probability values ​​for each of the multiple secondary action types. For example, if one of the secondary action types having the highest probability value among the probability values ​​for each of the multiple secondary action types is an action switching type, the first action classification unit (62) classifies the user's action into an action switching type.

[0052] Figure 5 is a conceptual diagram of user action classification in steps 262 and 264 illustrated in Figure 3. As shown in Figure 5, when action data for each period of the dataset is input into the CNN model, probability values ​​for each of the multiple primary action types are output from the LSTM model. Only when the primary action type with the highest probability value among the multiple primary action types is the walking type is position data for each period of the dataset input into the CNN model, and probability values ​​for each of the multiple secondary action types are output from the LSTM model. In Figure 5, solid arrows indicate the data processing path based on the input of action data for each period, and dotted arrows indicate the data processing path based on the input of position data for each period. Since position data for each period is used only when the primary action type is the walking type, daily actions such as standing, sitting, lying down, and walking, and dangerous actions such as action transitions and direction changes can be rapidly distinguished and recognized in real time.

[0053] The deep learning model (200) used by the first motion classification unit (62) and the second motion classification unit (63) is trained in the following manner. A large number of training datasets are prepared in advance before the execution of steps 23 to 26 in the same manner as described in steps 23 and 24. Labeling is performed by the user for each training dataset. For example, if a training dataset represents a standing motion type, a label corresponding to the standing motion type is assigned to that training dataset; if a training dataset represents a walking motion type, a label corresponding to the walking motion type is assigned to that training dataset; and if a training dataset represents a motion transition type, a label corresponding to the motion transition type is assigned to that training dataset.

[0054] In the same manner as described above, the training of the deep learning model (200) is performed by updating various training parameters, such as the weights of the CNN model and the LSTM model, so that the label assigned to each training dataset is output from the LSTM model while each training dataset is input into the CNN model. For example, while a training dataset representing a walking motion type is input into the CNN model, various training parameters, such as the weights of the CNN model and the LSTM model, are updated so that the label corresponding to the walking motion type is output from the LSTM model. When the training of the deep learning model (200) is completed for all of the large amount of training datasets, the training of the deep learning model (200) is completed.

[0055] In step 27, the combination generation unit (70) generates a spatial operation combination for each cycle consisting of the spatial information identified by the spatial identification unit (50) in step 25 and any one of the operation types corresponding to the classification result of the operation classification unit (60) in step 26. The spatial information identified by the spatial identification unit (50) in step 25 is the information of any one virtual space obtained by the spatial identification unit (50) in step 25. Accordingly, the combination generation unit (70) generates a spatial operation combination for each cycle consisting of the information of any one virtual space obtained by the spatial identification unit (50) in step 25 and any one of the operation types corresponding to the classification result of the operation classification unit (60) in step 26.

[0056] The information of a virtual space obtained by the space identification unit (50) in step 25 consists of the section information of the virtual space in a two-dimensional coordinate system representing a specific area of ​​the real world and the virtual space type. The combination generation unit (70) generates a spatial operation combination for each cycle consisting of the section information of the virtual space obtained by the space identification unit (50) in step 25 and a virtual space type, that is, the section information of a virtual space where the user is currently located among a plurality of virtual spaces set in a specific area of ​​the real world by the virtual space setting unit (20), a virtual space type, and a motion type corresponding to the classification result of the motion classification unit (60) in step 26.

[0057] In step 28, the counter (90) counts the number of times the user action is continuously classified into one action type by the action classification unit (60) whenever the user action is classified into one action type by the action classification unit (60) in step 26. For example, if the user action is classified into standing action type, walking action type, walking action type, walking action type, and standing action type by the action classification unit (60) during the process of repeating step 26 5 times, the counter (90) counts the standing action type 1 time, the walking action type 3 times, and the standing action type 1 time.

[0058] In steps 29 and 210, the behavior monitoring unit (80) monitors whether the spatial motion combination for each cycle generated by the combination generation unit (70) in step 27 satisfies one of the movement behavior conditions among the plurality of abnormal behavior conditions. In this embodiment, each of the plurality of abnormal behavior conditions consists of one of the plurality of virtual space types, a location change condition according to that virtual space type, and one of the plurality of operation types. The location change condition varies depending on the virtual space type. For example, if the type of a virtual space is a door type, the location change condition becomes a condition for determining whether the user goes out through the door, and if it is a restroom type, it becomes a condition for determining whether the user stays in the restroom for a long time.

[0059] In step 29, the behavior monitoring unit (80) identifies at least one abnormal behavior condition among a plurality of abnormal behavior conditions that includes a virtual space type identical to the virtual space type of the spatial motion combination for each period, and checks whether there exists an abnormal behavior condition among the at least one abnormal behavior condition identified in this way that includes a motion type that matches the motion type of the spatial motion combination for each period. If it is confirmed in step 29 that there exists an abnormal behavior condition that includes a motion type that matches the motion type of the spatial motion combination for each period, the process proceeds to step 210. If it is confirmed in step 29 that there does not exist an abnormal behavior condition that includes a motion type that matches the motion type of the spatial motion combination for each period, the process proceeds to step 210.

[0060] In step 210, if there is an abnormal behavior condition that includes a motion type matching the motion type of the spatial motion combination for each period, the behavior monitoring unit (80) detects a change in the user's location based on the location data for each period of the dataset generated by the dataset generation unit (40) in step 24, and monitors whether each period's spatial motion combination satisfies each of the multiple abnormal behavior conditions by checking whether at least one of the detected user location change and the count counted by the counter (90) in step 28 satisfies the location change condition according to the virtual space type of the abnormal behavior condition, that is, the virtual space type of the spatial motion combination for each period. The behavior monitoring unit (80) determines that the abnormal behavior condition is satisfied if at least one of the user location change and the counted count satisfies the location change condition according to the virtual space type of the spatial motion combination for each period.

[0061] In step 211, the user interface (110) outputs the results monitored by the behavior monitoring unit (80) in step 210 under the control of the control unit (100). The user interface (110) can output the results monitored by the behavior monitoring unit (80) in step 210 in various forms such as text, images, and voice. In step 210, if the control unit (100) determines by the behavior monitoring unit (80) that the spatial motion combination for each cycle generated by the combination generation unit (70) in step 27 satisfies any one of the multiple abnormal behavior conditions, the user interface (110) controls the user interface (110) to output content indicating that an abnormal behavior corresponding to that abnormal behavior condition has occurred. The control unit (100) controls the user interface (110) to output content indicating that no abnormal behavior occurred when it is determined by the behavior monitoring unit (80) in step 210 that the spatial motion combination for each cycle generated by the combination generation unit (70) in step 27 does not satisfy all of the multiple abnormal behavior conditions.

[0062] As steps 23 to 211 described above are repeatedly performed, the user's motion data and location data are collected in real time by the data collection unit (30), and then the user's motion data and location data are periodically sampled by the dataset generation unit (40) to generate a new dataset. Whenever a new dataset is generated, the user is monitored to determine whether they are performing abnormal behavior by considering the characteristics of the space where they are currently located, based on the dataset. According to this embodiment, since the user's current movements can be distinguished and recognized as everyday movements such as standing, sitting, lying down, and walking, and dangerous movements such as movement switching and direction switching, the user can be precisely monitored to determine whether they are performing abnormal behavior by considering the injury risk characteristics of the space where they are currently located. In a space that is relatively safe for the user, dangerous movements may be abnormal behavior, but in a space where the risk of injury to the user is high, even everyday movements may be abnormal behavior depending on the duration of the movement.

[0063] FIG. 6 is an example diagram of the case where the virtual space type of the spatial motion combination for each cycle is a door type. The behavior monitoring unit (80) identifies at least one abnormal behavior condition among a plurality of abnormal behavior conditions that includes a door type identical to the virtual space type of the spatial motion combination for each cycle. Subsequently, the behavior monitoring unit (80) checks whether there exists an abnormal behavior condition among the at least one abnormal behavior condition identified in this way that includes a walking type that matches the motion type of the spatial motion combination for each cycle. For example, if an abnormal behavior condition corresponding to user zone departure consists of a door type, a position change condition according to the door type, and a walking motion type, then there exists an abnormal behavior condition that includes a walking type that matches the motion type of the spatial motion combination for each cycle.

[0064] In the example illustrated in FIG. 6, the condition for a change in location according to the door type is satisfied in the case of leaving the virtual space corresponding to the door. If the change in user location detected by the behavior monitoring unit (80) in step 210 indicates a departure from the partitioned space according to the virtual space partition information of the spatial action combination for each cycle generated by the combination generation unit (70) in step 27, the behavior monitoring unit (80) determines that the spatial action combination for each cycle satisfies the movement behavior condition corresponding to the user leaving the area. In this case, the number of times counted by the counter (90) in step 28 is not used.

[0065] FIG. 7 is an example diagram of the case where the virtual space type of the spatial operation combination for each cycle is a lobby type. The behavior monitoring unit (80) identifies at least one abnormal behavior condition among a plurality of abnormal behavior conditions that includes a lobby type identical to the virtual space type of the spatial operation combination for each cycle. Subsequently, the behavior monitoring unit (80) checks whether there exists an abnormal behavior condition among the at least one abnormal behavior condition identified in this way that includes a sitting type that matches the operation type of the spatial operation combination for each cycle. For example, if the abnormal behavior condition corresponding to entry into the detection zone consists of a lobby type, a position change condition according to the lobby type, and a sitting operation type, then there exists an abnormal behavior condition that includes a sitting type that matches the operation type of the spatial operation combination for each cycle.

[0066] In the example illustrated in FIG. 7, the condition for a change in location according to the door type is satisfied when the lying-down type is counted continuously 10 or more times in the virtual space corresponding to the lobby. In step 210, the change in user location detected by the behavior monitoring unit (80) indicates that the user is staying within the partitioned space according to the virtual space partition information of the spatial motion combination for each cycle generated by the combination generation unit (70) in step 27, and in step 28, if the number counted by the counter (90) is 10 or more times, the behavior monitoring unit (80) determines that the spatial motion combination for each cycle satisfies the movement behavior condition corresponding to entry into the restricted area.

[0067] FIG. 8 is an example diagram of the case where the virtual space type of the spatial motion combination for each cycle is a toilet type. The behavior monitoring unit (80) identifies at least one abnormal behavior condition among a plurality of abnormal behavior conditions that includes a toilet type identical to the virtual space type of the spatial motion combination for each cycle. Subsequently, the behavior monitoring unit (80) checks whether there exists an abnormal behavior condition among the at least one abnormal behavior condition identified in this way that includes a lying type that matches the motion type of the spatial motion combination for each cycle. For example, if an abnormal behavior condition corresponding to a toilet fall consists of a toilet type, a position change condition according to the toilet type, and a lying motion type, then there exists an abnormal behavior condition that includes a lying type that matches the motion type of the spatial motion combination for each cycle.

[0068] In the example illustrated in FIG. 8, the condition for a change in location according to the toilet type is satisfied when the lying-down type is counted continuously 10 or more times in the virtual space corresponding to the toilet. In step 210, the change in user location detected by the behavior monitoring unit (80) indicates that the user is staying within the partitioned space according to the virtual space partition information of the spatial motion combination for each cycle generated by the combination generation unit (70) in step 27, and in step 28, if the number counted by the counter (90) is 10 or more times, the behavior monitoring unit (80) determines that the spatial motion combination for each cycle satisfies the movement behavior condition corresponding to a toilet fall.

[0069] Meanwhile, a user behavior monitoring method according to one embodiment of the present invention as described above can be written as a program executable on a computer processor and can be implemented on a computer that executes the program by recording it on a computer-readable recording medium. The computer includes all types of computers capable of executing the program, such as desktop computers, notebook computers, smartphones, and embedded type computers. In addition, the computer-readable recording medium includes storage media such as RAM, ROM, SSD (Solid State Drive), magnetic storage media (e.g., floppy disk, hard disk, etc.), and optical reading media (e.g., CD-ROM, DVD, etc.).

[0070] The present invention has been described above with reference to its preferred embodiments. Those skilled in the art will understand that the present invention may be embodied in modified forms without departing from the essential characteristics of the invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the invention is defined by the claims, not by the foregoing description, and all variations within the scope of the claims should be interpreted as being included in the invention. Explanation of the symbols

[0071] 10 ... Virtual Beacon Configuration Section 20 ... Virtual Space Configuration Section 30 ... Data Collection Department 40 ... dataset creation section 50 ... spatial identification unit 60 ... Action Classification Department 61 ... data partitioning section 62 ... 1st Action Classification Unit 63 ... 2nd Action Classification Unit 70 ... combination generation section 80 ... Behavior Monitoring Department 90 ... counter 100 ... control unit 110 ... User Interface 200 ... deep learning model

Claims

Claim 1 A method for monitoring user behavior, characterized by comprising: a step of collecting motion data representing the user's actions and location data representing the user's location in real time using at least one sensor attached to the user; a step of generating a dataset consisting of motion data and location data for each period by periodically sampling the motion data and location data collected in real time; a step of identifying the space where the user is currently located based on the location data for each period of the generated dataset; a step of classifying the user's actions into one of a plurality of motion types based on the motion data for each period of the generated dataset; a step of generating a spatial motion combination for each period consisting of the information of the identified space and the one of the motion types; and a step of monitoring whether the generated spatial motion combination for each period satisfies at least one of a plurality of abnormal behavior conditions. Claim 2 A user behavior monitoring method according to claim 1, wherein the identifying step determines one of a plurality of virtual spaces set in a specific area of ​​the real world where the user is currently located based on location data for each period of the generated dataset, and identifies the space where the user is currently located by obtaining information of the determined virtual space, and the step of generating a spatial operation combination for each period generates a spatial operation combination for each period consisting of the obtained virtual space information and one of the operation types. Claim 3 In claim 2, the monitoring step is characterized by identifying at least one abnormal behavior condition among the plurality of abnormal behavior conditions that includes a virtual space type identical to that of the spatial operation combination for each period, and monitoring whether the spatial operation combination for each period satisfies each of the plurality of abnormal behavior conditions by confirming whether there exists an abnormal behavior condition among the identified at least one abnormal behavior condition that includes an operation type matching the operation type of the spatial operation combination for each period. Claim 4 A user behavior monitoring method according to claim 3, wherein the monitoring step is characterized by detecting a change in user location in a virtual space corresponding to the space where the user is currently located based on the location data of each period of the generated dataset when there is an abnormal behavior condition including a motion type that matches the motion type of each periodic spatial motion combination, and checking whether each periodic spatial motion combination satisfies each of the plurality of abnormal behavior conditions by verifying whether the detected change in user location satisfies the condition for a change in location according to the virtual space type of each periodic spatial motion combination. Claim 5 A user behavior monitoring method according to claim 4, further comprising the step of counting the number of times the user is continuously classified into any one of the above-mentioned operation types whenever the user is classified into any one of the above-mentioned operation types, wherein the monitoring step monitors whether each of the spatial operation combinations for each period satisfies each of the plurality of abnormal behavior conditions by confirming whether at least one of the detected user location change and the counted number of times satisfies the location change condition according to the virtual space type of each periodic spatial operation combination. Claim 6 A user behavior monitoring method according to claim 2, further comprising: a step of setting a plurality of virtual beacons in a specific area based on a two-dimensional coordinate system of a plurality of actual beacons installed in the specific area; and a step of setting a plurality of virtual spaces for the specific area using at least one of the set plurality of virtual beacons. Claim 7 A user behavior monitoring method according to claim 6, wherein the step of setting the plurality of virtual spaces is characterized by partitioning each virtual space using at least one virtual beacon among the set plurality of virtual beacons, and setting the plurality of virtual spaces in the specific area by assigning one of the plurality of virtual space types to each partitioned virtual space. Claim 8 A user behavior monitoring method according to claim 7, wherein the step of setting the plurality of virtual spaces is characterized by partitioning a virtual boundary line, which is a straight virtual space with the two virtual beacons as ends, using two virtual beacons among the plurality of virtual beacons set in the specific area. Claim 9 A user behavior monitoring method according to claim 7, wherein the step of setting the plurality of virtual spaces is characterized by partitioning a circular virtual space with one of the virtual beacons set in the specific area as the center point using one of the virtual beacons. Claim 10 A user behavior monitoring method according to claim 7, wherein the step of setting the plurality of virtual spaces is characterized by partitioning a polygon-shaped virtual space with each of the at least three virtual beacons as corners using at least three virtual beacons among the plurality of virtual beacons set in the specific area. Claim 11 A computer-readable recording medium storing a program for executing the method of claim 1 on a computer. Claim 12 A user behavior monitoring device characterized by comprising: a data collection unit that collects motion data representing the user's actions and location data representing the user's location in real time using at least one sensor attached to the user; a dataset generation unit that generates a dataset consisting of motion data and location data for each period by periodically sampling the motion data and location data collected in real time; a space identification unit that identifies the space where the user is currently located based on the location data for each period of the generated dataset; a motion classification unit that classifies the user's actions into one of a plurality of motion types based on the motion data for each period of the generated dataset; a combination generation unit that generates a spatial motion combination for each period consisting of the information of the identified space and one of the motion types; and a behavior monitoring unit that monitors whether the generated spatial motion combination for each period satisfies at least one of a plurality of abnormal behavior conditions.

Images