Behavioral analysis system and behavioral analysis method

Abstract

To make it possible to analyze, even when the number of edge devices has increased, an action state of each person by coordination between a server device and each of the edge devices.SOLUTION: An action analysis system 100 includes an AI edge processor 10 including; an edge side analysis unit 12 for executing pre-processing of analysis processing for analyzing an action of an aged person 40 from data obtained by imaging the aged person 40; an analysis result extraction unit 13 for extracting data including information to be used for identifying the aged person 40; and an analysis result transfer unit 14 for transferring the data extracted by the analysis result extraction unit 13 to a cloud server 20. The cloud server 20 includes a server side analysis unit 23 that executes post-processing of the analysis processing on the basis of the data transferred from the analysis result transfer unit 14, and displays its analysis result on an analysis result display unit 24.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a behavior analysis system and a behavior analysis method. [Background technology]

[0002] With the dramatic improvement in computer performance, big data analysis systems have been proposed that have server devices capable of processing larger amounts of data than ever before. The large amounts of data to be analyzed are often collected remotely from Internet of Things (IoT) devices, which are things connected to the Internet, to server devices. In this way, cloud services that collect data using a large number of edge devices (edge ​​devices) and process the large amounts of data by concentrating it on a single server device via the Internet (cloud) are becoming widespread.

[0003] For example, Patent Document 1 describes an information output device that includes a first output unit that outputs acquired information acquired by a sensor, and a second output unit that converts personal information included in the acquired information into attribute information that does not identify an individual, and outputs the attribute information (Claim 1). This information output device calculates customer movement paths and behavior in front of the shelves captured in the collected image data, converts the calculation results into text and sends them to a server device, which analyzes them as marketing information (paragraph 0022). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2016 / 114392 Summary of the Invention [Problem to be solved by the invention]

[0005] When the number of edge devices increases, the load on the server device that collects and analyzes data from each edge device also increases. For example, if a large amount of data such as video data acquired by the edge device is sent directly to the server device, the communication load to the server device increases. In addition, when the server device performs analysis using AI (Artificial Intelligence) processing, which has a high processing load, the processing load of the server device increases according to the number of devices.

[0006] On the other hand, edge devices are small, low-performance computers such as wearable devices and smartphones for ease of use by end users, and it is difficult for an edge device alone to perform all of the analysis processing that is normally performed by a server device.

[0007] Therefore, a main object of the present invention is to enable each edge device to cooperate with a server device to analyze the behavioral state of each person, even if the number of edge devices increases. [Means for solving the problem]

[0008] In order to solve the above problems, the present invention Behavioral Analysis System has the following characteristics: The present invention provides a behavior analysis system including an edge device that analyzes a person's behavior and a server device that processes data transferred from the edge device, The edge device, The analysis process includes analyzing the behavior of the person from the image data of the person. A process of analyzing data showing the posture of the person at each time from the image data of the person, and a process of analyzing a falling state indicating that the person has fallen and a falling time in the falling state from the analyzed change in posture over time as data showing the state of the person. An edge side analysis unit that executes From the analysis result of the edge side analysis unit, The above Used to identify a person Rude Extract the data Execute the action An analysis result extraction unit; Data on the analysis results by the edge side analysis unit; an analysis result transfer unit that transfers the data extracted by the analysis result extraction unit to the server device, The server device, a database in which the data transferred from the analysis result transfer unit is stored for each person; Based on the data in the database, the analysis process From the analyzed state of the person, alarm information is analyzed in which the fewer other people around the person who are assisting the person, the higher the urgency is calculated, and the longer the duration of the fall, the higher the urgency is calculated. The system is characterized by having a server-side analysis unit that executes the processing and displays the analysis results on the analysis result display unit. Other means will be described later. Effect of the Invention

[0009] According to the present invention, even if the number of edge devices increases, each edge device can cooperate with a server device to analyze the behavioral state of each person. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a configuration diagram showing a behavior analysis system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a configuration diagram showing a first example of functional sharing between an edge-side analysis unit and a server-side analysis unit according to the present embodiment. [Diagram 3] FIG. 11 is a configuration diagram showing a second example of functional sharing between an edge-side analysis unit and a server-side analysis unit according to the present embodiment. [Figure 4] FIG. 13 is a configuration diagram showing a specific example of an edge side analysis unit according to the present embodiment. [Diagram 5] FIG. 5 is a configuration diagram showing a modified example of the edge side analysis unit in FIG. 4 according to the present embodiment. [Figure 6] 11 is a flowchart showing the operation of the behavior analysis system when focusing on walking pattern analysis according to the present embodiment. [Figure 7] FIG. 2 is an explanatory diagram of a process for detecting a person from a 2D video according to the present embodiment. [Figure 8] 8 is an explanatory diagram of a process for extracting a posture of the person detected in FIG. 7 according to this embodiment. FIG. [Figure 9] 1 is a table showing data processed by an AI edge processor according to this embodiment. [Figure 10] FIG. 11 is a first screen diagram showing the result output of gait analysis according to the present embodiment. [Figure 11] FIG. 13 is a second screen diagram showing the result output of gait analysis according to the present embodiment. [Figure 12] FIG. 13 is a third screen diagram showing the result output of gait analysis according to the present embodiment. [Figure 13] 13 is a flowchart showing the operation of the behavior analysis system when focusing on the fall state analysis unit according to the present embodiment. [Figure 14] FIG. 11 is a first screen diagram showing an alarm output by fall analysis according to the present embodiment. [Figure 15] FIG. 11 is a second screen diagram showing an alarm output by fall analysis according to the present embodiment. [Figure 16] FIG. 13 is a third screen diagram showing an alarm output by fall analysis according to the present embodiment. [Figure 17] FIG. 2 is a hardware configuration diagram of each device of the behavior analysis system according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] FIG. 1 is a configuration diagram showing a behavior analysis system 100. As shown in FIG. The behavior analysis system 100 is configured by connecting an AI edge processor (edge ​​device) 10 and a cloud server 20 via a network 50. The network 50 may be a LAN (Local Area Network), a WAN (Wide Area Network), or a network combining a LAN and a WAN. The cloud server 20 is an example of a server device, and a device other than the cloud server 20 may be used as the server device.

[0013] 1, the number of AI edge processors 10 is two, but may be one or more. Also, a cache server (not shown) may be provided in the network 50 between the AI ​​edge processor 10 and the cloud server 20. In this case, the cache server collects data from multiple AI edge processors 10 and transfers the data collectively to the cloud server 20, thereby reducing the amount of communication.

[0014] The AI ​​edge processor 10 is an edge-side device that senses the physical condition of an elderly person (person) 40. The AI ​​edge processor 10 can reduce development costs by developing it based on an AI developer kit, for example. As developer kits, for example, small single boards such as Jetson Nano (registered trademark) and Jetson Xavir NX (registered trademark), which are products of NVIDIA, are commercially available.

[0015] A camera 30 is connected to each AI edge processor 10 via a wired connection such as USB (Universal Serial Bus) or a wireless connection such as Bluetooth (registered trademark). Alternatively, a camera 30 built into the AI ​​edge processor 10 may be used. The camera 30 captures images of surrounding elderly people 40 as video data. In this embodiment, the subject of the camera 30 is the elderly person 40, but this is merely an example of applying the behavior analysis system 100 to a nursing care business, and the person to be photographed (analyzed) is not limited to the elderly person 40. In one example of the nursing care business, sensing the physical condition of the elderly person 40 from the camera 30 in real time can be used for a cloud service that monitors the elderly person 40 from a remote location.

[0016] The AI ​​edge processor 10 has a data acquisition unit 11, an edge-side analysis unit 12, an analysis result extraction unit 13, and an analysis result transfer unit 14. The cloud server 20 has a database 21, a data extraction unit 22, a server-side analysis unit 23, and an analysis result display unit 24. The data acquisition unit 11 acquires video data from the camera 30 . The edge-side analysis unit 12 and the server-side analysis unit 23 each have their own role to analyze the behavior of the elderly person 40 shown in the video data. That is, the edge-side analysis unit 12 uses the video data as input to create preprocessed data for analysis. The analysis result extraction unit 13 extracts data necessary for the analysis process of the server-side analysis unit 23 from the preprocessed data output from the edge-side analysis unit 12, and transfers the extracted data to the cloud server 20 via the analysis result transfer unit 14.

[0017] The cloud server 20 stores the data transferred from the analysis result transfer unit 14 in a database 21 for each person (for each AI edge processor 10). The data extraction unit 22 extracts data necessary for the analysis processing of the server-side analysis unit 23 from the data for each person stored in the database 21. The server-side analysis unit 23 receives the data extracted by the data extraction unit 22, analyzes (post-processes) the behavior of the elderly person 40, and displays the results on the analysis result display unit 24.

[0018] FIG. 2 is a configuration diagram showing a first example of the division of roles between the edge-side analysis unit 12 and the server-side analysis unit 23. As shown in FIG. The edge side analysis unit 12 includes a posture analysis unit 121 and a state analysis unit 122. The server side analysis unit 23 includes a behavior analysis unit 231. The posture analysis unit 121 analyzes data indicating the posture of the elderly person 40 at each time from image data of the elderly person 40. For example, the posture of the elderly person 40 while walking changes over time in the following order (see FIG. 8 for details). In this embodiment, skeletal data is exemplified as a data expression indicating a posture. The skeletal data is configured as position data of each of 17 joints in the human body (nose, right eye, left eye, right ear, left ear, right shoulder, left shoulder, right hand, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle).

[0019] The state analyzing unit 122 analyzes data indicating the state of the elderly person 40 from the change over time in the posture analyzed by the posture analyzing unit 121 (classifies the state). For example, when the posture changes from standing with the left foot forward → right foot forward → left foot forward, etc., the condition analysis unit 122 classifies the condition of the elderly person 40 as a walking condition. Or, when the posture changes in the following order: standing but leaning forward → right hand on the floor → whole body lying on the floor, etc., the state analysis unit 122 classifies the state of the elderly person 40 as a fallen state.

[0020] At least one of the posture analysis unit 121 and the state analysis unit 122 can be implemented as a TensorRT model that runs on the AI ​​edge processor 10 (Jetson Nano), enabling high-speed inference processing. TensorRT is a software development kit (SDK) provided by NVIDIA for purposes such as high-speed execution of deep learning inference.

[0021] The process of creating a TensorRT model is, for example, as follows. (Step 1) A human (acting as a teacher) adds tags (annotations) to the sample video data collected for learning, indicating the results of detecting and tracking people, as well as the classification results of the behavior of the tracked people. (Step 2) A trained model is created by performing machine learning (training) based on the video data from Step 1. The trained model is updated through model evaluation. (Step 3) The trained model from step 2 is converted (ported) into a TensorRT model to create an accelerated TensorRT-based inference model. (Step 4) The TensorRT model from step 3 is placed on the edge-side analysis unit 12. This makes it possible to input video data for inference into the TensorRT model, accelerating the process of detecting and tracking people in the video and classifying the postures and behaviors of the tracked people.

[0022] The behavior analysis unit 231 analyzes data useful for the elderly person 40 from the state of the elderly person 40 analyzed by the state analysis unit 122. That is, the behavior analysis unit 231 monitors health information of the elderly person 40 and creates information useful for nursing care based on the data in the database 21. For example, the behavior analysis unit 231 notifies the elderly person 40 of the results of analyzing the walking history of the elderly person 40 classified as being in a walking state by the state analysis unit 122 (details in Figs. 10 to 12). Alternatively, the behavior analysis unit 231 notifies a nearby caregiver to request rescue for the elderly person 40 who has been classified as having fallen by the state analysis unit 122 (see FIG. 14 to FIG. 16 for details).

[0023] FIG. 3 is a configuration diagram showing a second example of the division of roles between the edge-side analysis unit 12 and the server-side analysis unit 23. In FIG. The edge side analysis unit 12 has a posture analysis unit 121. The server side analysis unit 23 has a state analysis unit 232 and a behavior analysis unit 231. The behavior analysis unit 231 has the same function as the state analysis unit 122. That is, compared to the first example in Fig. 2, in the second example in Fig. 3, the state analysis unit 122 of the edge-side analysis unit 12 is moved to the behavior analysis unit 231 of the server-side analysis unit 23. As a result, in the second example, the processing load on the AI ​​edge processor 10 is reduced, so that even a low-performance computer such as a wearable device can be used as the AI ​​edge processor 10. On the other hand, in the first example in Fig. 2, the load on the cloud server 20 can be reduced by performing more pre-processing.

[0024] FIG. 4 is a configuration diagram showing a specific example of the edge side analysis unit 12. As shown in FIG. The state analysis unit 122 in one AI edge processor 10 has, as analysis units for each state, a walking state analysis unit 122A and a fall state analysis unit 122B. Of the AI ​​edge processor 10, the analysis result transfer unit 14 is omitted from the illustration. The walking state analysis unit 122A analyzes whether or not the elderly person 40 is in a walking state. The falling state analysis unit 122B analyzes whether or not the elderly person 40 is in a falling state.

[0025] In this way, accommodating multiple types of state analysis units in a single AI edge processor 10 makes it easy to deploy and carry the AI ​​edge processor 10. Furthermore, within a single AI edge processor 10, a walking detection mode in which the walking state analysis unit 122A is operated and a fall detection mode in which the fall state analysis unit 122B is operated may be switched by a switch operation or the like.

[0026] FIG. 5 is a configuration diagram showing a modified example of the edge side analyzer 12 of FIG. In FIG. 5, the AI ​​edge processor 10A accommodating the walking state analysis unit 122A and the AI ​​edge processor 10B accommodating the fall state analysis unit 122B are configured to be distributed in separate devices. Then, the AI ​​edge processor 10A and the AI ​​edge processor 10B individually transmit the status of the elderly person 40 that they have detected to the cloud server 20. The cloud server 20 integrates multiple statuses of the same elderly person 40 and registers them in the database 21.

[0027] In this way, by housing the analysis units for each state in separate AI edge processors 10, each AI edge processor 10 can detect the state it is responsible for with high accuracy. For example, the AI ​​edge processor 10A is installed in a hall or a common space of a facility for the elderly. This allows the walking state analysis unit 122A to analyze the walking state of an elderly person walking within the shooting range of the camera. On the other hand, the AI ​​edge processor 10B is installed in a private room where an elderly person is present, a shared space, etc. This enables the fall state analysis unit 122B to analyze the fall state of an elderly person who is within the shooting range of the camera. The analysis results of each of the AI ​​edge processors 10A and 10B are transferred to the cloud server 20 via the network 50. This enables the cloud server 20 to analyze the condition of the elderly person.

[0028] FIG. 6 is a flowchart showing the operation of the behavior analysis system 100 when focusing on the walking state analysis unit 122A. The data acquisition unit 11 acquires a 2D image of a person (elderly person 40) captured by the camera 30 (S11). Pose analysis unit 121 detects a person from the 2D image of S11 and tracks the person within the 2D image (S12). Then, posture analysis unit 121 extracts the posture of the person detected in S12 at each time (S13). Based on the 2D position of the posture extracted in S13, the state analysis unit 122 (walking state analysis unit 122A) extracts the 3D distance (distance in three-dimensional space, details in FIG. 8) traveled from the person's 2D position at a certain time to the person's 2D position at the next time (S14).

[0029] The state analysis unit 122 (walking state analysis unit 122A) calculates the walking speed and stride length of the same person based on the 3D distance of S14 (S15). The analysis result extraction unit 13 extracts information for identifying the person detected in S12, such as face information showing facial features, as person information, and extracts the walking speed and stride length of S15 as walking information (S16). The analysis result transfer unit 14 transfers the analysis results (person information, walking information) extracted in S16 to the cloud server 20 via the network 50 (S17). Note that the analysis results may include the posture of S13.

[0030] The cloud server 20 receives the analysis results (person information, walking information) of S17 and stores them in the database 21 (S21). The data extraction unit 22 extracts an area where a person is shown (a person area) in the 2D image indicated by the person information based on the person information read from the database 21, and extracts walking information from the database 21 (S22). The behavior analysis unit 231 recognizes who the person is based on the information captured in the person area in the 2D image extracted in S22, and specifies an ID for identifying the person as a result of the recognition (S23).

[0031] The behavior analysis unit 231 performs a walking analysis (S24) for each ID identified in S23, based on the walking information (walking speed, stride length) to obtain the distance walked by the person in a predetermined period, average speed, acceleration, walking ability, etc. The walking ability is calculated, for example, according to the Timed Up & Go Test (TUG), which is an index of musculoskeletal ambulatory disability symptom complex (MADS). The behavior analysis section 231 outputs the analysis result of S24 to the analysis result display section 24 (S25). An example of the output contents will be given below. As a result of the behavior analysis unit 231 analyzing the walking information (walking history), the history of walking characteristics and the distribution of walking features are output to the analysis result display unit 24. Based on the walking ability analyzed by the behavior analysis unit 231, predicted data on health risks is output to the analysis result display unit 24.

[0032] FIG. 7 is an explanatory diagram of the process (S12) of detecting a person from a 2D video image. Assume that the camera 30 captures an image of a person walking from the left side to the right side of the drawing at times t1, t2, and t3 in that order. The posture analysis unit 121 detects the position of the person at each of the times t1, t2, and t3 (a rectangle surrounding the person in the drawing) and tracks the person.

[0033] FIG. 8 is an explanatory diagram of the process (S13) of extracting a posture of the person detected in FIG. For the person detected at each of the times t1, t2, and t3, posture analysis unit 121 extracts, as a posture, skeletal data in which each joint is represented by a black dot and the lines connecting these black dots are represented by bones. Also, in S14, the state analysis unit 122 (walking state analysis unit 122A) calculates a 3D distance d12 from the person position at time t1 to the person position at time t2. Similarly, the state analysis unit 122 calculates a 3D distance d23 from the person position at time t2 to the person position at time t3.

[0034] Then, in S15, the state analyzing unit 122 calculates the walking speed of the person based on the 3D distances d12 and d23 and the elapsed time from time t1, t2, and t3. Furthermore, the state analyzing unit 122 calculates the stride length w1 of the person at time t1. In addition, in S16, the analysis result extraction unit 13 extracts a face image f1 and face information of the person as person information for identifying the detected person. The face information is authentication information extracted from the face image, and examples thereof include feature information for face authentication and iris information.

[0035] FIG. 9 is a table showing the data processed by the AI ​​edge processor 10. Table 201 is data generated inside the AI ​​edge processor 10 (S11-S15), and table 202 is data that the analysis result transfer unit 14 transfers to the cloud server 20 based on table 201 (S16, S17). 6. The table 201 associates the camera image of S11, the person detection result of S12, the posture information (2D position) of S13, the 3D distance of S14, and the walking information of S15 in FIG.

[0036] The camera image of S11 is acquired by the data acquisition unit 11 as a JPG (Joint Photographic Experts Group) image or a PNG (Portable Network Graphics) image. The result of person detection in S12 is a combination of the frame of the area in which the person is captured and the ID of the person. Note that at the time of S12, the ID of the person is not used to identify the person, but is used to track where the person has moved in 10 seconds, that is, to distinguish the person from other people.

[0037] The posture information in S13 is information indicating the 2D position (vertical position u, horizontal position v) in the image for each of the person's 17 joints. The 3D distance in S14 is information that converts the position information of each of a person's 17 joints from 2D positions (u, v) in the image to 3D positions (x, y, z) in real space, and the time difference of these 3D positions is the 3D distance. The walking information in S15 is, for example, the person's stride length and walking speed.

[0038] 6. The table 202 associates the person information of S16, the walking information of S16, and the posture information (the 2D position (u, v) of S13 and the 3D distance (x, y, z) of S14) in FIG. The person information in S16 includes, for example, facial features (contour, mouth size, etc.) and clothing features (color, type, etc.) as identification information of the elderly person f1 in FIG. The walking information of S16 includes, for example, the person's state=walking, stride=65 [cm], speed=1.0 [m / s], and acceleration=+0.1 [m / s squared].

[0039] FIG. 10 is a first screen shot showing the result output of the gait analysis (S25). The display screen 211 has a time series graph with the horizontal axis representing the date and the vertical axis representing the numerical value of the walking information, and a "Video" button 213B for starting the playback of video data corresponding to the date on the horizontal axis. In the time series graph, the walking information of S15 such as the person's stride length, walking speed, acceleration, etc. are displayed with distinction from each other (by using different colors or types of graph lines, etc.) The user can check this graph and press the "Video" button 213B of a date of interest to check the video data of the person taken on that date.

[0040] FIG. 11 is a second screen shot showing the results output of the gait analysis (S25). The display screen 212 displays in the form of a radar chart each of the person's abilities (judgment, memory, attention) including walking ability analyzed by the behavior analysis unit 231. Each ability is evaluated on a scale of 1 to 5, with the higher the value indicating the higher the ability. Note that the results of measurements of abilities other than walking ability have been measured in advance and are stored in the database 21. By correctly evaluating and presenting the life capabilities of an elderly person as in this display screen 212, it is possible to allow the elderly person to properly recognize their frailty state before they become dependent on care.

[0041] FIG. 12 is a third screen shot showing the result output of the gait analysis (S25). The display screen 220 has a time series graph 221 of walking ability analyzed by a behavior analysis section 231 , and a recommendation message 222 proposed by the behavior analysis section 231 based on changes in walking ability in the time series graph 221 . This will encourage elderly people to undergo appropriate rehabilitation before they reach a state where they require nursing care and are at risk of falling or breaking a bone, thereby supporting the elderly in maintaining their independence with dignity.

[0042] Fig. 13 is a flowchart showing the operation of the behavior analysis system 100 when focusing on the fall state analysis unit 122B. The same processes as those in Fig. 6 (the flowchart focusing on the walking state analysis unit 122A) are given the same reference numerals and the description thereof will be omitted. The processing (S11-S13) of the AI ​​edge processor 10 is as described in Fig. 6. The posture analysis unit 121 detects a person from the 2D image acquired in S11 and tracks the person within the 2D image (S12). Then, the posture analysis unit 121 extracts the posture at each time for the person detected in S12 (S13).

[0043] The state analyzing unit 122 recognizes (classifies) the state of the person from the change over time in the posture of the person extracted in S13 (S14B). The fallen state analysis unit 122B determines whether the state of the person recognized in S14B is a fallen state (S15B). If the state is a fallen state (Yes in S15B), the process proceeds to S16B, and if the state is another state such as a sleeping state (No in S15B), the process returns to S11.

[0044] The analysis result extraction unit 13 extracts the same person information as in S16 of Fig. 6 and information about the person in a fallen state (fall information) from the 2D image of S11 (S16B). The fall information is, for example, whether or not the person has fallen at each time, and if so, the duration of the fall (the time that has elapsed since the person fell without getting up). The analysis result transfer unit 14 transfers the person information of S16B and the fall information (which may include the posture information of S13) to the cloud server 20 via the network 50 (S17B).

[0045] Cloud server 20 receives the analysis results (person information, fall information) of S17B and stores them in database 21 (S21B). The data extraction unit 22 extracts an area in which a person is shown (a person area) in the 2D image indicated by the person information based on the person information read from the database 21, and extracts fall information from the database 21 (S22B). The behavior analysis unit 231 recognizes who the person is based on the information captured in the person area in the 2D image extracted in S22, and as a result of this recognition, specifies an ID for identifying the person (same as S23 in Figure 6).

[0046] The behavior analysis unit 231 uses the fall information extracted in S22B to analyze the urgency of rescue for the person who fell in S23 (S24B). The longer the fall time, the higher the urgency calculated by the behavior analysis unit 231. Also, the fewer other people around the person who has fallen to provide assistance (for example, in a situation where only one person has fallen in a private room), the higher the urgency calculated by the behavior analysis unit 231. The behavior analysis unit 231 outputs the analysis result of S24B to the analysis result display unit 24 (S25B). Note that the higher the urgency, the higher the degree of output of the warning information may be. A high degree of output may mean highlighting the information on the screen, playing an alarm sound in addition to displaying the information on the screen, adding the AI ​​edge processor 10 and the terminals of the elderly person's family members to the warning output destinations in addition to the cloud server 20, and so on.

[0047] FIG. 14 is a first screen shot showing the warning output (S25B) due to fall analysis. The screen image 220A includes a layout diagram section 221A showing the relative positions of the elderly person 40 who has fallen and other people around him, and an output section 222A for warning information which is the analysis result of the behavior analysis section 231. In the situation of this floor plan field 221A, the AI ​​edge processor 10 that analyzes the behavior of elderly person A detects that elderly person A has fallen, but also detects that a caregiver is nearby in the same room. Therefore, since it is expected that the caregiver will immediately assist elderly person A, the behavior analysis unit 231 outputs the analysis result "no notification is necessary" to the alarm information output field 222A.

[0048] FIG. 15 is a second screen shot showing the warning output (S25B) due to fall analysis. Similar to FIG. 14, the screen image 220B includes a layout view field 221B and an output field 222B for alarm information. In the situation of this floor plan field 221B, not only the fallen state of elderly person A but also the fallen state of elderly people A nearby is detected by the AI ​​edge processor 10. Meanwhile, it is also registered in advance in the database 21 that a caregiver is waiting near the room where the elderly people A and B are located. Therefore, the behavior analysis unit 231 outputs the analysis result "Elderly people A and B have fallen. A nearby caregiver will be automatically notified" to the warning information output field 222B in order to direct the caregiver to assist the elderly people A and B. The behavior analysis unit 231 also transmits the automatic notification to the terminal held by the caregiver.

[0049] FIG. 16 is a third screen shot showing the alarm output (S25B) due to fall analysis. Similar to FIG. 14, the screen image 220C includes a layout view field 221C and an output field 222C for alarm information. In the situation of this floor plan field 221C, the AI ​​edge processor 10 detects that the elderly person A has fallen in a private room. On the other hand, it is registered in the database 21 that there is no caregiver near the room where the elderly person A is. Therefore, in order to notify the family of the elderly A of the emergency, the behavior analysis unit 231 outputs the analysis result "Elderly A has fallen and there is no caregiver nearby. Please contact the emergency services" to the output field 222C of the alarm information. This allows the family of the elderly A to promptly request emergency services.

[0050] As explained above with reference to Figures 14 to 16, even if the same situation, "elderly person A has fallen," is detected by a single AI edge processor 10, the presence or absence of warning information output and the output contents can be changed depending on the relationship with other people around him / her. This is because the server-side analysis unit 23 determines the warning information by referring to the fallen state of elderly person A and the state of the people around him / her. To achieve this, in addition to a configuration in which the AI ​​edge processor 10 independently analyzes the fall status of each elderly person, it is also effective to configure a cloud server 20 to manage the status of multiple people in a database 21 as a cloud service.

[0051] FIG. 17 is a hardware configuration diagram of each device (AI edge processor 10, cloud server 20) of the behavior analysis system 100. Each device of the behavior analysis system 100 is configured as a computer 900 having a CPU 901 , a RAM 902 , a ROM 903 , a HDD 904 , a communication I / F 905 , an input / output I / F 906 , and a media I / F 907 . The communication I / F 905 is connected to an external communication device 915. The input / output I / F 906 is connected to an input / output device 916. The media I / F 907 reads and writes data from a recording medium 917. Furthermore, the CPU 901 controls each processing unit by executing a program (also called an application or its abbreviation, an app) loaded into the RAM 902. This program can be distributed via a communication line, or can be recorded on a recording medium 917 such as a CD-ROM and distributed.

[0052] In the embodiment described above, the large-volume video data from the camera 30 is transmitted as is from the AI ​​edge processor 10 to the cloud server 20, reducing the load on the cloud service in which the cloud server 20 performs all analysis processing. In other words, the edge-side analysis unit 12 of the AI ​​edge processor 10 preprocesses the video data to reduce the amount of data, and then transmits it to the cloud server 20 to be handed over to post-processing by the server-side analysis unit 23. This will enable the provision of cloud services that sense a person's physical condition remotely in real time, at low cost and with low load.

[0053] The load can be distributed between the AI ​​edge processor 10 and the cloud server 20 as follows. Since the network 50 transmits pre-processed data with reduced data volume rather than large-volume video data, the communication load can be reduced. The AI ​​edge processor 10 does not handle all analytical processing, but only some of the preprocessing, so it can run on small, low-performance computers such as wearable devices and smartphones. The cloud server 20 receives data preprocessed by the AI ​​edge processor 10 and only needs to perform post-processing, reducing the processing load. [Explanation of symbols]

[0054] 10 AI edge processor (edge ​​device) 11 Data Acquisition Section 12 Edge side analysis unit 13 Analysis result extraction part 14 Analysis results transfer section 20 Cloud server (server equipment) 21 Database 22 Data Extraction Section 23 Server-side analysis 24 Analysis result display section 30 Camera 40 Elderly people (people) 50 Network 100 Behavioral Analysis System 121 Posture Analysis Department 122 Status Analysis Unit 122A Walking state analysis unit 122B Falling Condition Analysis Unit 231 Behavior Analysis Department 232 Status Analysis Unit

Claims

1. A behavior analysis system including an edge device that analyzes a person's behavior and a server device that processes data transferred from the edge device, The edge device includes: an edge side analysis unit that executes a process of analyzing data indicating the posture of the person at each time from the image data of the person, in an analysis process of analyzing the behavior of the person from the image data of the person, and a process of analyzing a falling state indicating that the person has fallen and a falling time in the falling state from a change in the analyzed posture over time as data indicating the state of the person; an analysis result extraction unit that executes a process of extracting data used for identifying the person from the analysis result of the edge side analysis unit; an analysis result transfer unit that transfers data of the analysis result by the edge side analysis unit and data extracted by the analysis result extraction unit to the server device, The server device includes: a database in which the data transferred from the analysis result transfer unit is stored for each person; The system further comprises a server-side analysis unit that executes a process of analyzing alarm information based on the data in the database, the data being useful to the person from the analyzed state of the person in the analysis process, the fewer the number of other people around the person providing care, the higher the urgency is calculated, and the longer the duration of the fall, the higher the urgency is calculated, and the server-side analysis unit displays the analysis results on an analysis result display unit. Behavioral analysis system.

2. A behavior analysis system including an edge device that analyzes a person's behavior and a server device that processes data transferred from the edge device, The edge device includes: an edge side analysis unit that executes a process of analyzing data indicating a posture of the person at each time from the image data of the person, in an analysis process of analyzing a behavior of the person from the image data of the person; an analysis result extraction unit that executes a process of extracting data used for identifying the person from the analysis result of the edge side analysis unit; an analysis result transfer unit that transfers data of the analysis result by the edge side analysis unit and data extracted by the analysis result extraction unit to the server device, The server device includes: a database in which the data transferred from the analysis result transfer unit is stored for each person; The system further comprises a server-side analysis unit that executes a process of analyzing a state of a fall indicating that the person has fallen and a duration of the fall in the fallen state as data indicating the state of the person from the change in posture over time analyzed by the edge-side analysis unit based on the data in the database, and a process of analyzing alarm information that calculates a higher urgency as the number of other people providing care around the person is reduced and a higher urgency as the duration of the fall is longer as data useful to the person from the analyzed state of the person, and displays the analysis results on an analysis result display unit. Behavioral analysis system.

3. The data indicating the state of the person further includes a walking state indicating that the person is walking, and walking information indicating a walking speed and a stride length in the walking state; The data pertaining to the person further includes at least one of the following: a distance walked by the person in a given period of time, an average speed, and a walking ability. The behavioral analysis system according to claim 1 or 2.

4. The server device displays a graph showing a time-dependent change in an analysis result of the walking information and a recommendation message corresponding to the time-dependent change on the analysis result display unit. The behavior analysis system according to claim 3 .

5. The server device displays, on the analysis result display unit, a floor plan showing a positional relationship between the person in the fallen state and the other person present around the person, and warning information corresponding to the positional relationship. The behavioral analysis system according to claim 1 or 2.

6. A behavior analysis method executed by a behavior analysis system including an edge device that analyzes a person's behavior and a server device that processes data transferred from the edge device, comprising: The edge device includes: an edge side analysis unit that executes a process of analyzing data indicating the posture of the person at each time from the image data of the person, in an analysis process of analyzing the behavior of the person from the image data of the person, and a process of analyzing a falling state indicating that the person has fallen and a falling time in the falling state from a change in the analyzed posture over time as data indicating the state of the person; an analysis result extraction unit that executes a process of extracting data used for identifying the person from the analysis result of the edge side analysis unit; an analysis result transfer unit that transfers data of the analysis result by the edge side analysis unit and data extracted by the analysis result extraction unit to the server device, The server device includes: a database in which the data transferred from the analysis result transfer unit is stored for each person; The system further comprises a server-side analysis unit that executes a process of analyzing alarm information based on the data in the database, the data being useful to the person from the analyzed state of the person in the analysis process, the fewer the number of other people around the person providing care, the higher the urgency is calculated, and the longer the duration of the fall, the higher the urgency is calculated, and the server-side analysis unit displays the analysis results on an analysis result display unit. Behavior analysis methods.

7. A behavior analysis method executed by a behavior analysis system including an edge device that analyzes a person's behavior and a server device that processes data transferred from the edge device, comprising: The edge device includes: an edge side analysis unit that executes a process of analyzing data indicating a posture of the person at each time from the image data of the person, in an analysis process of analyzing a behavior of the person from the image data of the person; an analysis result extraction unit that executes a process of extracting data used for identifying the person from the analysis result of the edge side analysis unit; an analysis result transfer unit that transfers data of the analysis result by the edge side analysis unit and data extracted by the analysis result extraction unit to the server device, The server device includes: a database in which the data transferred from the analysis result transfer unit is stored for each person; The system further comprises a server-side analysis unit that executes a process of analyzing a state of a fall indicating that the person has fallen and a duration of the fall in the fallen state as data indicating the state of the person from the change in posture over time analyzed by the edge-side analysis unit based on the data in the database, and a process of analyzing alarm information that calculates a higher urgency as the number of other people providing care around the person is reduced and a higher urgency as the duration of the fall is longer as data useful to the person from the analyzed state of the person, and displays the analysis results on an analysis result display unit. Behavior analysis methods.

Images