Behavior analysis system and behavior analysis method

Abstract

The present application provides a behavior analysis system and a behavior analysis method, which can analyze the behavior state of each person through the cooperation of each edge device and a server device even if the number of edge devices increases. An AI edge processor (10) of a behavior analysis system (100) has: an edge-side analysis section (12) that performs preprocessing in analysis processing of analyzing the behavior of an elderly person (40) from data obtained by photographing the elderly person (40); an analysis result extraction section (13) that extracts data containing information for determining the elderly person (40); and an analysis result transmission section (14) that transmits the data extracted by the analysis result extraction section (13) to a cloud server (20). The cloud server (20) has a server-side analysis section (23) that performs post-processing in the analysis processing from the data transmitted from the analysis result transmission section (14) and displays the analysis result on an analysis result display section (24).

Description

Technical Field

[0001] This invention relates to behavior analysis systems and methods. Background Technology

[0002] With the leap in computing performance, big data analytics systems have emerged, featuring server devices capable of processing massive amounts of data. Large volumes of data being analyzed are frequently collected remotely from Internet of Things (IoT) devices connected to the internet and then processed on a single server device. This has led to the proliferation of cloud services where data is collected from multiple edge devices and then centralized on a single server device via the internet (cloud) to handle the massive amounts of data.

[0003] For example, Patent Document 1 describes an information output device (claim 1) comprising: a first output unit that outputs acquired information obtained by a sensor, and a second output unit that converts personal information contained in the acquired information into attribute information that the individual cannot determine and outputs the attribute information.

[0004] The information output device calculates the customer's movement routes and behavior in front of the shelves reflected in the collected image data, and sends the calculation results to the server device after textualization, so that the server device can analyze them as marketing information (paragraph 0022).

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: International Publication No. 2016 / 114392 Summary of the Invention

[0008] The problem the invention aims to solve

[0009] As the number of edge devices increases, the load on the server devices that collect and analyze data from each edge device also increases. For example, if large volumes of data, such as dynamic image data acquired at the edge device, are sent directly to the server device, the communication load to the server device increases. Furthermore, if the server device uses high-processing AI (Artificial Intelligence) processing for analysis, the processing load on the server device increases depending on the number of devices.

[0010] On the other hand, for the convenience of end users, edge devices are small and low-performance computers such as wearable devices or smartphones, which are difficult to replace all the analysis and processing performed by server devices in a single edge device.

[0011] Therefore, the main objective of this invention is to enable the analysis of individual behavior states through collaboration between the edge devices and the server device, even as the number of edge devices increases.

[0012] Technical means to solve the problem

[0013] To address the aforementioned problems, the behavior analysis system of the present invention has the following features.

[0014] This invention is a behavior analysis system comprising an edge device for analyzing human behavior and a server device for processing data transmitted from the edge device. The behavior analysis system is characterized in that…

[0015] The edge device includes:

[0016] The edge-side analysis unit performs preprocessing in the analysis process of analyzing the behavior of the person based on the data obtained from photographing the person;

[0017] The analysis result extraction unit extracts data containing information for identifying the person by performing post-processing in the analysis process based on the analysis results of the edge-side analysis unit; and

[0018] The analysis result transmission unit transmits the data extracted by the analysis result extraction unit to the server device.

[0019] The server device includes:

[0020] A database that stores data transmitted from the analysis result transmission unit for each of the aforementioned individuals; and

[0021] The server-side analysis unit performs the post-processing in the analysis process based on the data in the database, and displays the analysis results on the analysis result display unit.

[0022] Other methods will be described later.

[0023] The effects of the invention

[0024] According to the present invention, even if the number of edge devices increases, the behavioral state of each person can be analyzed through the collaboration of each edge device and the server device. Attached Figure Description

[0025] Figure 1 This is a diagram illustrating the configuration of the behavior analysis system in this embodiment.

[0026] Figure 2 This is a configuration diagram showing the first example of the division of functions between the edge-side analysis unit and the server-side analysis unit in this embodiment.

[0027] Figure 3This is a configuration diagram showing a second example of the division of functions between the edge-side analysis unit and the server-side analysis unit in this embodiment.

[0028] Figure 4 This is a configuration diagram showing a specific example of the edge-side analysis unit in this embodiment.

[0029] Figure 5 This refers to the implementation method. Figure 4 A diagram illustrating the configuration of a modified example of the edge-side analysis section.

[0030] Figure 6 This is a flowchart illustrating the actions of the behavior analysis system when performing walking state analysis in this embodiment.

[0031] Figure 7 This is an explanatory diagram of the process for detecting a person from a 2D image in this embodiment.

[0032] Figure 8 This is about the implementation method. Figure 7 An explanatory diagram of the processing of human poses detected in the image.

[0033] Figure 9 This is a table representing the data processed by the AI ​​edge processor in this embodiment.

[0034] Figure 10 This is the first screen showing the output of the walk analysis results in this embodiment.

[0035] Figure 11 This is the second screen showing the output of the walk analysis results in this embodiment.

[0036] Figure 12 This is the third screen showing the output of the walk analysis results in this embodiment.

[0037] Figure 13 This is a flowchart illustrating the actions of the behavior analysis system when focusing on the fall state analysis unit of this embodiment.

[0038] Figure 14 This is the first screen image showing the alarm output based on fall analysis in this embodiment.

[0039] Figure 15 This is the second screen showing the alarm output based on fall analysis in this embodiment.

[0040] Figure 16 This is the third screen image showing the alarm output based on fall analysis in this embodiment.

[0041] Figure 17 This is a hardware configuration diagram of each device in the behavior analysis system of this embodiment. Detailed Implementation

[0042] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0043] Figure 1 This is a diagram showing the configuration of the behavior analysis system 100.

[0044] The AI ​​edge processor (edge ​​device) 10 and the cloud server 20 are connected via a network 50 to form a behavior analysis system 100. The network 50 can be a LAN (Local Area Network), a WAN (Wide Area Network), or a combination of both. Furthermore, the cloud server 20 is an example of a server device; other devices besides the cloud server 20 can also be used as server devices.

[0045] In addition, the number of AI edge processors 10 is Figure 1 Two units are required, but more than one unit is sufficient. Alternatively, a cache server (illustration omitted) can be set up within the network 50 between the AI ​​edge processor 10 and the cloud server 20. In this case, the cache server collects data from multiple AI edge processors 10 and centrally transmits this data to the cloud server 20, thereby reducing communication volume.

[0046] The AI ​​edge processor 10 is an edge-side device for sensing the physical state of an elderly person (person) 40. This AI edge processor 10 can reduce development costs, for example, by being developed based on an AI developer kit. Commercially available developer kits include, for example, small single-board products from NVIDIA such as the Jetson Nano (registered trademark) and Jetson Xavier NX (registered trademark).

[0047] Each AI edge processor 10 is connected to a camera 30 via a wired connection such as USB (Universal Serial Bus) or a wireless connection such as Bluetooth. Alternatively, a camera 30 built into the AI ​​edge processor 10 can also be used. The camera 30 captures images of the elderly people 40 in the surrounding area as dynamic image data.

[0048] Furthermore, in this embodiment, the subject of the camera 30 is set to an elderly person 40, but this is only one example of applying the behavior analysis system 100 to the nursing care industry, and the subject of the camera (analysis subject) is not limited to the elderly person 40. In one example in the nursing care industry, by sensing the physical condition of the elderly person 40 in real time from the camera 30, it can be utilized for cloud services that remotely care for the elderly person 40.

[0049] The AI ​​edge processor 10 includes a data acquisition unit 11, an edge-side analysis unit 12, an analysis result extraction unit 13, and an analysis result transmission unit 14. The cloud server 20 includes a database 21, a data extraction unit 22, a server-side analysis unit 23, and an analysis result display unit 24.

[0050] The data acquisition unit 11 acquires dynamic image data from the camera 30.

[0051] The edge-side analysis unit 12 and the server-side analysis unit 23 each perform their respective functions, analyzing the behavior of the elderly person 40 reflected in the dynamic image data. That is, the edge-side analysis unit 12 takes the dynamic image data as input and generates pre-processed data for analysis. The analysis result extraction unit 13 extracts the data required for analysis processing by the server-side analysis unit 23 from the pre-processed data output by the edge-side analysis unit 12, and transmits the extracted data to the cloud server 20 via the analysis result transmission unit 14.

[0052] The cloud server 20 stores the data transmitted from the analysis result transmission unit 14 in the database 21 for each individual (each AI edge processor 10). The data extraction unit 22 extracts the data required for analysis and processing by the server-side analysis unit 23 from the data of each individual stored in the database 21.

[0053] The server-side analysis unit 23 takes the data extracted by the data extraction unit 22 as input, analyzes the behavior of the elderly 40 (performs post-processing), and displays the results on the analysis result display unit 24.

[0054] Figure 2 This is a configuration diagram showing the first example of the division of functions between the edge-side analysis unit 12 and the server-side analysis unit 23.

[0055] The edge-side analysis unit 12 has a posture analysis unit 121 and a state analysis unit 122. The server-side analysis unit 23 has a behavior analysis unit 231.

[0056] The posture analysis unit 121 analyzes data representing the posture of the elderly person 40 at various times based on the data obtained from photographing the elderly person 40. For example, the posture of the elderly person 40 while walking changes over time in the order of standing with the left foot extended forward → standing with the right foot extended forward → standing with the left foot extended forward… (details are as follows). Figure 8 ).

[0057] In this embodiment, skeletal data is used as an example to represent posture. The skeletal data consists of the positional data of the 17 joints of the human body (nose, right eye, left eye, right ear, left ear, right shoulder, left shoulder, left hand, right hand, left wrist, right wrist, left waist, right waist, left knee, right knee, left ankle, and right ankle).

[0058] The state analysis unit 122 analyzes (classifies) the data representing the state of the elderly person 40 based on the time changes in posture analyzed by the posture analysis unit 121.

[0059] For example, when the posture changes in the order of standing with the left foot extended forward → standing with the right foot extended forward → standing with the left foot extended forward… the state analysis unit 122 classifies the state of the elderly person 40 as a walking state.

[0060] Alternatively, when the state changes in the order of standing but leaning forward → right hand on the ground → whole body lying on the floor, the state analysis unit 122 classifies the state of the elderly person 40 as a fall.

[0061] Additionally, at least one of the pose analysis unit 121 and the state analysis unit 122 is installed as a TensorRT model operating on the AI ​​edge processor 10 (Jetson Nano), thereby enabling high-speed inference processing. TensorRT is a software development kit (SDK) provided by NVIDIA for rapidly performing deep learning inference.

[0062] The creation process of a TensorRT model involves the following steps.

[0063] (Step 1) In the dynamic image data collected for learning, the human (supervisor) assigns labels to the results of detecting and tracking the human, and labels to the classification results of the tracked human behavior (annotations).

[0064] (Step 2) Based on the dynamic image data from Step 1, perform machine learning (training) to create a learned model. Furthermore, the learned model is updated through model evaluation.

[0065] (Step 3) Convert (transfer) the learning model from Step 2 to a TensorRT model to create a high-speed inference model based on TensorRT.

[0066] (Step 4) Configure the TensorRT model from Step 3 on the edge-side analysis unit 12. As a result, by inputting the dynamic image data for inference into the TensorRT model, the processing of detecting and tracking people reflected in the dynamic image and classifying the poses or behaviors of the tracked people can be accelerated.

[0067] The behavior analysis unit 231 analyzes data useful to the elderly person 40 based on the status analyzed by the state analysis unit 122. That is, the behavior analysis unit 231 monitors the health information of the elderly person 40 based on data in the database 21 and generates information helpful for care. For example, for the elderly person 40 classified as having a walking status by the state analysis unit 122, the behavior analysis unit 231 notifies the elderly person 40 of the analysis results of their walking history (details are as follows). Figures 10-12 ).

[0068] Alternatively, behavioral analysis department 231 may, based on the classification of elderly person 40 by state analysis department 122 as having fallen, notify a nearby nurse requesting assistance (details to follow). Figures 14-16 ).

[0069] Figure 3 This is a configuration diagram representing the second example of the division of functions between the edge-side analysis unit 12 and the server-side analysis unit 23.

[0070] The edge-side analysis unit 12 has a posture analysis unit 121. The server-side analysis unit 23 has a state analysis unit 232 and a behavior analysis unit 231. The behavior analysis unit 231 has the same function as the state analysis unit 122.

[0071] That is, with Figure 2 Compared to the first case, in Figure 3 In the second example, the state analysis unit 122 of the edge-side analysis unit 12 is moved to the behavior analysis unit 231 of the server-side analysis unit 23. Therefore, in the second example, by reducing the processing burden on the AI ​​edge processor 10, even low-performance computers such as wearable devices can be used with the AI ​​edge processor 10. On the other hand, in Figure 2 In the first example, by adding preprocessing, the burden on cloud server 20 can be reduced.

[0072] Figure 4 This is a configuration diagram showing a specific example of the edge-side analysis unit 12.

[0073] Within the state analysis unit 122 of the AI ​​edge processor 10, there are walking state analysis units 122A and fall state analysis units 122B, which serve as analysis units for each state. The illustration of the analysis result transmission unit 14 in the AI ​​edge processor 10 is omitted.

[0074] Walking status analysis unit 122A analyzes whether the elderly person 40 is in a walking state. Fall status analysis unit 122B analyzes whether the elderly person 40 is in a fall state.

[0075] In this way, by incorporating multiple state analysis units within a single AI edge processor 10, the configuration and transportation of the AI ​​edge processor 10 become easier. Furthermore, within a single AI edge processor 10, the walking detection mode that activates the walking state analysis unit 122A and the fall detection mode that activates the fall state analysis unit 122B can be switched via a switch operation or similar means.

[0076] Figure 5 It means Figure 4 A structural diagram of a modified example of the edge-side analysis section 12.

[0077] exist Figure 5 In this configuration, the AI ​​edge processor 10A, which houses the walking state analysis unit 122A, and the AI ​​edge processor 10, which houses the fall state analysis unit 122, are distributed in different devices.

[0078] Then, AI edge processors 10A and 10B respectively send the status of the elderly person 40 they detected to cloud server 20. Cloud server 20 integrates multiple statuses of the same elderly person 40 and registers them in database 21.

[0079] In this way, by housing the analysis unit for each state in a different AI edge processor 10, each AI edge processor 10 can detect the state it is responsible for with high precision.

[0080] For example, the AI ​​edge processor 10A is installed in the lobby or common areas of facilities for the elderly. Thus, the walking status analysis unit 122A can analyze the walking status of elderly people walking within the camera's field of view.

[0081] On the other hand, the AI ​​edge processor 10B is installed in a private room or shared space where the elderly reside. Therefore, the fall analysis unit 122B is able to analyze the fall status of the elderly within the camera's field of view.

[0082] The analysis results from AI edge processors 10A and 10B are transmitted to cloud server 20 via network 50. This allows cloud server 20 to analyze the condition of the elderly.

[0083] Figure 6 This is a flowchart showing the actions of the behavior analysis system 100 when focusing on the walking state analysis unit 122A.

[0084] The data acquisition unit 11 acquires 2D images of a person (elderly person 40) captured by the camera 30 (S11).

[0085] The pose analysis unit 121 detects a person from the 2D image in S11 and tracks the person within the 2D image (S12). Then, the pose analysis unit 121 extracts the pose of the person detected in S12 at each moment (S13).

[0086] The state analysis unit 122 (walking state analysis unit 122A) extracts the 3D distance (distance in three-dimensional space, details of which are as follows) from the 2D position of the person at a certain moment to the 2D position of the person at the next moment, based on the 2D position of the posture extracted in S13. Figure 8 (S14).

[0087] The state analysis unit 122 (walking state analysis unit 122A) calculates the walking speed and stride length of the same person based on the 3D distance in S14 (S15). The analysis result extraction unit 13 extracts information used to identify the person detected in S12, such as facial information representing facial features, as person information, and extracts the walking speed and stride length in S15 as walking information (S16).

[0088] The analysis result transmission unit 14 transmits the analysis results (person information, walking information) extracted in S16 to the cloud server 20 via the network 50 (S17). In addition, the posture in S13 may also be included in the analysis results.

[0089] The cloud server 20 receives the analysis results (person information, walking information) from S17 and stores them in the database 21 (S21).

[0090] The data extraction unit 22 extracts the area (the area of ​​the person) that is reflected in the 2D image represented by the person information read from the database 21, and extracts walking information from the database 21 (S22).

[0091] The behavior analysis unit 231 identifies the person reflected in the area of ​​the person in the 2D image extracted in S22 based on the information reflected in the area of ​​the person, and determines the ID used to identify the person as the identification result (S23).

[0092] The behavior analysis unit 231 performs a walking analysis (S24) on each ID identified in S23, based on walking information (walking speed, stride), to measure the distance walked, average speed, acceleration, walking ability, etc., within a specified period. Furthermore, walking ability is calculated, for example, using the TUG (Timed Up & Go Test), an indicator of Musculoskeletal Ambulation Disability Symptom Complex (MADS).

[0093] The behavior analysis unit 231 outputs the analysis results of S24 to the analysis result display unit 24 (S25). An example of the output content is given below.

[0094] • The behavior analysis unit 231 analyzes the results of walking information (walking history) and outputs the history of walking characteristics and the distribution of walking features to the analysis results display unit 24.

[0095] • Based on the walking ability analyzed by the behavior analysis unit 231, the predicted data of health risks are output to the analysis results display unit 24.

[0096] Figure 7 This is an explanatory diagram of the processing of human detection from 2D images (S12).

[0097] Camera 30 captured images of a person walking from left to right in the diagram, in the order of time points t1, t2, and t3. The posture analysis unit 121 detected the position of the person at each time point t1, t2, and t3 (represented by a quadrilateral around the person in the diagram) and tracked the person.

[0098] Figure 8 Yes Figure 7 An explanatory diagram of the processing (S13) of human pose extraction detected in the image.

[0099] The posture analysis unit 121 extracts the posture data of the character detected at each time t1, t2, t3 by setting each joint as a black point and the line connecting these black points as a skeleton.

[0100] Additionally, as S14, the state analysis unit 122 (walking state analysis unit 122A) calculates the 3D distance d12 from the character's position at time t1 to the character's position at time t2. Similarly, the state analysis unit 122 calculates the 3D distance d23 from the character's position at time t2 to the character's position at time t3.

[0101] Then, as S15, the state analysis unit 122 calculates the person's walking speed based on the 3D distances d12 and d23 and the elapsed time at times t1, t2, and t3. Furthermore, the state analysis unit 122 calculates the person's stride length w1 at a certain time t1.

[0102] Additionally, as S16, the analysis result extraction unit 13 extracts the facial image f1 or facial information of the person as person information for determining the detected person. Facial information refers to authentication information extracted from the facial image, such as feature information for facial authentication, iris information of the eyes, etc.

[0103] Figure 9 This is a table representing the data processed by the AI ​​edge processor 10.

[0104] Table 201 is the data generated inside the AI ​​edge processor 10 (S11-S15), and Table 202 is the data transmitted by the analysis result transmission unit 14 to the cloud server 20 based on Table 201 (S16, S17).

[0105] Table 201 will Figure 6 The camera images in S11, the human detection results in S12, the posture information (2D position) in S13, the 3D distance in S14, and the walking information in S15 are correlated.

[0106] The camera images of S11 are acquired by the data acquisition unit 11 as JPG (Joint Photographic Experts Group) images and PNG (Portable Network Graphics) images.

[0107] The detection result for each person in S12 is a combination of data: a bounding box representing the person's region and the person's ID. Furthermore, at time S12, the person's ID is not used to identify who the person is, but rather to track where the person will move to 10 seconds later, i.e., to distinguish them from other people.

[0108] The pose information of S13 is based on a total of 17 joints of the character, representing the 2D position (vertical position u, horizontal position v) within the image.

[0109] The 3D distance of S14 is the information of replacing the position information of each of the 17 joints of the character from the 2D position (u, v) in the image with the 3D position (x, y, z) in the actual space. The time difference of the 3D position is divided into 3D distance.

[0110] S15 walking information includes, for example, a person's stride or walking speed.

[0111] Table 202 will Figure 6 The information about the person in S16, the walking information and the posture information in S16 (2D position (u, v) in S13, 3D distance (x, y, z) in S14) are mapped together.

[0112] S16's personal information, such as... Figure 8 The identification information for the elderly person f1 includes facial features (outline, mouth size, etc.) and clothing features (color, type, etc.).

[0113] S16 walking information includes, for example, the person's status = walking, stride = 65 [cm], speed = 1.0 [m / s], and acceleration = +0.1 [m / s squared].

[0114] Figure 10This is the first screen showing the output of the walking analysis results (S25).

[0115] Display screen 211 has a timeline graph with the horizontal axis set to the date and the vertical axis set to the values ​​of walking information, and an "Image" button 213B for starting the reproduction of image data corresponding to the date on the horizontal axis.

[0116] In the time-series graph, walking information such as stride length, walking speed, and acceleration of a person (S15) are displayed in a differentiated manner (by using different colors or types of graph lines). By confirming the graph and pressing the "Image" button 213B for the date of interest, the user can confirm the image data of the person taken on that date.

[0117] Figure 11 This is the second screen showing the output of the walk analysis results (S25).

[0118] Display screen 212 shows, in radar chart form, the various abilities (judgment, memory, attention) of a person whose walking ability has been analyzed by the behavior analysis unit 231. Each ability is evaluated on a five-level scale from 1 to 5, with higher values ​​indicating higher abilities. In addition, the results of pre-measured ability values ​​other than walking ability are stored in database 21.

[0119] As shown in screen 212, by properly assessing and prompting the elderly person’s ability to live independently, it is possible to appropriately identify the frailty state that is about to become a state requiring care.

[0120] Figure 12 This is the third screen showing the output of the walk analysis results (S25).

[0121] Display screen 220 includes a timing graph 221 of walking ability analyzed by behavior analysis unit 231 and a recommendation message 222 proposed by behavior analysis unit 231 based on changes in walking ability in timing graph 221.

[0122] Therefore, by urging older adults to undergo appropriate rehabilitation treatment before they reach a state requiring care and are at risk of falls or fractures, it is possible to help older adults maintain their dignity and independence.

[0123] Figure 13 This is a flowchart illustrating the actions of the behavior analysis system 100 when focusing on the fall state analysis unit 122B. (Regarding...) Figure 6 (Focusing on the flowchart of the walking state analysis unit 122A) The same process is used, the same symbols are used and the explanation is omitted.

[0124] The processing of AI edge processor 10 (S11-S13) and Figure 6The same applies as described in the previous section. The pose analysis unit 121 detects the person from the 2D image acquired in S11 and tracks the person within the 2D image (S12). Then, the pose analysis unit 121 extracts the poses of the person detected in S12 at each moment (S13).

[0125] The state analysis unit 122 identifies (classifies) the state of the person based on the temporal changes in the person's posture extracted in S13 (S14B).

[0126] The fall status analysis unit 122B determines whether the state of the person identified in S14B is a fall state (S15B). If it is a fall state (yes in S15B), it proceeds to S16B; if it is a sleeping state or other state (no in S15B), it returns to S11.

[0127] Analysis result extraction unit 13 extracts the results from the 2D image of S11 and... Figure 6 The same information as S16 regarding the person and information about the person in a fallen state (fall information) (S16B). Fall information refers to, for example, whether a person has fallen at any given moment, and if so, the time elapsed since falling (the time elapsed from when the person fell without getting up).

[0128] The analysis result transmission unit 14 transmits the person information and fall information (which may also include the posture information of S13) of S16B to the cloud server 20 via the network 50 (S17B).

[0129] The cloud server 20 receives the analysis results (person information, fall information) from S17B and stores them in the database 21 (S21B).

[0130] The data extraction unit 22 extracts the area (the area of ​​the person) that is reflected in the 2D image represented by the person information read from the database 21, and extracts the fall information from the database 21 (S22B).

[0131] The behavior analysis unit 231 identifies the person reflected in the area of ​​the person extracted in S22 based on the information reflected in the 2D image, and determines the ID (and related information) used to identify the person as the identification result. Figure 6 (Same as S23).

[0132] The Behavior Analysis Unit 231 uses the fall information extracted in S22B to analyze the urgency of rescuing the person who fell in S23 (S24B). The longer the fall lasts, the higher the urgency calculated by the Behavior Analysis Unit 231. In addition, the fewer other people around the fallen person who need help (for example, the more likely it is that a person has fallen in a single room), the higher the urgency calculated by the Behavior Analysis Unit 231.

[0133] The behavior analysis unit 231 outputs the analysis results of S24B to the analysis result display unit 24 (S25B). Furthermore, the higher the urgency, the greater the degree of alarm information output. A high degree of output means emphasizing the display on the screen, or reproducing the alarm sound (alarm) based on the screen display. In addition to the cloud server 20, the AI ​​edge processor 10 and the terminals of the elderly person's family members are also added as output destinations for the alarm.

[0134] Figure 14 This is the first screen image representing the alarm output (S25B) based on fall analysis.

[0135] The screen 220A includes a schematic diagram 221A showing the positional relationship between the elderly person 40 who has fallen and other people around them, and an output screen 222A showing alarm information as the analysis result of the behavior analysis unit 231.

[0136] In the case of the schematic diagram 221A, the AI ​​edge processor 10, which analyzes the behavior of the elderly person A, detects that the elderly person A has fallen, but also detects that a nurse is nearby in the same room. Therefore, since it can be expected that the nurse will immediately help the elderly person A, the behavior analysis unit 231 outputs the analysis result of "no notification required" to the alarm information output column 222A.

[0137] Figure 15 This is the second screen image representing the alarm output (S25B) based on fall analysis.

[0138] Image 220B and Figure 14 Similarly, it includes a schematic diagram section 221B and an alarm information output section 222B.

[0139] In the scenario described in diagram 221B, the AI ​​edge processor 10 not only detected the fall of elderly person A, but also detected the fall of nearby elderly person A. Meanwhile, the presence of nurses waiting near the rooms where elderly persons A and B were located was also pre-registered in database 21.

[0140] Therefore, in order to redirect nurses to assist elderly individuals A and B, the behavior analysis unit 231 outputs the analysis result "Elderly individuals A and B have fallen. Automatically notify nearby nurses" to the alarm information output field 222B. Additionally, the behavior analysis unit 231 sends this automatic notification to the terminal held by the nurse.

[0141] Figure 16 This is the third screen showing the alarm output (S25B) based on fall analysis. Screen 220C and... Figure 14 Similarly, it includes a schematic diagram section 221C and an alarm information output section 222C.

[0142] In the case of schematic diagram 221C, the AI ​​edge processor 10 detects that elderly person A has fallen in a single room. On the other hand, the database 21 records that a nurse is not near the room where elderly person A is located.

[0143] Therefore, in order to notify the family members of elderly person A of the emergency, the behavior analysis unit 231 outputs the analysis result "Elderly person A has fallen, and there is no nurse nearby. Please contact emergency services immediately" to the alarm information output field 222C. Thus, the family members of elderly person A can quickly request emergency medical assistance.

[0144] The above, as in Figures 14-16 As explained, even if the same situation of "elderly person A has fallen" is detected by an AI edge processor 10, the presence or absence of alarm information and the content of the output can be changed based on the relationship with other people around them. This is because the server-side analysis unit 23 takes into account both the elderly person A's fall state and the state of the people around them when determining the alarm information.

[0145] Therefore, as a cloud service, in addition to the AI ​​edge processor 10 analyzing the composition of each elderly person's fall state separately, the cloud server 20 also pre-builds the database 21 to manage the composition of the states of multiple people, which is effective.

[0146] Figure 17 This is a hardware configuration diagram of the various devices (AI edge processor 10, cloud server 20) of the behavior analysis system 100.

[0147] The devices of the behavior analysis system 100 are configured as a computer 900 having a CPU 901, RAM 902, ROM 903, HDD 904, communication I / F 905, input / output I / F 906 and media I / F 907.

[0148] Communication I / F 905 is connected to an external communication device 915. Input / output I / F 906 is connected to input / output device 916. Media I / F 907 reads and writes data from recording medium 917. Furthermore, CPU 901 controls each processing unit by executing a program (also called an application program or its abbreviation) read into RAM 902. This program can also be distributed via communication lines or recorded on recording medium 917 such as CD-ROM.

[0149] In the embodiment described above, the load of cloud services that directly transmit large-capacity camera 30 image data from AI edge processor 10 to cloud server 20 for full analysis and processing is reduced. That is, the edge-side analysis unit 12 of AI edge processor 10 preprocesses the image data and reduces the data volume before transmitting it to cloud server 20, where the server-side analysis unit 23 performs post-processing.

[0150] This enables cloud services that can achieve real-time and remote sensing of a person's physical condition at low cost and with low load.

[0151] In addition, the load can be distributed in the AI ​​edge processor 10 and cloud server 20 in the following ways.

[0152] • Since it transmits pre-processed data with reduced data volume instead of large-capacity image data to network 50, it can reduce communication load.

[0153] The AI ​​edge processor 10 does not handle all the analysis and processing, but only a portion of the pre-processing, so it can operate even in small and low-performance computers such as wearable devices or smartphones.

[0154] The cloud server 20 receives data that has been preprocessed by the AI ​​edge processor 10 and only needs to perform post-processing, thus reducing the processing load.

[0155] Symbol Explanation

[0156] 10 AI Edge Processors (Edge Devices)

[0157] 11 Data Acquisition Department

[0158] 12 Edge-side Analysis Section

[0159] 13. Analytical Results Extraction Section

[0160] 14 Analysis Results Transmission Department

[0161] 20. Cloud Servers (Server Devices)

[0162] 21 Databases

[0163] 22 Data Extraction Department

[0164] 23 Server-side Analysis Department

[0165] 24. Analysis results show that...

[0166] 30 cameras

[0167] 40-year-old (person)

[0168] 50 Networks

[0169] 100 Behavioral Analysis System

[0170] 121 Posture Analysis Department

[0171] 122 Status Analysis Department

[0172] 122A Walking Status Analysis Department

[0173] 122B Fall Status Analysis Department

[0174] 231 Behavioral Analysis Department

[0175] 232 Status Analysis Department.

Claims

1. A behavior analysis system comprising an edge device for analyzing human behavior and a server device for processing data transmitted from the edge device, characterized in that... The edge device includes: The edge-side analysis unit performs preprocessing in the analysis process of analyzing the behavior of the person based on the data obtained from filming the person. The preprocessing includes: analyzing data representing the posture of the person at various times based on the data obtained from filming the person; and analyzing data representing the state of the person based on the time changes of the analyzed posture. The data representing the state of the person includes the falling state of the person and the falling time in the falling state. The analysis result extraction unit, based on the analysis results of the edge-side analysis unit, performs data extraction processing in the analysis process, including data used to determine the person; and The analysis result transmission unit transmits the analysis result data from the edge-side analysis unit and the data extracted by the analysis result extraction unit to the server device. The server device includes: A database that stores data transmitted from the analysis result transmission unit for each of the aforementioned individuals; and The server-side analysis unit performs post-processing based on the data in the database. This post-processing includes calculating the urgency of rescue based on the person's fall status and time, and whether there are other people nearby offering assistance. It then generates an alarm message corresponding to this urgency as useful data for the person and displays the alarm message on the analysis results display unit. The fewer other people around the person offering assistance, the higher the urgency of the calculation; the longer the fall lasts, the higher the urgency of the calculation.

2. The behavior analysis system according to claim 1, characterized in that, The data representing the person's state also includes the person's walking state, and the walking speed and stride length in that walking state as walking information. Data useful to the person also includes at least one of the person's walking distance, average speed, and walking ability during the specified period.

3. The behavior analysis system according to claim 2, characterized in that, The server-side analysis unit displays a graph showing the time-varying changes in the analysis results of the walking information and a corresponding recommendation message on the analysis results display unit.

4. The behavior analysis system according to claim 1, characterized in that, The server-side analysis unit displays a schematic diagram on the analysis result display unit showing the positional relationship between the person who has fallen and other people around them, as well as alarm information corresponding to the positional relationship.

5. A behavior analysis method executed by a behavior analysis system, the behavior analysis system comprising an edge device for analyzing human behavior and a server device for processing data transmitted from the edge device, the edge device including an edge-side analysis unit, an analysis result extraction unit, and an analysis result transmission unit, the server device including a database and a server-side analysis unit, the behavior analysis method being characterized in that: The edge-side analysis unit performs preprocessing in the analysis process of analyzing the behavior of the person based on the data obtained from photographing the person. This preprocessing includes: Analyze the data obtained from photographing the person to represent the person's posture at various times; And based on the analyzed changes in posture over time, data representing the state of the person is analyzed, including the state of the person falling and the time of the fall in that state. The analysis result extraction unit performs data extraction, which includes information used to identify the person, based on the analysis results from the edge-side analysis unit. The analysis result transmission unit transmits the analysis result data from the edge-side analysis unit and the data extracted by the analysis result extraction unit to the server device. The database stores data transmitted from the analysis result transmission unit for each of the aforementioned individuals. The server-side analysis unit performs post-processing based on the data in the database. This post-processing includes calculating the urgency of rescuing the person based on their fall status and time, and whether there are other people nearby offering assistance. It then generates an alarm message corresponding to this urgency as useful data for the person and displays the alarm message on the analysis results display unit. The fewer other people around the person offering assistance, the higher the urgency of the calculation; the longer the fall lasts, the higher the urgency of the calculation.

Images