A person analysis system, a person analysis method, and a person analysis program.

Abstract

Even when multiple individuals are present in the monitoring area, the system efficiently monitors individuals and their biometric information or status. [Solution] The person analysis system comprises a radar device installed in the monitoring area at least once and outputting observation data including the results of observing the monitoring area using a radar system, and a person analysis device that performs the following based on the observation data: detecting the location of each of the multiple people present in the monitoring area and estimating the vital signs of each of the detected multiple people, and displays an area map corresponding to the monitoring area on a predetermined display device, displays pictograms at each of the detected multiple people on the area map, and displays information regarding the vital signs or behavior of a representative person among the multiple people.

Description

Technical Field

[0001] The present disclosure relates to a person analysis system, a person analysis method, and a person analysis program.

Background Art

[0002] As a sensor capable of detecting a person present in a room while protecting privacy, a sensor that detects a person using microwaves is known.

[0003] In Patent Document 1, using an FMCW (Frequency Modulated Continuous Wave) sensor, the sum of the intensities of at least some of the frequency components in the frequency spectrum of the generated beat signal is calculated, and when the sum of the intensities is smaller than a predetermined value, it is determined that an object is present in the room. Also, based on the spectral distribution of the generated beat signal, the position of the person is detected and the state of the person (for example, the state where the person has fallen) is estimated.

[0004] In Patent Document 2, a vital sensor that transmits a modulated wave in the microwave band or millimeter wave band, compares the transmitted wave with the received wave reflected from an object, and detects minute fluctuations such as the respiration and heartbeat of a living body is disclosed.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] Conventional technologies allow for highly accurate person detection and acquisition of biometric information such as respiration or heart rate, or information regarding a person's state, while maintaining privacy. Furthermore, in monitoring spaces where multiple people are expected to be present, it is conceivable to display multiple biometric data points (vital data) corresponding to each individual. In such cases, as the number of people present increases, it becomes more difficult to verify the biometric information or status information corresponding to each person. Therefore, it is desirable to be able to change the display format to one preferred by the user, thereby enabling more efficient monitoring of people and biometric information. However, conventional technologies have not taken any consideration of these needs.

[0007] Therefore, this disclosure aims to provide a technology that enables efficient monitoring of individuals and their biometric information or status, even when multiple individuals are present in the monitoring area. [Means for solving the problem]

[0008] One aspect of the present disclosure provides a person analysis system comprising: a radar device installed in a monitoring area and outputting observation data including the results of observing the monitoring area by radar; and a person analysis device that performs the following based on the observation data: detecting the location of each of a plurality of persons present in the monitoring area and estimating the vital signs of each of the detected persons; displaying an area map corresponding to the monitoring area on a predetermined display device; displaying pictograms at the locations of each of the detected persons on the area map; and displaying information regarding the vital signs of a representative person among the plurality of persons.

[0009] One aspect of the present disclosure provides a person analysis method that includes acquiring observation data, including the results of observing the monitoring area by radar, from at least one radar device installed in the monitoring area; performing a process to detect the location of each of a plurality of persons present in the monitoring area based on the observation data; and estimating the vital signs of each of the detected persons; displaying an area map corresponding to the monitoring area on a predetermined display device; displaying pictograms at each of the detected persons on the area map; and displaying information regarding the vital signs of a representative person among the plurality of persons.

[0010] One aspect of the present disclosure provides a person analysis program that causes a computer to perform the following actions: acquire observation data including the results of observing the monitoring area by radar from at least one radar device installed in the monitoring area; perform the following based on the observation data: detect the location of each of the multiple persons present in the monitoring area; estimate the vital signs of each of the detected multiple persons; display an area map corresponding to the monitoring area on a predetermined display device; display pictograms at the locations of the detected multiple persons on the area map; and display information regarding the vital signs of a representative person among the multiple persons.

[0011] One aspect of the present disclosure provides a person analysis system comprising: a radar device installed in a monitoring area and outputting observation data including the results of observing the monitoring area by radar; and a person analysis device that performs the following based on the observation data: detecting the location of each of a plurality of persons present in the monitoring area and estimating the actions of each of the detected persons, and displays an area map corresponding to the monitoring area on a predetermined display device, displays pictograms at the locations of each of the detected persons on the area map, and displays information regarding the actions of a representative person among the plurality of persons.

[0012] One aspect of the present disclosure provides a person analysis method that includes acquiring observation data, including the results of observing the monitoring area by radar, from at least one radar device installed in the monitoring area; performing a process to detect the location of each of a plurality of persons present in the monitoring area based on the observation data; and performing a process to estimate the actions of each of the detected persons; displaying an area map corresponding to the monitoring area on a predetermined display device; displaying pictograms at each of the detected persons on the area map; and displaying information regarding the actions of a representative person among the plurality of persons.

[0013] One aspect of the present disclosure provides a person analysis program that causes a computer to perform the following actions: acquire observation data including the results of observing the monitoring area by radar from at least one radar device installed in the monitoring area; perform the following based on the observation data: detect the location of each of the multiple persons present in the monitoring area; estimate the actions of each of the detected multiple persons; display an area map corresponding to the monitoring area on a predetermined display device; display pictograms at each of the detected multiple persons on the area map; and display information regarding the actions of a representative person among the multiple persons.

[0014] These comprehensive or specific embodiments may be implemented as systems, devices, methods, integrated circuits, computer programs, or recording media, or as any combination of systems, devices, methods, integrated circuits, computer programs, and recording media. [Effects of the Invention]

[0015] According to this disclosure, even when multiple people are present in the monitoring area, it is possible to efficiently monitor individuals and their biometric information. [Brief explanation of the drawing]

[0016] [Figure 1] A diagram showing an example configuration of the person analysis system according to this embodiment. [Figure 2] Diagram showing a display example of the display device according to this embodiment [Figure 3] Diagram showing a configuration example of the person analysis device according to this embodiment [Figure 4] Diagram showing an example of information included in each data according to this embodiment [Figure 5] Diagram showing a first modification example of the display of the display device according to this embodiment [Figure 6] Diagram showing a second modification example of the display of the display device according to this embodiment [Figure 7] Diagram showing a hardware configuration example of an information processing device (computer) that realizes the functional blocks of the person analysis device according to the present disclosure by a computer program

Embodiments for Carrying Out the Invention

[0017] Hereinafter, embodiments of the present disclosure will be described in detail with appropriate reference to the drawings. However, a more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters and duplicate descriptions of substantially the same configurations may be omitted. This is to avoid making the following description unnecessarily redundant and to facilitate the understanding of those skilled in the art. Note that the attached drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0018] (This embodiment) <Configuration of the person analysis system> FIG. 1 is a diagram showing a configuration example of a person analysis system 10 according to this embodiment. FIG. 2 is a diagram showing a display example of the display device 41 according to this embodiment.

[0019] As shown in FIG. 1, the person analysis system 10 includes a radar device 11, a person analysis device 20, an input device 40, a display device 41, a transmitting device 42, and a user terminal 43.

[0020] The person analysis device 20 may be able to send and receive data with at least one radar device 11 via the communication network 14. The person analysis device 20 may also be able to send and receive information with at least one user terminal 43 via the communication network 14. Examples of the communication network 14 include a wired LAN (Local Area Network), wireless LAN (Wi-Fi), the Internet, or a mobile communication network. Examples of the user terminal 43 include a smartphone, tablet, smartwatch, or mobile PC.

[0021] The person analysis device 20 is connected to an input device 40, a display device 41, and an alarm device 42. Examples of the input device 40 include a keyboard, mouse, touchpad, microphone, or tablet terminal. Examples of the display device 41 include a liquid crystal display, organic EL display, or tablet terminal. Examples of the alarm device 42 include a stacked signal light, indicator light, rotating light, or speaker.

[0022] The radar device 11 is an example of a radar-based sensor and is installed in the monitoring area 1 at least once. The radar device 11 observes the position of a person 2 or object (not shown) in the monitoring area 1 by irradiating radio waves (radar) and receiving the reflected waves, and generates observation data including the observed results. For example, the radar device 11 transmits radio waves (radar) modulated with FMCW (Frequency Modulated Continuous Wave) in the millimeter wave band (e.g., 60 GHz band or 79 GHz band) using a MIMO (Multi Input Multi Output) antenna. The radar device 11 then receives the reflected waves of the transmitted radio waves reflected by the person or object using the MIMO antenna. Based on the difference signal between the transmitted signal and the received signal, the radar device 11 generates a beat signal, which is an example of observation data. The beat signal may be configured as IQ (In-phase / Quadrature-phase) data. The radar device 11 transmits IQ data to the person analysis device 20. The radar device 11 may also be a LiDAR (Light Detection and Ranging) system.

[0023] The person analysis device 20 performs three processes based on observation data received from the radar device 11: a first process to detect the location of each of the multiple persons 2 present in the monitoring area 1; a second process to estimate the actions of each of the detected persons 2; and a third process to estimate the vital signs of each of the detected persons 2. As shown in Figure 2, the person analysis device 20 displays an area map 51 corresponding to the monitoring area 1 on the display device 41, displays pictograms 60 indicating the estimated actions of each of the detected persons 2 at their respective locations on the area map 51, and displays information regarding the vital signs of a representative person 2A among the multiple persons 2. This allows a user viewing the display device 41 to see at a glance where each person 2 present in the monitoring area 1 is located and what actions they are taking, as well as information regarding the vital signs of the representative person 2A. Details of the person analysis device 20 and the information displayed on the display device 41 as exemplified in Figure 2 will be described later.

[0024] The following example describes a scenario where the "monitoring area" is a warehouse, the "person" is a worker in the warehouse, and the "action" is a typical task performed by the worker in the warehouse. Examples of actions include transporting goods (hereinafter referred to as "transporting"), collecting goods (hereinafter referred to as "collecting"), packing goods (hereinafter referred to as "packing"), and inspecting goods (hereinafter referred to as "inspection"). Furthermore, actions may include "walking," "standing," "sitting," "falling," "lying down," etc.

[0025] In this case, as shown in Figure 2, the person analysis device 20 displays a pictogram 60 indicating the actions of person 2 at the location where person 2 is located on the warehouse area map 51. For example, if the person analysis device 20 estimates that person 2's action is "transportation," it displays a pictogram 60A indicating "transportation" at the location where person 2 is located on the warehouse area map 51. If the person analysis device 20 estimates that person 2's action is "collection," it displays a pictogram 60B indicating "collection" at the location where person 2 is located on the warehouse area map 51. If the person analysis device 20 identifies that person 2's action is "packaging," it displays a pictogram 60C indicating "packaging" at the location where person 2 is located on the warehouse area map 51. If the person analysis device 20 identifies that person 2's action is "inspection," it displays a pictogram 60D indicating "inspection" at the location where person 2 is located on the warehouse area map 51.

[0026] Furthermore, if the person analysis device 20 estimates that a person 2's behavior is "falling," it may display a pictogram 60 (not shown) indicating "falling" at the location of person 2 on the area map 51. If the person analysis device 20 estimates that a person 2's behavior is "lying down," it may display a pictogram 60 (not shown) indicating "lying down" at the location of person 2 on the area map 51.

[0027] This allows users viewing the display device 41 to easily see the pictograms 60 displayed on the display device 41 and determine the location and actions of each person 2 working in the warehouse. The pictograms 60 showing the actions of person 2 are not limited to these; for example, avatars or 3D models corresponding to the people could be set and displayed as animations showing their movements.

[0028] Furthermore, surveillance area 1 is not limited to warehouses; it may be, for example, a factory, shop, hospital, nursing home, elderly care facility, convenience store, department store, office, school, home, or room. Also, the type of behavior can be any typical behavior performed by a person in surveillance area 1, and the type of behavior may differ if surveillance area 1 is different.

[0029] The person analysis system 10 according to this embodiment will be described in more detail below.

[0030] <Configuration of the Human Analysis Device> Figure 3 shows an example of the configuration of the person analysis device 20 according to this embodiment. Figure 4 shows an example of the information contained in each data according to this embodiment.

[0031] The person analysis device 20 includes, as functions, an IQ data storage unit 21, a point cloud data generation unit 24, a point cloud data storage unit 25, a target data generation unit 26, a target data storage unit 27, a behavior data generation unit 28, a behavior data storage unit 29, a vital data generation unit 30, a vital data storage unit 31, and a monitoring control unit 32. As shown in Figure 5, the person analysis device 20 includes at least a processor 1001 and a memory 1002, and the functions of the person analysis device 20 described above may be realized by the processor 1001 reading a predetermined computer program from the memory 1002 and executing it.

[0032] The IQ data storage unit 21 stores the IQ data that is periodically transmitted from the radar device 11.

[0033] <Point Cloud Data Generation Unit> The point cloud data generation unit 24 generates three-dimensional point cloud data using the IQ data in the IQ data storage unit 21 and stores it in the point cloud data storage unit 25. As shown in Figure 4, the point cloud data may include the measured time and frame number, the position where the point cloud data was measured (i.e., the position of the radar device 11), the position coordinates (x, y, z) of each point, the reflection intensity of each point, and the direction and speed of movement of each point. Each point included in the point cloud data may be a point whose reflection intensity is above a predetermined threshold (i.e., a point that is presumed not to be noise), or a moving point whose speed is above a predetermined threshold. One point indicates the position coordinates (x, y, z) of a reflection point on the surface of a person 2 or an object (not shown). The direction and speed of movement of a point can be determined by repeatedly irradiating it with modulated radio waves and capturing the reflected waves from multiple directions for a certain period of time, and measuring the Doppler velocity. Also, when LiDAR is used in the radar device 11, the speed of movement of a point can be determined by modulating a laser and measuring the Doppler velocity. Information including the position coordinates of each point may be expressed as coordinate information. Information including the direction and speed of movement of each point may be expressed as motion information.

[0034] <Target Data Generation Unit> The target data generation unit 26 uses the point cloud data in the point cloud data storage unit 25 to detect the location of each person in the warehouse, the posture and orientation of each person 2, and the density of each person 2. Based on the detection results, the target data generation unit 26 generates target data and stores it in the target data storage unit 27. As shown in Figure 4, the target data may include the time of measurement, a person ID for identifying person 2, the position coordinates (x, y, z) of person 2 in the warehouse, the posture of person 2, the orientation of person 2, and the density of person 2. The density of person 2 is an indicator of the distance between person 2 and other person 2. Examples of posture types include "standing," "sitting," and "lying down." Examples of orientation types include "forward," "sideways," and "diagonally." Examples of density types include "one person," "multiple at a distance," and "multiple nearby." A density of "one person" indicates that no other person 2 is within a predetermined distance range from person 2. The "Multiple distant" density indicates that other persons 2 exist within a predetermined distance range from person 2, but the distance to these other persons 2 is above a predetermined threshold. The "Multiple close" density indicates that other persons 2 exist within a predetermined distance range from person 2, but the distance to these other persons 2 is below a predetermined threshold.

[0035] For example, the target data generation unit 26 inputs point cloud data into a 3D point cloud DNN (Deep Neural Network) and estimates the position coordinates, posture, and orientation of each person 2 based on the output from the 3D point cloud DNN. The 3D point cloud DNN may be pre-trained using multiple training data sets that associate the point cloud data with the correct position coordinates, posture, and orientation of person 2 for the distribution of the point cloud shown by the point cloud data. Examples of 3D point cloud DNNs include PointNet, VoxelNet, or PointPillars. The processing of the target data generation unit 26 may be reinterpreted as the first processing. The 3D point cloud DNN may be reinterpreted as the first neural network. The target data generation unit 26 includes the estimated position coordinates, posture, and orientation in the target data. However, the target data generation unit 26 may estimate the position coordinates, posture, and orientation of each person 2 from the point cloud data without using a DNN.

[0036] The target data generation unit 26 may assign a person ID to each person 2 located at each estimated position coordinate and include that person ID in the target data. The target data generation unit 26 may calculate the distance between each person 2 from the estimated position coordinates of each person 2 and determine the density of each person 2 based on the calculated distance. The target data generation unit 26 may then include the result of this density determination in the target data.

[0037] <Behavioral Data Generation Department> The behavior data generation unit 28 estimates the behavior of person 2 using the target data in the target data storage unit 27 and the point cloud data in the point cloud data storage unit 25. Based on the estimation result, the behavior data generation unit 28 generates behavior data and stores it in the behavior data storage unit 29. For example, as shown in Figure 4, the behavior data may include the time of measurement, a person ID for identifying person 2, the estimated behavior of person 2, and a score indicating the likelihood of the estimation (e.g., estimation accuracy or likelihood). In this embodiment, a larger score indicates a higher probability that the estimated behavior is correct, and a smaller score indicates a lower probability that the estimated behavior is correct. In this embodiment, as mentioned above, "transportation," "collection," "packaging," and "inspection" are given as examples of types of behavior.

[0038] For example, the behavior data generation unit 28 identifies the position of person 2 and the body parts of person 2 (e.g., hands, arms, head, legs, etc.) based on target data, and generates pseudo-image data corresponding to each part of the point cloud data of person 2. The behavior data generation unit 28 then inputs the pseudo-image data in time series into a recurrent DNN and estimates the type of action of person 2 based on the output results from the recurrent DNN. For example, the recurrent DNN outputs a score for each type of action as an output result, and the behavior data generation unit 28 estimates the type of action with the highest output score as the action of person 2.

[0039] The recurrent DNN may be pre-trained using multiple training datasets that associate the temporal changes of pseudo-image data with the correct human actions for those temporal changes in the pseudo-image data. Examples of recurrent DNNs include LSTM (Long Short Term Memory) or GRU (Gated Recurrent Unit). The processing of the action data generation unit 28 may be interpreted as the second processing. The recurrent DNN may be interpreted as the second neural network. However, the action data generation unit 28 may estimate the actions of person 2 from the target data in the target data storage unit 27 and the point cloud data in the point cloud data storage unit 25 without using a DNN.

[0040] <Vital Data Generation Unit> The vital data generation unit 30 estimates the vitals of person 2 using the target data in the target data storage unit 27 and the IQ data in the IQ data storage unit 21. Based on the estimation results, the vital data generation unit 30 generates vital data and stores it in the vital data storage unit 31. The processing of the vital data generation unit 30 may be interpreted as the third processing. The vital data may include, as shown in Figure 4, the time of measurement, a person ID for identifying person 2, an estimated respiratory rate, an estimated heart rate, and a score indicating the accuracy of the estimation (e.g., estimation precision or likelihood).

[0041] For example, the vital data generation unit 30 identifies the position of each person 2 based on the target data and identifies the chest area of ​​the person 2 located at that position. The vital data generation unit 30 then calculates a slight periodic motion of the body surface of person 2 based on the time change of the phase information of the portion of the IQ data corresponding to the chest area, and estimates the respiratory rate and heart rate based on that periodic motion. The vital data generation unit 30 may perform the estimation of the respiratory rate and heart rate by performing spectrum analysis and / or deep learning analysis.

[0042] For example, the vital data generation unit 30 may calculate a score for the estimated respiratory rate and / or heart rate based on the signal-to-noise ratio (SNR) of the IQ data. For example, the vital data generation unit 30 increases the score as the SNR increases and decreases the score as the SNR decreases. This is because a low SNR indicates a large noise component, which can reduce the estimation accuracy. The vital data generation unit 30 includes the calculated score in the vital data.

[0043] The vital data generation unit 30 may also estimate blood pressure, etc., using a similar method. In this case, the vital data shown in Figure 4 may include blood pressure, etc.

[0044] <Monitoring and Control Unit> The monitoring control unit 32 generates a monitoring screen 200 as shown in Figure 2, using at least one of the target data in the target data storage unit 27, the action data in the action data storage unit 29, and the vital data in the vital data storage unit 31, and displays it on the display device 41. The monitoring control unit 32 also controls the display of the monitoring screen 200 based on instructions input from the user through the input device 40. The monitoring screen 200 will be described in detail below.

[0045] As shown in Figure 2, the monitoring screen 200 has a monitoring area display area 210, a vital signs display area 220, a vital signs history display area 230, and an activity history display area 240.

[0046] The monitoring control unit 32 displays an area map 51 corresponding to the monitoring area 1 in the monitoring area display area 210.

[0047] The monitoring control unit 32 identifies the location and actions of each person 2 based on the target data stored in the target data storage unit 27 and the action data stored in the action data storage unit 29. The monitoring control unit 32 then displays a pictogram 60 indicating the action of each person 2 at the location of each person 2 on the area map 51 of the monitoring area display area 210. This allows users viewing the monitoring screen 200 to see at a glance where each person 2 is and what actions they are taking in the monitoring area 1. In particular, since the actions of person 2 are represented by pictograms 60, it is preferable to change the display form of the pictograms 60 using data on the posture of the people included in the target data.

[0048] The monitoring control unit 32 displays the pictogram 60 corresponding to the representative person 2A among the multiple pictograms 60 of people 2 displayed in the monitoring area display area 210 in an identifiable manner. For example, as shown in Figure 2, the monitoring control unit 32 displays a predetermined icon (hereinafter referred to as the representative person icon 300) above the head of the pictogram 60B corresponding to the representative person 2A. Alternatively, the monitoring control unit 32 may display the pictogram 60B corresponding to the representative person 2A in a different color from the other pictograms 60.

[0049] The monitoring control unit 32 identifies the vital data of representative person 2A based on the vital data stored in the vital data storage unit 31. The monitoring control unit 32 then displays the vital data of representative person 2A (e.g., respiratory rate and heart rate) in the vital display area 220. As a result, a user viewing the monitoring screen 200 can immediately recognize that the vital data displayed in the vital display area 220 belongs to representative person 2A, indicated by the representative person icon 300. In other words, even when multiple people 2 are present in the monitoring area 1, the monitoring of people 2 and their vital data (biometric information) can be performed efficiently.

[0050] The monitoring control unit 32 identifies the vital data history (time-dependent changes) of representative person 2A based on the vital data history stored in the vital data storage unit 31. Then, based on that vital data history, the monitoring control unit 32 displays the vital data history of representative person 2A in the vital history display area 230. For example, as shown in Figure 2, the monitoring control unit 32 displays a graph 231 showing the time-dependent changes in respiratory rate, which is an example of vital data history, in the vital history display area 230. For example, as shown in Figure 2, the monitoring control unit 32 displays a graph 232 showing the time-dependent changes in heart rate, which is another example of vital data history, in the vital history display area 230. As a result, a user viewing the monitoring screen 200 can immediately recognize that the vital data history displayed in the vital history display area 230 belongs to representative person 2A, indicated by the representative person icon 300. Furthermore, as shown in Figure 2, by displaying the time-dependent changes in vital data in a graph, users can determine, for example, whether a vital abnormality detected in representative person 2A is due to a temporary error or noise in the observation data, or whether it is a genuine vital abnormality.

[0051] The monitoring control unit 32 identifies the behavioral history of representative person 2A based on the history of target data stored in the target data storage unit 27 and the history of behavioral data stored in the behavioral data storage unit 29. Then, based on that behavioral history, the monitoring control unit 32 displays the behavioral history of representative person 2A in the behavioral history display area 240. For example, as shown in Figure 2, the monitoring control unit 32 displays a graph 241 in the behavioral history display area 240 that shows the change in behavior over time, which is an example of behavioral history. For example, as shown in Figure 2, the monitoring control unit 32 displays a graph 252 in the behavioral history display area 240 that shows the change in location (place) where representative person 2A was staying, which is an example of behavioral history. As a result, a user viewing the monitoring screen 200 can immediately recognize that the behavioral history displayed in the behavioral history display area 240 belongs to representative person 2A, indicated by the representative person icon 300. Furthermore, as shown in Figure 2, by displaying the changes in behavior and location over time in graphs 241 and 242, users can see what kind of actions representative person 2A took in different locations in the past.

[0052] The user can select a pictogram 60 to be designated as representative person 2A from among several pictograms 60 displayed in the monitoring area display region 210 of the monitoring screen 200 via the input device 40. The monitoring control unit 32 moves the representative person icon 300 above the pictogram 60 selected by the user. That is, by dragging the representative person icon 300 near the destination pictogram 60, the representative person icon 300 can be positioned in a fixed location and the display of vital data, etc., can be switched. In this way, the monitoring control unit 32 can make the user aware that the representative person 2A has been switched. Furthermore, when the representative person 2A has been switched, the monitoring control unit 32 changes the contents of the vital display region 220, the vital history display region 230, and the behavior history display region 240 to reflect the contents of the switched representative person 2A. This allows users to easily check the vital data, vital history, and behavioral history of a specific person 2, even when multiple people 2 are present in the monitoring area 1, by selecting the pictogram 60 displayed in the monitoring area display area 210. In other words, even when multiple people 2 are present in the monitoring area 1, monitoring of people 2 and their vital data (biometric information) can be performed efficiently.

[0053] The monitoring control unit 32 may automatically select a pictogram 60 to be designated as the representative person 2A from among the multiple pictograms 60 displayed in the monitoring area display area 210, based on pre-set conditions.

[0054] For example, the monitoring and control unit 32 selects the pictogram 60 of person 2 whose vital signs meet the pre-set conditions (set vital condition) as the pictogram 60 of representative person 2A.

[0055] As a set vital condition, the monitoring control unit 32 automatically selects the pictogram 60 of person 2 whose respiratory rate is outside the preset normal range (i.e., person 2 with an abnormal respiratory rate) as the pictogram 60 for representative person 2A. As a result, on the monitoring screen 200, the representative person icon 300 is switched to be placed above the pictogram 60 of person 2 with an abnormal respiratory rate, and the abnormal respiratory rate of person 2 is automatically displayed in the vital display area 220 and the vital history display area 230. If the respiratory rate is abnormal, the monitoring control unit 32 may display a predetermined mark (e.g., a "!" mark) indicating that the respiratory rate is abnormal in the vital display area 220 and / or the vital history display area 230. In addition, the monitoring control unit 32 may display the pictogram 60 of representative person 2A with an abnormal respiratory rate in a different manner (e.g., a different color) than the pictogram 60 when normal. This allows users viewing the monitoring screen 200 to quickly notice the presence of person 2 with an abnormal respiratory rate.

[0056] Furthermore, as a set vital condition, the monitoring control unit 32 automatically selects the pictogram 60 of a person whose heart rate is outside the preset normal range (i.e., person 2 with an abnormal heart rate) as the pictogram for representative person 2A. As a result, on the monitoring screen 200, the representative person icon 300 is switched to be placed above the pictogram 60 of person 2 with an abnormal heart rate, and the abnormal heart rate of person 2 is automatically displayed in the vital display area 220 and the vital history display area 230. If the heart rate is abnormal, the monitoring control unit 32 may also display a predetermined mark (for example, a "!" mark) in the vital display area 220 and / or the vital history display area 230 to indicate that the heart rate is abnormal. This allows a user viewing the monitoring screen 200 to quickly notice the presence of person 2 with an abnormal heart rate.

[0057] For example, the monitoring and control unit 32 selects the pictogram 60 of person 2 whose behavior (set behavior condition) matches a preset condition as the pictogram for representative person 2A.

[0058] As a set behavioral condition, the monitoring control unit 32 automatically selects the pictogram 60 of person 2 (i.e., person 2 with abnormal behavior) in which the pre-set behavior of "falling" has been detected, as the pictogram 60 of representative person 2A. As a result, on the monitoring screen 200, the representative person icon 300 is switched to be placed above the pictogram 60 of person 2 with abnormal behavior, and the abnormal behavior of person 2 is automatically displayed in the behavior history display area 240. The monitoring control unit 32 may also display a predetermined mark (e.g., a "!" mark) in the behavior history display area 240 to indicate that the behavior is abnormal if the behavior is abnormal. Furthermore, the monitoring control unit 32 may display the pictogram 60 of representative person 2A with abnormal behavior in a different manner (e.g., a different color) than the pictogram 60 when the behavior is normal. This allows users viewing the monitoring screen 200 to quickly notice the presence of person 2 with abnormal behavior.

[0059] For example, the monitoring and control unit 32 selects the pictogram 60 of person 2, whose presence meets pre-set conditions (set presence conditions), as the pictogram for representative person 2A.

[0060] As a set dwelling condition, the monitoring control unit 32 automatically selects the pictogram 60 of person 2 (i.e., person 2 dwelling in the monitored location) who is detected to have been dwelling in a pre-set location (such as a monitored location) for a specified time or longer, as the pictogram 60 of representative person 2A. As a result, on the monitoring screen 200, the representative person icon 300 is switched to be placed above the pictogram 60 of person 2 dwelling in the monitored location, and the location of person 2 is automatically displayed in the activity history display area 240. If the dwelling location is abnormal, the monitoring control unit 32 may display a predetermined mark (for example, a "!" mark) in the activity history display area 240 to indicate that the dwelling is abnormal. In addition, the monitoring control unit 32 may display the pictogram 60 of representative person 2A whose dwelling location is abnormal in a different manner (for example, a different color) than the pictogram 60 when it is normal. This allows users viewing the monitoring screen 200 to quickly notice the presence of person 2 whose dwelling location is abnormal.

[0061] The monitoring control unit 32 can set three conditions as pre-configured conditions: set vital conditions, set behavior conditions, and set dwell time conditions. If multiple conditions from these three conditions are detected at the same time, the control unit may control the detection according to the priority of each condition and select the pictogram 60 of person 2 detected by the priority condition as the pictogram of representative person 2A.

[0062] The monitoring control unit 32 may issue an alarm from the alarm device 42 if it detects an abnormality in the vital data, behavior, or dwell time of at least one person 2 in the monitoring area 1. If the alarm device 42 is a stacked signal light, indicator light, or rotating light, the alarm may be issued by turning on or flashing the stacked signal light, indicator light, or rotating light. If the alarm device 42 is a speaker, the alarm may be issued by outputting an audio message from the speaker indicating that an abnormality has occurred. This allows the user to quickly notice that an abnormality has occurred with person 2 staying in the monitoring area 1, even without looking at the monitoring screen 200.

[0063] Furthermore, the monitoring and control unit 32 stores records of alarm occurrences (alarm logs) in the storage device of the person analysis device 20 (memory 1002 in Figure 5), allowing the monitor to review the alarm logs afterward to verify the alarm status. This is particularly useful for verifying the alarm status when multiple alarms targeting different people occur simultaneously.

[0064] The monitoring control unit 32 may notify the user terminal 43 held by the user if it detects an abnormality in the vital data or behavior of at least one person 2 in the monitoring area 1. This allows the user to quickly become aware of an abnormality occurring in person 2 staying in the monitoring area 1, even if they are in a location where they cannot see the monitoring screen 200.

[0065] Figure 5 shows a first modified example of the display of the display device 41 according to this embodiment.

[0066] The monitoring control unit 32 may generate a monitoring screen 200 as shown in Figure 5 and display it on the display device 41. That is, the monitoring screen 200 has an action display area 221 in addition to the components shown in Figure 2. The monitoring control unit 32 displays a pictogram 60B indicating the current action (person status) of the representative person 2A in the action display area 221. When the representative person 2A is switched, the monitoring control unit 32 also switches the display in the action display area 221 accordingly. This allows for efficient monitoring of the representative person 2A's current action and vital data (biometric information).

[0067] Figure 6 shows a second modified example of the display of the display device 41 according to this embodiment.

[0068] The monitoring control unit 23 may generate a monitoring screen 200 as shown in Figure 6 and display it on the display device 41. Specifically, the monitoring screen 200 displays a vital information display area 220 in a callout format for the pictogram 60B of the representative person 2A. The monitoring control unit 32 displays the current vital data (e.g., respiratory rate and heart rate) of the representative person 2A in the callout-format vital information display area 220. When the representative person 2A is switched, the monitoring control unit 32 also switches the destination of the callout in the vital information display area 220 accordingly. This allows for efficient monitoring of the current actions of the representative person 2A and their vital data (biological information).

[0069] In the above embodiment, the representative person 2A is limited to one person, but the representative person 2A may be multiple people (for example, two people with different representative person icons 300 displayed to each other) as long as the display does not become cluttered, and they will be extracted according to the display conditions as in the case where there is one representative person 2A, and each representative person icon 300 can be manipulated.

[0070] (Hardware configuration) The functional blocks of the aforementioned human analysis device 20 can be implemented by a computer program.

[0071] Figure 7 shows an example of the hardware configuration of an information processing device (computer) that implements the functional blocks of the person analysis device 20 according to this disclosure using a computer program.

[0072] The information processing device 1000 includes a processor 1001, memory 1002, storage 1003, input interface 1004, output interface 1005, communication interface 1006, graphics processing unit (GPU) 1007, read interface 1008, and bus 1009.

[0073] The processor 1001, memory 1002, storage 1003, input I / F 1004, output I / F 1005, communication I / F 1006, GPU (Graphics Processing Unit) 1007, and read I / F 1008 are connected to bus 1009, and can send and receive data bidirectionally via bus 1009.

[0074] The processor 1001 is a device that executes the computer program stored in the memory 1002 and realizes the functional blocks described above. Examples of the processor 1001 include a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a controller, an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field-Programmable Gate Array).

[0075] Memory 1002 is a device for storing computer programs and data handled by the information processing device 1000. Memory 1002 may include ROM (Read-Only Memory) and RAM (Random Access Memory).

[0076] The storage device 1003 is composed of a non-volatile storage medium and is a device that stores computer programs and data handled by the information processing device 1000. Examples of storage devices 1003 include HDDs (Hard Disk Drives) and SSDs (Solid State Drives).

[0077] The input interface 1004 is connected to an input device 40 that receives input from the user, and transmits the data received from the input device 40 to the processor 1001.

[0078] Output I / F 1005 is connected to the display device 41 and transmits data received from the processor 1001 to the display device 41.

[0079] The communication interface 1006 is connected to the communication network 14 and transmits and receives data with other devices via the communication network 14. The communication interface 1006 may support either wired communication or wireless communication. An example of wired communication is Ethernet®. Examples of wireless communication include Wi-Fi®, Bluetooth®, LTE (Long Term Evolution), 4G, and 5G.

[0080] The GPU 1007 is a device that processes images at high speed. The GPU 1007 may also be used for AI (Artificial Intelligence) processing (for example, deep learning processing). The information processing device 1000 may also include an NPU (Neural Network Processing Unit) in place of, or together with, the GPU 1007.

[0081] The Read I / F 1008 reads data from an external storage medium that is connected to it. Examples of external storage media include DVD-ROM (Digital Versatile Disk Read Only Memory) and USB (Universal Serial Bus) memory.

[0082] Furthermore, the functional blocks of the human analysis device 20 may be implemented as integrated circuits (LSIs). These functional blocks may be individually integrated onto a single chip, or some or all of them may be integrated onto a single chip. Here, we refer to them as LSIs, but depending on the degree of integration, they may also be called ICs, system LSIs, super LSIs, or ultra LSIs. Moreover, if an integrated circuit technology that replaces LSIs emerges due to advances in semiconductor technology or other derived technologies, it is certainly possible to integrate the functional blocks using that technology.

[0083] (Summary of this embodiment) Based on the above description of embodiments, the following technologies are disclosed. <Technology 1> A person analysis system (10) according to one embodiment includes a radar device (11) installed in a monitoring area (1) that outputs observation data including the results of observing the monitoring area using a radar system, and a person analysis device (20) that performs the following based on the observation data: detecting the location of each of a plurality of persons (2) present in the monitoring area, and estimating the vital signs of each of the detected persons, and displays an area map (51) corresponding to the monitoring area on a predetermined display device (41), displays pictograms (60) at each of the detected persons on the area map, and displays information regarding the vital signs of a representative person (2A) among the plurality of persons. This allows the user to efficiently monitor the vital signs of multiple individuals by displaying information about the vital signs of a representative person among several people present in the monitoring area.

[0084] <Technology 2> In the person analysis system described in Technology 1, the person analysis device displays the pictogram corresponding to the representative person from among a plurality of pictograms in a manner that allows for identification. This allows users to recognize the pictogram of the representative person, enabling them to instantly identify which of the multiple pictograms corresponds to the displayed vital signs information.

[0085] <Technology 3> In the person analysis system described in Technology 2, the person analysis device displays a predetermined icon above the head of the pictogram corresponding to the representative person. This allows users to instantly recognize which of the multiple pictograms displays the vital signs associated with the pictogram that has a designated icon above its head.

[0086] <Technology 4> In the person analysis system described in Technology 2, the person analysis device displays the pictogram corresponding to the representative person in a different color from the other pictograms. This allows users to instantly recognize that the displayed vital signs correspond to different colored pictograms among multiple pictograms.

[0087] <Technology 5> In the person analysis system described in any one of the technologies 1 to 4, the person analysis device further performs a process to estimate the behavior of each of the detected persons based on the observation data, and displays a pictogram showing the estimated behavior in the pictogram display. This allows users to instantly see the actions of multiple individuals within a monitoring area based on the appearance of their pictograms.

[0088] <Technology 6> In the person analysis system described in Technical 5, the person analysis device displays information on the display device that shows the temporal changes in the behavior of the representative person. This allows users to instantly recognize which of the multiple pictograms represents the information indicating the time-dependent changes in the actions of the displayed person.

[0089] <Technology 7> In the person analysis system described in any one of the technologies 1 to 6, the person analysis device displays information on the display device that shows the time change of the vital signs of the representative person as information related to the vital signs of the representative person. This allows users to instantly recognize which of the multiple pictograms represents the information indicating the time-dependent changes in the displayed person's vital signs.

[0090] <Technology 8> In the person analysis system described in any one of the technologies 1 to 7, the person analysis device designates the person selected by the user as the representative person. This allows users to manually switch (select) the representative.

[0091] <Technology 9> In the person analysis system described in any one of the technologies 1 to 8, the person analysis device designates a person whose vital signs meet pre-set conditions as the representative person. This allows the person analysis device to automatically select a representative person whose vital signs meet pre-set conditions. For example, a person whose vital signs are determined to be abnormal will be automatically selected as a representative person, and information regarding that representative person's vital signs will be automatically displayed. Therefore, the user can efficiently monitor the vital signs of multiple people.

[0092] <Technology 10> In the person analysis system described in Technical 5, the person analysis device designates a person who has performed actions that conform to pre-set conditions as the representative person. This allows the person analysis device to automatically select a representative person based on actions that meet pre-set criteria. For example, a person whose actions are determined to be "falling" is automatically selected as the representative person, and information regarding that representative person's vital signs is automatically displayed. Therefore, the user can efficiently monitor the vital signs of multiple people.

[0093] <Technology 11> In the person analysis system described in Technical 5, the person analysis device designates a person who is present and meets pre-set conditions as the representative person. This allows the person analysis device to automatically select a person who meets pre-set criteria as a representative. This enables users to quickly notice the presence of individuals whose presence in a location is unusual.

[0094] <Technology 12> A person analysis method according to one embodiment involves acquiring observation data, including the results of observing the monitoring area using a radar system, from at least one radar device installed in the monitoring area; performing a process to detect the location of each of the multiple persons present in the monitoring area based on the observation data; and estimating the vital signs of each of the detected multiple persons; displaying an area map corresponding to the monitoring area on a predetermined display device; displaying pictograms at each of the detected multiple persons on the area map; and displaying information regarding the vital signs of a representative person among the multiple persons. This allows the user to efficiently monitor the vital signs of multiple individuals by displaying information about the vital signs of a representative person among several people present in the monitoring area.

[0095] <Technology 13> A person analysis program according to one embodiment involves obtaining observation data, including the results of observing the monitoring area using a radar system, from at least one radar device installed in the monitoring area; performing the following processes based on the observation data: detecting the location of each of the multiple persons present in the monitoring area; estimating the vital signs of each of the detected multiple persons; displaying an area map corresponding to the monitoring area on a predetermined display device; displaying pictograms at the locations of the detected multiple persons on the area map; and displaying information regarding the vital signs of a representative person among the multiple persons. These processes are then performed by a computer (e.g., an information processing device 1000). This allows the user to efficiently monitor the vital signs of multiple individuals by displaying information about the vital signs of a representative person among several people present in the monitoring area.

[0096] <Technology 14> A person analysis system according to one embodiment includes: a radar device installed in a monitoring area and outputting observation data including the results of observing the monitoring area using a radar system; and a person analysis device that performs the following based on the observation data: detecting the location of each of a plurality of persons present in the monitoring area and estimating the actions of each of the detected persons; displaying an area map corresponding to the monitoring area on a predetermined display device; displaying pictograms at each of the detected persons on the area map; and displaying information regarding the actions of a representative person among the plurality of persons. This allows information about the actions of a representative person among multiple individuals present in the monitoring area to be displayed, enabling users to efficiently monitor the actions of multiple individuals.

[0097] <Technology 15> A person analysis method according to one embodiment involves acquiring observation data, including the results of observing the surveillance area using a radar system, from at least one radar device installed in the surveillance area; performing a process to detect the location of each of the multiple persons present in the surveillance area based on the observation data; and performing a process to estimate the actions of each of the detected multiple persons; displaying an area map corresponding to the surveillance area on a predetermined display device; displaying pictograms at each of the detected multiple persons on the area map; and displaying information regarding the actions of a representative person among the multiple persons. This allows information about the actions of a representative person among multiple individuals present in the monitoring area to be displayed, enabling users to efficiently monitor the actions of multiple individuals.

[0098] <Technology 16> A person analysis program according to one embodiment involves obtaining observation data, including the results of observing the monitoring area using a radar system, from at least one radar device installed in the monitoring area; performing the following processes based on the observation data: detecting the location of each of the multiple persons present in the monitoring area; estimating the actions of each of the detected multiple persons; displaying an area map corresponding to the monitoring area on a predetermined display device; displaying pictograms at the locations of the detected multiple persons on the area map; and displaying information regarding the actions of a representative person among the multiple persons. These actions are then performed by a computer (e.g., an information processing device 1000). This allows information about the actions of a representative person among multiple individuals present in the monitoring area to be displayed, enabling users to efficiently monitor the actions of multiple individuals.

[0099] While embodiments have been described above with reference to the attached drawings, this disclosure is not limited to such examples. It is clear to those skilled in the art that various modifications, alterations, substitutions, additions, deletions, and equivalents can be conceived within the scope of the claims, and these are also understood to fall within the technical scope of this disclosure. Furthermore, the components of the embodiments described above can be combined in any way without departing from the spirit of the invention. [Industrial applicability]

[0100] The technology disclosed herein is useful for devices and methods for monitoring the vital information of multiple individuals while protecting their privacy. [Explanation of Symbols]

[0101] 1. Surveillance Area 2 people 2A Representative person 10. Person Analysis System 11 Radar equipment 14. Communication Networks 20 Person analysis device 21 IQ Data Storage Unit 24-point cloud data generation unit 25 Point cloud data storage unit 26 Target Data Generation Unit 27 Target Data Storage Unit 28 Behavioral Data Generation Unit 29 Action Data Storage Unit 30 Vital Data Generation Unit 31 Vital Data Storage Unit 32 Monitoring and Control Unit 40 Input devices 41 Display device 42 Alarm device 43 User terminals 51 Area Map 60, 60A, 60B, 60C, 60D pictograms 200 Monitoring screen 210 Monitoring area display area 220 Vital Signs Display Area 221 Behavior display area 230 Vitals history display area 240 Activity history display area 300 Representative People Icons

Claims

1. A radar device installed in the monitoring area, which outputs observation data including the results of observing the monitoring area using a radar system, The person analysis device includes a process for detecting the location of each of several people present in the monitoring area based on the observation data, and a process for estimating the vital signs of each of the detected people, and displays an area map corresponding to the monitoring area on a predetermined display device, displays pictograms at the locations of each of the detected people on the area map, and displays information regarding the vital signs of a representative person among the people. A character analysis system.

2. The aforementioned person analysis device, Among multiple pictograms, the pictogram corresponding to the aforementioned representative person is displayed in an identifiable manner. The person analysis system according to claim 1.

3. The aforementioned person analysis device, A predetermined icon is displayed above the pictogram corresponding to the aforementioned representative person. The person analysis system according to claim 2.

4. The aforementioned person analysis device, The pictogram corresponding to the aforementioned representative person is displayed in a different color from the other pictograms. The person analysis system according to claim 2.

5. The aforementioned person analysis device, Based on the aforementioned observation data, a process is further performed to estimate the actions of each of the detected individuals. In the display of the aforementioned pictogram, a pictogram representing the estimated action is displayed. The person analysis system according to claim 1.

6. The aforementioned person analysis device, The display device displays information showing the temporal changes in the actions of the aforementioned representative person. The person analysis system according to claim 5.

7. The aforementioned person analysis device, As information regarding the vital signs of the aforementioned representative person, the display device will show information indicating the time-dependent changes in the vital signs of the aforementioned representative person. A person analysis system according to any one of claims 1 to 6.

8. The aforementioned person analysis device, The person selected by the user shall be designated as the representative person. The person analysis system according to claim 1.

9. The aforementioned person analysis device, The person whose vital signs meet the pre-set conditions will be designated as the representative person. The person analysis system according to claim 1.

10. The aforementioned person analysis device, The person who performed an action that conformed to the predetermined conditions shall be designated as the representative person. The person analysis system according to claim 5.

11. The aforementioned person analysis device, A person who resides in the area and meets the pre-set conditions will be designated as the representative person. The person analysis system according to claim 5.

12. Observation data, including the results of observing the monitoring area using a radar system, is acquired from at least one radar device installed in the monitoring area. Based on the observation data, the following processes are performed: detecting the location of each of the multiple persons present in the monitoring area, and estimating the vital signs of each of the detected persons. A predetermined display device displays an area map corresponding to the monitoring area, displays pictograms at the locations of the multiple persons detected on the area map, and displays information regarding the vital signs of a representative person among the multiple persons. Person analysis method.

13. Observation data, including the results of observing the monitoring area using a radar system, is acquired from at least one radar device installed in the monitoring area. Based on the observation data, the following processes are performed: detecting the location of each of the multiple persons present in the monitoring area, and estimating the vital signs of each of the detected persons. A predetermined display device displays an area map corresponding to the monitoring area, displays pictograms at the locations of the multiple persons detected on the area map, and displays information regarding the vital signs of a representative person among the multiple persons. A character analysis program that has a computer perform a task.

14. A radar device installed in the monitoring area, which outputs observation data including the results of observing the monitoring area using a radar system, The person analysis device includes a process for detecting the location of each of several people present in the monitoring area based on the observation data, and a process for estimating the actions of each of the detected people, and displays an area map corresponding to the monitoring area on a predetermined display device, displays pictograms at the locations of each of the detected people on the area map, and displays information regarding the actions of a representative person among the people. A character analysis system.

15. Observation data, including the results of observing the monitoring area using a radar system, is acquired from at least one radar device installed in the monitoring area. Based on the observation data, the following processes are performed: detecting the location of each of the multiple persons present in the monitoring area, and estimating the actions of each of the detected persons. A predetermined display device displays an area map corresponding to the monitoring area, displays pictograms at the locations of the multiple persons detected on the area map, and displays information regarding the actions of a representative person among the multiple persons. Person analysis method.

16. Observation data, including the results of observing the monitoring area using a radar system, is acquired from at least one radar device installed in the monitoring area. Based on the observation data, the following processes are performed: detecting the location of each of the multiple persons present in the monitoring area, and estimating the actions of each of the detected persons. A predetermined display device displays an area map corresponding to the monitoring area, displays pictograms at the locations of the multiple persons detected on the area map, and displays information regarding the actions of a representative person among the multiple persons. A character analysis program that has a computer perform a task.

Images