Condition monitoring system and condition monitoring method

Abstract

To grasp a situation of a person from a viewpoint different from an appearance.SOLUTION: A monitoring system includes an acquisition unit that acquires vital data which is a measurement result of a vital sensor from a person present in a monitoring target area, an emotion estimation unit that estimates emotion data indicating an emotion corresponding to the vital data, a presence area detection unit that detects a detection area in which the person whose vital data is acquired is present in the monitoring target area, and an output unit that outputs a message corresponding to a relationship between the estimated emotion data and the detection area.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a situation monitoring system and a situation monitoring method. [Background technology]

[0002] There are systems that take images of facilities used by users using cameras or the like and provide various information based on the image capture results. For example, there is a system that takes images of people's behavior within a facility using a camera, detects whether the behavior is suspicious based on the captured images, and if the behavior is suspicious, outputs an alert indicating that a suspicious person may have entered the facility (for example, Patent Document 1). Furthermore, if images taken by a camera capturing images inside the facility detect that a person is behaving in a way that appears to be in distress, an alert can be sent to a management center, etc., so that staff can take action such as assisting the person in distress. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-211514 Summary of the Invention [Problem to be solved by the invention]

[0004] However, while such systems can determine whether a person is suspicious based on their movements and appearance, if the person's movements are smooth or they behave in the same way as a non-suspicious person, it can be difficult to determine whether they are suspicious based on their appearance. Also, if a person's actions do not clearly indicate that they are in distress, it can be difficult to determine whether they are in distress based on their appearance.

[0005] The present invention has been made in consideration of the above circumstances, and its purpose is to provide a situation monitoring system and a situation monitoring method that are capable of grasping a person's situation from a perspective other than appearance. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, one aspect of the present invention is a situation monitoring system having an acquisition unit that acquires vital data, which is the measurement result of a vital sensor, from a person present in a monitored area; an emotion estimation unit that estimates emotion data that expresses an emotion corresponding to the vital data; a presence area detection unit that detects a detection area within the monitored area in which the person from whom the vital data was acquired is present; and an output unit that outputs a message corresponding to the relationship between the estimated emotion data and the detection area.

[0007] In order to solve the above-mentioned problems, one aspect of the present invention is a situation monitoring method executed by a computer, the situation monitoring method including: acquiring vital data, which is the measurement result of a vital sensor, from a person present in a monitored area; estimating emotion data expressing an emotion corresponding to the vital data; detecting a detection area within the monitored area in which the person from whom the vital data was acquired is located; and outputting a message corresponding to the relationship between the estimated emotion data and the detection area. [Effects of the Invention]

[0008] As described above, according to the present invention, it is possible to grasp the situation of a person from a perspective other than the appearance. Furthermore, according to this invention, a message is output according to the relationship between a person's emotions and psychology and the area in which the person is located, so that it is possible to understand the situation of a person in an area from a perspective other than the person's behavior, appearance, etc. [Brief explanation of the drawings]

[0009] [Figure 1]1 is a schematic block diagram showing the configuration of a situation monitoring system S according to an embodiment of the present invention. [Figure 2] 2 is a functional block diagram showing the general functions of a situation monitoring device 40. FIG. [Figure 3] 10 is a diagram showing an example of a message rule stored in a message rule storage unit 4022. FIG. [Figure 4] 10 is a flowchart illustrating a flow of generating and using emotion estimation data. [Figure 5] 2 is a conceptual diagram illustrating the flow of operation of the situation monitoring system S. FIG. [Figure 6] 10 is a flowchart illustrating the operation of the situation monitoring device 40 in the execution phase. [Figure 7] 10 is a diagram showing another example of message rules stored in message rule storage unit 4022. FIG. [Figure 8] FIG. 1 is a diagram illustrating the configuration of a situation monitoring system S including a camera. DETAILED DESCRIPTION OF THE INVENTION

[0010] A situation monitoring system according to an embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a schematic block diagram showing the configuration of a situation monitoring system S according to an embodiment of the present invention. The situation monitoring system S includes a plurality of terminal devices 10 (terminal device 10a, terminal device 10b, ... terminal device 10n), a plurality of vital sensors 15 (vital sensor 15a, vital sensor 15b, ... vital sensor 15n), a learning device 30, a situation monitoring device 40, an output device 50, and a network NW.

[0011] The multiple terminal devices 10 (terminal device 10a, terminal device 10b, ... terminal device 10n) are used by different users, and may be, for example, any of smartphones, tablets, personal computers, etc. Each of the multiple terminal devices 10 is communicably connected to a network NW and communicates with other devices connected to the network NW. Different terminal identification information is assigned to each of the terminal devices 10.

[0012] The multiple vital sensors 15 (vital sensor 15a, vital sensor 15b, ... vital sensor 15n) are used by different users, and perform biological sensing of the pulse, breathing, etc. from the movements of the body surface of the user, and generate vital data representing the sensing results. The vital data is data in which values ​​representing the measurement results are arranged in chronological order.

[0013] Vital sensor 15 may be a contact type that senses the user's biological activity by contacting a part of the user's body, or a non-contact type that senses the user's biological activity without contacting the user. If vital sensor 15 is a non-contact type, it senses the user's biological activity by, for example, irradiating the user with microwaves using a Doppler sensor and detecting the biological activity based on the reflected waves. The non-contact type vital sensor 15 can measure the user from any distance, but a model that can perform measurements from a distance of, for example, about 10 cm to several meters may be used. When the non-contact vital sensor is installed indoors, for example, it may be attached to a wall, ceiling, furniture, stand, etc., and may measure the person being measured.

[0014] The contact-type vital sensor may be, for example, an existing electronic device that is built into a wristwatch and sold for sale. Such wristwatch-type vital sensors are widely available, and the number of users who already own them is increasing. Furthermore, even if the contact-type vital sensor is not built into a wristwatch, it may be a wearable type that can be worn on the wrist, arm, chest, abdomen, or the like. When the non-contact vital sensor is installed outdoors, for example, it may be attached to a wall, fence, roof, eaves, mounting base, etc., and used to measure the person being measured. Non-contact vital signs sensors are increasingly being installed in businesses and homes, creating an environment where they can be easily used. In this way, at least one of a contact type and a non-contact type may be used as the vital sensor.

[0015] Terminal device 10a and vital sensor 15a, terminal device 10b and vital sensor 15b, and terminal device 10n and vital sensor 15n are each installed in an area to be monitored. The area to be monitored may be any of a store, a school classroom, a cram school, a theme park, a commercial facility, a station, an airport, etc. Furthermore, a plurality of these terminal devices and vital sensors may be installed in the area to be monitored, and each may measure at a different location. The number of combinations of terminal devices 10 and vital sensors 15 may be determined according to the size of the area to be monitored. Vital sensor 15a is connected to terminal device 10a via wireless communication by performing a pairing process, and vital data, which is the measurement result, may be transmitted to terminal device 10a wirelessly or via a wired connection. Similarly, vital sensor 15b and terminal device 10b, and vital sensor 15n and terminal device 10n are connected to each other wirelessly or via a wired connection so that they can communicate with each other.

[0016] The non-contact vital sensor 15 may acquire vital data of the person to be measured from a position away from the person, and transmit the vital data to the situation monitoring device 40 from a terminal device 10 installed near the vital sensor 15. When the non-contact vital sensor 15 is used, the terminal device 10 that is the communication partner may be one that is personally owned by the user, or may be one that is installed in a facility where the non-contact vital sensor 15 is installed. The contact-type vital sensor 15 may be attached to the person to be measured, and the terminal device 10 may be placed near the person wearing the sensor, and the vital data may be transmitted to the condition monitoring device 40. Here, if each person entering the monitored area wears a contact vital sensor, vital data within the monitored area can be acquired, but not everyone necessarily wears a sensor. In such cases, by installing at least one non-contact vital sensor within the monitored area, vital data of people within the area can be acquired.

[0017] The vital sensor 15 may be communicably connected to the network NW without going through the terminal device 10, and may be configured to transmit vital data to other devices connected to the network NW (e.g., the learning device 30, the situation monitoring device 40).

[0018] The learning device 30 performs learning using training data and generates a trained model. The learning device 30 generates an emotion estimation model (trained model) by learning a combination of vital data measured while performing an action specified as the measurement target (such as a sport, game, or TV appearance) and the results of a questionnaire answered about emotions at the time the vital data was obtained, as training data, to generate a trained model.

[0019] The learning method performed by the learning device 30 may be machine learning or deep learning. The learning device 30 is communicatively connected to a network NW and transmits the generated trained model to the situation monitoring device 40 via the network NW.

[0020] The situation monitoring device 40 is communicably connected to the terminal device 10, the learning device 30, the output device 50, etc. via the network NW. The situation monitoring device 40 may also be communicably connected to the vital sensor 15 via the network NW. The situation monitoring device 40 may be a physical server or a cloud server provided by a cloud computing service.

[0021] The output device 50 outputs a message. For example, the output device 50 may be either a display device or a speaker, or may be a device including a display device and a speaker. The output device 50 is used by a manager who manages the area to be monitored. The output device 50 may be installed in a management center that manages the area to be monitored, or may be a terminal device (smartphone, tablet, etc.) carried by the manager.

[0022] The network NW may be the Internet, a LAN (Local Area Network), or a combination of these. The network NW may also include a communication network for communication using a television communication method.

[0023] FIG. 2 is a functional block diagram showing an outline of the functions of the situation monitoring device 40. As shown in FIG. The situation monitoring device 40 includes a communication unit 401 , a storage unit 402 , an acquisition unit 403 , a feeling estimation unit 404 , a presence area detection unit 405 , an output unit 406 , and a control unit 407 . The communication unit 401 communicates with external devices via the network NW.

[0024] The storage unit 402 stores various types of data. The memory unit 402 is configured by a storage medium, such as a hard disk drive (HDD), flash memory, electrically erasable programmable read-only memory (EEPROM), random access read / write memory (RAM), read-only memory (ROM), or any combination of these storage media. The storage unit 402 may be, for example, a nonvolatile memory.

[0025] For example, the storage unit 402 stores an emotion estimation data storage unit 4021 and a message rule storage unit 4022 . The emotion estimation data storage unit 4021 stores emotion estimation data that indicates the correspondence between vital data and emotions. The emotion estimation data may be an emotion estimation model, which will be described later. The emotion estimation data storage unit 4021 may store the emotion estimation model generated by the learning device 30. The message rule storage unit 4022 stores data representing message rules.

[0026] Fig. 3 is a diagram showing an example of message rules stored in message rule storage unit 4022. Fig. 3 also shows an example of message rules in the case where the area to be monitored is a store. The message rule is data in which a detection area, emotion data, and a message are associated with each other. The detection area indicates the area to which the location where the vital data was measured belongs. The emotion data indicates emotion data corresponding to the measured vital data. The message indicates the content to be notified to an administrator or the like. The message may include at least one of characters, symbols, audio, images, videos, etc. For example, in the data on the first line, the detection area "in front of the ATM" is associated with the emotion data "nervous" or "nervous" and the message "need to check whether it is a bank transfer scam." In the data on the fourth line, the detection area "queue at the cash register" is associated with the emotion data "irritated" and the message "need help at the cash register."

[0027] The acquisition unit 403 acquires vital data, which is the measurement result of a vital sensor, from a person present in a target area.

[0028] The emotion estimation unit 404 estimates emotion data representing an emotion according to the vital data acquired by the acquisition unit 403. For example, the emotion estimation unit 404 may estimate emotion data by scoring based on the transition of time-series data of vital data that belongs to a certain period of time from the time when the emotion is estimated.

[0029] The presence area detection unit 405 detects an area within the area to be monitored based on the location of the person whose vital data has been acquired. For example, if the vital sensor 15 is non-contact type, the presence area detection unit 405 can determine the area to be monitored based on the terminal identification information assigned to the terminal device 10 connected to the vital sensor 15. Here, a table storing the correspondence between the area to be monitored and the terminal identification information is stored in the storage unit 402, and the presence area detection unit 405 can identify the area to be monitored by referring to this table. Furthermore, the presence area detection unit 405 estimates the position of the person to be measured based on the direction and distance to the person, using the location where the vital sensor 15 is installed as a reference. The presence area detection unit 405 determines which of the divided areas into which the monitored area is divided the estimated position belongs, and acquires the divided area obtained as a result of the determination as the detection area. Here, the range of coordinates belonging to the divided area is stored in advance in the storage unit 402 as coordinate data, and the presence area detection unit 405 refers to this coordinate data to determine which divided area the estimated position belongs to, and acquires the divided area obtained as a result of the determination as the detection area. Furthermore, when vital data is obtained by a contact-type vital sensor, the presence area detection unit 405 may acquire location information indicating the location where the contact-type vital sensor is located from a positioning function (e.g., GNSS (Global Navigation Satellite System)) installed in the contact-type vital sensor. Then, the presence area detection unit 405 acquires, as a detection area, a divided area to which the location indicated by the acquired location information belongs, based on the coordinate data.

[0030] The output unit 406 outputs a message according to the relationship between the emotion data estimated by the emotion estimation unit 404 and the detection area detected by the presence area detection unit 405. For example, the output unit 406 obtains a message corresponding to the combination of the estimated emotion data and the detected detection area by reading out the message rule stored in the message rule storage unit 4022, and outputs the obtained message. The destination to which the output unit 406 outputs the message is, for example, the output device 50, and the communication unit 401 transmits the message via the network NW.

[0031] The control unit 407 controls each unit of the situation monitoring device 40 .

[0032] The communication unit 401, acquisition unit 403, emotion estimation unit 404, presence area detection unit 405, output unit 406, and control unit 407 of the situation monitoring device 40 may be configured as a processing unit such as a CPU (Central Processing Unit) or a dedicated electronic circuit.

[0033] FIG. 4 is a flow diagram illustrating the flow of generating and using emotion estimation data. Emotion estimation data is data that represents the correspondence between vital data and emotions. A user U1 who can provide vital data used to generate emotion estimation data is targeted. The user U1 is asked to perform a predetermined behavior, and vital data is acquired by having a vital sensor 15 that measures the user U1 perform biometric sensing during the period in which the behavior is being performed. This behavior may be viewing content, having a conversation with a third party, performing a predetermined task, or the like.

[0034] User U1 may be a person who may come to the area to be monitored, such as a student if the area to be monitored is a school, or a person who may come to the store as a customer if the area to be monitored is a store. Furthermore, a questionnaire about the emotions felt while performing the behavior is conducted on the person who performed the behavior, and the results of the questionnaire are input from the terminal device 10 or a separately provided management device, etc. Then, the vital data of the user U1 while performing the behavior specified as the measurement target and the emotions resulting from the questionnaire are associated and stored as emotion estimation data in the emotion estimation data storage unit 4021. Here, the user to be measured may be a single person, user U1, or measurements may be taken of multiple different users.

[0035] Furthermore, using as training data the relationship between the vital data of user U1 while performing the behavior specified as the measurement target and the emotions of user U1 ascertained through a questionnaire, the learning device 30 learns the relationship between the vital data and emotions, thereby generating a trained model for estimating emotions from the vital data, and transmits the trained model to the situation monitoring device 40. The control unit 407 of the situation monitoring device 40 may receive the emotion estimation model and store it in the emotion estimation data storage unit 4021. In this way, by using the emotion estimation model, the situation monitoring device 40 can estimate emotion data from trends in changes in vital data such as heart rate and pulse rate.

[0036] This makes it possible to generate emotion estimation data or an emotion estimation model that takes into account the causal relationship between vital data and emotions by using the correspondence between vital data and actual emotions based on a questionnaire. Alternatively, an existing estimation engine that estimates emotions from vital data may be used.

[0037] After the emotion estimation data is stored in the emotion estimation data storage unit 4021 in this way, when vital data of a user U2 who is a person arriving in the area to be monitored (for example, a student if the area to be monitored is a school, or a customer if the area to be monitored is a store) is obtained, the emotion estimation unit 404 reads out emotion data corresponding to this vital data by referring to the emotion estimation data storage unit 4021, thereby obtaining emotion data corresponding to the vital data. Furthermore, the emotion estimation unit 404 may obtain emotion data by inputting vital data into an emotion estimation model.

[0038] Here, the types of emotional data obtained from vital data include, for example, positive (pleasant) and negative (unpleasant). Positive categories include "joy," "relaxation," and "calmness." Negative categories include "irritation," "stress," and "fatigue." The emotion estimation unit 404 may extract any one of these emotion types as the emotion corresponding to the vital data. Alternatively, the emotion estimation unit 404 may extract the emotion type corresponding to the vital data and obtain a value representing the degree of the emotion by scoring the emotion type.

[0039] The scoring process performed by the emotion estimation unit 404 may be, for example, based on the frequency with which the emotion corresponding to the index of the target to be evaluated is expressed in the user to be evaluated during an evaluation interval (e.g., a range of a certain period from the time of measurement).

[0040] More specifically, the degree of emotion corresponding to the index of the evaluation target may be determined by evaluating the emotion based on the cumulative time during which the emotion, which indicates good or bad, is expressed. Alternatively, the degree may be determined by making an evaluation based on the integral value (area) of the region where a positive or negative emotion is expressed.

[0041] Next, the operation of the situation monitoring system S will be described. FIG. 5 is a conceptual diagram illustrating the flow of operations of the situation monitoring system S. The operation of the situation monitoring system S can be broadly divided into a "preparation phase" and an "execution phase."

[0042] Preparation Phase The preparation phase is a phase in which emotion estimation data (or emotion estimation model) is generated and stored in emotion estimation data storage unit 4021. Here, multiple users (for example, people who may arrive in the area to be monitored) are targeted, and while they are engaged in activities that have been determined to be the subject of measurement (such as classes, breaks, and lunchtimes if the area to be monitored is a classroom, or walking around the store or paying at the cash register if the area to be monitored is a store) (step S1), vital data is measured for each user using vital sensor 15. By performing measurements using vital sensor 15, it is possible to obtain vital data, which is time-series data that changes according to emotions.

[0043] When vital data is measured by the vital sensor 15, the terminal device 10 acquires the vital data from the vital sensor 15. The terminal device 10 asks the user to enter answers to a questionnaire about emotions during the measurement period by the vital sensor 15 along with the acquired vital data, and transmits the vital data and the emotions resulting from the questionnaire to the learning device 30. The learning device 30 acquires data in which such vital data and emotions are associated with each other from each of the multiple users (step S2), and stores the data in a storage device within the learning device 30. Furthermore, the behavior determined as the measurement target may be viewing content. The content may include scenes that are likely to evoke various emotions (positive emotions, negative emotions, etc.). For example, scenes that evoke joy, scenes that are relaxing, scenes that irritate, scenes that evoke stress, scenes that evoke fatigue, etc.

[0044] The learning device 30 generates an emotion estimation model by learning a combination of vital data and the results of a questionnaire answered about emotions at the time the vital data was obtained as training data, thereby generating a trained model. After generating the emotion estimation model, the learning device 30 stores the generated emotion estimation model in a storage device (step S3). Then, the learning device 30 transmits the generated emotion estimation model to the situation monitoring device 40. The situation monitoring device 40 stores the emotion estimation model transmitted from the learning device 30 in the emotion estimation data storage unit 4021. Here, you can learn the relationship between various emotions and vital data.

[0045] Execution Phase In the execution phase, the acquisition unit 403 of the situation monitoring device 40 acquires vital data of a person present in the area to be monitored (step S5). This vital data may be measured using a non-contact vital sensor. The emotion estimation unit 404 inputs the acquired vital data into an emotion estimation model and obtains emotion data derived from the emotion estimation model, thereby estimating the emotion of the person (step S6). The presence area detection unit 405 detects a detection area corresponding to the position where the vital data was acquired. The output unit 406 reads a message corresponding to the combination of the emotion data and the detection area from the message rule storage unit 4022, and outputs the message to the output device 50 via the network NW by the communication unit 401. As a result, the message is output from the output device 50.

[0046] Next, the operation of the situation monitoring device 40 will be described. FIG. 6 is a flowchart illustrating the operation of the situation monitoring device 40 in the execution phase. Here, emotion estimation data (emotion estimation model) is generated in advance and stored in emotion estimation data storage unit 4021. When monitoring of the monitored area begins, the vital sensor 15 measures vital data of people present in the monitored area and transmits the measurement results to the terminal device 10. The terminal device 10 transmits the vital data transmitted from the vital sensor 15 to the situation monitoring device 40 together with terminal identification information.

[0047] The acquisition unit 403 acquires vital data measured by the vital sensor 15 and terminal identification information (step S101). The emotion estimation unit 404 acquires emotion data by inputting the vital data acquired by the acquisition unit 403 into an emotion estimation model (step S102). The presence area detection unit 405 detects a detection area according to the position where the vital data was acquired (step S103). Output unit 406 reads out a message corresponding to the combination of the estimated emotion data and the detected detection area from message rule storage unit 4022 (step S104), and outputs the read message to output device 50 (step S105). If no message corresponding to the combination of the estimated emotion data and the detected detection area is stored in message rule storage unit 4022, output unit 406 does not need to output a message because there is no message to read, and may instead output "no message" to output device 50, indicating that there is no message. The output unit 406 can output messages from the output device 50 in approximately real time in accordance with the timing at which the vital data is measured. Therefore, for example, it is possible to respond quickly in response to the occurrence of negative emotions.

[0048] <Example 1: Store> The above-mentioned situation monitoring system S will be described in the case where a store is the monitoring target. When monitoring of a store begins, vital data is measured for people in the monitored area using vital sensor 15. If there are multiple people in the monitored area, the person being monitored is changed at regular intervals, and vital data is measured for each person. The acquisition unit 403 of the situation monitoring device 40 acquires the vital data measured by the vital sensor 15 via the terminal device 10. The situation monitoring device 40 outputs to the output device 50 a message according to a combination of emotion data corresponding to the measured vital data and the detection area in which the vital data was detected.

[0049] For example, if emotion data with a negative emotional type, such as "nervous" or "nervousness," is obtained near the location where an ATM is installed (e.g., in front of the ATM), a message warning about a transfer, such as "Please confirm whether this is a transfer fraud," is output from the output device 50. This allows a manager at the management center that manages the store to see this message and take action, such as going to the ATM to check the situation. In this case, it is possible that a person who is feeling nervous or nervous in front of the ATM has fallen victim to a transfer fraud and is attempting to operate the ATM in an abnormal psychological state. In such a case, the manager at the management center instructs a store employee to go to the ATM. This allows the employee to take action, such as calling out to the customer in front of the ATM and checking the situation.

[0050] Furthermore, for example, if emotion data with a negative emotion type such as "irritation" is obtained in an area near the checkout registers that corresponds to the queue for checkouts (the area corresponding to the checkout queue), a message regarding the congestion at the checkout registers, such as a message such as "Assistance needed at the register," is output from the output device 50. This allows a manager at the management center that manages the store to take action such as instructing the cashier in charge of the register to go and handle the checkout. In this case, if there is an irritated customer in the checkout queue, it is possible that the customer is feeling negative about the long wait time at the register or the fact that the wait time is likely to become long. Therefore, the cashier can go to the register, check the status of the queue, and start handling customers at an empty register.

[0051] Furthermore, for example, when emotion data of "joy" belonging to the positive category is obtained in an area where the detection area is near a product shelf in a store (for example, in front of the product shelf), a message such as "Product selection may be good" is output from the output device 50 as a message indicating that the product inventory status on the product shelf and the product layout on the shelf are good. This allows the manager of the management center that manages the store to understand that a customer who has visited the store may be happy to have found the product they were looking for on the product shelf. In this way, by recording the factors that caused the emotion of joy (positive emotion) as a log, it can be used as a reference when selecting products to sell in the store. Furthermore, when the detection area is in front of an area near a product shelf in a store (for example, a product shelf) and emotion data representing a negative emotion such as "disappointment" or "stress" is obtained, a message regarding product display, such as "check the product display status," is output from the output device 50. This allows the manager to understand that the customer may be feeling "disappointment" or "stress" because the product they were trying to purchase was sold out or the product was placed in a difficult-to-find location on the shelf. This allows them to improve the product display status by restocking the product on the shelf, reviewing the shelf layout, checking the product ordering status, etc.

[0052] FIG. 7 is a diagram showing another example of message rules stored in message rule storage unit 4022. FIG. 7 also shows an example of message rules when the management target area is a school. In this example, the message rule is data in which a detection area, a detection time period, emotion data, and a message are associated with each other. What differs from the items included in the message rule shown in FIG. 3 is that a detection time period is included. In this example, the detection time period includes the school timetable and time periods obtained by further dividing the time period indicated by the timetable. For example, for the detection area "Classroom A," the detection time period "fourth period (arithmetic)" is divided into time periods such as "first time period," "second time period," etc., and the emotional data "tension" or "irritation" is associated with the "first time period," as well as the message "possibility of not understanding the explanation about XX." Here, the detection time period may be stored as a time period according to the timetable, such as "xxth period," and may not be set (stored) for the "first time period" etc., which is a further division of the timetable. Here, the control unit 407 of the situation monitoring device 40 may have a time zone detection function that detects which time zone in the schedule determined for which educational facility the time when the vital data was obtained falls in. For example, the control unit 407 determines which time zone in the timetable of the school being monitored the time when the vital data was obtained falls into.

[0053] For example, if the monitored area is an educational facility (e.g., a school), and the time slot corresponding to the time when vital data was measured in the detection area "Classroom A" is determined based on the timetable, and if the emotion type expressed by the obtained emotional data is "tension" or "irritation," which belong to the negative category, a message appropriate to the detected time slot is output, informing the student that action is required, such as "It is possible that the explanation about XX is not being understood." For example, if the time period in which negative emotions were detected is the first time period, and it is a lesson in which the teacher is explaining how to do a certain math calculation to the students, then by outputting a message indicating that the cause of the negative emotions may be related to the level of understanding in the lesson, it is possible to determine that the students may not have understood the explanation of how to do the calculation, and it can be confirmed whether or not improvements to the teaching method or follow-up education are necessary. In this way, it is also possible to understand the level of understanding of the lesson. Furthermore, for example, if emotion data "stress" or "fatigue" is detected in the detection area "Classroom A," the detection time period "fourth period (math)," and the "second time period," a message "Practice problem A is difficult to solve" is output. If practice problems are given in this second time period and students are solving them, it is possible to understand that students are finding solving the practice problems difficult. This makes it possible to take measures such as providing additional explanations on how to solve the problems.

[0054] Furthermore, for example, if negative emotion data "sad" is detected in the detection area "Classroom A" and the detection time period is "Lunchtime," a message indicating that the cause of the negative emotion may be the school lunch menu, such as a "School Lunch Menu" message, may be output. This makes it possible to understand that the contents of the school lunch or the ingredients used may not have been to the students' liking. This makes it possible to consider ways to make the school lunch easier to eat. This makes it possible to improve school lunches so that they are more appealing to students, and also reduces food waste.

[0055] Furthermore, for example, if emotion data of "sad" is detected in the detection area "corridor" and the detection time period "recess," a message indicating the occurrence of a negative emotion may be output. This allows the situation in the corridor to be checked. This also makes it possible to check whether bullying is occurring. It may be difficult for students to report bullying to teachers or to speak up about it, but by measuring vital signs, it is possible to check the situation even if such a report is not made.

[0056] In the above-described embodiment, the situation monitoring system S shown in FIG. 1 may further include a camera as a component. FIG. 8 is a diagram illustrating the configuration of a situation monitoring system S including a camera. The camera 20 captures an image of an imaging target region that includes the monitoring target area Ar. When the vital data of a person present in the monitored area Ar is measured by the vital sensor 15, emotional data corresponding to the vital data is estimated (analyzed) (reference numeral 200), and the emotional data is obtained (reference numeral 210). The control unit 407 of the situation monitoring device 40 performs a process of integrating the monitored area included in the image data captured by the camera 20 with the detection area (reference numeral 220). In the integration process, the control unit 407 extracts an image region of the monitored area included in the image data captured by the camera 20 that corresponds to the detection area from which the vital data was detected. This not only enables the detection area to be detected, but also enables the region of the monitored area from which the vital data from which the emotional data was obtained to be measured to be estimated. Here, a character string representing the emotional data may be displayed in the image data obtained from the camera 20 near the position where the emotional data was detected (reference numeral 230). This allows confirmation on the image that the emotional data was obtained from a person present in the estimated region. As a result, when a message corresponding to the emotional data and the detection area is output, the administrator can confirm the person from whom the emotional data was obtained in the data obtained from the camera 20 and take appropriate action. Data expressing emotion data (which may be characters, images, symbols, etc.) may be displayed on such a camera image, and a message may be superimposed on the image or displayed near the image. The correspondence between the position in the image obtained from the camera and the position where the vital data was measured may be mapped in advance, so that a character string of emotional data or the like is displayed on the image according to the position where the vital data was measured.

[0057] In the above-described embodiment, the monitored area is a school classroom, a store, or the like. However, the monitored area may also be a facility such as an amusement park. For example, divided areas are set up within the facility, and vital data of visitors is measured using a vital sensor 15 installed within the facility. If negative emotion data is obtained, the divided area containing the location where the emotion data was obtained is detected as a detection area. A message indicating that a negative emotion has been detected is then output to an output device at the management center. This allows the management center administrator to instruct a facility staff member to go to the detection area and check the situation. This allows the management center administrator to check whether visitors are enjoying themselves within the facility. If the message indicates that some visitors are not enjoying themselves, the management center administrator can check the situation and investigate the cause. For example, if it is determined that the cause is a long, overcrowded line or equipment malfunction, the necessary measures can be taken to improve the facility so that visitors can enjoy themselves.

[0058] According to the embodiment described above, a message is output according to a combination of the attributes of the area corresponding to the position where the vital data was obtained and the emotion data corresponding to the vital data, so that an event occurring in that area can be estimated. Here, even if a person's condition cannot be determined from their appearance, it can be detected based on the vital data and appropriate action can be taken. Furthermore, if a message is output in response to the occurrence of negative emotions, it is possible to identify the locations in the monitored area where negative emotions are occurring and to confirm the situation occurring in those locations.

[0059] The situation monitoring device 40 in the above-described embodiment may be implemented by a computer. In this case, a program for implementing the functions may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" may also include media that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over a network such as the Internet or over a communication line such as a telephone line, or media that store programs for a fixed period of time, such as volatile memory within a computer system serving as a server or client. The program may be a program that implements only a portion of the functions described above, or may be a program that can implement the functions in combination with a program already stored in the computer system, or may be implemented using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0060] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0061] 10, 10a, 10b, 10n terminal equipment 15, 15a, 15b, 15n Vital Sensors 20 Camera 30 Learning Device 40 Situation Monitoring Device 50 Output Device 401 Communications Department 402 Storage section 403 Acquisition Department 404 Emotion estimation section 405 Existence area detection unit 406 Output section 407 Control Unit 4021 Emotion estimation data storage unit 4022 Message rule memory unit Ar Surveillance Area NW Network S Situation Monitoring System

Claims

1. an acquisition unit that acquires vital data, which is a measurement result of a vital sensor, from a person present in a monitoring area; an emotion estimation unit that estimates emotion data representing an emotion according to the vital data; a presence area detection unit that detects a detection area in the monitoring target area where the person whose vital data has been acquired is present; an output unit that outputs a message according to the relationship between the estimated emotion data and the detection area; A situation monitoring system having:

2. The output unit A message is output according to a combination of the type of emotion represented by the estimated emotion data and the detection area. The situation monitoring system according to claim 1 .

3. The output unit If the type of emotion expressed by the emotion data is a type that indicates negativity and the detection area is an area near a location where an ATM (Automated Teller Machine) is installed, a message warning about a transfer is output. The situation monitoring system according to claim 2 .

4. The output unit If the emotion type represented by the emotion data is a negative type and the detection area is an area near a cash register that corresponds to a queue for waiting at the cash register, a message about congestion at the cash register is output. The situation monitoring system according to claim 2 .

5. The output unit If the type of emotion expressed by the emotion data is a type that indicates negativity and the detection area is an area near product shelves in a store, a message regarding product displays is output. The situation monitoring system according to claim 2 .

6. The output unit If the type of emotion expressed by the emotion data is a type that indicates negativity and the detected area is an area corresponding to an educational facility, a message is output to the student informing them that action is required. The situation monitoring system according to claim 2 .

7. a time zone detection unit that detects which time zone in a schedule determined for which educational facility the time when the vital data was obtained falls; The output unit If the type of emotion represented by the emotion data is a type representing negative emotion and the detected area is an area corresponding to an educational facility, a message corresponding to the detected time period is output. The situation monitoring system according to claim 6 .

8. The time period is a meal time, The output unit outputs a message indicating that the cause of the negative emotion may be the school lunch menu. The condition monitoring system according to claim 7.

9. The time period is a class time, The output unit outputs a message indicating that the cause of the negative emotion may be related to the level of understanding in the lesson. The condition monitoring system according to claim 7.

10. a camera for capturing an image of the area to be monitored; The output unit displays data representing emotion data in the vicinity of a position corresponding to a position where emotion data was detected in the image obtained from the camera, and outputs data for displaying the message.

10. A situation monitoring system according to claim 1.

11. 1. A computer-implemented method for monitoring a condition, comprising: Vital data, which is the measurement result of a vital sensor, is acquired from a person present in the monitored area, Estimating emotion data representing an emotion according to the vital data; Detecting a detection area in which the person whose vital data has been acquired is present within the monitoring target area; A message is output according to the relationship between the estimated emotion data and the detection area. A situation monitoring method including:

Images