Estimation device and estimation method

Abstract

This estimation device comprises: a biological information acquisition unit that acquires biological information based on biological measurements taken by a sensor that measures biological phenomena of a person; an environmental information acquisition unit that acquires environmental information, including video and / or audio of the person's surroundings; an activity status estimation unit that estimates an activity status of the person on the basis of the biological information and the environmental information; and a psychological state estimation unit that estimates a psychological state of the person on the basis of the biological information and the activity status.

Description

Estimation device and estimation method

[0001] This application claims priority to PCT / JP2023 / 045502, filed in Japan on December 19, 2023, the contents of which are incorporated herein by reference.

[0002] Companies and other organizations are expected to understand the impact that workplace communication has on the psychological state of employees, promote positive communication, and take measures to improve communication that has a negative impact. In particular, problems arising in communication between employees in the workplace can cause employees to fall into a psychological state different from their normal state (e.g., a state of high stress). Examples of problems include harassment such as power harassment, moral harassment, and sexual harassment. In such cases, the manager (e.g., supervisor) of an employee who has fallen into a psychological state different from their normal state needs to fully understand the situation in order to improve it.

[0003] This is not limited to problems that arise in communication between employees, but also applies to problems that arise in communication between managers and employees, etc. Furthermore, this is not limited to problems that arise in communication in the workplace, but also applies to problems that arise in communication between students in educational settings, or between teachers and students, etc.

[0004] For example, in the workplace, understanding and responding to the psychological and health states of subordinates is an important task for organizations and supervisors. Similarly, understanding and responding to whether supervisors are making problematic comments to their subordinates or communicating appropriately is also an important task for managers in higher-level organizations (hereinafter referred to as "superordinate organizational managers"). In the educational field, understanding and responding to instances of "bullying" among students is also an important task for teachers. However, due to the diversification of communication styles in recent years and the busyness of work and daily life, supervisors, managers in higher-level organizations, and teachers often do not have the time to understand the psychological impact of such communication and respond accordingly.

[0005] Given this background, there is a need these days to create a system that allows supervisors and managers of higher-level organizations to identify and respond to communication problems, even in situations where they cannot directly check on the state of employees or the state of communication in the workplace.

[0006] JP 2012-8683 A Japanese Patent No. 6388824 A

[0007] Hossein Hamidi shishavan et al., "Wearable Technology and Pulse Transit Time (PTT) Used to Assess Workplace Violence Incidents in Nursing", Proceedings of the 2022 HFES 66th International Annual Meeting, 66(1), pp.1648-1652, 2022, [Retrieved December 15, 2023], Internet (URL: https: / / doi.org / 10.1177 / 1071181322661267) Takahiro Ueno and Masayoshi Ohashi, "Estimating Heart Rate Variability Characteristics Based on Differences in Subjective Stress During Harassment", Information Processing Society of Japan SIG-BTI Symposium, 2023, "Launch of an AI Text Estimation Service to Detect Fraudulent Transactions and Harassment Risks from Internal Emails and Chats", Eltes Co., Ltd. Homepage, March 22, 2023, [Retrieved December 15, 2023], Internet (URL: https: / / prtimes.jp / main / html / rd / p / 000000265.000004487.html)

[0008] For example, Patent Literature 1 describes a technology that analyzes the movements of a subject obtained by a sensor, verbalizes the movements, and adds the verbalized information to a video as metadata. This makes it possible to search for videos using the movements of the subject as search terms, for example.

[0009] In fact, for example, a person's state of high stress (hereinafter referred to as a "high stress state") may manifest itself as behavior such as fidgeting or a high-pitched voice. However, a person's high stress state does not necessarily manifest itself as such outward behavior. Furthermore, the degree to which a person's high stress state manifests itself generally varies greatly from person to person. While the technology described in Patent Document 1 can detect the subject's behavior, it does not include a configuration for estimating a person's psychological state. Therefore, simply using the technology described in Patent Document 1 that analyzes a person's behavior makes it difficult to accurately detect a psychological state that differs from normal, for example, that of an employee.

[0010] Furthermore, Patent Literature 2 describes a technology for estimating emotions according to various physical states by learning the relationship between biometric information and emotional information for each physical state. This method estimates a physical state from the position and biometric information by learning in advance the relationship between the physical state and the position, biometric information, etc., and then estimates emotions according to that physical state. However, there are limitations to the physical state that can be estimated using only the position and biometric information, and it is difficult to accurately distinguish whether a change in biometric information is due to a change in the physical state or a change in emotional state using only the position and biometric information.

[0011] On the other hand, it has recently become known that psychological states, such as high stress levels, can be detected from biological information such as heart rate measured by biosensors. For example, Non-Patent Document 1 (2013) shows research results that indicate that changes in biological information such as heart rate variability (HRV) and pulse transmission time (PTT) occur when violent incidents occur. These biological information items are recorded, for example, 24 hours a day by sensors attached to a wearable device worn by the subject. Non-Patent Document 1 also suggests that such a method may be used to objectively measure stress factors in the workplace.

[0012] Furthermore, for example, Non-Patent Document 2 describes a study that estimates a victim's stress state based on their heart rate by performing machine learning using measured values ​​of their heart rate. The training data used for the machine learning is data that associates measured heart rate values ​​obtained when subjects are made to experience simulated acts of harassment using virtual reality (VR) with the results of a questionnaire filled out by the subjects about the subjective level of stress they felt at the time. This makes it possible to estimate the level of subjective stress based on measured heart rate values.

[0013] However, even if the same trouble occurs, each individual experiences stress differently and its impact on biometric information. Furthermore, there are significant individual differences in the degree to which psychological states are reflected in biometric values ​​such as heart rate. For example, Non-Patent Document 2 shows that even if the same virtual reality (VR) video is used to experience harassment, there are individual differences and differences depending on the user's attributes in how stress is perceived. Therefore, it is considered necessary to take individual differences into account when estimating a person's psychological state.

[0014] For the above reasons, it is difficult to appropriately estimate the psychological state of each individual by simply applying the techniques described in Non-Patent Documents 1 and 2, for example.

[0015] On the other hand, a technology different from the above-mentioned technology using biometric information is disclosed in, for example, Non-Patent Document 3. Non-Patent Document 3 describes a technology for analyzing text (utterances) in communication. For example, the technology described in Non-Patent Document 3 can detect conduct risks such as fraudulent billing, inappropriate sales, embezzlement, misappropriation, and insider trading by analyzing the text of emails and chats. Furthermore, for example, the technology described in Non-Patent Document 3 can also detect harassment acts in the workplace, such as sexual harassment and power harassment, and customer harassment caused by significant nuisance behavior by customers, by inferring the text of emails and chats.

[0016] However, the technology described in Non-Patent Document 3 requires analyzing sentences (utterances) within communication, and therefore cannot detect non-verbal harassment. Non-verbal harassment here refers to, for example, ignoring someone, making intimidating or unpleasant gestures or facial expressions, and making intimidatingly loud noises. Furthermore, even the same sentence (utterance) can be perceived very differently depending on the flow of the conversation, the context before and after, and the relationship between the parties to date. Therefore, simply analyzing sentences (utterances) as with the technology described in Non-Patent Document 3 is difficult to accurately determine whether a sentence constitutes trouble, such as harassment, and is therefore difficult to accurately estimate an employee's psychological state, such as high stress. Furthermore, the technology described in Non-Patent Document 3 also makes it difficult to grasp the actual level of stress felt by the target.

[0017] In view of the above circumstances, an object of the present invention is to provide a technique that can detect a person's psychological state with higher accuracy.

[0018] One aspect of the present invention is an estimation device comprising: a biometric information acquisition unit that acquires biometric information based on biometric measurement values ​​measured by a sensor that measures a biological phenomenon of a person; an environmental information acquisition unit that acquires environmental information including at least one of video and audio around the person; an activity status estimation unit that estimates the activity status of the person based on the biometric information and the environmental information; and a psychological state estimation unit that estimates the psychological state of the person based on the biometric information and the activity status.

[0019] Another aspect of the present invention is a computer-based estimation method, comprising: a biometric information acquisition step of acquiring biometric information based on biometric measurements taken by a sensor that measures a biological phenomenon of a person; an environmental information acquisition step of acquiring environmental information including at least one of video and audio around the person; an activity status estimation step of estimating the activity status of the person based on the biometric information and the environmental information; and a psychological state estimation step of estimating the psychological state of the person based on the biometric information and the activity status.

[0020] The present invention makes it possible to provide a technique that can detect a person's psychological state with higher accuracy.

[0021] FIG. 1 is a schematic diagram showing the overall configuration of a notification system 1 according to a first embodiment of the present invention. FIG. 2 is a block diagram showing the functional configuration of an estimation device 10 according to the first embodiment of the present invention. FIG. 3 is a diagram showing an example of analysis data calculation by a biometric information analysis unit 102 according to the first embodiment of the present invention. FIG. 4 is a diagram showing an example of data stored in a biometric information storage unit 103 according to the first embodiment of the present invention. FIG. 5 is a diagram showing a schematic diagram showing data analysis processing by a user situation analysis unit 105 according to the first embodiment of the present invention. FIG. 6 is a diagram showing an example of user activity situation analysis processing by a user situation analysis unit 105 according to the first embodiment of the present invention. FIG. 7 is a diagram showing an example of environmental information stored in an environmental information storage unit 108 according to the first embodiment of the present invention. FIG. 8 is a diagram showing an example of environmental information stored in the environmental information storage unit 108 according to the first embodiment of the present invention. FIG. 9 is a diagram showing an example of communication situations stored in a communication situation storage unit 106 according to the first embodiment of the present invention. FIG. 10 is a diagram showing an example of data collection by an estimation parameter analysis unit 109 according to the first embodiment of the present invention. FIG. 11 is a diagram showing an example of estimation parameters stored in an estimation parameter storage unit 110 according to the first embodiment of the present invention. FIG. 12 is a block diagram showing an example of data stored in a psychological state storage unit 113 according to the first embodiment of the present invention. FIG. 13 is a block diagram showing the functional configuration of a notification control device 40 according to the first embodiment of the present invention. FIG. 1 is a diagram showing an example of notification control processing by a notification control device 40 in the first embodiment of the present invention. FIG. 2 is a flowchart showing the operation of the estimation device 10 in the first embodiment of the present invention. FIG. 3 is a schematic diagram showing the overall configuration of a notification system 1a in a modified example of the first embodiment of the present invention. FIG. 4 is a schematic diagram for explaining an example of organization information stored in an organization information storage unit 404b in the second embodiment of the present invention. FIG. 5 is a diagram showing an example of data of hierarchical information included in organization information stored in an organization information storage unit 404b in the second embodiment of the present invention. FIG. 6 is a diagram showing an example of data of user information included in organization information stored in an organization information storage unit 404b in the second embodiment of the present invention. FIG. 7 is a diagram showing an example of data collection by an estimation parameter analysis unit 109c in the third embodiment of the present invention.It is a diagram showing an example of the estimated parameters stored in the estimated parameter storage unit 110c in the third embodiment of the present invention. It is a block diagram showing the functional configuration of the estimation device 10d in the first modification of the third embodiment of the present invention. It is a diagram showing an example of data collection by the teacher data generation unit 115 in the first modification of the third embodiment of the present invention. It is a block diagram showing the functional configuration of the estimation device 10e in the second modification of the third embodiment of the present invention. It is a schematic diagram showing the data analysis process by the user situation analysis unit 105e in the second modification of the third embodiment of the present invention. It is a diagram showing an example of the aggregation process and analysis process of environmental information by the user situation analysis unit 105g in the fifth embodiment of the present invention. It is a diagram showing an example of data collection by the estimated parameter analysis unit 109g in the fifth embodiment of the present invention. It is a diagram showing an example of the aggregation process of environmental information by the user situation analysis unit 105h in the modification of the fifth embodiment of the present invention. It is a diagram showing an example of data collection by the teacher data generation unit 115h in the modification of the fifth embodiment of the present invention. It is a block diagram showing the functional configuration of the estimation device 10i in the sixth embodiment of the present invention. It is a diagram showing an example of the data stored in the user attribute storage unit 119 of the estimation device 10i in the sixth embodiment of the present invention. It is a diagram showing an example of the data stored in the user attribute storage unit 119 of the estimation device 10i in the sixth embodiment of the present invention. It is a diagram showing an example of a new user when the estimation parameter analysis unit 109 of the estimation device 10i in the sixth embodiment of the present invention performs a search process for similar users. It is a diagram showing an example of a user corresponding to the search process for similar users by the estimation parameter analysis unit 109 of the estimation device 10i in the sixth embodiment of the present invention. It is a diagram showing an example of data for each user and activity situation input to the estimation parameter analysis unit 109 of the estimation device 10i in the sixth embodiment of the present invention. It is a diagram showing an example of data for each attribute and activity situation input to the estimation parameter analysis unit 109 of the estimation device 10i in the sixth embodiment of the present invention. It is a diagram for explaining the flow of the process by the estimation parameter analysis unit 109 of the estimation device 10i in the sixth embodiment of the present invention. It is a diagram for explaining the flow of the process by the estimation parameter analysis unit 109 of the estimation device 10i in the sixth embodiment of the present invention.FIG. for explaining the processing flow by the estimation parameter analysis unit 109 of the estimation device 10i in the sixth embodiment of the present invention. FIG. for explaining the classification using the user attributes by the estimation parameter analysis unit 109 of the estimation device 10i in the sixth embodiment of the present invention. FIG. for explaining the classification using the user attributes by the estimation parameter analysis unit 109 of the estimation device 10i in the sixth embodiment of the present invention. FIG. for explaining the classification using the user attributes by the estimation parameter analysis unit 109 of the estimation device 10i in the sixth embodiment of the present invention. FIG. for explaining the estimation process using the data of users whose tendencies and ways of change of activity status × biological information are similar by the estimation parameter analysis unit 109 of the estimation device 10i in the sixth embodiment of the present invention. FIG. for explaining the estimation process using the data of users whose tendencies and ways of change of activity status × biological information are similar by the estimation parameter analysis unit 109 of the estimation device 10i in the sixth embodiment of the present invention. FIG. for explaining the estimation process using the data of users whose tendencies and ways of change of activity status × biological information are similar by the estimation parameter analysis unit 109 of the estimation device 10i in the sixth embodiment of the present invention. FIG. for explaining the clustering process by the estimation parameter analysis unit 109 of the estimation device 10i in the sixth embodiment of the present invention. FIG. for explaining the clustering process by the estimation parameter analysis unit 109 of the estimation device 10i in the sixth embodiment of the present invention. FIG. for explaining the clustering process by the estimation parameter analysis unit 109 of the estimation device 10i in the sixth embodiment of the present invention. Flowchart showing the operation of the estimation device 10i in the sixth embodiment of the present invention. Flowchart showing the operation of the estimation device 10i in the sixth embodiment of the present invention. FIG. for explaining the method of using the data of users whose values and tendencies of environment (location attribute) × biological information are similar by the estimation parameter analysis unit 109 of the estimation device 10i in the sixth embodiment of the present invention. FIG. for explaining the method of using the data of users whose values and tendencies of environment (location attribute) × biological information are similar by the estimation parameter analysis unit 109 of the estimation device 10i in the sixth embodiment of the present invention.

[0022] Hereinafter, an embodiment of an estimation device and an estimation method of the present invention will be described in detail with reference to the drawings.

[0023] [Overview] The notification system in each embodiment described below is an information processing system having an estimation device according to one embodiment of the present invention. The notification system executes notification processing using an estimation method according to one embodiment of the present invention. The notification system in each embodiment described below acquires biometric values ​​of participants in a meeting, for example, at a workplace, using a sensor or the like. The notification system also acquires video data and audio data indicating the participants' words, actions, and behavior using an environmental information terminal, for example, a camera or microphone, and estimates the activity status of the participants. The notification system estimates the participants' psychological states based on the acquired biometric values ​​and the estimated activity status. The notification system is configured to perform notification processing to appropriate notification recipients when the estimated psychological state of the participants corresponds to a predetermined psychological state.

[0024] In the following embodiments, a notification system is described, as an example, that notifies a suitable person (e.g., a superior of a participant or a manager of a higher organization) when a communication problem (e.g., harassment) that occurs mainly in the workplace causes a meeting participant to experience a psychological state different from normal (e.g., a high-stress state). However, notification systems to which the estimation device and estimation method of the present invention can be applied are not limited to such workplace notification systems. For example, the estimation device and estimation method of the present invention can also be used as a notification system that notifies a suitable person (e.g., a teacher) when a communication problem (e.g., a problem between students or between a teacher and a student) that occurs in an educational institution such as a school causes a student to experience a psychological state different from normal.

[0025] The estimation device and estimation method of the present invention are not limited to use in the notification system described above, but can also be used for other purposes or used standalone as an estimation device specialized in the function of estimating psychological states.

[0026] First Embodiment A first embodiment of the present invention will be described below.

[0027] [Overall Configuration of Notification System] The overall configuration of the notification system 1 in the first embodiment will now be described. Fig. 1 is a schematic diagram showing the overall configuration of the notification system 1 in the first embodiment of the present invention. The notification system 1 in the first embodiment described below is a system including an estimation device 10 that estimates the psychological states of participants in a face-to-face meeting, for example, gathered in a conference room in an office.

[0028] 1 shows an example of a meeting being held in a conference room with three participants: a boss B and his subordinates A and D.

[0029] Hereinafter, such a face-to-face meeting will also be referred to as an "offline meeting." Meanwhile, a meeting in which participants join from various locations via communication lines using, for example, a web conferencing system will also be referred to as an "online meeting." Furthermore, a meeting in which some participants gather in a conference room or the like and some participants join via communication lines will also be referred to as a "mixed online and offline meeting." In addition, in a variation of the first embodiment described below, a notification system 1a including an estimation device 10a that estimates the psychological states of participants in a mixed online and offline meeting will be described.

[0030] As shown in Figure 1, the notification system 1 is composed of an estimation device 10, at least one personal terminal 20, an environmental information terminal 30a (office camera), an environmental information terminal 30b (office microphone), a notification control device 40, and a notification device 50.

[0031] The personal terminal 20 is a terminal device carried by each meeting participant. For example, the personal terminal 20 is a terminal device worn on the wrist (e.g., a smart watch) or a terminal device worn on a finger. The personal terminal 20 is equipped with sensors capable of acquiring biometric values ​​of the participants, such as a heart rate sensor, a pulse wave sensor, and an electrodermal activity sensor. The personal terminal 20 is also equipped with sensors capable of acquiring location information and activity status of the participants, such as a positioning sensor and an acceleration / angular velocity sensor.

[0032] The biometric values ​​are values ​​measured using the above-mentioned various sensors. For example, when a reflective optical pulse wave sensor is used, the biometric values ​​are values ​​indicating the level of reflected light from an LED at a specific time point. When an electrodermal activity sensor for detecting sweating is used, the biometric values ​​are voltage values.

[0033] The environmental information terminal 30a is a camera (office camera) installed in the conference room that captures the facial expressions, full-body appearance, and movements of participants during the meeting, as well as the overall state of the meeting. The environmental information terminal 30b is a microphone (office microphone) installed in the conference room that records the speech of participants during the meeting and other sounds generated. Note that multiple cameras and microphones may be installed in the conference room. In addition to the cameras and microphones, an environmental information terminal (not shown) that can measure temperature, humidity, illuminance, etc. may also be installed.

[0034] The estimation device 10 estimates the psychological states of meeting participants by combining biometric measurements, location information, and activity status acquired by the personal terminal 20, video data acquired by the environmental information terminal 30a, and audio data acquired by the environmental information terminal 30b. The estimation device 10 is an information processing device such as a general-purpose computer.

[0035] The notification control device 40 controls the notification processing by the notification device 50. The notification control device 40 determines notification destinations based on the psychological states of the meeting participants estimated by the estimation device 10, and causes the notification device 50 to issue a notification. The notification control device 40 is, for example, an information processing device such as a general-purpose computer.

[0036] The notification device 50 executes notification processing under the control of the notification control device 40. The notification device 50 is, for example, an information processing device such as a general-purpose computer.

[0037] The personal terminal 20 transmits biometric measurements, location information, and the like to the estimation device 10 via a communication network (not shown). For example, a wireless LAN (Local Area Network) router (not shown) is installed in a corner of a conference room, and the personal terminal 20 communicates with the wireless LAN router connected to the communication network via a wireless connection via Wi-Fi (registered trademark). This allows the personal terminal 20 to communicate with the notification control device 40 connected to the connected communication network.

[0038] The communication connection between the personal terminal 20 and the estimation device 10 is not limited to the above-described configuration in which they are connected using Wi-Fi (registered trademark). For example, the personal terminal 20 and the estimation device 10 may be connected by communication based on a short-range wireless communication standard such as Bluetooth (registered trademark). Alternatively, the personal terminal 20 and the estimation device 10 may be connected by communication based on a mobile phone communication standard such as LTE (Long Term Evolution) (registered trademark). Alternatively, the personal terminal 20 and the estimation device 10 may be connected by a combination of communication using the above-described multiple communication methods.

[0039] The personal terminal 20 and the estimation device 10 may be configured to be communicatively connected via a mobile information terminal (not shown), such as a smartphone, carried by a participant. In this case, for example, the personal terminal 20 and the smartphone may be connected via Bluetooth (registered trademark), and the smartphone and the estimation device 10 may be connected via Wi-Fi (registered trademark) or LTE (registered trademark).

[0040] [Configuration of Estimation Device] The configuration of the estimation device 10 will be described in more detail below.

[0041] 2 is a block diagram showing the functional configuration of the estimation device 10 according to the first embodiment of the present invention. As shown in Fig. 2, the estimation device 10 includes a personal data acquisition unit 101, a biometric information analysis unit 102, a biometric information storage unit 103, an environmental information acquisition unit 104, a user situation analysis unit 105, a communication situation storage unit 106, an activity situation storage unit 107, an environmental information storage unit 108, an estimation parameter analysis unit 109, an estimation parameter storage unit 110, an estimation parameter setting unit 111, a psychological state estimation unit 112, and a psychological state storage unit 113.

[0042] The personal data acquisition unit 101 acquires various data from the personal terminal 20 and outputs it to the biometric information analysis unit 102 and the user status analysis unit 105. The personal data acquisition unit 101 is a communication interface for connecting to and communicating with the personal terminal 20 via a communication network.

[0043] The biometric information analysis unit 102 acquires biometric values ​​that can be used to estimate psychological states, along with time and a user identifier, from the personal data acquisition unit 101. For example, the biometric values ​​include heart rate sensor data, pulse wave sensor data, and electrodermal activity sensor data. For example, the heart rate sensor data is data such as an instantaneous current value or voltage value. For example, the pulse wave sensor data is data such as a measurement value of instantaneous reflected light intensity. For example, the electrodermal activity sensor data is data such as an instantaneous current value or voltage value.

[0044] Note that if sensor data usable for estimating psychological states can be obtained from a device other than the personal terminal 20 (for example, a device not worn by the user, such as a non-contact pulse wave sensor), the biometric information analysis unit 102 may acquire the sensor data from such a device. For example, if data obtained by the environmental information terminals 30a and 30b can be used for estimating psychological states, the biometric information analysis unit 102 may also acquire sensor data from the environmental information acquisition unit 104, which will be described later.

[0045] The biological information analysis unit 102 analyzes the acquired biological measurements and calculates analysis data that can be used to analyze the psychological state. For example, the analysis data is data related to heart rate, pulse, or electrodermal activity. For example, the data related to heart rate is the heart rate, heartbeat interval, and fluctuation of the heartbeat interval obtained as a result of analyzing the electrocardiogram data. For example, the data related to pulse is the pulse rate, pulse wave interval, and fluctuation of the pulse wave interval. For example, the data related to electrodermal activity is information indicating the degree of increase or decrease in sweat level, such as the difference of the measured value from the average for an arbitrary period of time in the past (n hours), the amount of fluctuation of the measured value in a recent arbitrary period of time (n seconds), or the rate of fluctuation of the measured value in a recent arbitrary period of time (n seconds).

[0046] 3 is a diagram showing an example of calculation of analysis data by the biological information analysis unit 102 in the first embodiment of the present invention. As shown in FIG. 3, for example, the personal terminal 20 includes a heart rate sensor, a pulse wave sensor, a skin resistance sensor, a positioning sensor, and an acceleration / angular velocity sensor. The biological information analysis unit 102 acquires biological measurement values ​​measured by the heart rate sensor, pulse wave sensor, and skin resistance sensor.

[0047] 3 shows, as an example, biometric values ​​consisting of pulse wave sensor values ​​and biometric values ​​consisting of electrodermal activity sensor values. Also, as shown in FIG. 3, biometric values ​​obtained from the personal terminal 20 are associated with a user identifier (user ID) and a time. Based on this data, it is possible to recognize from which user, at what time, what kind of pulse wave sensor value or electrodermal activity sensor value was obtained.

[0048] 3 shows an example of analysis data generated by the biological information analysis unit 102 based on the above-mentioned biological measurement values, etc. As shown in FIG. 3, for example, the pulse wave analysis data is data in which values ​​indicating a user identifier (user ID), time, pulse rate, pulse interval, and fluctuation in pulse interval are associated with each other. Based on this analysis data, it is possible to recognize which user had what pulse rate, pulse interval, and fluctuation in pulse interval at what time.

[0049] 3, for example, the analysis data of electrical skin resistance is data in which values ​​indicating a user identifier (user ID), time, average ratio over the past hour, 10-second fluctuation interval, and 30-second fluctuation interval are associated with each other. Based on this analysis data, it is possible to identify which user, at what time, what average ratio over the past hour, 10-second fluctuation interval, and 30-second fluctuation interval of electrical skin resistance were.

[0050] The biological information analysis unit 102 stores the acquired biological measurement values ​​and the calculated analysis data in the biological information storage unit 103. Hereinafter, data including both the biological measurement values ​​and the analysis data will be collectively referred to as "biological information." Note that the biological information analysis unit 102 may calculate new analysis data using past biological measurement values ​​and analysis data stored in the biological information storage unit 103.

[0051] The biometric information storage unit 103 stores biometric values ​​acquired by the biometric information analysis unit 102 and analysis data calculated by the biometric information analysis unit 102. The biometric information storage unit 103 provides the biometric values ​​and analysis data in response to a request from an estimation parameter analysis unit 109 (described later) for calculating parameters for estimating a psychological state. The biometric information storage unit 103 also provides the biometric values ​​and analysis data at a specified time in response to a request from a psychological state estimation unit 112 (described later) for estimating a psychological state at each time. The biometric information storage unit 103 also provides the biometric values ​​and analysis data in response to a request from the biometric information analysis unit 102.

[0052] Fig. 4 is a diagram showing an example of data stored in the biometric information storage unit 103 in the first embodiment of the present invention. As shown in Fig. 4, the biometric measurement values ​​and analysis data shown in Fig. 3 are stored as biometric information in the biometric information storage unit 103. The biometric information storage unit 103 stores a data group of biometric measurement values ​​and analysis data as biometric information in a data format that enables the biometric measurement values ​​and analysis data to be provided according to a specified user identifier (user ID) and time (or time zone).

[0053] The environmental information acquisition unit 104 acquires video data from the environmental information terminal 30a (office camera) and audio data from the environmental information terminal 30b (office microphone), and outputs the data to the user situation analysis unit 105. The environmental information acquisition unit 104 may also acquire various sensor data from other environmental information terminals (not shown), such as temperature sensors and humidity sensors, and output the data to the user situation analysis unit 105. The environmental information acquisition unit 104 is a communication interface for connecting and communicating with the environmental information terminals 30a and 30b via a communication network.

[0054] The user situation analysis unit 105 analyzes various data obtained from the personal terminal 20 and the environmental information terminals 30a and 30b, etc., and generates communication situation history data indicating the history of the communication situation between the meeting participants, activity situation history data indicating the history of the activity situation of each meeting participant, and environmental information history data indicating the history of environmental information. Figure 5 is a schematic diagram showing the data analysis process by the user situation analysis unit 105 in the first embodiment of the present invention.

[0055] The user situation analysis unit 105 acquires data relating to the user's communication and activity status, and data relating to the user's environment, together with the time and the user identifier, from the personal data acquisition unit 101 and the environment information acquisition unit 104 .

[0056] For example, the data that the user situation analysis unit 105 acquires from the personal data acquisition unit 101 is data obtained by the personal terminal 20, such as location data and activity information.

[0057] For example, the location data is location information obtained by a satellite positioning system such as a GPS (Global Positioning System). Furthermore, for example, the activity information is data indicating the user's movement speed, the user's exercise type, and the exercise load level, calculated from the sensor values ​​of a positioning sensor and an acceleration / angular velocity sensor. The user's exercise type here refers to, for example, data indicating the user's exercise state, such as standing, walking, or running, and data indicating the user's posture, such as sitting or standing. Furthermore, for example, the activity information also includes data indicating various activity situations (such as desk work, non-desk work tasks, meetings, and travel) obtained by integrating the user's speech and communication status with the surroundings estimated from a microphone, and the movement, posture, and speech status estimated from the acceleration / angular velocity sensor status.

[0058] For example, the data acquired by the user situation analysis unit 105 from the environmental information acquisition unit 104 is data obtained by the environmental information terminals 30a and 30b, such as video data, audio data, and environmental data.

[0059] For example, video data is data showing a video of the situation during communication. For example, audio data is data showing audio accompanying the video, or data showing audio recorded separately from the video, such as a conversation during a meeting. For example, environmental data is information showing the environmental conditions of the place where the communication took place (e.g., a conference room), such as temperature, humidity, illuminance, and noise level.

[0060] The user situation analysis unit 105 analyzes the people and communication situations included in the video data and audio data. In the case of an offline meeting, the user situation analysis unit 105 identifies the people (user IDs) included in the video data and audio data, for example, by the following method. Note that a method for identifying users in the case of an offline meeting will be described later in a modified example of the first embodiment.

[0061] For example, when a person appears in the video, the user situation analysis unit 105 converts the coordinates of the person in the video into relative position information within the conference room (or building) or global position information based on the position and direction of the camera and the camera's angle of view. Note that information indicating the position and direction of the camera and the camera's angle of view is stored in advance in, for example, a storage medium (not shown) included in the estimation device 10. The user situation analysis unit 105 identifies the person (the user ID) in the video by comparing the converted position information with position information obtained from the personal terminal 20.

[0062] Alternatively, the user status analysis unit 105 identifies (the user ID of) a person in the video by, for example, comparing a facial image of the person in the video with a facial image of a previously prepared employee, etc. In this case, data associating the facial image of the employee, etc. with the user ID is stored in advance in, for example, a storage medium (not shown) included in the estimation device 10.

[0063] Alternatively, the user status analysis unit 105 may identify the position of the speaker based on, for example, audio information contained in multiple microphones. The user status analysis unit 105 then compares the identified position information with position information obtained from the personal terminal 20 to identify the person (the user ID) in the video.

[0064] Alternatively, the user status analysis unit 105 may identify (the user ID of) a person in the video by, for example, comparing the voiceprint of the person in the video with a previously prepared voiceprint of an employee, etc. In this case, data associating the voiceprint of the employee, etc. with the user ID is stored in advance in, for example, a storage medium (not shown) included in the estimation device 10.

[0065] Alternatively, the user status analysis unit 105 identifies (the user ID of) a person in the video based on data obtained from an entry / exit management system (not shown) that can identify the user ID of a person when the person enters or leaves a conference room. That is, the user status analysis unit 105 estimates that a person identified as being in the conference room is a person who appears in the video or a person whose speech is recorded in the audio.

[0066] Alternatively, the user status analysis unit 105 acquires, for example, location information of an area in a conference room where video images can be captured by a camera and audio can be captured by a microphone. When the user status analysis unit 105 confirms from the video that a person has entered the area, it compares the location information of the area with the location information obtained from the personal terminal 20 to identify the person (the user ID) in the video.

[0067] Alternatively, the user status analysis unit 105 acquires conference information including information about meeting participants from another system (not shown), such as an in-house conference system, and identifies the people (user IDs) in the video by identifying the meeting participants included in the conference information.

[0068] The user situation analysis unit 105 estimates communication between users and generates communication groups. For example, the user situation analysis unit 105 analyzes video and audio, and estimates users who are communicating with each other based on the users' positions, postures, facial orientations, speech situations, etc., and generates communication groups.

[0069] Alternatively, the user situation analysis unit 105 identifies users who are in the same conference room or booth, for example, from the results of video analysis or data obtained from a conference system (not shown), and generates a communication group.

[0070] Alternatively, the user status analysis unit 105 acquires information indicating employee schedules from a system (not shown), such as a scheduler, identifies users who have the same scheduled appointment (for example, the same meeting), and generates a communication group.

[0071] The user status analysis unit 105 estimates the user's exercise and behavior based on various data acquired by the personal terminal 20 and the environmental information terminals 30a and 30b. The user status analysis unit 105 integrates the estimated user's exercise and behavior and classifies them into a defined activity status. Figure 6 is a schematic diagram showing the analysis process of the user activity status by the user status analysis unit 105 in the first embodiment of the present invention.

[0072] The user situation analysis unit 105 analyzes data indicating the position, speed, and acceleration acquired by, for example, an acceleration / angular velocity sensor and a positioning sensor provided in the personal terminal 20, and estimates the user's posture and movement state.

[0073] The user status analysis unit 105 analyzes the user's posture and behavior from video data acquired by the environmental information terminal 30a, for example, and associates the user being analyzed with a user ID based on location information obtained from a personal terminal 20, etc.

[0074] The user status analysis unit 105 estimates and classifies the user's activity status by integrating information obtained from, for example, a positioning sensor, acceleration / angular velocity sensor, etc. of the personal terminal 20, information obtained from environmental information terminals such as the environmental information terminal 30a (office camera) and the environmental information terminal 30b (office microphone), and information obtained from the conference system and scheduler. The user status analysis unit 105 then generates activity status history data for each user. FIG. 6 shows an example of the activity status history data. As shown in FIG. 6, the activity status history data is data in which, for example, a user identifier (user ID), the start time of an activity, the end time of the activity, and information indicating the activity status are associated with each other.

[0075] The user situation analysis unit 105 aggregates and analyzes the environmental information output from the environmental information acquisition unit 104. The user situation analysis unit 105 aggregates data such as temperature, humidity, and illuminance (in a conference room, etc.) that may affect biometric information, and generates history data by associating the data with time information. Note that the user situation analysis unit 105 converts the data, as necessary, into numerical values ​​and data classifications that are easy to use in subsequent analysis processing. For example, the user situation analysis unit 105 can convert data indicating temperature and humidity into more easily usable data such as an "uncomfort index."

[0076] The activity status storage unit 107 stores the activity status history data for each user generated by the user status analysis unit 105. In response to a request from the estimation parameter analysis unit 109, the activity status storage unit 107 provides information indicating a time period corresponding to a specified activity status of a specified user. In addition, in response to a request from the estimation parameter setting unit 111, the activity status storage unit 107 provides information indicating the activity status of a specified user during a specified time period.

[0077] The environmental information storage unit 108 stores the environmental information aggregated by the user status analysis unit 105. For example, the environmental information storage unit 108 stores data such as temperature, humidity, and illuminance in chronological order for each user. This allows the environmental information storage unit 108 to provide environmental information for the user at each time.

[0078] 7 is a diagram showing an example of environmental information stored in the environmental information storage unit 108 in the first embodiment of the present invention. The environmental information shown in FIG. 7 is an example of data in the case where environmental information is stored as is for each user. As shown in FIG. 7, the environmental information storage unit 108 stores environmental information in which, for example, a user ID, a time, a temperature, a humidity, and an illuminance are associated with each other.

[0079] Alternatively, for example, the environmental information storage unit 108 stores location information for each user in chronological order. Separately, the environmental information storage unit 108 stores environmental information for each location in chronological order. This allows the environmental information storage unit 108 to provide the user with environmental information at each time.

[0080] FIG. 8 is a diagram showing an example of environmental information stored in the environmental information storage unit 108 in the first embodiment of the present invention. The environmental information shown in FIG. 8 is an example of data in the case where location information of a user's location is recorded and environmental information is stored for each location. As shown in FIG. 8, the environmental information storage unit 108 stores environmental information in which, for example, a user ID, a start time (stay start time), an end time (stay end time), and a location are associated with temperature. In addition, the environmental information storage unit 108 also stores environmental information in which, for example, a location, a time, a temperature, humidity, and illuminance are associated with each other.

[0081] The communication situation storage unit 106 stores the video data from the camera and the audio data from the microphone, etc., acquired by the environment information acquisition unit 104, together with information on communication groups, which are the communication situations between users, estimated by the user situation analysis unit 105. The communication situation storage unit 106 stores the communication situation content in a form that can be searched by specifying a user, time, etc.

[0082] 9 is a diagram showing an example of a communication situation stored in the communication situation storage unit 106 according to the first embodiment of the present invention. As shown in Fig. 9, the communication situation storage unit 106 stores, in addition to a group of communication situation contents, communication situation data in which correspondences between, for example, a communication group, a start time (communication start time), an end time (communication end time), participants (user ID groups), and communication situation contents are recorded.

[0083] The estimated parameter analysis unit 109 acquires past biometric values ​​and analysis data for each activity status. The estimated parameter analysis unit 109 acquires information indicating users and time periods in each activity status from the activity status storage unit 107. The estimated parameter analysis unit 109 also acquires biometric values ​​and analysis data for each user and time period from the biometric information storage unit 103, and compiles the acquired biometric values ​​and analysis data for each activity status to generate a history data group.

[0084] 10 is a diagram showing an example of data collection by the estimation parameter analysis unit 109 in the first embodiment of the present invention. As shown in Fig. 10, the estimation parameter analysis unit 109 first acquires an activity status list. The activity status list is a list of information indicating activity status such as "desk work" and "movement (walking)".

[0085] 10 , the estimated parameter analysis unit 109 acquires information associating an activity status in the activity status list with a user ID and a time period from the activity status storage unit 107. Next, as shown in FIG. 10 , the estimated parameter analysis unit 109 acquires information associating an activity status in the activity status list with various types of biological information (values ​​such as pulse rate, pulse wave peak interval, and skin electrical resistance) from the biological information storage unit 103.

[0086] The history data group may be generated by acquiring all relevant data, but in practice, this would result in a huge amount of data. Therefore, the history data group may be generated by acquiring only a certain number of recent data or only data from a certain period of time. The biometric values ​​acquired here are not data for each user, but data for each activity status of all users. In a third embodiment described below, the biometric values ​​acquired here are data for each user.

[0087] The estimated parameter analysis unit 109 generates estimated parameters of the psychological state from past biological information for each activity situation. An example of the generated estimated parameters is given below.

[0088] For example, it is generally known that when a person is feeling anger or shame, the pulse wave peak interval remains short. Therefore, for example, the estimation parameter analysis unit 109 estimates "anger / shame" when the pulse wave peak interval remains at or below -1 standard deviation for 30 seconds or more. For example, the estimation parameter storage unit 110, which will be described later, stores values ​​of the pulse wave peak interval (values ​​of -1 standard deviation) that differ for each activity status as estimation parameters for anger / shame (state).

[0089] It is also generally known that, for example, when a person is in a state of fear, disgust, or anger, a decrease in electrical skin resistance occurs. Therefore, for example, the estimation parameter analysis unit 109 estimates that the state is "fear, disgust, or anger" when the standard deviation of the electrical skin resistance over 10 seconds is equal to or less than a certain value. For example, the estimation parameter storage unit 110, which will be described later, stores the rate of change (rate of decrease in resistance value) of electrical skin resistance over 10 seconds, which differs for each activity state, as an estimation parameter for fear, disgust, or anger.

[0090] The estimation parameter may be a parameter for estimating the degree (level) of a psychological state, such as a "high stress state," instead of simply estimating the psychological state. Alternatively, the estimation parameter may be a combination of multiple parameters for estimating both the psychological state and its level. The estimation parameter may be, for example, a parameter for estimating the level of a high stress state from 0% to 100%. Alternatively, the estimation parameter may be, for example, a parameter for estimating the level of a high stress state in three levels: 40% to 60%, 60 to 80%, and 80% or more.

[0091] The estimation parameter storage unit 110 stores estimation parameters of the psychological state for each activity situation. The estimation parameter storage unit 110 provides the estimation parameters in response to a request from the estimation parameter setting unit 111 when estimating the psychological state.

[0092] 11 is a diagram showing an example of estimated parameters stored in the estimated parameter storage unit 110 according to the first embodiment of the present invention. As shown in Fig. 11, the estimated parameter storage unit 110 stores data in which, for example, activity situations are associated with estimated parameters for each psychological state (i.e., for each psychological state such as "high stress," "fear / tension," and "relaxed").

[0093] The estimation parameter setting unit 111 acquires data indicating the user's activity status during a time period for which the psychological state is to be estimated from the activity status storage unit 107. The estimation parameter setting unit 111 also acquires estimation parameters for the psychological state from the estimation parameter storage unit 110 based on the acquired data indicating the activity status.

[0094] The psychological state estimation unit 112 applies the estimation parameters acquired by the estimation parameter setting unit 111 to an estimation algorithm for the psychological state to estimate the psychological state.

[0095] For example, the psychological state estimation unit 112 uses a psychological state algorithm that estimates a high stress state when the pulse wave peak interval is less than a predetermined threshold for 30 seconds or more. In this case, for example, if the target user's activity state during a certain estimation period is "desk work" and the estimation parameter for that activity state is 0.75, the value of 0.75 is used as the predetermined threshold in the estimation algorithm. The psychological state estimation unit 112 then acquires biometric measurements for a time period for which the psychological state is to be estimated from the biometric information storage unit 103, and determines whether the user during that time period is in the psychological state to be estimated (here, "high stress state") based on the estimation algorithm.

[0096] The psychological state storage unit 113 stores data indicating the psychological state estimated by the psychological state estimation unit 112, together with data indicating a user ID and a time (period). Fig. 12 is a diagram showing an example of data stored in the psychological state storage unit 113 in the first embodiment of the present invention. As shown in Fig. 12, the psychological state storage unit 113 stores data in which, for example, a user ID, a start time (the time when the estimated psychological state was reached), an end time (the time when the estimated psychological state was no longer reached), and information indicating the estimated psychological state are associated with each other.

[0097] [Configuration of Notification Control Device] The configuration of the notification control device 40 will be described in more detail below.

[0098] 13 is a block diagram showing the functional configuration of the notification control device 40 according to the first embodiment of the present invention. As shown in FIG. 13, the notification control device 40 includes a notification determination unit 401, a notification information generation unit 402, a notification destination selection unit 403, an organization information storage unit 404, and a notification control unit 405.

[0099] FIG. 14 is a diagram showing an example of a notification control process performed by the notification control device 40 according to the first embodiment of the present invention.

[0100] The notification determination unit 401 acquires data to be notified for each user from the psychological state storage unit 113 of the estimation device 10. As shown in Fig. 14 , for example, the notification determination unit 401 acquires data in which a user ID, a start time (the time when the estimated psychological state was reached), an end time (the time when the estimated psychological state was no longer reached), and information indicating the estimated psychological state are associated with each other from the psychological state storage unit 113. If data to be notified exists, the notification determination unit 401 outputs data such as the corresponding user, time period, and estimated psychological state to the notification information generation unit 402.

[0101] The notification information generation unit 402 acquires communication situation content for a relevant user and time period from the communication situation storage unit 106 of the estimation device 10. As shown in Fig. 14 , for example, the notification information generation unit 402 acquires from the communication situation storage unit 106 a group of communication situation contents in which a communication group, a start time (communication start time), an end time (communication end time), participants (user ID groups), and communication situation content are associated with each other.

[0102] Instead of acquiring the communication situation content, the notification information generating unit 402 may acquire information indicating a means of accessing the communication situation content stored on a server or the like (for example, a link such as a URL).

[0103] The notification information generation unit 402 generates notification content by extracting a relevant portion of the acquired communication situation content as necessary. Note that content extraction may not be limited to the content for the relevant time period, but may also include content for a period that includes a certain period before and after the relevant time period. Alternatively, all content for the duration of the relevant communication group may be extracted.

[0104] The notification content may be stored in a content server (not shown) or the like. In this case, the notification device 50 performs notification by transmitting information indicating a means of accessing the notification content (for example, a link such as a URL). Alternatively, the notification device 50 may perform notification by transmitting an email or the like to which the notification content itself is attached.

[0105] The notification destination selection unit 403 acquires notification destination information from the organization information storage unit 404. As shown in Fig. 14, for example, the notification destination selection unit 403 acquires notification destination information from the organization information storage unit 404, in which information indicating a user (such as a name or a user ID), information indicating the user's job title, a means of contacting the user, and an address (such as an email address) that is a contact point for the user are associated with each other.

[0106] The organization information storage unit 404 stores in advance a single notification destination information that is used in common for all users as a notification destination. In a second embodiment described later, the notification destination is determined taking into consideration the organization information.

[0107] The notification control unit 405 transmits notification information to the notification device 50 based on the notification destination information acquired by the notification destination selection unit 403 .

[0108] [Operation of Estimation Device] The following describes the operation of the estimation device 10. Fig. 15 is a flowchart showing the operation of the estimation device 10 in the first embodiment of the present invention. Note that the operation of the estimation device 10 shown in the flowchart of Fig. 15 is an example, and the order of some of the operation steps can be changed.

[0109] First, the personal data acquisition unit 101 acquires various data from the personal terminal 20 and outputs it to the biometric information analysis unit 102 and the user situation analysis unit 105. The biometric information analysis unit 102 acquires biometric measurements that can be used to estimate the psychological state, together with the time and the user identifier, from the personal data acquisition unit 101 (step S001).

[0110] Next, the user status analysis unit 105 acquires activity information (e.g., data indicating the user's movement speed, the user's exercise type, and exercise load level, etc., calculated from the sensor values ​​of the positioning sensor and the acceleration / angular velocity sensor) that can be used to analyze the user's activity status from the personal data acquisition unit 101 (step S002).

[0111] Next, the environmental information acquisition unit 104 acquires video data from the environmental information terminal 30a (office camera) and audio data from the environmental information terminal 30b (office microphone), and outputs them to the user status analysis unit 105 (steps S003 and S004). Next, the environmental information acquisition unit 104 acquires various sensor data, such as temperature sensors and humidity sensors, from other environmental information terminals (not shown), and outputs them to the user status analysis unit 105 (step S005).

[0112] Next, the biometric information analysis unit 102 analyzes the acquired biometric measurements and calculates analysis data that can be used to analyze psychological states (step S006). Next, the user status analysis unit 105 analyzes various data obtained from the personal terminal 20 and the environmental information terminals 30a and 30b, etc., to estimate the communication status between the meeting participants (step S007). Next, the user status analysis unit 105 outputs a combination of the user's communication status and video / audio data (step S008).

[0113] Next, the user status analysis unit 105 estimates the activity status of each meeting participant based on a combination of the activity information, user communication status, and video and audio data acquired in step S002 (step S009). Next, the user status analysis unit 105 estimates environmental information based on the information acquired from the environmental information acquisition unit 104 (step S010). Next, if it is time to generate estimated parameters (step S011, YES), the estimated parameter analysis unit 109 acquires the accumulated activity status and, if necessary, environmental information and biometric information history (step S012). Then, for each acquired activity status (if necessary, environmental information), it generates estimated parameters using the biometric information (step S013). The generated estimated parameters are stored in the estimated parameter storage unit (step S014).

[0114] Next, if it is time to estimate the psychological state (step S015, YES), the estimation parameter for the psychological state is acquired according to the activity status (and environmental information) (step S016), and the psychological state is estimated from the biological information (step S017). First, the estimation parameter setting unit 111 acquires the estimation parameter for the psychological state according to the activity status and environmental information (step S016). The psychological state estimation unit 112 applies the estimation parameter acquired by the estimation parameter setting unit 111 to an estimation algorithm for the psychological state to estimate the psychological state (step S017).

[0115] Next, the psychological state estimation unit 112 uses, for example, the psychological state algorithm described above, which estimates a high stress state when the pulse wave peak interval is less than a predetermined threshold for 30 seconds or more. For example, if the target user's activity state during a certain estimation period is "desk work" and the estimation parameter for that activity state is 0.75, the value of 0.75 is used as the predetermined threshold in the estimation algorithm. The psychological state estimation unit 112 then acquires biometric measurements for the time period for which the psychological state is to be estimated from the biometric information storage unit 103, and determines whether the user during that time period is in the psychological state to be estimated (here, "high stress state") based on the estimation algorithm (step S018).

[0116] Next, if the psychological state estimation unit 112 determines that the user is not in a high stress state (step S019), the estimation device 10 ends the operation shown in the flowchart of Fig. 15. On the other hand, if the psychological state estimation unit 112 determines that the user is in a high stress state (step S020), the estimation device 10 outputs the user ID, time information, and psychological state (step S021), and the estimation device 10 ends the operation shown in the flowchart of Fig. 15.

[0117] As described above, the estimation device 10 according to the first embodiment of the present invention estimates a psychological state of a person using biometric values ​​acquired from a sensor and the person's activity status estimated from video data and audio data. The estimation device 10 is configured to notify a notification destination when a predefined psychological state is detected in a situation where communication such as a meeting is taking place.

[0118] In this way, the estimation device 10 in the first embodiment of the present invention is configured to estimate a person's activity status based on biometric information obtained from the personal terminal 20, video data obtained from the environmental information terminal 30a, and audio data obtained from 30b, and then to estimate the person's psychological state based on the biometric information and activity status, thereby making it possible to estimate a person's psychological state with greater accuracy than conventional technology.

[0119] <Modification of First Embodiment> A modification of the first embodiment of the present invention will now be described.

[0120] [Overall Configuration of Notification System] The overall configuration of a notification system 1a in a modified example of the first embodiment will now be described. Fig. 16 is a schematic diagram showing the overall configuration of a notification system 1a in a modified example of the first embodiment of the present invention. The notification system 1a in the modified example of the first embodiment described below is a system including an estimation device 10a that can estimate the psychological states of participants in a meeting that is a mixture of online and offline, where some participants gather in a conference room, for example, and some participants join via a communication line.

[0121] 16 shows an example of a meeting being held between two participants gathered in a conference room and two participants participating online from a remote environment. The four participants in FIG. 16 are Boss B and his subordinates A, D, and E.

[0122] As shown in FIG. 16, the notification system 1a includes an estimation device 10, at least one personal terminal 20, an environmental information terminal 30a (office camera), an environmental information terminal 30b (office microphone), at least one environmental information terminal 30c (PC camera for web conferences), at least one environmental information terminal 30d (PC microphone for web conferences), a notification control device 40, and a notification device 50.

[0123] As shown in Fig. 16, the notification system 1a in the modification of the first embodiment has a configuration in which at least one environmental information terminal 30c (web conference PC camera) and at least one environmental information terminal 30d (web conference PC microphone) are further added to the configuration of the notification system 1 in the modification of the first embodiment shown in Fig. 1. In the system configuration illustrated in Fig. 16, the environmental information terminals 30c and 30d are connected not only to a PC used for the web conference in the conference room (office), but also to PCs used for the web conference in the remote environments of each participant participating online.

[0124] Similar to the personal terminals 20 of participants gathered in the conference room, the personal terminals 20 of participants participating in the online meeting also transmit biometric measurements, location information, and the like to the estimation device 10 via a communication network (not shown). For example, a wireless LAN router (not shown) is installed in a corner of a remote environment (e.g., a participant's home), and the personal terminal 20 communicates with the wireless LAN router connected to the communication network via a wireless connection via Wi-Fi (registered trademark). This allows the personal terminal 20 to communicate with the notification control device 40 connected to the connected communication network.

[0125] As mentioned above, the communication connection between the personal terminal 20 and the estimation device 10 is not limited to a configuration in which they are connected using Wi-Fi (registered trademark) as described above. For example, the personal terminal 20 and the estimation device 10 may be connected by communication based on a short-range wireless communication standard such as Bluetooth (registered trademark). Alternatively, the personal terminal 20 and the estimation device 10 may be connected by communication based on a mobile phone communication standard such as LTE (registered trademark). Alternatively, the personal terminal 20 and the estimation device 10 may be connected by a combination of communication using multiple communication methods such as those described above.

[0126] As described above, the personal terminal 20 and the estimation device 10 may be configured to be communicatively connected via a mobile information terminal (not shown), such as a smartphone, carried by a participant. In this case, for example, the personal terminal 20 and the smartphone may be connected via Bluetooth (registered trademark), and the smartphone and the estimation device 10 may be connected via Wi-Fi (registered trademark) or LTE (registered trademark).

[0127] 14, an environmental information terminal 30c (web conference PC camera) and an environmental information terminal 30d (web conference PC microphone) are connected to a web conference personal computer (PC). Video data representing images captured by the environmental information terminal 30c and audio data representing audio recorded by the environmental information terminal 30 are transmitted to the estimation device 10 via the web conference PC and a communication network.

[0128] The estimation device 10 can estimate the psychological states of the meeting participants by further using the video data acquired from the environmental information terminal 30c and the audio data acquired from the environmental information terminal 30d. Therefore, the estimation device 10 can also estimate the psychological states of participants who participate in a meeting online. This also allows the estimation device 10 to more accurately estimate the psychological states of participants gathered in a conference room.

[0129] In the case of participants who join a meeting online, the estimation device 10 can easily identify the person by acquiring a user identifier (user ID) input into the web conference system.

[0130] Second Embodiment A second embodiment of the present invention will now be described.

[0131] In the notification control device 40 of the notification system 1 in the first embodiment described above, the organization information storage unit 404 pre-stores single notification destination information that is used commonly as a notification destination for all users, and the notification control device 40 transmits notification information to the notification device 50 based on the single notification destination information. In other words, in the notification system 1 in the first embodiment described above, the notification device 50 is configured to always send notifications to the same single notification destination.

[0132] In contrast, a notification control device (hereinafter referred to as a "notification control device 40b") of a notification system (hereinafter referred to as a "notification system 1b") in a second embodiment described below determines notification destinations based on the psychological states of meeting participants estimated by the estimation device 10, taking into account correlations between people (person correlations), and causes the notification device 50 to issue notifications. The person correlations referred to here refer to relationships between people in a hierarchical structure, such as subordinates, superiors, and upper-level organizational managers, within an organization such as a company. However, person correlations may also be relationships between people that do not have a hierarchical structure.

[0133] The following description focuses on the differences in the configuration of notification system 1b in the second embodiment from notification system 1 in the first embodiment. As with the first embodiment, notification system 1b in the second embodiment can also be configured like notification system 1a in the modified version of the first embodiment, which includes an estimation device that estimates the psychological states of participants in a meeting that includes both online and offline activities.

[0134] [Configuration of Notification Control Device] The configuration of the notification control device 40b in the second embodiment will be described below.

[0135] The block diagram showing the functional configuration of the notification control device 40b in the second embodiment is similar to the block diagram showing the functional configuration of the notification control device 40 in the first embodiment shown in Fig. 13 described above. The notification control device 40b in the second embodiment described below differs from the notification control device 40 in the first embodiment described above in the configurations of the notification destination selection unit and the organization information storage unit. Hereinafter, the notification destination selection unit and the organization information storage unit of the notification control device 40b in the second embodiment will be referred to as the "notification destination selection unit 403b" and the "organization information storage unit 404b," respectively.

[0136] The notification destination selection unit 403b acquires notification destination information from the organization information storage unit 404b based on the user (user ID) to be notified and the psychological state estimated by the estimation device 10. For example, if the estimation device 10 estimates that a certain user is in a "high stress state," the notification destination selection unit 403b acquires notification destination information of the user's superior or the notification destination information of a manager of a higher-level organization.

[0137] The organization information storage unit 404b stores organization information in advance. The organization information includes information indicating the hierarchical structure (person correlation) of people within an organization or a team (hereinafter also referred to as "hierarchical information") and information about each person (hereinafter also referred to as "user information"). Fig. 17 is a schematic diagram for explaining an example of organization information stored in the organization information storage unit 404b in the second embodiment of the present invention.

[0138] 17 is a visual representation of an example of an organizational structure and personnel allocation identified based on the above organizational information. In Fig. 17, the leaders of each organization are marked with a star, organizations with hierarchical relationships are connected by solid lines, and each organization is connected to its employees by dashed lines.

[0139] 17 includes seven organizations: "Head Office," "Sales Department," "Sales Group 1," "Sales Group 2," "Development Department," "Development Section 1," and "Development Section 2." For example, employees 2 to 4 belong to the Sales Department, and employees 11 and 12 belong to Development Section 1.

[0140] In the organizational information exemplified in FIG. 17, all organizations except the highest-level organization ("Headquarters" in FIG. 17) always have a higher-level organization. Also, one leader is assigned to each organization. A leader is a person designated to be aware of problems in the organization and is a person who can be a recipient of notifications from the notification device 50. Note that the organizational information may include an organization that has multiple higher-level organizations. Also, the organizational information may include an organization in which multiple leaders are set as notification recipients.

[0141] The structure of the hierarchical information included in the organizational information will be described below. Fig. 18 is a diagram showing an example of the data structure of the hierarchical information included in the organizational information stored in the organizational information storage unit 404b according to the second embodiment of the present invention.

[0142] As shown in FIG. 18 , the hierarchical information is data in which, for example, an "organization ID," an "organization name," a "superordinate organization ID," and a "leader user ID" are associated with each other. Information about one organization is set in each record of the hierarchical information. For example, as shown in FIG. 18 , it is set that the organization name of the organization with the organization ID "002" is "Sales Department," the organization ID of the upper organization of the organization with the organization ID "002" is "001" (i.e., "Headquarters"), and the user ID of the leader of the organization with the organization ID "002" is "000002."

[0143] 18, it is identified that the superior organization of the Sales 1 Group (organization ID: 004) is the Sales Department (organization ID: 002), and that the user ID of the leader of the Sales 1 Group is "000005." Furthermore, it is identified that the superior organization of the Sales Department (organization ID: 002), which is the superior organization, is the Headquarters (organization ID: 001), and that the user ID of the leader of the Sales Department is "000002." Based on such hierarchical information, the notification destination selection unit 403b can identify the superior organization of a certain organization and the leader of that superior organization.

[0144] The configuration of user information included in the organization information will be described below. Fig. 19 is a diagram showing an example of the data configuration of user information included in the organization information stored in the organization information storage unit 404b according to the second embodiment of the present invention.

[0145] As shown in FIG. 19 , the user information is data in which, for example, a "user ID," a "name," an "organization ID," a "job title," and an "email address" are associated with each other. Each record of the user information contains information about one person (user). For example, as shown in FIG. 19 , the name of the person with the user ID "000001" is "(name of employee 1)," the organization ID of the organization to which the person with the user ID "000001" belongs is "001," the job title of the person with the user ID "000001" is "general manager," and the email address of the person with the user ID "000001" is registered.

[0146] 19, it is identified that, for example, Employee 6 (User ID: 000006) belongs to Sales 1 Group (Organization ID: 0004). By combining this with the hierarchical information illustrated in Fig. 18, it is identified that Employee 6's superior is Employee 5 (User ID: 000005), who is the leader of Sales 1 Group, and that the leader of that higher-level organization is Employee 2 (User ID: 000002), and that the leader of that higher-level organization is Employee 1 (User ID: 000001), and so on.

[0147] In this way, by referencing the hierarchical information and the user information together, it becomes possible to specify information related to organizations and people (users) in a chain. Furthermore, since the notification destination information (e.g., email address) of each person (each user) is set in the user information, the notification destination selection unit 403b can also specify the notification destination information of the leader who will be the notification destination.

[0148] The organization information storage unit 404b provides notification destination information in response to a request from the notification destination selection unit 403b. For example, when "notifying the superior of a participant" is to be performed in accordance with the estimated psychological state of the participant of the meeting, the notification destination selection unit 403b notifies the leader (= superior) of the organization to which the user (participant) belongs as the notification destination as a basic notification policy.

[0149] Furthermore, the notification destination selection unit 403b, for example, when "the communication situation is such that the participant's superior is included in the communication group (i.e., there is a possibility that one of the people communicating with the participant is the participant's superior)," sends a notification to the leader of the upper organization (= upper organization administrator) of the organization to which the user (participant) belongs as the notification destination as a basic notification policy.

[0150] In addition, for example, the notification destination selection unit 403b can set, as a basic notification policy, the notification destination to be the leaders (=supervisors) of the organizations to which all users (participants) included in the communication group belong.In addition, for example, the notification destination selection unit 403b can set, as a basic notification policy, the notification destination to be the leaders (=administrators of the upper organizations) of the organizations n levels above the organizations to which all users (participants) included in the communication group belong.

[0151] In this way, for example, organizational information including hierarchical information in the data format shown in Figure 18 and user information in the data format shown in Figure 19 is stored in advance in the organizational information storage unit 404b, so that the notification system 1b in the second embodiment can flexibly perform notification processing in accordance with various notification policies defined, for example, as described above.

[0152] Third Embodiment A third embodiment of the present invention will now be described.

[0153] In the estimation device 10 of the notification system 1 in the first embodiment described above, the estimation parameter analysis unit 109 is configured to acquire past biometric measurements and analysis data for each activity status and generate estimation parameters for the psychological state from the past biometric information for each activity status. The estimation parameter setting unit 111 is configured to acquire data indicating the user's activity status during a time period for which the psychological state is to be estimated and acquire estimation parameters for the psychological state based on the acquired data indicating the activity status. The psychological state estimation unit 112 is configured to apply the estimation parameters acquired by the estimation parameter setting unit 111 to a psychological state estimation algorithm to estimate the psychological state.

[0154] As described above, the estimation device 10 of the notification system 1 in the first embodiment is configured to estimate a user's psychological state based on biometric information. However, the estimation device 10 in the first embodiment does not take into account individual differences in biometric measurements. It is generally known that even if similar changes in psychological state occur, the changes manifested as biometric values ​​vary from person to person. Because the estimation device 10 in the first embodiment estimates a user's psychological state without taking into account individual differences in biometric measurements, the estimation accuracy may be low.

[0155] In contrast, an estimation device (hereinafter referred to as "estimation device 10c") of a notification system (hereinafter referred to as "notification system 1c") according to a third embodiment described below is configured to set a threshold value for biometric values ​​used to estimate a psychological state for each user and estimate the user's psychological state taking individual differences into consideration. This enables the estimation device 10c according to the third embodiment to estimate the user's psychological state with higher accuracy.

[0156] The following description focuses on the differences in the configuration of the notification system 1c in the third embodiment from the notification system 1 in the first embodiment. As with the first embodiment, the notification system 1c in the third embodiment can also be configured like the notification system 1a in the modified version of the first embodiment, which includes an estimation device that estimates the psychological states of participants in a meeting that includes both online and offline activities.

[0157] [Configuration of Estimation Device] The configuration of the estimation device 10c according to the third embodiment will be described below.

[0158] Note that the block diagram showing the functional configuration of the estimation device 10c in the third embodiment is similar to the block diagram showing the functional configuration of the estimation device 10 in the first embodiment shown in FIG. 2. The estimation device 10c in the third embodiment described below differs from the estimation device 10 in the first embodiment described above in the configurations of an estimation parameter analysis unit, an estimation parameter storage unit, and an estimation parameter setting unit. Hereinafter, the estimation parameter analysis unit, the estimation parameter storage unit, and the estimation parameter setting unit of the estimation device 10c in the third embodiment will be referred to as an "estimation parameter analysis unit 109c," an "estimation parameter storage unit 110c," and an "estimation parameter setting unit 111c," respectively.

[0159] The estimated parameter analysis unit 109c acquires past biometric values ​​and analysis data for each combination of a user and an activity status (hereinafter also referred to as "user x activity status"). The estimated parameter analysis unit 109c acquires information indicating past time periods corresponding to the combination of a user and an activity status from the activity status storage unit 107. The estimated parameter analysis unit 109c also acquires biometric values ​​and analysis data for each user x activity status from the biometric information storage unit 103, and generates a history data group by compiling the acquired biometric values ​​and analysis data for each user x activity status.

[0160] 20 is a diagram showing an example of data collection by the estimation parameter analysis unit 109c in the third embodiment of the present invention. As shown in Fig. 20, the estimation parameter analysis unit 109c first acquires a list of users x activity situations. The user x activity situation list is a list of combinations of "user ID" and information indicating an activity situation, such as "desk work" and "movement (walking)."

[0161] 20 , the estimated parameter analysis unit 109c acquires information associating a combination of a user ID and an activity status in the user x activity status list with a time period from the activity status storage unit 107. Next, as shown in FIG. 20 , the estimated parameter analysis unit 109c acquires information associating a user x activity status in the user x activity status list with various types of biometric information (values ​​such as pulse rate, pulse wave peak interval, and skin electrical resistance) from the biometric information storage unit 103.

[0162] The biometric information history data group may be generated by acquiring all relevant data, but in practice, the amount of data would be enormous. Therefore, the biometric information history data group may be generated by acquiring only a certain number of recent data or only data from a certain period of time. The biometric information acquired here is not data for each activity status of all users as in the first embodiment, but data for each user.

[0163] The estimated parameter analysis unit 109c generates estimated parameters of the psychological state from the past biological information for each user and activity situation. An example of the generated estimated parameters is given below.

[0164] For example, it is generally known that when a person is feeling anger or shame, the pulse wave peak interval remains short. Therefore, for example, the estimation parameter analysis unit 109c estimates "anger / shame" when the pulse wave peak interval remains equal to or smaller than the standard deviation minus one for 30 seconds or more. For example, the estimation parameter storage unit 110c, which will be described later, stores a pulse wave peak interval value (value of the standard deviation minus one) that differs for each user and activity status as an estimation parameter for anger / shame (state).

[0165] It is also generally known that, for example, when a person is in a state of fear, disgust, or anger, a decrease in electrical skin resistance occurs. Therefore, for example, the estimation parameter analysis unit 109c estimates that the state is "fear, disgust, or anger" when the standard deviation of the electrical skin resistance over 10 seconds is equal to or less than a certain value. For example, the estimation parameter storage unit 110c, which will be described later, stores, as estimation parameters for fear, disgust, and anger, the rate of change (rate of decrease in resistance value) of electrical skin resistance over 10 seconds, which differs for each user and activity status.

[0166] As in the first embodiment, the estimation parameter may be a parameter for estimating the degree (level) of a psychological state, rather than a parameter for simply estimating a psychological state such as a "high stress state." Alternatively, the estimation parameter may be a combination of multiple parameters for estimating both the psychological state and its level. The estimation parameter may be, for example, a parameter for estimating the level of a high stress state from 0% to 100%. Alternatively, the estimation parameter may be, for example, a parameter for estimating the level of a high stress state in three levels: 40% to 60%, 60 to 80%, and 80% or more.

[0167] The estimation parameter storage unit 110c stores estimation parameters of the psychological state for each user and activity situation. When estimating the psychological state, the estimation parameter storage unit 110c provides the estimation parameters in response to a request from the estimation parameter setting unit 111c.

[0168] 21 is a diagram showing an example of estimated parameters stored in the estimated parameter storage unit 110c according to the third embodiment of the present invention. As shown in Fig. 21, the estimated parameter storage unit 110c stores data in which the user's activity status is associated with estimated parameters for each psychological state (i.e., for each psychological state such as "high stress," "fear / tension," and "relaxed").

[0169] The estimation parameter setting unit 111c acquires data indicating the user's activity status during a time period for which the psychological state is to be estimated from the activity status storage unit 107. The estimation parameter setting unit 111c also acquires estimation parameters for the psychological state from the estimation parameter storage unit 110c based on the acquired data indicating the user x activity status.

[0170] As in the first embodiment, the psychological state estimation unit 112 applies the estimation parameters acquired by the estimation parameter setting unit 111c to the psychological state estimation algorithm to estimate the psychological state.

[0171] Two variations of the third embodiment will be described below. In addition to the configuration of the estimation device 10c in the third embodiment, each of the estimation devices in the notification systems further includes a configuration for performing machine learning on individual differences in biometric measurements to estimate the psychological state of the user.

[0172] <First Modification of Third Embodiment> A first modification of the third embodiment of the present invention will now be described.

[0173] In a notification system according to a first modified example of the third embodiment (hereinafter referred to as a "notification system 1d"), a user inputs information indicating a psychological state. Then, an estimation device according to a first modified example of the third embodiment (hereinafter referred to as an "estimation device 10d") generates training data using the psychological state based on the input information as a correct answer label, and performs supervised machine learning.

[0174] For example, if the personal terminal 20 is provided with an input interface (e.g., input buttons, etc.) that accepts input of information indicating a psychological state, the user himself / herself can input information indicating the psychological state. For example, when the user performs an input operation (e.g., pressing an input button, etc.) on the personal terminal 20, training data can be generated by associating biometric values ​​for a certain period immediately before the input operation with the information indicating the psychological state input by the user, and the training data can be used for machine learning.

[0175] In general, it is difficult to constantly request a user to input information indicating a psychological state that will be used as a correct label for the training data. Therefore, the estimation device 10d can generate training data using the most recent biometric measurement values ​​when the user performs an input operation, as described above, and use the generated training data for machine learning.

[0176] The input button does not have to be a physical button, but may be an input button displayed on the screen. Furthermore, the input operation on the personal terminal 20 may be performed by a specific gesture or the like. Furthermore, the input operation on the personal terminal 20 may be initiated by the user himself / herself, or the user may be prompted to initiate the input operation by receiving a notification from a notification device such as the notification device 50 and displaying information on the screen of the personal terminal 20 that prompts the user to input their psychological state.

[0177] The following description focuses on the differences in the configuration of the notification system 1d in the first modified example of the third embodiment from the notification system 1 in the first embodiment described above. As in the first embodiment described above, the notification system 1d in the first modified example of the third embodiment can also be configured like the notification system 1a in the modified example of the first embodiment, which has an estimation device that estimates the psychological states of participants in a meeting that includes a mixture of online and offline activities.

[0178] [Configuration of Estimation Device] The configuration of the estimation device 10d will be described below.

[0179] 22 is a block diagram showing the functional configuration of an estimation device 10d according to a first modified example of the third embodiment of the present invention. As shown in FIG. 22, the estimation device 10d includes a personal data acquisition unit 101, a biometric information analysis unit 102, a biometric information storage unit 103, an environmental information acquisition unit 104, a user situation analysis unit 105, a communication situation storage unit 106, an activity situation storage unit 107, an environmental information storage unit 108, a mental state estimation unit 112, a mental state storage unit 113, a label mental state storage unit 114, a teacher data generation unit 115, a learning execution unit 116, a learning model storage unit 117, and a learning model calling unit 118.

[0180] As shown in Figure 22, the functional configuration of the estimation device 10d in the first variant of the third embodiment differs from the functional configuration of the estimation device 10 in the first embodiment described above in that the estimation parameter analysis unit 109, the estimation parameter storage unit 110, and the estimation parameter setting unit 111 are omitted, and instead a label psychological state storage unit 114, a teacher data generation unit 115, a learning execution unit 116, a learning model storage unit 117, and a learning model calling unit 118 are added.

[0181] When information indicating a psychological state is input by a user's input operation using, for example, an input button or the like provided on the personal terminal 20, the label psychological state storage unit 114 stores data in which, for example, a "user ID," a "psychological state" (i.e., a correct label), and a "time period" (i.e., a time period during which the user is estimated to have been in the above-mentioned psychological state) are associated with each other. In other words, the data stored in the label psychological state storage unit 114 means that when a user inputs information indicating a psychological state, the user is considered to have been in that psychological state for a certain period immediately prior to the input timing.

[0182] Although the estimation device 10d is configured to automatically estimate the time period during which the user was in a certain psychological state based on the timing of the user's input operation, the present invention is not limited to this configuration. For example, the estimation device 10d may be configured so that the user himself / herself inputs the time period during which the user was in a certain psychological state using the personal terminal 20 or the like.

[0183] The label psychological state storage unit 114 provides the above data in response to a request from the teacher data generation unit 115.

[0184] The teacher data generation unit 115 acquires information indicating a corresponding time period for each user's psychological state from the label psychological state storage unit 114. The teacher data generation unit 115 also acquires information indicating an activity status for each acquired user x psychological state and for each time period from the activity status storage unit 107, and acquires biometric values ​​from the biometric information storage unit 103. The teacher data generation unit 115 compiles the acquired data and generates data in which the user (user ID), information indicating the activity status, information indicating the psychological state, time period, and biometric values ​​are associated with each other.

[0185] The learning execution unit 116 performs machine learning using training data in which psychological states and activity situations x biometric values ​​are associated for each user, based on the data generated by the training data generation unit 115. A plurality of trained learning models generated for each user by the execution of machine learning are stored in the learning model storage unit 117 in a format that can be used by specifying a user.

[0186] 23 is a diagram showing an example of data collection by the teacher data generation unit 115 in the first modified example of the third embodiment of the present invention. As shown in Fig. 23, the teacher data generation unit 115 first acquires information on activity status x time period for each user from the activity status storage unit 107, and generates a list of users x activity status x time period.

[0187] 23 , the teacher data generation unit 115 acquires a psychological state from the label psychological state storage unit 114 based on the user ID and time period in the list of user x activity status x time period. The teacher data generation unit 115 also acquires biometric information from the biometric information storage unit 103.

[0188] The learning model calling unit 118 acquires the learning model generated for each user from the learning model storage unit 117 .

[0189] The psychological state estimation unit 112 acquires biometric values ​​for a time period to be estimated from the biometric information storage unit 103. The psychological state estimation unit 112 estimates the psychological state by inputting the biometric values ​​into the learning model acquired by the learning model calling unit 118.

[0190] <Second Modification of Third Embodiment> A second modification of the third embodiment of the present invention will now be described.

[0191] In some cases, a psychological state can be estimated from facial expressions included in video data acquired by a camera and voice quality and tone of voice acquired by a microphone. However, in many cases, a psychological state cannot be estimated from video data or audio data. In contrast, a notification system in a second modified example of the third embodiment described below (hereinafter referred to as "notification system 1e") is configured to estimate a psychological state from biometric information (even when it cannot be estimated from video or audio) by using the psychological state as a label when it can be estimated from video and audio data and using the label together with biometric information at that time as training data.

[0192] In the notification system 1e in the second variant of the third embodiment, the estimation device (hereinafter referred to as the "estimation device 10e") generates training data in which the psychological state estimated based on, for example, video data obtained from an environmental information terminal 30a, which is an office camera, and audio data obtained from an environmental information terminal 30b, which is an office microphone, is used as a correct answer label, and performs supervised machine learning.

[0193] In addition, the estimation device 10e may generate training data in which the correct answer label is a psychological state estimated based on, for example, video data obtained from the environmental information terminal 30c, which is a PC camera for web conferencing, and audio data obtained from the environmental information terminal 30d, which is a PC microphone for web conferencing.

[0194] The following description focuses on the differences in the configuration of notification system 1e in the second modified example of the third embodiment from notification system 1d in the first modified example of the third embodiment. As with the first embodiment, notification system 1e in the second modified example of the third embodiment can also be configured like notification system 1a in the modified example of the first embodiment, which includes an estimation device that estimates the psychological states of participants in a meeting that includes a mixture of online and offline activities.

[0195] [Configuration of Estimation Device] The configuration of the estimation device 10e will be described below.

[0196] 24 is a block diagram showing the functional configuration of an estimation device 10e according to a second modified example of the third embodiment of the present invention. As shown in FIG. 24, the estimation device 10e includes a personal data acquisition unit 101, a biometric information analysis unit 102, a biometric information storage unit 103, an environmental information acquisition unit 104, a user situation analysis unit 105e, a communication situation storage unit 106, an activity situation storage unit 107, an environmental information storage unit 108, a mental state estimation unit 112, a mental state storage unit 113, a label mental state storage unit 114e, a teacher data generation unit 115, a learning execution unit 116, a learning model storage unit 117, and a learning model calling unit 118.

[0197] As shown in FIG. 24, the functional configuration of the estimation device 10e in the second modified example of the third embodiment differs from the functional configuration of the estimation device 10d in the first modified example of the third embodiment in that the user situation analysis unit 105 and the label psychological state storage unit 114 are replaced with a user situation analysis unit 105e and a label psychological state storage unit 114e, respectively.

[0198] The user state analysis unit 105e estimates a psychological state that can be used as a correct label for machine learning, based on the video data and audio data acquired from the environment information acquisition unit 104. Fig. 25 is a schematic diagram showing data analysis processing by the user state analysis unit 105e in a second modified example of the third embodiment of the present invention.

[0199] The user status analysis unit 105e estimates the psychological state based on video data of a scene in which the facial expressions of meeting participants (users) are clearly visible. Any existing psychological state estimation technology using facial image analysis can be used for the psychological state estimation process based on this video data. The user status analysis unit 105e also estimates the psychological state based on audio data of a scene in which the speech of the meeting participants (users) is clearly recorded. Any technology using, for example, audio psychology can be used for the psychological state estimation process based on this audio data. Any other technology can also be used, such as a technology that estimates the psychological state from posture, gestures, and unconscious body movements.

[0200] The label psychological state storage unit 114e stores information indicating a psychological state to be used as a correct label for training data in supervised machine learning. Similar to the label psychological state storage unit 114 in the first modified example of the third embodiment described above, the label psychological state storage unit 114e stores data in which, for example, a "user ID," a "psychological state" (i.e., a correct label), and a "time period" (i.e., a time period during which the user is estimated to have been in the above-mentioned psychological state) are associated with each other.

[0201] The label psychological state storage unit 114e provides the above data in response to a request from the teacher data generation unit 115.

[0202] Fourth Embodiment A fourth embodiment of the present invention will now be described.

[0203] The estimation device 10 of the notification system 1 in the first embodiment is configured to estimate the psychological state of a specific person (e.g., one participant in a meeting) based on biometric measurements and an estimated activity status of the person. However, for example, in a meeting, if only one participant becomes highly stressed, the situation of the trouble that has occurred may be different from that in a meeting where many participants become highly stressed, and the severity of the problem may also be different.

[0204] In contrast, the estimation device (hereinafter referred to as "estimation device 10f") and notification control device (hereinafter referred to as "notification control device 40f") of the notification system (hereinafter referred to as "notification system 1f") in the fourth embodiment described below are configured to determine an appropriate notification destination in accordance with predetermined notification destination determination rules depending on the importance of the event in which a communication problem occurred (for example, the occurrence of a high-stress state), which is determined based on specified conditions.

[0205] The conditions for determining importance here include, for example, the number of participants involved, the degree of high stress, frequency of occurrence, etc. For example, conditions for determining importance include the number of participants in a meeting that became highly stressful, the number of times that a transition to a high stress state occurred within a given period of time (during a single meeting), the length of time that the high stress state continued, and the number of times that a high stress state occurred repeatedly due to the same stressor (including in past meetings).

[0206] Furthermore, the estimation device 10f and the notification control device 40f of the notification system 1f in a fourth embodiment described below are configured to further tag the video data and audio data included in the notification to the person to be notified (notification recipient) with information indicating the determined importance level. This allows the notified person to more easily grasp the seriousness of a communication problem, such as harassment that occurred in a meeting.

[0207] In the following explanation, a situation in which a participant becomes highly stressed due to communication problems such as harassment occurring in a meeting will be described as an example, but the situation is not limited to this. For example, the notification system 1f in the fourth embodiment may be configured to evaluate the degree of goodness (importance) of communication based on the user's positive psychological state, such as a relaxed state, and issue a notification.

[0208] The estimation device 10f detects meeting participants who were in a high-stress state. The estimation device 10f determines the importance of the event based on the number of meeting participants who were in a high-stress state, the degree of high-stress state a participant experienced during the meeting, the frequency (number of times) that a participant experienced a high-stress state, and the duration of the high-stress state a participant experienced during the meeting. The notification control device 40f then determines notification recipients based on the importance determined by the estimation device 10f and causes the notification device 50 to perform notification processing.

[0209] The rules for determining the importance can be set arbitrarily, but may include, for example, rules such as "if three or more participants enter a high stress state at the same time, the importance is high," "if two participants enter a high stress state multiple times in one minute, the importance is high," "if two participants enter a high stress state only once in one minute, the importance is medium," "if only one participant enters a high stress state multiple times in one minute, the importance is medium," and "if only one participant enters a high stress state only once in one minute, the importance is low."

[0210] The importance determination rules given above as examples are all rules that determine importance based on the number of participants who became highly stressed and the number of times they transitioned to a highly stressed state within a specified period (frequency of occurrence of a highly stressed state). However, the importance determination rules are not limited to these rules, and any rules can be used.

[0211] The importance determination may be performed by the estimation device 10f as described above, but other configurations may also be adopted, such as having the notification control device 40f make a comprehensive determination of the importance based on information from the stress state analysis by the estimation device 10f.

[0212] As with the first embodiment described above, the notification system 1f in the fourth embodiment can also be configured like the notification system 1a in the modified version of the first embodiment, which has an estimation device that estimates the psychological states of participants in a meeting that includes a mixture of online and offline activities.

[0213] As described above, the estimation device 10f included in the notification system 1f according to the fourth embodiment of the present invention is configured to determine the importance level based on, for example, the severity of the trouble that has occurred (e.g., the severity of the high stress state of the meeting participants). The notification control device 40f is configured to determine appropriate notification destinations in accordance with predetermined notification destination determination rules for each determined importance level. For example, if the notification control device 40f determines that the importance level is high, the notification control device 40f may control the notification so that a notification is sent not only to the superiors of the meeting participants but also to a manager of a higher organization, etc.

[0214] Fifth Embodiment A fifth embodiment of the present invention will now be described.

[0215] The estimation device 10 of the notification system 1 in the first embodiment is configured to estimate the psychological state of a person (e.g., a meeting participant) based on biometric measurements of the person and an estimated activity status. This configuration is based on the premise that the cause of a change in the psychological state of a person is communication between people (e.g., trouble during a meeting) or activity.

[0216] However, in general, a person's biometric values ​​change not only due to interpersonal communication and activities but also due to, for example, the environment, etc. The environment here refers to, for example, temperature, humidity, illuminance, noise, etc.

[0217] In contrast, the estimation device (hereinafter referred to as "estimation device 10g") of the notification system (hereinafter referred to as "notification system 1g") in the fifth embodiment described below is configured to estimate a person's psychological state while also taking into account the impact of the environment on biometric measurements.

[0218] The following description focuses on the differences in the configuration of the notification system 1g in the fifth embodiment from the notification system 1 in the first embodiment. As with the first embodiment, the notification system 1g in the fifth embodiment can also be configured like the notification system 1a in the modified version of the first embodiment, which includes an estimation device that estimates the psychological states of participants in a meeting that includes both online and offline activities.

[0219] [Configuration of Estimation Device] The configuration of an estimation device 10g according to the fifth embodiment will be described below.

[0220] Note that the block diagram showing the functional configuration of the estimation device 10g in the fifth embodiment is similar to the block diagram showing the functional configuration of the estimation device 10 in the first embodiment shown in FIG. 2 described above. The estimation device 10g in the fifth embodiment described below differs from the estimation device 10 in the first embodiment described above in the configurations of a user situation analysis unit, an environmental information storage unit, and an estimation parameter analysis unit. Hereinafter, the user situation analysis unit, the environmental information storage unit, and the estimation parameter analysis unit of the estimation device 10g in the fifth embodiment will be referred to as a "user situation analysis unit 105g," an "environment information storage unit 108g," and an "estimation parameter analysis unit 109g," respectively.

[0221] The user status analysis unit 105g aggregates and analyzes environmental information. The user status analysis unit 105g sets one or more categories for, for example, temperature, humidity, illuminance, and noise level, and grades the data. The user status analysis unit 105g stores the graded data for each category in the environmental information storage unit 108g.

[0222] The environmental information storage unit 108g stores the above data generated by the user situation analysis unit 105g. In response to a request from the estimation parameter analysis unit 109g, the environmental information storage unit 108g provides environmental information for a specified user at a specified time. In response to a request from the estimation parameter setting unit 111, the environmental information storage unit 108g provides environmental information for a specified user during a specified time period.

[0223] An example of the environmental information aggregation process and analysis process performed by the user status analysis unit 105g will be described below. Fig. 26 is a diagram showing an example of the environmental information aggregation process and analysis process performed by the user status analysis unit 105g according to the fifth embodiment of the present invention.

[0224] 26 , the user status analysis unit 105g acquires data such as temperature, humidity, illuminance, and noise levels in the conference room from, for example, an environmental information terminal. Based on the acquired data, the user status analysis unit 105g sets categories such as a "sweating influence index" and a "heart rate influence index." The user status analysis unit 105g then defines a scale of 1 to 5 for each of the established categories of "sweating influence index" and "heart rate influence index."

[0225] The user status analysis unit 105g analyzes the acquired data and determines a level in each category for each user and time period. The user status analysis unit 105g stores the level values ​​in each category for each user and time period in the environmental information storage unit 108g. As shown in FIG. 26 , for example, the data stored in the environmental information storage unit 108g is data in which a "user ID," a "time," and a level value for each category (i.e., a "sweating influence index" and a "heart rate influence index") are associated with each other.

[0226] The estimated parameter analysis unit 109g acquires past biometric measurements and analysis data for each combination of activity status and environmental information (hereinafter also referred to as "activity status × environmental information"). First, the estimated parameter analysis unit 109g acquires information indicating the activity status and past user × time period from the activity status storage unit 107. Next, the estimated parameter analysis unit 109g acquires the environmental information for the user × time period from the environmental information storage unit 108g, acquires biometric measurements and analysis data for each activity status × environmental information from the biometric information storage unit 103, and generates a history data group by compiling the acquired biometric measurements and analysis data for each activity status × environmental information.

[0227] The history data group may be generated by acquiring all relevant data, but in practice, the amount of data would be enormous. Therefore, the history data group may be generated by acquiring only a certain number of recent data or only data from a certain period of time. The biometric values ​​acquired here are not data for each user, but data for each activity status of all users, as in the first embodiment described above.

[0228] 27 is a diagram showing an example of data collection by the estimation parameter analysis unit 109 g in the fifth embodiment of the present invention. As shown in Fig. 27, the estimation parameter analysis unit 109 g first acquires a list of activity situations and a list of categories and their stage values ​​set based on environmental information.

[0229] Next, the estimated parameter analysis unit 109g acquires information associating a user ID with a time period for each activity in the activity status list from the activity status storage unit 107. Next, the estimated parameter analysis unit 109g acquires environmental information for each user and time period for each activity status from the environmental information storage unit 108g, and generates user ID and time period information for the activity status and environmental information, as shown in FIG. 27 . Next, the estimated parameter analysis unit 109g acquires various biometric information (such as pulse rate, pulse wave peak interval, and skin electrical resistance) for the user and time period in the activity status and environmental information list from the biometric information storage unit 103. The estimated parameter analysis unit 109g then compiles the acquired data to generate data indicating an “activity status,” an environmental information category (i.e., a “sweating influence index” and a “heart rate influence index”), and the biometric information. The above-mentioned environmental information category refers to a category obtained by classifying each type of environmental information into several categories.

[0230] The estimated parameter analysis unit 109g acquires past biometric measurements and analysis data for each combination of activity status and environmental information (hereinafter also referred to as "activity status x environmental information"). First, the estimated parameter analysis unit 109g acquires information indicating the activity status and past user x time period from the activity status storage unit 107. Next, the estimated parameter analysis unit 109g acquires the environmental information for the user x time period from the environmental information storage unit 108, acquires biometric measurements and analysis data for each activity status x environmental information from the biometric information storage unit 103, and generates a history data group by compiling the acquired biometric measurements and analysis data for each activity status x environmental information.

[0231] The estimated parameter analysis unit 109g generates estimated parameters of the psychological state from the past biological information for each activity situation x environmental information. Note that the method for generating the estimated parameters can be the same as that of the third embodiment, for example.

[0232] <Modification of Fifth Embodiment> A modification of the fifth embodiment of the present invention will now be described.

[0233] In the notification system 1g according to the fifth embodiment, the user situation analysis unit 105g of the estimation device 10g aggregates and analyzes environmental information, and the user's psychological state is estimated based on the analysis results.

[0234] In contrast, the estimation device (hereinafter referred to as the "estimation device 10h") of the notification system (hereinafter referred to as the "notification system 1h") in a variant of the fifth embodiment described below has a configuration that estimates the user's psychological state using machine learning, similar to, for example, the first variant of the third embodiment described above.

[0235] The following description focuses on the differences in the configuration of notification system 1h according to the modification of the fifth embodiment from notification system 1g according to the aforementioned fifth embodiment. As with the aforementioned first embodiment, notification system 1h according to the modification of the fifth embodiment can also be configured like notification system 1a according to the modification of the first embodiment, which includes an estimation device that estimates the psychological states of participants in a meeting that includes both online and offline activities.

[0236] [Configuration of Estimation Device] The configuration of an estimation device 10h according to a modification of the fifth embodiment will be described below.

[0237] The block diagram showing the functional configuration of the estimation device 10h in the modified fifth embodiment is basically the same as the block diagram showing the functional configuration of the estimation device 10d in the first modified third embodiment shown in FIG. 22 described above. The estimation device 10h in the modified fifth embodiment described below differs from the estimation device 10d in the first modified third embodiment described above in the configurations of the user situation analysis unit, the environmental information storage unit, the teacher data generation unit, and the learning execution unit. Hereinafter, the user situation analysis unit, the environmental information storage unit, the teacher data generation unit, and the learning execution unit of the estimation device 10h in the modified fifth embodiment will be referred to as the "user situation analysis unit 105h," the "environment information storage unit 108h," the "teacher data generation unit 115h," and the "learning execution unit 116h," respectively.

[0238] The user status analysis unit 105h stores each acquired piece of environmental information as is, unlike the user status analysis unit 105g in the fifth embodiment described above. Fig. 28 is a diagram showing an example of the environmental information aggregation process performed by the user status analysis unit 105h in a modified example of the fifth embodiment of the present invention.

[0239] As shown in FIG. 28 , the user status analysis unit 105h acquires data such as temperature, humidity, illuminance, and noise levels in a conference room from an environmental information terminal or the like. The user status analysis unit 105h organizes the acquired data by user and time period. The user status analysis unit 105h stores data indicating environmental information for each user and time period in the environmental information storage unit 108h. As shown in FIG. 28 , for example, the data stored in the environmental information storage unit 108h is data in which a "user ID" is associated with each environmental information category (i.e., "temperature," "humidity," "illuminance," "noise," etc.). Note that instead of using environmental information categories, each measurement item of environmental information and the measurement value itself can also be retained and used.

[0240] The teacher data generation unit 115h acquires the activity status for each user from the activity status storage unit 107. Based on the acquired user x time period information, the teacher data generation unit 115h also acquires the psychological state from the psychological state storage unit 113, the environmental information from the environmental information storage unit 108h, and the biometric information from the biometric information storage unit 103. The teacher data generation unit 115h compiles the acquired data and generates data in which the user (user ID), information indicating the activity status, information indicating the psychological state, a category level value based on the environmental information, and the biometric information are associated with each other.

[0241] The learning execution unit 116h performs machine learning using teacher data in which the psychological state is associated with activity information, environmental information, and biological information for each user, based on the data generated by the teacher data generation unit 115h. The multiple trained learning models generated for each user by the execution of machine learning are stored in the learning model storage unit 117 in a format that can be used by specifying the user.

[0242] 29 is a diagram showing an example of data collection by the teacher data generation unit 115h in a modified example of the fifth embodiment of the present invention. As shown in Fig. 29, the teacher data generation unit 115h first acquires information on activity status x time period for each user from the activity status storage unit 107, and generates a list of users x activity status x time period.

[0243] Next, as shown in Fig. 29 , the teacher data generation unit 115h acquires the activity status for each user from the activity status storage unit 107. Also, as shown in Fig. 29 , based on the acquired user x time period information, the teacher data generation unit 115h acquires the psychological state from the psychological state storage unit 113, the environmental information from the environmental information storage unit 108h, and the biometric information from the biometric information storage unit 103. Furthermore, the teacher data generation unit 115h compiles the acquired data and generates data in which the user (user ID), information indicating the activity status, information indicating the psychological state, a category level value based on the environmental information, and the biometric information are associated with each other.

[0244] The learning execution unit 116h performs machine learning using teacher data in which activity status, mental state, environmental information, and biological information are associated for each user, based on the data generated by the teacher data generation unit 115h.

[0245] Sixth Embodiment The estimation device 10c of the notification system 1c in the third embodiment described above is configured to set a threshold value of a biometric value used for estimating a psychological state for each user and generate estimation parameters for each user, thereby enabling the estimation device 10c in the third embodiment to estimate the psychological state of a user with higher accuracy, taking into account individual differences.

[0246] However, generating such estimation parameters for each user requires biometric information for each user (or biometric information for each user and activity status). Furthermore, when there is a large variation in the biometric information to be collected, such as biometric information for each user and activity status, the amount of data required becomes enormous. Therefore, the estimation device 10c in the third embodiment has a problem in that it takes a long time to collect biometric information, and cannot immediately start estimating the psychological state of a user for whom sufficient biometric information has not been collected in advance.

[0247] In contrast, an estimation device (hereinafter referred to as "estimation device 10i") of a notification system (hereinafter referred to as "notification system i") according to a sixth embodiment described below is configured to compensate for a lack of data for a user for whom sufficient biometric information has not been collected in advance by instead using biometric information of a user similar to the user (hereinafter referred to as "similar user"). As a result, the estimation device 10i according to the sixth embodiment can immediately start estimating the psychological state of a user for whom sufficient biometric information has not been collected in advance, while ensuring a certain degree of estimation accuracy.

[0248] As described above, the estimation device 10i in the sixth embodiment is configured to generate estimated parameters for a user for whom sufficient biometric information has not been collected in advance using biometric information of similar users. Alternatively, the estimation device 10i may be configured to use estimated parameters that have already been generated for similar users as they are for the user.

[0249] [Configuration of Estimation Device] The configuration of the estimation device 10i according to the sixth embodiment will be described below. Fig. 30 is a block diagram showing the functional configuration of the estimation device 10i according to the sixth embodiment of the present invention. As shown in Fig. 30, the estimation device 10i includes a personal data acquisition unit 101, a biometric information analysis unit 102, a biometric information storage unit 103, an environmental information acquisition unit 104, a user situation analysis unit 105, a communication situation storage unit 106, an activity situation storage unit 107, an environmental information storage unit 108, an estimation parameter analysis unit 109, an estimation parameter storage unit 110, an estimation parameter setting unit 111, a psychological state estimation unit 112, a psychological state storage unit 113, and a user attribute storage unit 119.

[0250] 30 , the estimation device 10i according to the sixth embodiment differs from the estimation device 10c according to the third embodiment in that it further includes a user attribute storage unit 119. The following description will focus on the differences in the configuration of the notification system 1i according to the sixth embodiment from the notification system 1c according to the third embodiment.

[0251] The estimated parameter analysis unit 109 queries the biometric information storage unit 103 and the activity status storage unit 107 to refer to the biometric information (past information) of the target user that has already been stored and is to be used to generate estimated parameters. As a result of the data reference, it is assumed that the amount of data required to generate estimated parameters has not yet been stored, for example, because the user is a new user or is performing an activity state for the first time. In this way, when the user data is insufficient, the estimated parameter analysis unit 109 queries the user attribute storage unit 119 to identify the user IDs of users similar to this user.

[0252] The similar users referred to here are, for example, users whose attribute information is similar to that of a user whose psychological state is to be estimated. Note that the configuration of the estimation device 10 i in the sixth embodiment is based on the assumption that users whose attribute information is similar will also have similar biometric information.

[0253] The estimated parameter analysis unit 109 uses the user ID of the identified similar user to acquire the biometric information of the similar user for each activity status from the biometric information storage unit 103. The estimated parameter analysis unit 109 regards, for example, the average value or median value of the acquired biometric information as the biometric information of the user whose psychological state is to be estimated, and generates estimated parameters.

[0254] 31 and 32 are diagrams showing an example of data stored in the user attribute storage unit 119 of the estimation device 10 i according to the sixth embodiment of the present invention.

[0255] Immutable attribute information (e.g., date of birth, gender, etc.) is registered by each user when they first use the system. Date of birth is used, for example, to calculate age. Attribute information that may change (e.g., affiliation, job title, etc.) is, for example, periodically acquired and updated.

[0256] If the notification system 1i is a system used by a company or the like, for example, when each user uses the system for the first time, new information about the affiliation and job title is registered by the user. After that, when the affiliation and job title are updated, or periodically, the updated information about the affiliation and job title is obtained, for example, from the organizational information storage unit 404 of the notification control device 40, and is reflected in the notification system 1i.

[0257] Furthermore, if the notification system 1i is a system used by a company, school, or the like, data such as medical history and illnesses currently being treated may be obtained from, for example, the results of annual health checkups. To estimate psychological states such as tension, data related to skin electrodermal activity (e.g., skin resistance response (SRR)) indicating the degree of increase or decrease in sweating level is used, as described above. The sensor criteria and responses for users suffering from palmar hyperhidrosis and the like differ from those for users without the condition, so using similar criteria may result in poor accuracy in estimating psychological states. Therefore, it is considered useful to use medical history (medical history) as information for classifying users when estimating psychological states.

[0258] For example, in a company, even if users have the same job content and the same position, it is possible that there will be individual differences in their internal state depending on their level of experience. For example, it is possible that users with fewer years of experience will be more likely to become nervous. Therefore, if the notification system 1i is a system used in a company, for example, the number of years in the position (the number of years of experience in the current position) may be obtained from the organizational information storage unit 404 of the notification control device 40 and used to estimate the psychological state, as shown in FIG. 32 .

[0259] 32, if each user has a role outside of work (for example, at home), information indicating that role may also be used to estimate the psychological state. For example, this is because unmarried users or users with children are likely to experience similar changes in their internal states in response to the same event.

[0260] Fig. 33 is a diagram showing an example of a new user when the similar user search process is performed by the estimation parameter analysis unit 109 of the estimation device 10i according to the sixth embodiment of the present invention. Also, Fig. 34 is a diagram showing an example of a user found in the similar user search process by the estimation parameter analysis unit 109 of the estimation device 10i according to the sixth embodiment of the present invention.

[0261] As shown in Figure 33, for example, assume that a person with a user ID of "1112," age 45, male, general employee, and no medical history is newly registered as a user of the notification system 1i. Note that attributes other than those shown in Figure 33 may also be registered. However, since it is desirable that the data input to the estimation parameter analysis unit 109 be only data related to attributes necessary for analyzing the parameters to be estimated, data related to attributes that have an extremely low correlation with the biometric information of the estimation target is considered unnecessary.

[0262] Here, it is assumed that user ID "1112" is a user ID newly registered in the notification system 1i. Therefore, it is assumed that sufficient biological information to calculate estimation parameters for estimating a psychological state has not yet been accumulated. In addition, here, for example, pulse rate data of the user with user ID "1112" is available in real time, and it is desired to estimate the psychological state of the user from the pulse rate.

[0263] In this case, the estimated parameter analysis unit 109 selects similar users whose attributes are similar to those of the user with user ID "1112" by searching the data stored in the user attribute storage unit 119 using, for example, the following search conditions: Age, gender, and medical history are specified as search conditions because they are assumed to be correlated with pulse rate, while job title is not specified as a search condition because it is assumed to have a low correlation with pulse rate.

[0264] Age: 45±5 years old Gender: Male Position: Not specified Medical history: None

[0265] As a result of the search, the estimated parameter analysis unit 109 obtains a list of user IDs and attribute information of similar users, as shown in FIG. 34 . Next, the estimated parameter analysis unit 109 obtains biometric information corresponding to the selected user ID from the biometric information storage unit 103. At this time, the estimated parameter analysis unit 109 analyzes the estimated parameters by regarding the average or median value of the biometric information of similar users during the same activity status as the initial biometric information value of user ID "1112", depending on the current activity status of the user ID "1112" to be estimated. Then, until the amount of biometric information of the user with user ID "1112" required for analyzing the estimated parameters has been accumulated, the estimated parameter analysis unit 109 may analyze the estimated parameters using the biometric information of similar users identified by the search described above.

[0266] Fig. 35 is a diagram showing an example of data for each user and activity situation input to the estimation parameter analysis unit 109 of the estimation device 10i according to the sixth embodiment of the present invention. Fig. 36 is a diagram showing an example of data for each attribute and activity situation input to the estimation parameter analysis unit 109 of the estimation device 10i according to the sixth embodiment of the present invention. As shown in Fig. 35 , biometric information for each activity situation of similar users is input to the estimation parameter analysis unit 109. When the amount of biometric information data of a user whose psychological state is to be estimated is insufficient for generating estimated parameters, the estimation device 10i according to the sixth embodiment generates estimated parameters by instead using, for example, biometric information for each activity situation of similar users as shown in Fig. 35 .

[0267] In addition, in order to avoid using data of similar users when an abnormality occurs more than necessary, data when an abnormality determination continues for a certain period of time or more may be excluded from use in generating estimated parameters.

[0268] In addition, when only information of general attributes retrieved from a user ID is used, data of average or median values ​​of biometric information calculated for each attribute and activity status may be stored in advance, as shown in Fig. 36. For example, the estimation device 10i may be further configured to include an attribute-specific average value storage unit (not shown) that stores the above-mentioned average or median value data calculated with reference to the user attribute storage unit 119, the biometric information storage unit 103, and the activity status storage unit 107.

[0269] 37 to 39 are diagrams illustrating the flow of processing by the estimation parameter analysis unit 109 of the estimation device 10i according to the sixth embodiment of the present invention. For example, assume that information indicating the real-time activity status and biometric information of a new user (i.e., a user for whom past data such as biometric information has not yet been accumulated) has been acquired. The estimation parameter analysis unit 109 further selects, from the selected similar users, only similar users for whom data such as biometric information whose activity status matches that of the new user (the user whose psychological state is to be estimated) has been accumulated.

[0270] FIG. 38 shows a case where the activity status of a new user is, for example, desk work. FIG. 39 shows how data of similar users who have performed desk work is selected. The estimation parameter analysis unit 109 uses the median or average value of the biometric information of similar users who have similar attributes and matching activity status as the reference value. For example, in the examples shown in FIGS. 37 to 39, the user with user ID "1112" is estimated to have emotions such as fear or disgust because, for example, his pulse rate is higher than that of similar users with matching activity status.

[0271] 40 to 42 are diagrams illustrating classification using user attributes by the estimation parameter analysis unit 109 of the estimation device 10i according to the sixth embodiment of the present invention. Below, classification using the attributes of users whose past biological information has been accumulated will be described for specific activity situations.

[0272] For example, as shown in Figures 40 and 41, it is assumed that a new user with user ID "1115" has already accumulated sufficient biometric information when their activity status is "moving (walking)" (for example, by wearing a sensor-equipped device when moving for several days). It is also assumed that a similar user with user ID "1026" as shown in Figure 42 exists as a user whose biometric information when their activity status is "moving (walking)" is similar to that of the new user with user ID "1115."

[0273] In this case, the estimated parameter analysis unit 109 treats the biometric information value of the similar user with user ID "1026" in another activity situation (an activity situation other than "moving (walking)") as the reference value of the biometric information of the new user with user ID "1115" in the same activity situation. For example, when the new user with user ID "1115" performs desk work, the estimated parameter analysis unit 109 uses the biometric information value of the similar user with user ID "1026" during desk work as the reference value.

[0274] By using such a technique, the estimation parameter analysis unit 109 can set the initial values ​​of the estimation parameters even if, as a result of referring to the user attribute storage unit 119, there are no other users with the same or similar attributes.

[0275] It is to be noted that using such a method is effective even when there are many other users with the same or similar attributes. For example, the estimation parameter analysis unit 109 can further improve estimation accuracy by using only values ​​of biometric information of similar users whose biometric information is more similar to that of the user whose psychological state is to be estimated.

[0276] 43 to 45 are diagrams illustrating an estimation process performed by the estimation parameter analysis unit 109 of the estimation device 10i according to the sixth embodiment of the present invention, using data of users with similar trends and changes in activity status and biological information. In the processes shown in FIGS. 40 to 42, the estimation parameter analysis unit 109 was configured to estimate the psychological state of the target user based on the degree of similarity of the values ​​of the biological information. In contrast, in the processes shown in FIGS. 43 to 45, when multiple pairs of activity status and biological information are known, the estimation parameter analysis unit 109 estimates the psychological state based on the trends of the values ​​of the biological information.

[0277] For example, Figure 44 shows a case where the relationship between activity status and pulse rate values ​​is equal between a user whose psychological state is to be estimated and a similar user. That is, the pulse rates of a user whose psychological state is to be estimated and whose user ID is "1115" are "90," "100," and "110" when "moving (walking)," "attending a meeting with 2 to 4 people," and "attending a meeting with 5 to 10 people," respectively, with a ratio of 9:10:11. Also, the pulse rates of a similar user whose user ID is "1026" when "moving (walking)," "attending a meeting with 2 to 4 people," and "attending a meeting with 5 to 10 people" are "70," "77," and "85," respectively, with a ratio of approximately 9:10:11.

[0278] That is, it can be said that the relationship between the activity status and the pulse rate value is approximately equal between the estimation target user with user ID "1115" shown in Fig. 44 and the similar user with user ID "1026" shown in Fig. 45. In this way, even if the values ​​of the biometric information do not match, if the relationship between the values ​​of the biometric information in a plurality of activity statuses matches (here, equal, proportional), the estimation parameter analysis unit 109 can utilize the values ​​of the biometric information of the similar user.

[0279] For example, as shown in FIG. 45 , the pulse value of the similar user with user ID "1026" during "desk work" is "60." Therefore, the ratio of pulse values ​​of the similar user with user ID "1026" during "movement (walking)," "participation in a meeting with 2-4 people," "participation in a meeting with 5-10 people," and "desk work" is approximately 9:10:11:7.7-7.8. When these ratios are applied to the biometric information values ​​of the inferred user with user ID "1115," it can be estimated that the pulse value of the inferred user with user ID "1115" during "desk work" is approximately "77" to "78."

[0280] In reality, there are multiple values ​​of biological information for each activity status, so the average or median of these multiple values ​​is used for the pulse value, etc. shown in Figures 44 and 45, for example.

[0281] Although the method of utilizing the biometric information of similar users having similar ratios between a plurality of activity situations and biometric information values ​​has been described here, the present invention is not limited to this. For example, as another method, a method of utilizing the biometric information of similar users having the same variance when pulse rates are distributed in a certain activity situation (similar users having similar distribution shapes of biometric values) may be used.

[0282] In addition, if there are many new users, a configuration may be adopted in which clustering is used. Figures 46 to 48 are diagrams for explaining the clustering process performed by the estimation parameter analysis unit 109 of the estimation device 10i according to the sixth embodiment of the present invention.

[0283] For example, consider a case where a large number of users suddenly joins a company, school, or the like at a certain time. For example, it is likely that most new employees share some attributes, such as age and job title. In such a case, the estimation parameter analysis unit 109 may refer to attributes representative of multiple users obtained by clustering, rather than referring to the attributes of each individual user. Using such a clustering technique enables initial operation with reduced computational costs and data volume.

[0284] For example, as shown in Fig. 46, the estimated parameter analysis unit 109 divides the data of all new employees into three groups based on gender. Also, as shown in Fig. 47 and Fig. 48, it is assumed that all employees have the same age and position, as described above. The estimated parameter analysis unit 109 generates estimated parameters using reference value data created in advance based on the biometric information of new employees from the previous year.

[0285] [Operation of Estimation Device] An example of the operation of the estimation device 10i will now be described. Figures 49 and 50 are flowcharts showing the operation of the estimation device 10i according to the sixth embodiment of the present invention.

[0286] The operations of steps S101 to S110 and steps S114 to S120 in the flowchart shown in Fig. 49 are the same as the operations of steps S001 to S010 and steps S015 to S019 in the flowchart shown in Fig. 15. The following description will focus on the parts that differ from the flowchart shown in Fig. 15.

[0287] 49 , after the user situation analysis unit 105 estimates environmental information based on information acquired from the environmental information acquisition unit 104 (step S110), the estimated parameter analysis unit 109 checks whether there are estimated parameters generated in the past (step S111). If there are no estimated parameters generated in the past (or if there are insufficient estimated parameters) (step S111, NO), the estimated parameter analysis unit 109 executes an estimation process using the attributes of similar users (step S113). After the estimation process in step S113 is completed, the process proceeds to step S114.

[0288] If there are no estimated parameters generated in the past (or if there is insufficient data) (step S111, YES), the estimated parameter analysis unit 109 checks whether it is time to generate estimated parameters (step S112). If it is time to generate estimated parameters (step S112, YES), the estimated parameter analysis unit 109 executes a psychological state estimation process using the attributes of similar users (step S113). After the estimation process in step S113 is completed, the process proceeds to step S114.

[0289] If it is not the timing to generate estimated parameters (step S112: NO), the estimated parameter analysis unit 109 proceeds to the process of step S114 without performing the estimation process of step S113.

[0290] The following describes the estimation process using the attributes of similar users, which is the process in step S113 of the flowchart shown in Fig. 49. The operation of the estimation device 10i in the estimation process will be described with reference to Fig. 50.

[0291] The estimated parameter analysis unit 109 checks whether estimated parameters have been generated or updated for all situations (activity situations + environmental information) by referring to the estimated parameter storage unit 110 (step S201). If estimated parameters have been generated or updated for all situations (activity situations + environmental information) (step S201: YES), the estimation process shown in the flowchart of Fig. 50 ends, and the process proceeds to step S114 of the flowchart shown in Fig. 49.

[0292] If estimated parameters have not been generated or updated for at least one situation (activity status + environmental information) (step S201, NO), the estimated parameter analysis unit 109 checks whether there is a sufficient amount of biometric information of the user to be estimated to generate estimated parameters even in a situation where estimated parameters have not been generated or updated (step S202).

[0293] Even if estimated parameters have not yet been generated or updated, if there is a sufficient amount of biometric information of the user to be estimated to generate estimated parameters (step S202, YES), the estimated parameter analysis unit 109 acquires the accumulated activity status and, if necessary, environmental information and biometric information history (step S203).Then, the estimated parameter analysis unit 109 generates estimated parameters from the biometric information history for each acquired activity status (and, if necessary, environmental information) (step S204).The generated estimated parameters are stored in the estimated parameter storage unit 110 (step S205).Then, the process returns to step S201.

[0294] If the amount of biometric information of the user to be estimated is not sufficient to generate estimated parameters (step S202, NO), the estimated parameter analysis unit 109 checks whether there is another user (similar user) with similar user attributes (step S206).If there is another user with similar user attributes (step S206, YES), the estimated parameter analysis unit 109 checks whether the biometric information for each other situation held by both the user and the similar user is similar (step S207).

[0295] If there is a similar user whose biometric information for other situations is similar to that of the user and the similar user (step S207, YES), the estimated parameter analysis unit 109 acquires the activity situations and biometric information history of the similar user who has the same attributes and similar biometric information (or its tendency) for other activity situations (step S208). The estimated parameter analysis unit 109 then generates estimated parameters from the biometric information history for each acquired activity situation (and environmental information, if necessary) (step S204). The generated estimated parameters are stored in the estimated parameter storage unit 110 (step S205). The process then returns to step S201.

[0296] If there is no similar user whose biometric information for each situation is similar to that of the user and the similar user (step S207, NO), the estimated parameter analysis unit 109 acquires biometric information histories for each activity situation of users whose user attributes match those of the user (step S209).Then, the estimated parameter analysis unit 109 generates estimated parameters from the biometric information histories for each acquired activity situation (and environmental information, if necessary) (step S204).The generated estimated parameters are stored in the estimated parameter storage unit 110 (step S205).Then, the process returns to step S201.

[0297] If there is no other user with similar user attributes (step S206, NO), the estimated parameter analysis unit 109 checks whether there is a user with similar biometric information in another situation (step S210). If there is a user with similar biometric information in another situation (step S210, YES), the estimated parameter analysis unit 109 acquires the activity status and biometric information history of a similar user whose biometric information (or its tendency) is similar in another activity status (step S211). Then, the estimated parameter analysis unit 109 generates estimated parameters from the biometric information history for each acquired activity status (and environmental information, if necessary) (step S204). The generated estimated parameters are stored in the estimated parameter storage unit 110 (step S205). Then, the process returns to step S201.

[0298] If there is a user with similar biometric information in another situation (step S210, NO), the estimated parameter analysis unit 109 uses initial values ​​(such as the average values ​​of all users) prepared in advance by the notification system i as estimated parameters (step S212). The generated estimated parameters are stored in the estimated parameter storage unit 110 (step S205). Then, the process returns to step S201.

[0299] 15, the processing flow was described assuming that estimated parameters already exist. However, in reality, in the case of a new user, estimated parameters have never been generated in the past, so if it is determined that it is not time to generate estimated parameters, psychological state estimation will be performed without estimated parameters. Therefore, in this embodiment, before determining that it is time to generate estimated parameters, it is checked whether estimated parameters (which can be used to estimate the current psychological state) generated in the past exist, and if not, estimated parameters are generated.

[0300] However, the present invention is not limited to such a configuration, and for example, a configuration may be adopted in which estimated parameters for all patterns are generated in advance to prevent a situation in which estimated parameters are not generated. If generating estimated parameters for all patterns requires a large amount of calculation or memory, a configuration may be adopted in which estimated parameters are generated in advance only for actions performed by similar users or only for locations where similar users have performed actions.

[0301] The estimation device 10i in this embodiment can acquire the user's location and environmental information using environmental sensors (mainly cameras) and positioning sensors, and can treat this information as a type of user attribute. By considering the match between the activity status and biometric information and the environmental information (location attribute), the accuracy of the psychological state estimation can be further improved.

[0302] 51 and 52 are diagrams illustrating a method for using data of users with similar values ​​or tendencies of environment (location attribute) x biometric information by the estimation parameter analysis unit 109 of the estimation device 10i according to the sixth embodiment of the present invention. For example, in a conference room that is not used often or a conference room that is used only for important meetings, the user may feel more tense. In other words, there may be a relationship between psychological state and location.

[0303] For example, as shown in Fig. 51, for the user with user ID "1115" whose psychological state is to be estimated, judging only from the biological information for each activity status, the pulse rate in conference room B is estimated to be about 100. This is because conference room B is a similar conference room to conference room A, allowing "3 to 5 people to participate in the conference," and the pulse rate in conference room A is also about 100.

[0304] However, for example, as shown in Figure 52, suppose there is a similar user with user ID "1108" whose data taking into account environmental information is very similar to that of the user with user ID "1115" whose psychological state is to be estimated (here, only one similar user is listed, but in reality, it is thought that multiple people will often be referenced).

[0305] By referencing the activity status x environmental information x biological information data of a similar user with user ID "1108," the estimated parameter analysis unit 109 can recognize that the pulse rate in conference room B tends to be higher than the pulse rate in conference room A, even if the activity status is the same. This allows the estimated parameter analysis unit 109 to estimate that, for example, a user with user ID "1115," a pulse rate of 110 or less (rather than 100 or less) is within the normal range. Note that although the similarity (match) of values ​​is given here as an example, the similarity of value changes may also be used.

[0306] According to the above-described embodiment, the estimation device includes a biometric information acquisition unit, an environmental information acquisition unit, an activity status estimation unit, and a psychological state estimation unit. For example, the estimation device is the estimation device 10, 10a to 10h in the embodiment, the biometric information acquisition unit is the personal data acquisition unit 101 in the embodiment, the environmental information acquisition unit is the environmental information acquisition unit 104 in the embodiment, the activity status estimation unit is the user status analysis unit 105, 105e in the embodiment, and the psychological state estimation unit is the psychological state estimation unit 112 in the embodiment.

[0307] The biometric information acquisition unit acquires biometric information based on biometric values ​​measured by a sensor that measures a person's biological phenomena. For example, the biometric information is the biometric values ​​and analysis data in the embodiment. The environmental information acquisition unit acquires environmental information including at least one of video and audio around the person. The activity status estimation unit estimates the person's activity status based on the biometric information and environmental information. The mental state estimation unit estimates the person's mental state based on the biometric information and activity status.

[0308] The estimation device may further include a psychological state storage unit. For example, the psychological state storage unit is the psychological state storage unit 113 in the embodiment. The psychological state storage unit stores psychological state information indicating characteristics of each of a plurality of predetermined psychological states. The psychological state estimation unit estimates the psychological state of the person based on a comparison between the characteristics of the person's psychological state identified based on the biometric information and the activity status and the characteristics of the psychological state included in the psychological state information.

[0309] In the above estimation device, if the amount of biometric information of the person required to estimate the psychological state is not obtained, the psychological state estimation unit may estimate the psychological state of the person using biometric information of another person who matches or is similar to the person in at least one of the specified user attributes, biometric information, and biometric information tendencies.

[0310] The estimation device may further include a biometric information storage unit. For example, the biometric information storage unit is the biometric information storage unit 103 in the embodiment. The biometric information storage unit stores biometric information for each person and for each activity status based on biometric measurements for each person and for each activity status. The psychological state estimation unit estimates the psychological state of the person based on the biometric information for each person and for each activity status.

[0311] In the above-described embodiments, the estimation devices 10, 10a-10h, the notification control devices 40, 40a, and the notification device 50 may be partially or entirely implemented by a computer. In this case, a program for implementing this function may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, the term "computer-readable recording medium" may also include media that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or communication lines such as telephone lines, or media that store programs for a certain period of time, such as volatile memory within the computer system serving as the server or client. Furthermore, the above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in a computer system, or may be one that is realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0312] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the present invention that do not deviate from the gist of the present invention.

[0313] REFERENCE SIGNS LIST 1, 1a to 1i Notification system 10, 10a to 10i Estimation device 20 Personal terminal 30, 30a to 30d Environmental information terminal 40, 40a, 40b, 40f Notification control device 50 Notification device 101 Personal data acquisition unit 102 Biometric information analysis unit 103 Biometric information storage unit 104 Environmental information acquisition unit 105, 105e, 105g, 105h User situation analysis unit 106 Communication situation storage unit 107 Activity situation storage unit 108, 108g, 108h Environmental information storage unit 109, 109c, 109g Estimation parameter analysis unit 110, 110c Estimation parameter storage unit 111, 111c Estimation parameter setting unit 112 Mental state estimation unit 113 Mental state storage unit 114, 114e Label mental state storage unit 115, 115h Teacher data generation unit 116, 116h Learning execution unit 117 Learning model storage unit 118 Learning model calling unit 119 User attribute storage unit 401 Notification determination unit 402 Notification information generation unit 403, 403b Notification destination selection unit 404, 404b Organization information storage unit 405 Notification control unit

Claims

1. An estimation device comprising: a biometric information acquisition unit that acquires biometric information based on biometric measurements taken by a sensor that measures a person's biological phenomenon; an environmental information acquisition unit that acquires environmental information including at least one of video and audio around the person; an activity status estimation unit that estimates the activity status of the person based on the biometric information and the environmental information; and a psychological state estimation unit that estimates the psychological state of the person based on the biometric information and the activity status.

2. The estimation device as described in claim 1, further comprising a psychological state memory unit that stores psychological state information indicating characteristics of each of a plurality of predetermined psychological states, wherein the psychological state estimation unit estimates the psychological state of the person based on a comparison between the characteristics of the psychological state of the person identified based on the bio-information and the activity status and the characteristics of the psychological state included in the psychological state information.

3. The estimation device as described in claim 1, further comprising a bio-information storage unit that stores the bio-information for each person and for each activity status based on the bio-measurement values ​​for each person and for each activity status, wherein the psychological state estimation unit estimates the psychological state of the person based on the bio-information for each person and for each activity status.

4. A computer-implemented estimation method comprising: a biometric information acquisition step of acquiring biometric information based on biometric measurements taken by a sensor that measures a biological phenomenon of a person; an environmental information acquisition step of acquiring environmental information including at least one of video and audio around the person; an activity status estimation step of estimating the activity status of the person based on the biometric information and the environmental information; and a psychological state estimation step of estimating the psychological state of the person based on the biometric information and the activity status.

5. The estimation device described in claim 3, wherein, when the biometric information of the person with the amount of data required to estimate the psychological state is not obtained, the psychological state estimation unit estimates the psychological state of the person using the biometric information of another person who has the same or similar at least one of a predetermined user attribute, the biometric information, and a tendency of the biometric information as the person.

Images