Determination device, learning device, determination system, determination method, learning method, and computer program

Abstract

In the present invention, a learning unit learns, by supervised learning or reinforcement learning, a determination model for calculating labor productivity-related information indicating an index relating to labor productivity of a user by using one or more pieces of input information among objective information obtained by measuring the organism of the user when the user is reacting to a question, subjective information obtained by subjective selection of the user, attribute information of the user, temporal change information of the objective information, temporal change information of the subjective information, and temporal change information of the attribute information. A determination unit obtains an estimation result of the labor productivity-related information of an estimation target person by inputting input information obtained for the estimation target person into the trained determination model.

Description

Determination device, learning device, determination system, determination method, learning method, and computer program

[0001] This application claims priority to Japanese Patent Application No. 2024-149958, filed on August 30, 2024, the contents of which are incorporated herein by reference.

[0002] Conventionally, there are technologies for measuring and managing the mental and physical health status of a user based on biological activity data, which is the user's subjective health status and the objective time required to perform simple tasks (see, for example, Patent Document 1), and technologies for estimating a subject's psychological state from their facial expressions, voice, and biological signals (see, for example, Patent Document 2). There are also technologies for estimating presenteeism and calculating economic losses by having users individually answer questionnaire-style check items stored on a server (see, for example, Patent Document 3), and technologies for calculating economic losses, such as the risk of taking time off work, using vital data related to the number of steps taken and speech, and for taking appropriate measures, such as sending an email to a manager (see, for example, Patent Document 4).

[0003] JP 2011-238215 A JP 2022-114906 A JP 2020-42547 A JP 2022-154769 A

[0004] Improving labor productivity while maintaining workers' physical and mental health benefits both workers and employers. Therefore, estimating labor productivity indicators, such as presenteeism and absenteeism, is considered as a way to detect potential mental stress issues among workers early. However, the technologies of Patent Documents 1 and 2 do not calculate economic losses, such as labor productivity. The technology of Patent Document 3 calculates economic losses by estimating presenteeism, but has issues in that it does not aim to measure workers' emotional states, which change during daily work, and instead relies on the subjective assessment of respondents. Furthermore, the technology of Patent Document 4 calculates economic losses using vital signs, requiring workers to wear measuring devices at all times.

[0005] In view of the above circumstances, the present disclosure provides a technology that enables labor productivity to be estimated accurately on a daily basis while reducing the burden on measurements.

[0006] One aspect of the present disclosure is a determination device that includes a determination unit that obtains an estimation result of labor productivity related information of an estimation subject, who is a user to be estimated, by inputting the input information obtained about the estimation subject, who is the user to be estimated, into a determination model that calculates labor productivity related information that indicates an index related to the labor productivity of the user, using one or more input information from among objective information obtained by measuring the user's biological activity while the user is responding to a question, subjective information obtained by the user's subjective selection, attribute information that represents the user's attributes, information on changes over time in the objective information, information on changes over time in the subjective information, and information on changes over time in the attribute information.

[0007] One aspect of the present disclosure is the above-mentioned determination device, wherein the input information includes at least the objective information, and the objective information includes at least information about the living body obtained by a non-contact sensor.

[0008] One aspect of the present disclosure is the above-mentioned determination device, wherein the labor productivity related information includes at least any of the user's presenteeism, absenteeism, health status, psychological state, performance that expresses an evaluation of the labor productivity using multiple levels of values, engagement, work performance, sales performance, and personnel evaluation.

[0009] One aspect of the present disclosure is the above-mentioned determination device, wherein the objective information includes one or more of the user's facial expression, voice features, posture, sleep time, sleep quality, working hours, and workload; the subjective information includes one or more of the user's emotional state and speech content; the attribute information includes one or more of the user's age, gender, personality tendencies, occupation, and annual income; and the labor productivity-related information includes one or more of the user's presenteeism, absenteeism, health condition, psychological state, performance that expresses an evaluation of the labor productivity using multiple values, engagement, work performance, sales performance, and personnel evaluation; and when the emotional state is not included in the subjective information, the labor productivity-related information may include the psychological state.

[0010] One aspect of the present disclosure is the above-mentioned determination device, wherein the question is a question that requires metacognition regarding the state of the user.

[0011] One aspect of the present disclosure is a learning device that includes a learning unit that learns, by supervised learning using training data including the input information and correct labor productivity-related information or by reinforcement learning, a determination model that calculates labor productivity-related information that indicates an index regarding the user's labor productivity using one or more input information selected from the group consisting of objective information obtained by measuring the user's biological activity while the user is responding to a question, subjective information obtained by the user's subjective selection, attribute information that indicates the user's attributes, information on changes over time in the objective information, information on changes over time in the subjective information, and information on changes over time in the attribute information.

[0012] One aspect of the present disclosure is a determination system including: a learning unit that learns, by supervised learning using training data including the input information and correct labor productivity-related information or by reinforcement learning, a determination model that calculates labor productivity-related information that indicates an index related to the user's labor productivity using one or more input information selected from objective information obtained by measuring the user's biological activity while the user is responding to a question, subjective information obtained from the user's subjective selection, attribute information that represents the user's attributes, information on changes over time of the objective information, information on changes over time of the subjective information, and information on changes over time of the attribute information; and a determination unit that obtains an estimation result of the labor productivity-related information of an estimation target person, who is the user to be estimated, by inputting the input information obtained about the estimation target person into the determination model.

[0013] One aspect of the present disclosure is the above-mentioned determination system, wherein the user from whom the training data was obtained and the estimated subject are different users or the same user.

[0014] One aspect of the present disclosure is a determination method including an estimation step of obtaining an estimated result of labor productivity-related information of an estimation subject, who is a user to be estimated, by inputting the input information obtained about the estimation subject, who is the user to be estimated, into a determination model that calculates labor productivity-related information that indicates an index related to the labor productivity of the user, using one or more input information from among objective information obtained by measuring the user's biological activity while the user is responding to a question, subjective information obtained by the user's subjective selection, attribute information that represents the user's attributes, information on changes over time of the objective information, information on changes over time of the subjective information, and information on changes over time of the attribute information.

[0015] One aspect of the present disclosure is a learning method including a learning step of learning, by supervised learning using training data including the input information and the correct labor productivity-related information or by reinforcement learning, a determination model that calculates labor productivity-related information that indicates an index regarding the labor productivity of the user, using input information of one or more of objective information obtained by measuring the user's biological activity while the user is responding to a question, subjective information obtained by the user's subjective selection, attribute information that indicates the user's attributes, information on changes over time of the objective information, information on changes over time of the subjective information, and information on changes over time of the attribute information.

[0016] One aspect of the present disclosure is a computer program for causing a computer to function as the above-described determination device.

[0017] One aspect of the present disclosure is a computer program for causing a computer to function as the learning device described above.

[0018] According to the present disclosure, for example, it becomes possible to reduce the burden of measurement and accurately estimate labor productivity on a daily basis.

[0019] FIG. 1 is a schematic block diagram showing a system configuration of a determination system according to an embodiment of the present disclosure. FIG. 1 is a diagram showing measurement of a person to be estimated using the determination system according to the embodiment. FIG. 2 is a diagram showing an example of the flow of a dialogue with a dialogue agent using a measurement terminal device according to the embodiment. FIG. 2 is a schematic block diagram showing an example of the functional configuration of a learning device according to the embodiment. FIG. 3 is a diagram showing an example of teacher data according to the embodiment. FIG. 4 is a diagram showing an example of preprocessed teacher data according to the embodiment. FIG. 3 is a schematic block diagram showing an example of the functional configuration of a measurement terminal device according to the embodiment. FIG. 4 is a schematic block diagram showing an example of the functional configuration of a determination device according to the embodiment. FIG. 5 is a schematic block diagram showing an example of the functional configuration of a manager terminal device according to the embodiment. FIG. 6 is a flowchart showing an example of processing of a learning device according to the embodiment. FIG. 7 is a flowchart showing an example of processing of a determination device according to the embodiment. FIG. 8 is a diagram showing an example of emotional state information used in a demonstration experiment of the determination system according to the embodiment. FIG. 9 is a diagram showing results of a demonstration experiment of the determination system according to the embodiment. FIG. 10 is a diagram showing an outline of an example of the hardware configuration of an information processing device according to the embodiment. FIG. 11 is a diagram showing a modified example of the determination device according to the embodiment. FIG. 12 is a diagram showing a modified example of the determination device according to the embodiment.

[0020] An embodiment of the present disclosure will be described below with reference to the drawings. In this embodiment, a determination system targets a worker user as a target. The determination system estimates not only a simple emotional state at the time of index measurement but also labor productivity based on objective indices that do not rely on the target's subjective information. Estimating labor productivity makes it possible to quantitatively evaluate economic losses and the improvement effects of intervention. Furthermore, by utilizing the knowledge of, for example, doctors or psychologists as a criterion for estimation, it becomes possible to objectively and accurately estimate labor productivity and provide appropriate guidance to those who require intervention.

[0021] FIG. 1 is a schematic block diagram illustrating a system configuration of a determination system 100 according to an embodiment of the present disclosure. The determination system 100 includes a learning device 10, a measurement terminal device 20, a determination device 30, and an administrator terminal device 40. While FIG. 1 illustrates one measurement terminal device 20 and one administrator terminal device 40, the determination system 100 may include multiple measurement terminal devices 20 and multiple administrator terminal devices 40. The learning device 10, the measurement terminal device 20, the determination device 30, and the administrator terminal device 40 may be communicatively connected via a network 70. The determination device 30, the measurement terminal device 20, and the administrator terminal device 40 may be communicatively connected via the network 70. The network 70 may be a network using wireless communication or a network using wired communication. The network 70 may be configured using a public network such as the Internet, or a private network such as a local area network (LAN). The network 70 may be configured by combining multiple networks.

[0022] FIG. 2 is a diagram showing how a person to be estimated is measured using the determination system 100. The measurement terminal device 20 is an example of a device equipped with a built-in sensor capable of measuring vital information of the person to be estimated. A non-contact sensor such as a camera or a microphone is used as this sensor. The person to be estimated responds to questions from the dialogue agent output as images and audio using the display and speaker of the measurement terminal device 20. The measurement terminal device 20 may display an image of the dialogue agent without outputting the audio of the dialogue agent, or may not display an image of the dialogue agent but output the audio of the dialogue agent. The measurement terminal device 20 acquires audiovisual information, including video data and audio data, while the person to be estimated is interacting with the dialogue agent in response to the questions from the dialogue agent, and transmits the acquired information to the determination device 30.

[0023] The determination device 30 extracts the response reactions of the estimated subject 80, such as response time, gaze, facial expression, and tone of voice, as feature quantities based on the audiovisual information received from the measurement terminal device 20, and estimates the labor productivity of the estimated subject 80 using the extracted feature quantities. These response reactions are vital information and are therefore objective information that is not dependent on the subjective judgment of the estimated subject. The response reactions used for estimation are the combination of response reactions that provided the highest estimation accuracy in demonstration experiments. Furthermore, in addition to or instead of the response reactions, subjective information such as the content of the speech of the estimated subject when responding, or attribute information of the estimated subject may also be used for estimation.

[0024] FIG. 3 is a diagram showing an example of the flow of a dialogue with a dialogue agent using the measurement terminal device 20. First, the measurement terminal device 20 outputs Question 1 by the dialogue agent. The person to be estimated answers Question 1 (Response 1). Next, the measurement terminal device 20 outputs Question 2 by the dialogue agent. The person to be estimated answers Question 2 (Response 2). Next, the measurement terminal device 20 outputs Question 3 by the dialogue agent. The person to be estimated answers Question 3 (Response 3). The measurement terminal device 20 ends the measurement. The questions are intended to observe the user's reaction to some kind of stimulus. The number of questions is arbitrary, and for example, it may be just one. Furthermore, the content of the questions can be arbitrary. However, it is desirable that the dialogue include questions that require metacognition regarding the user's own condition, etc., rather than questions that can be answered immediately based on knowledge (such as a question that asks the user's name), such as "How are you feeling today?". Examples of such questions include "If you were to compare your mood to a color?" and "If you were to compare your mood to the weather, how would you describe it?" Metacognition refers to, for example, the ability to objectively grasp one's own cognitive activities and to recognize the state of one's own cognition.

[0025] Let us assume that Question 1 starts at t1 and ends at t2, Answer 1 starts at t3 and ends at t4, Question 2 starts at t5 and ends at t6, Answer 2 starts at t7 and ends at t8, Question 3 starts at t9 and ends at t10, and Answer 3 starts at t11 ​​and ends at t12. The measurement terminal device 20 transmits audiovisual information for all or a portion of the section from t1 to t12 to the determination device 30. The audiovisual information for the portion of the section may be audiovisual information for multiple discontinuous sections, and the video data section and the audio data section may be different sections. The determination device 30 extracts features from each of the video data and audio data contained in the received audiovisual information. For example, the determination device 30 obtains facial expression information from the video data from t1 to t4 and response time and prosodic information such as pitch and tone representing tone of voice from the audio data from t2 to t4 as features. The determination device 30 may further acquire other feature quantities. Note that even if the information is of the same type, if the information is obtained from different sections of video data or audio data, the feature quantities are different. For example, facial expression information acquired from video data from times t1 to t4 and facial expression information acquired from video data from times t5 to t8 are different feature quantities. The determination device 30 estimates labor productivity using the acquired feature quantities.

[0026] The determination system 100 measures the responses of the subject and estimates the labor productivity, for example, daily, and monitors the mental health by observing changes in the estimation results over a long period of time. The determination system 100 can also provide diagnostic support by intervening with the subject to encourage them to see an industrial physician based on the daily estimation results of labor productivity and changes in the estimation results over a long period of time.

[0027] 4 is a schematic block diagram showing a specific example of the functional configuration of learning device 10. Learning device 10 is configured using an information processing device such as a personal computer or a server device. Learning device 10 includes a communication unit 11, a storage unit 12, and a control unit 13.

[0028] The communication unit 11 is a communication device. The communication unit 11 may be configured as, for example, a network interface. The communication unit 11 communicates data with other devices via the network 70 in accordance with the control of the control unit 13. The communication unit 11 may be a device that performs wireless communication or a device that performs wired communication.

[0029] The storage unit 12 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 12 stores data used by the control unit 13. The storage unit 12 may function as, for example, a teacher data storage unit 121, a preprocessed teacher data storage unit 122, and a trained model storage unit 123.

[0030] The teacher data storage unit 121 stores teacher data used in the learning process executed by the learning device 10. The teacher data stored in the teacher data storage unit 121 includes video data and audio data of the subject being measured during a dialogue with a dialogue agent, as well as the subject's correct labor productivity-related information. The subject being measured is a worker and may be the same user as the estimated subject or a different user. The teacher data may also include one or more of the following: objective information of the subject being measured that cannot be obtained through a dialogue with the dialogue agent (e.g., sleep time, sleep quality, working hours, and workload) and changes in that objective information over time; subjective information answered by the subject being measured separately from the dialogue (e.g., emotional state) and changes in that subjective information over time; and attribute information of the subject being measured and changes in that attribute information over time. The labor productivity-related information is information related to labor productivity and includes one or more of a presenteeism value, an absenteeism value, and a performance value. In addition to or instead of these values, the labor productivity-related information may include one or more of information representing psychological state, information representing health state, engagement information, work performance information, sales performance information, and personnel evaluation information. Performance is information that expresses labor productivity evaluations, such as presenteeism, absenteeism, psychological state, health state, engagement, work performance, sales performance, and personnel evaluation, using multiple levels of values. Note that the emotional state of the subjective information and the psychological state of the labor productivity-related information are similar information, so if the emotional state is included in the subjective information, the psychological state is not included in the labor productivity-related information. For example, presenteeism indicates a state in which an employee is present at work but has reduced labor productivity due to physical or mental illness, while absenteeism indicates a state in which physical or mental illness makes it difficult to work. Engagement indicates, for example, the employee's relationship with the workplace, or their attachment or attachment to the workplace. Changes in objective information, subjective information, and attribute information over time indicate the progression of changes in the information over a specified period, such as a year, a month, or a few weeks. Furthermore, in the case of objective information or subjective information obtained from video data, the change over time may be a transition of the information according to the elapsed time from the start to the end of the video data.

[0031] The preprocessed teacher data storage unit 122 stores preprocessed teacher data. The preprocessed teacher data is teacher data that includes information obtained by preprocessing the teacher data. For example, the preprocessed teacher data may be teacher data to which predetermined information obtained from the teacher data through preprocessing has been added, or may be data in which some or all of the values ​​included in the teacher data have been replaced with other values ​​through preprocessing. The predetermined information obtained through preprocessing is a feature used to estimate labor productivity-related information. The predetermined information obtained through preprocessing may include, for example, information such as facial expression and posture obtained from video data, information such as reaction time, intonation, volume, sound pressure level, acoustic spectrum, and speaking time obtained from audio data, as well as changes in such information over time.

[0032] The trained model storage unit 123 stores a trained model obtained by a learning process using the preprocessed teacher data stored in the preprocessed teacher data storage unit 122.

[0033] The control unit 13 is configured using a processor such as a CPU (Central Processing Unit) and a memory (main storage device). The control unit 13 functions as an information control unit 131, a preprocessing control unit 132, and a learning control unit 133 by the processor executing a program. Note that all or part of the functions of the control unit 13 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The above program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as a flexible disk, a magneto-optical disk, a ROM (Read Only Memory), a CD-ROM, and a semiconductor storage device (e.g., an SSD: Solid State Drive), as well as storage devices such as a hard disk or semiconductor storage device built into a computer system. The above program may be transmitted via a telecommunications line.

[0034] The information control unit 131 controls the input and output of information. For example, the information control unit 131 acquires training data from another device (such as an information processing device or a storage medium) and stores it in the training data storage unit 121. The information control unit 131 may receive audiovisual information, including video data and audio data, of the subject engaged in a dialogue with a dialogue agent from the measurement terminal device 20. The information control unit 131 may generate training data by adding to the received audiovisual information objective information and its changes over time, subjective information and its changes over time, attribute information and its changes over time, and correct labor productivity-related information for the subject. The added information may be received from another device, such as the administrator terminal device 40, read from a recording medium, or input via an input unit (not shown) included in the learning device 10. The video data, audio data, objective information and its changes over time, subjective information and its changes over time, and attribute information and its changes over time are used as explanatory variables for estimating labor productivity evaluation index-related information. In addition, the information control unit 131 transmits the learned model stored in the learned model storage unit 123 to another device (e.g., the determination device 30).

[0035] The preprocessing control unit 132 generates preprocessed teacher data by performing a predetermined preprocessing on the teacher data. For example, the preprocessing control unit 132 acquires response feature quantities from each piece of video data and audio data included in the teacher data, and generates preprocessed teacher data by adding the acquired feature quantities to the teacher data or by replacing the acquired feature quantities with the video data and audio data of the teacher data. The feature quantities acquired from the video data are objective information such as facial expression and posture, while the feature quantities acquired from the audio data are objective information such as reaction time, intonation, volume, sound pressure level, acoustic spectrum, and speaking time, as well as subjective information such as speech content. These feature quantities can be acquired using any existing technology. Furthermore, changes in these feature quantities over time may also be used as feature quantities.

[0036] The learning control unit 133 performs a learning process using the preprocessed training data stored in the preprocessed training data storage unit 122. Specific examples of such learning processes include supervised learning using neural networks, decision trees, and support vector machines (SVMs). The learning control unit 133 performs supervised learning using the preprocessed training data, using as input features at least some of objective information obtained by measuring the user's biological activity while interacting with the dialogue agent, subjective information obtained from the user's subjective selection, the user's attribute information, and information on changes in the information over time. The learning control unit 133 generates a trained model for outputting labor productivity-related information, including one or more of the user's presenteeism value, absenteeism value, performance value, psychological state, health state, engagement, work performance, sales performance, and personnel evaluation. The learning control unit 133 records the generated trained model in the trained model storage unit 123. The trained model obtained by the learning control unit 133 may be transmitted to the determination device 30 and recorded in the determination model storage unit 321 of the determination device 30, which will be described later and is shown in FIG. 8 . Such a trained model uses as input the subjective information, subjective information, attribute information of the person to be estimated, and some information obtained from changes in that information over time as features, and is capable of calculating labor productivity-related information including one or more of the subject's presenteeism value, absenteeism value, performance value, psychological state, health state, engagement, work performance, sales performance, and personnel evaluation.

[0037] When acquiring information similar to the preprocessed teacher data as the teacher data, the learning device 10 does not need to have the preprocessed teacher data storage unit 122 and the preprocessing control unit 132. The learning control unit 133 executes the learning process using the teacher data stored in the teacher data storage unit 121.

[0038] FIG. 5 is a diagram showing an example of training data. The training data includes video data and audio data of the subject being measured during a dialogue with the dialogue agent, and correct labor productivity-related information. The video data and audio data are obtained, for example, by the measurement terminal device 20. The training data may include only one of the video data and the audio data. The training data may further include some or all of objective information, subjective information, attribute information, information on changes in objective information over time, information on changes in subjective information over time, and information on changes in attribute information over time.

[0039] The objective information includes, for example, one or more of sleep duration, sleep quality, working hours, and workload. Values ​​measured by any existing device can be used as information on sleep quality. When the sleep quality perceived by the user is used, the sleep quality information is included in the subjective information. The subjective information includes, for example, an emotional state representing an emotion selected by the subject. The emotional state is expressed by a line segment of an emotion plotted on Russell's circumplex model or a grid. The attribute information includes, for example, one or more of age, gender, personality tendency, occupation, age, and annual income.

[0040] The labor productivity related information is, for example, information acquired at the time of a consultation by a psychiatrist. The labor productivity related information includes information on presenteeism value, absenteeism value, psychological state, health state, performance value, engagement, work performance, sales performance, and personnel assessment, but may also include some of these pieces of information. For example, the labor productivity related information may include at least one of the presenteeism value, absenteeism value, and performance value.

[0041] The presenteeism value can be the one-item version of the Presenteeism Scale, but values ​​from other indicators can also be used. The absenteeism value can be the number of days taken off sick in a specified period, such as one year. The psychological state is represented by plotting on Russell's circumplex model. The health state can be represented by any indicator that quantitatively represents the health state. Examples of indicators of the health state include depression level and stress level. Examples of indicators of depression level include, but are not limited to, the Quick Inventory of Depressive Symptomatology (QIDS-J) and the Self-Rating Depression Scale (SDS). The stress level can be a value from a stress check, etc. The performance value is a value determined by the findings of a psychiatrist, psychologist, etc., or an estimated value based on the findings of a psychiatrist, psychologist, etc. Performance is represented by any of a number of levels. For example, performance may be represented by two-level label data, "dangerous" and "normal," or by three-level label data, "blue (normal), yellow (caution), and red (danger)," corresponding to multiple colors. For example, a psychiatrist may determine a performance value by referring to video and audio data of the subject interacting with a dialogue agent, subjective information such as emotional state, attribute information, and the subject's mental workload and free comments that are not included in the training data. Engagement can be measured using an engagement score that quantifies the relationship between the employee and the company. Values ​​of any indicator used by the company, etc., can be used for work performance, sales performance, and personnel evaluation.

[0042] The emotional state is used when the output data of the learning model is a value of presenteeism, but is not included in the training data when the labor productivity-related information includes a psychological state. For example, the emotional state is used when estimating a value of presenteeism.

[0043] FIG. 6 is a diagram showing an example of preprocessed training data. The preprocessed training data shown in FIG. 6 is data obtained by adding various features obtained by preprocessing the training data shown in FIG. 5. Specifically, at least some of the facial expression, posture, and emotional state information obtained from video data and audio features such as reaction time, intonation, volume, sound pressure level, acoustic spectrum, speech time, and speech content obtained from audio data are acquired as features and added to the training data. Facial expression, posture, and audio features are objective information, while the emotional state and speech content are subjective information. Furthermore, one or more of information on changes in objective information over time and information on changes in subjective information over time may be added to the preprocessed training data.

[0044] The facial expression information uses action units, which are a series of movements of each facial feature. Specific examples include raising and lowering of eyebrows, wrinkles between the eyebrows, opening and closing of eyes, raising and lowering of corners of the mouth, and opening and closing of the mouth. The posture information uses values ​​representing the tilt of the head along three axes: front, back, left, right, up, and down. The facial expression information and posture information may be obtained from, for example, all or part of the video data from the start time t1 of Question 1 to the end time t4 of Answer 1, the video data from the start time t5 of Question 2 to the end time t8 of Answer 2, the video data from the start time t9 of Question 3 to the end time t12 of Answer 3, the video data from the start time t3 to the end time t4 of Answer 1, the video data from the start time t7 to the end time t8 of Answer 2, and the video data from the start time t11 to the end time t12 of Answer 3. The feature amount may include the type of detected action unit, the number of times a specific action unit is detected, etc.

[0045] The reaction time is the time from when the dialogue agent finishes asking a question until the subject begins speaking. For example, it is the time from time t2 to time t3, the time from time t6 to time t7, and the time from time t10 to time t11 in FIG. 3 . All or some of these may be used as feature quantities, or the result of a predetermined calculation such as the average of all these times may be used as feature quantities. The feature quantities may include changes in reaction time over time according to the elapsed time of the video data.

[0046] Furthermore, intonation is a time-series change in pitch obtained from the audio data. Volume is a tone obtained from the audio data. The speech content is obtained by speech recognition of the audio data. The intonation, volume, sound pressure level, acoustic spectrum, and speech content obtained from the audio data from start time t3 to end time t4 of answer 1, the audio data from start time t7 to end time t8 of answer 2, and the audio data from start time t11 to end time t12 of answer 3 may all be used, or some of them may be used. The feature amount may include changes over time in intonation, volume, sound pressure level, and acoustic spectrum according to the elapsed time of the video data.

[0047] 7 is a schematic block diagram showing a specific example of the functional configuration of the measurement terminal device 20. The measurement terminal device 20 is, for example, an information processing device such as a smartphone, tablet, personal computer, or dedicated device of the estimated subject. Alternatively, the measurement terminal device 20 may be an information processing device such as a tablet terminal, personal computer, or dedicated device installed in a facility used by the estimated subject. The measurement terminal device 20 includes a communication unit 21, an input unit 22, a display unit 23, an audio output unit 24, an imaging unit 25, a sound collection unit 26, a control unit 27, and a storage unit 28.

[0048] The communication unit 21 is a communication device. The communication unit 21 may be configured as, for example, a network interface. The communication unit 21 communicates data with other devices via the network 70 in accordance with the control of the control unit 27. The communication unit 21 may be a device that performs wireless communication or a device that performs wired communication.

[0049] The input unit 22 is a keyboard, a mouse, a button, a touch panel, etc., and receives information input by a user's operation. Various types of information are input by the input unit 22.

[0050] The display unit 23 is an image display device such as a liquid crystal display or an organic EL (Electro Luminescence) display. The display unit 23 may be an interface for connecting the image display device to the measurement terminal device 20. In this case, the display unit 23 generates a video signal for displaying image data and outputs the video signal to the image display device connected to the display unit 23. The display unit 23 may be configured as a touch panel integrated with the input unit 22.

[0051] The audio output unit 24 is a device that outputs sound, such as a speaker. The audio output unit 24 may be an interface for connecting an audio output device, such as a speaker or headphones, to the measurement terminal device 20. In this case, the audio output unit 24 generates an audio signal for reproducing audio data and outputs the audio signal to the audio output device connected to the audio output unit 24.

[0052] The image capturing unit 25 is a camera that captures video images of the user. The sound collecting unit 26 is a microphone that converts the collected voice of the user into sound data.

[0053] The control unit 27 is configured using a processor such as a CPU and a memory. The control unit 27 functions when the processor executes a program. Note that all or part of the functions of the control unit 27 may be realized using hardware such as an ASIC, a PLD, or an FPGA. The above program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, and a semiconductor storage device (e.g., an SSD), as well as storage devices such as a hard disk or semiconductor storage device built into a computer system. The above program may be transmitted via a telecommunications line.

[0054] The control unit 27 may execute, for example, an application installed on its own device. A specific example of such an application is an application provided to the measurement terminal device 20 as a dedicated application for the determination system 100. Such an application may be pre-installed on the measurement terminal device 20, or may be downloaded each time an estimation of labor productivity-related information of an estimation subject is performed. For example, when implemented as a web browser application, the measurement terminal device 20 may download and execute the application from a device specified by the web server (e.g., the web server itself or another server) in response to the measurement terminal device 20 connecting to the specific web server. The specific web server may be the determination device 30. The control unit 27 operates according to the program of the application being executed.

[0055] The control unit 27 displays video data of the dialogue agent asking a predetermined question on the display unit 23, and outputs audio data of the question corresponding to the video data from the audio output unit 24. The control unit 27 acquires video data captured by the imaging unit 25 and audio data collected by the audio collection unit 26 while the dialogue agent is outputting the question and while the presumed subject is answering the question. The control unit 27 transmits audiovisual information, in which the acquired video data and audio data are added with identification information of the presumed subject, to the determination device 30. Note that the control unit 27 may include, in the audiovisual information, the sleep time, sleep quality, working hours, work load, emotional state, attribute information, etc., input by the presumed subject via the input unit 22. Note that the identification information and attribute information of the presumed subject may be stored in advance in the storage unit 28.

[0056] The storage unit 28 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 28 stores various information used by the control unit 27. For example, the storage unit 28 stores a dialogue application in which the dialogue agent asks questions to the person to be estimated. The storage unit 28 also stores identification information, attribute information, etc. of the person to be estimated.

[0057] 8 is a schematic block diagram showing a specific example of the functional configuration of the determination device 30. The determination device 30 is configured using an information processing device such as a personal computer or a server device. The determination device 30 includes a communication unit 31, a storage unit 32, and a control unit 33.

[0058] The communication unit 31 is a communication device. The communication unit 31 may be configured as, for example, a network interface. The communication unit 31 communicates data with other devices via the network 70 in accordance with the control of the control unit 33. The communication unit 31 may be a device that performs wireless communication or a device that performs wired communication.

[0059] The storage unit 32 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 32 stores data used by the control unit 33. The storage unit 32 may function as, for example, a determination model storage unit 321, an attribute information storage unit 322, a measurement information storage unit 323, a preprocessed data storage unit 324, and a history information storage unit 325.

[0060] The judgment model storage unit 321 stores a judgment model used by the judgment unit 333 when performing the judgment process. The judgment model may be configured using information of a trained model generated in advance by a learning process, for example. Such a learning process may be executed by, for example, another device (e.g., the learning device 10) or by the device itself (the judgment device 30). The judgment model does not necessarily have to be generated by a learning process.

[0061] The attribute information storage unit 322 stores the identification information of the estimated subject, the attribute information of the estimated subject, and information on changes in the attribute information over time in association with each other. The measurement information storage unit 323 stores the audiovisual information received from the measurement terminal device 20. The audiovisual information may be supplemented with objective information such as the estimated subject's sleep time, sleep quality, working hours, and work load, as well as information on changes in the same over time, subjective information such as the estimated subject's emotional state, as well as information on changes in the same over time, and the estimated subject's attribute information and information on changes in the same over time.

[0062] The preprocessed data storage unit 324 stores preprocessed data including the audiovisual information of the estimation target, objective information and information on its changes over time, subjective information and information on its changes over time, attribute information and information on its changes over time, and feature quantities obtained from the audiovisual information. The feature quantities obtained from the audiovisual information include objective information such as facial expression, reaction time, intonation, volume, sound pressure level, acoustic spectrum, speaking time, and posture, as well as information on its changes over time, and subjective information such as speech content.

[0063] The history information storage unit 325 stores history information indicating the history of the estimation results of the labor productivity related information of the estimation subject. The history information is information that associates the measurement date and time with the labor productivity related information estimated using the audiovisual information of the estimation subject obtained at that measurement date and time.

[0064] The control unit 33 is configured using a processor such as a CPU and a memory. The control unit 33 functions as an information control unit 331, a preprocessing control unit 332, and a determination unit 333 by the processor executing a program. Note that all or part of the functions of the control unit 33 may be realized using hardware such as an ASIC, a PLD, or an FPGA. The above program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, and a semiconductor storage device (e.g., an SSD), as well as storage devices such as a hard disk or semiconductor storage device built into a computer system. The above program may be transmitted via a telecommunications line.

[0065] The information control unit 331 acquires from the measurement terminal device 20 audiovisual information containing video data and audio data of the estimated subject engaged in a dialogue with the dialogue agent, as well as the estimated subject's identification information. The audiovisual information may further include one or more of objective information such as sleep duration, sleep quality, working hours, and workload, information on changes over time therein, subjective information such as emotional state, information on changes over time therein, and attribute information and information on changes over time therein. The information control unit 331 may extract the estimated subject's identification information from the audiovisual information, read the attribute information identified by the extracted identification information and information on changes over time therein from the attribute information storage unit 322, and add it to the audiovisual information. The information control unit 331 also transmits information indicating the estimation results of the labor productivity-related information obtained by the determination unit 333 and information based on the estimation results to other devices, such as the measurement terminal device 20 and the manager terminal device 40. Such information exchange between the information control unit 331 and other devices may be performed, for example, via the communication unit 31.

[0066] The preprocessing control unit 332 acquires features by performing predetermined preprocessing on the video data and audio data included in the audiovisual information acquired by the information control unit 331, and generates preprocessed data by adding the acquired features to the audiovisual information. Note that if the audiovisual information contains features other than the video data and audio data, the features are set as they are in the preprocessed data. The preprocessing performed by the preprocessing control unit 332 is similar to the preprocessing performed when generating a trained model used by the determination unit 333. In other words, the processing performed by the preprocessing control unit 332 on the video data and audio data received via the audiovisual information is the same as the processing performed by the preprocessing control unit 132 on the video data and audio data of the training data. The preprocessing control unit 332 stores the generated preprocessed data in the preprocessed data storage unit 324.

[0067] The determination unit 333 performs a determination process using the determination model stored in the determination model storage unit 321 and feature quantities acquired from the preprocessed data stored in the preprocessed data storage unit 324. The determination process obtains an estimation result of the labor productivity related information of the estimation target person. The determination unit 333 adds the estimation result to the history information stored in the history information storage unit 325.

[0068] 9 is a schematic block diagram showing a specific example of the functional configuration of the administrator terminal device 40. The administrator terminal device 40 is configured using, for example, a smartphone, a tablet, or a personal computer. The administrator terminal device 40 includes a communication unit 41, an input unit 42, an output unit 43, a storage unit 44, and a control unit 45.

[0069] The communication unit 41 is a communication device. The communication unit 41 may be configured as, for example, a network interface. The communication unit 41 communicates data with other devices via the network 70 in accordance with the control of the control unit 45. The communication unit 41 may be a device that performs wireless communication or a device that performs wired communication.

[0070] The input unit 42 is a keyboard, mouse, button, touch panel, etc., and receives information input by user operation. The input unit 42 is used to input the activity type.

[0071] The output unit 43 outputs information in a form that can be recognized by the user. The output unit 43 may be, for example, an image display device such as a liquid crystal display or an organic EL display. The output unit 43 may be an interface for connecting an image display device to the manager terminal device 40. In this case, the output unit 43 generates a video signal for displaying image data and outputs the video signal to the image display device connected to the output unit 43. The output unit 43 may be a device for outputting sound, such as a speaker. The output unit 43 may be an interface for connecting an audio output device, such as a speaker or headphones, to the manager terminal device 40. In this case, the output unit 43 generates an audio signal for reproducing audio data and outputs the audio signal to the audio output device connected to the output unit 43. The output unit 43 may be configured as a touch panel integrated with the input unit 42.

[0072] The storage unit 44 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 44 stores data used by the control unit 45. The storage unit 44 stores data required when the control unit 45 performs processing.

[0073] The control unit 45 is configured using a processor such as a CPU and a memory. The control unit 45 functions when the processor executes a program. Note that all or part of the functions of the control unit 45 may be realized using hardware such as an ASIC, a PLD, or an FPGA. The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, and a semiconductor storage device (e.g., an SSD), as well as storage devices such as a hard disk or semiconductor storage device built into a computer system. The program may be transmitted via a telecommunications line.

[0074] The control unit 45 may execute, for example, an application installed on its own device (the administrator terminal device 40). A specific example of such an application is an application provided to the administrator terminal device 40 as a dedicated application for the determination system 100. Another specific example of such an application is a web browser application. Such an application may be pre-installed on the administrator terminal device 40 or may be downloaded each time the determination process is executed. For example, when implemented as a web browser application, the administrator terminal device 40 may download and execute the application from a device specified by the web server (e.g., the web server itself or another server) in response to the administrator terminal device 40 connecting to the specific web server. The specific web server may be the determination device 30. The control unit 45 operates according to the program of the application currently being executed.

[0075] The control unit 45 controls the administrator terminal device 40 in accordance with the operation of the operator and information received from the determination device 30. For example, when information transmitted from the determination device 30 is received by the communication unit 41 via the network 70, the control unit 45 generates screen data based on the received information and causes the output unit 43 to display the screen data. Such screen data includes images and text indicating the information transmitted from the determination device 30. For example, when information transmitted from the determination device 30 is received by the communication unit 41 via the network 70, the control unit 45 generates audio data based on the received information and causes the output unit 43 to output the audio data.

[0076] Next, the operation of the assessment system 100 will be described. First, training data is prepared for the learning process using the learning device 10. For example, the subject of measurement is a white-collar office worker. The subject of measurement answers questions regarding sleep time, sleep quality, working hours, workload, attribute information, emotional information (plotted using Russell's circular ring model), and psychological workload. In addition, the subject of measurement engages in a dialogue with a dialogue agent, including questions about their physical condition, such as "How are you feeling today?" as shown in FIG. 3, and their responses are recorded on video. An industrial physician specializing in psychiatry reviews the video and audio of the recorded video, and furthermore, while observing the responses to questions regarding psychological workload, labels the performance into multiple levels, such as two levels (risk, normal) or three levels (risk, caution, normal). Training data is generated based on the information obtained from the subject of measurement and the labeled information.

[0077] FIG. 10 is a flowchart showing a specific example of processing by the learning device 10. First, the information control unit 131 acquires teacher data and writes it to the teacher data storage unit 121 (step S101). The teacher data may be, for example, input by a user, acquired via communication from another information device, or acquired from a recording medium connected to the learning device 10. The information control unit 131 may also receive audiovisual information, including video data and audio data from a dialogue with the dialogue agent, from the measurement terminal device 20 and generate teacher data by adding information received from another device, such as the manager terminal device 40, to the received audiovisual information. The added information includes objective information such as sleep time, sleep quality, working hours, and work load; information on changes in the objective information; subjective information such as emotional state; information on changes in the subjective information over time; attribute information; information on changes in the attribute information over time; and labor productivity-related information.

[0078] The preprocessing control unit 132 performs predetermined preprocessing on each of the multiple pieces of training data to generate preprocessed training data (step S102). Specifically, the preprocessing control unit 132 acquires objective information, such as facial expression and posture, from one or more predetermined sections of video data. Furthermore, the preprocessing control unit 132 acquires objective information, such as reaction time, intonation, volume, sound pressure level, acoustic spectrum, and speech duration, as well as subjective information, such as speech content, from one or more predetermined sections of audio data. The preprocessing control unit 132 adds the acquired information to the training data to generate preprocessed training data, which is then written to the preprocessed training data storage unit 122.

[0079] The learning control unit 133 executes a learning process using multiple preprocessed teacher data stored in the preprocessed teacher data storage unit 122 (step S103). For example, the learning control unit 133 trains a trained model using multiple pairs of feature values ​​and labor productivity-related information read from the preprocessed teacher data. The trained model receives as input, as features, at least some of the following information: information obtained by measuring the user's biological activity while interacting with the dialogue agent; objective information such as sleep time, sleep quality, working hours, and workload; subjective information obtained from the user's subjective selection; user attribute information; and information on changes in these information over time. The trained model outputs labor productivity-related information including one or more of the user's presenteeism value, absenteeism value, performance value, psychological state, health state, engagement, work performance, sales performance, and personnel evaluation. The output performance value is an estimated value based on a psychiatrist's findings. The learning control unit 133 records the generated trained model in the trained model storage unit 123.

[0080] It is assumed that the types of features to be used in learning are given in advance. The training data acquired in step S102 only needs to include information for obtaining the features to be used in learning. In addition, the preprocessing control unit 132 only needs to generate at least the features to be used in learning in the preprocessing of step S102, and does not need to generate features not to be used in learning.

[0081] The learning device 10 may perform the process of FIG. 10 by changing the combination of feature types used. The learning control unit 133 of the learning device 10 inputs features obtained from preprocessed teacher data not used in learning into trained models generated by the process of FIG. 10 using each different combination of feature types, and compares the resulting estimation results with the correct answer set in the labor productivity-related information of the preprocessed teacher data. The learning control unit 133 of the learning device 10 may calculate the accuracy rate of the estimation result for each trained model using multiple preprocessed teacher data, and select the trained model with the highest accuracy rate or an accuracy rate higher than a predetermined value as the trained model to be output to the determination device 30. In this case, the learning control unit 133 may add information on the types of feature types used in the selected trained model to the selected trained model. The information control unit 131 reads the selected trained model from the trained model storage unit 123 and outputs it to the determination device 30. The information control unit 331 of the determination device 30 writes the learned model acquired from the learning device 10 as a determination model in the determination model storage unit 321.

[0082] Next, the operation of the labor productivity-related information estimation process performed by the determination system 100 will be described. The person to be estimated inputs a dialogue start signal via the input unit 22 of the measurement terminal device 20. Furthermore, the person to be estimated may input, before or after the dialogue, their own identification information, objective information, information on changes in the objective information over time, subjective information, changes in the subjective information over time, attribute information, and information on changes in the attribute information over time. For example, the objective information is information on sleep time, sleep quality, work hours, and workload; the information on changes in the objective information over time is information on changes in sleep time, sleep quality, work hours, and workload over time; the subjective information is information on emotional state; and the information on changes in the subjective information over time is information on changes in the emotional state over time. The control unit 27 executes a dialogue application and outputs video data and audio data of the dialogue agent via the display unit 23 and audio output unit 24, respectively. The control unit 27 acquires video data and audio data from the dialogue agent's first question to the person to be estimated to the time when the person to be estimated answers the last question via the imaging unit 25 and audio collection unit 26, respectively. The control unit 27 adds the input identification information of the person to be estimated, objective information, information on changes in the objective information over time, subjective information, changes in the subjective information over time, attribute information, information on changes in the attribute information over time, and the measurement time to the audiovisual information including the acquired video data and audio data, and transmits the resulting information to the determination device 30. Some of the information added to the audiovisual information may be information that has been stored in advance in the storage unit 28. Furthermore, the control unit 27 may read information on working hours, work load, and changes therein over time from an attendance management system of a company or the like.

[0083] 11 is a flowchart showing a specific example of processing by the determination device 30. The information control unit 331 of the determination device 30 receives audiovisual information from the measurement terminal device 20 (step S201). The preprocessing control unit 332 may acquire identification information of the subject from the audiovisual information, read attribute information specified by the acquired identification information and information on changes in the attribute information over time from the attribute information storage unit 322, and add these to the audiovisual information. The information control unit 331 writes the audiovisual information to the measurement information storage unit 323.

[0084] The preprocessing control unit 332 performs preprocessing on the audiovisual information in the same manner as the preprocessing control unit 132 to generate preprocessed data (step S202). That is, the preprocessing control unit 332 acquires objective information such as facial expression and posture from the video data, and acquires objective information such as reaction time, intonation, volume, sound pressure level, acoustic spectrum, and speech time, as well as subjective information such as speech content, from the audio data. The preprocessing control unit 332 may acquire the type of feature to be generated from information added to the determination model. The preprocessing control unit 332 adds the acquired information to the audiovisual information to generate preprocessed data, which is then written to the preprocessed data storage unit 324.

[0085] The determination unit 333 inputs the feature values ​​acquired from the preprocessed data generated in step S202 into the determination model to obtain an estimation result of the labor productivity-related information of the estimation target (step S203). The feature values ​​include at least some of the following information: information obtained by measuring the biological activity of the estimation target while the target is interacting with the dialogue agent; objective information such as sleep time, sleep quality, working hours, and workload; subjective information obtained from the estimation target's subjective selection; attribute information of the estimation target; and information on changes in these information over time. The determination unit 333 may also acquire the type of feature value to input into the determination model from information added to the determination model. The determination unit 333 associates the identification information of the estimation target, the estimation result, and the measurement time and adds them to the history information stored in the history information storage unit 325. The estimation result of the labor productivity-related information may include estimated values ​​of each index representing labor productivity and an estimated value of performance indicating an evaluation of labor productivity based on estimated values ​​of multiple indexes. For example, the labor productivity related information may include one or more estimates of presenteeism, absenteeism, performance, psychological state, health state, engagement, work performance, sales performance, and personnel evaluations, or a combination thereof, which represent indicators of labor productivity.

[0086] The information control unit 331 outputs the estimation result to one or both of the measurement terminal device 20 and the manager terminal device 40 (step S204). For example, the information control unit 331 determines whether or not a warning is necessary based on the estimation result. To determine whether or not a warning is necessary, the information control unit 331 reads out past estimation results stored in association with the identification information of the person being estimated from the history information storage unit 325. The information control unit 331 determines whether or not the current estimation result estimated in step S203 satisfies a predetermined condition indicating low labor productivity, and whether or not a change in labor productivity-related information from the past estimation result to the current estimation result satisfies a predetermined condition indicating a deterioration in labor productivity. Predetermined conditions indicating low labor productivity may include, for example, at least one of the presenteeism, engagement, work performance, sales performance, or personnel evaluation values ​​in the current estimation result being lower than the respective predetermined standards; the absenteeism value in the current estimation result being higher than the predetermined standard; the psychological state in the current estimation result being within a predetermined region in Russell's Circumplex model; the health state in the current estimation result being consistent with a cautionary or poor state; or the performance in the current estimation result being red (dangerous). Predetermined conditions indicating a deterioration in labor productivity may include a rate of decline in at least one of the presenteeism, engagement, work performance, sales performance, or personnel evaluation values ​​being equal to or greater than the respective predetermined standard; an increase in the absenteeism value being equal to or greater than a predetermined level; the psychological state approaching a predetermined region in Russell's Circumplex model by a predetermined amount or more; the health state being in a cautionary state for a predetermined number of consecutive times; or the percentage of performance values ​​in the most recent estimation results being yellow (caution) being equal to or greater than a predetermined level. Note that these are merely examples, and the predetermined conditions may be set arbitrarily. If any of the conditions is met, the information control unit 331 determines that a warning is necessary.

[0087] If the information control unit 331 determines that no warning is necessary, it transmits result notification information including a message indicating that the result is normal to the measurement terminal device 20. If it determines that warning is necessary, it transmits result notification information including a message recommending an interview with a doctor or industrial health staff to the measurement terminal device 20. The control unit 27 of the measurement terminal device 20 outputs the result notification information received by the communication unit 21 from the determination device 30 via the display unit 23 or the audio output unit 24. Furthermore, if the information control unit 331 determines that warning is necessary, it may transmit information about a person to be warned to the manager's terminal device 40 of the manager. The information about a person to be warned includes, for example, the name of the person to be warned, the estimation result, the measurement date and time, and the history of the estimation result. The control unit 45 of the manager's terminal device 40 outputs the information about a person to be warned received by the communication unit 41 from the determination device 30 to the output unit 43.

[0088] The audiovisual information received in step S201 only needs to include information for obtaining features to be used in the determination process in step S203. In step S202, the preprocessing control unit 332 only needs to generate features to be input to the determination model in step S203, and does not need to generate features that are not input to the determination model.

[0089] The determination unit 333 of the determination device 30 may perform estimation using a trained model trained using past data of the estimation target as training data as the determination model, or may perform estimation using a trained model trained using training data of a measurement target different from the estimation target as the determination model. Furthermore, the determination unit 333 of the determination device 30 may combine, as a final estimation result, an estimation result obtained using a trained model trained using past data of the estimation target as training data as the determination model, with an estimation result obtained using a trained model trained using training data of a measurement target different from the estimation target as the determination model. Examples of combinations include calculating an average value and performing weighted addition.

[0090] Furthermore, the determination device 30 may have the learning control unit 133 of the learning device 10. The learning control unit 133 of the determination device 30 receives the correct labor productivity related information input by the input unit 22 of the measurement terminal device 20 or the input unit 42 of the manager terminal device 40, and learns (updates) the determination model using the feature amounts input to the determination model in step S203 and the received correct labor productivity related information as training data.

[0091] Next, a demonstration experiment using the above-described assessment system 100 will be described. Approximately 100 healthy adult workers participated. Participants answered questions displayed on the display unit 23 (monitor) of the measurement terminal device 20 according to a prescribed schedule, and their responses were recorded as video data including video and audio data using the imaging unit 25 and audio recording unit 26. The survey period was two weeks. At baseline, survey items included each participant's basic attributes (age, gender, job title, annual income, and educational background), commute time, working hours, sleep duration, the Athens Insomnia Scale, personality traits (Japanese version of the Ten-Item Personality Inventory), and psychological distress (K6). Daily survey items included each participant's emotional state (at least twice a day), mental workload (NASA-TLX) (once a day), the presenteeism scale (one-item version), working hours, a sleep meter (for 10 days), free comments, and video and audio data during interactions with a conversational agent, obtained via a PC (personal computer) built-in camera or an external webcam.

[0092] Fig. 12 shows an example of emotional state information. The emotional state information was obtained by plotting the location on Russell's circular ring model in response to the question, "Please show how you are feeling right now on a diagram." Emotional states were plotted at least twice a day.

[0093] In the dialogue with the conversational agent, questions 1 to 3 were asked as in Figure 3, and the responses to each question were recorded. Question 1 was "How are you feeling today?" Video data was then taken of the participants' facial expressions, limb movements, and posture.

[0094] Furthermore, a psychiatric industrial physician assessed participants' performance on a three-point scale: blue (normal), yellow (caution), or red (danger) based on the recorded video data, mental workload, working hours, and free comments. The results were used as training data. A trained model was trained using facial expressions, limb movements, and posture information obtained from video data from times t1 to t4, audio features (reaction time, intonation, volume, and speaking time) obtained from audio data from times t2 to t4, time-series changes in these facial expressions, posture, and audio features, and working hours as inputs, and presenteeism and performance values ​​were output. Approximately 1,500 video data sets were recorded. Presenteeism and performance values ​​were estimated using the data of some participants and those not used in training as estimation targets. The degree of decline in work performance was used as the performance value.

[0095] 13 and 14 are diagrams showing experimental results. Fig. 13 is a diagram showing the relationship between the presenteeism estimation result estimated by the determination device 30 and the true value of presenteeism. Fig. 14 is a diagram showing the relationship between the performance value estimated by the determination device 30 and the true value of the performance judgment by a psychiatrist. The true value of the performance judgment by the psychiatrist in Fig. 14 is represented as Low (blue), Middle (yellow), and High (red). It can be seen from these diagrams that high estimation accuracy is achieved by the determination system 100 of this embodiment.

[0096] In the above-described embodiment, for example, a worker's smartphone terminal can be used as the measurement terminal device 20 to perform measurements at any time and any place, and the results obtained by the determination device 30 can be sent to a manager's terminal device 40 of a supervisor or other manager for sharing. The manager can then provide labor management and rest guidance to the worker based on the estimation results.

[0097] In addition, in environments where the terminal of the person being estimated cannot be used, it is also possible to use a terminal such as a smart mirror installed in a shared space as a measurement terminal device 20 to estimate the physical condition and labor productivity of the person being estimated.

[0098] It is also possible to use in-vehicle cameras and microphones to take measurements inside the vehicle and use the results to assess driving risks.

[0099] Furthermore, in the above-described embodiment, the learning device 10 has learned the learned model through supervised learning, but the learned model may also be learned through reinforcement learning based on feedback from, for example, a doctor, psychologist, etc.

[0100] Furthermore, the questions in the dialogue of the dialogue agent may be prepared in advance or may be generated by AI (artificial intelligence) or the like. For example, the generation AI may be instructed to generate a question requiring metacognition regarding the user's state, and the question obtained as the answer may be used in the dialogue with the measurement subject or the estimation subject. Furthermore, the information control unit 131 of the learning device 10 may acquire information on the question used when the teacher data was obtained, in addition to the teacher data, and the learning control unit 133 may add the information on the question to the trained model. The control unit 27 of the measurement terminal device 20 may acquire, from the determination device 30, a question assigned to the trained model used as a judgment model, and control the dialogue agent to present the acquired question to the user. Furthermore, the control unit 27 of the measurement terminal device 20 may instruct the generation AI to generate a question requiring metacognition regarding the user's state, and control the dialogue agent to present the question obtained as the answer to the user.

[0101] As described above, the advantage of estimating labor productivity-related information using features such as facial expressions obtained from video data and features obtained from audio data is that it can be carried out easily and objectively by users. Conventional stress checks conducted annually by companies and other organizations involve a long number of questions, which require time to answer. Furthermore, because the questionnaires are self-administered, there is a risk that users may lie and obtain results that do not reflect their actual mental health state. In this embodiment, users only need to respond to questions from the system in a short time, such as a few seconds to a dozen seconds, and estimations are made using objective information, so the problems associated with conventional stress checks do not occur.

[0102] FIG. 15 is a diagram illustrating an outline of an example hardware configuration of an information processing device 90 applied to this embodiment. The information processing device 90 includes a processor 91, a main memory device 92, a communication interface 93, an auxiliary memory device 94, an input / output interface 95, and an internal bus 96. The processor 91, the main memory device 92, the communication interface 93, the auxiliary memory device 94, and the input / output interface 95 are communicably connected to each other via the internal bus 96. The information processing device 90 may be applied to, for example, the learning device 10, the measurement terminal device 20, and the determination device 30. In this case, for example, the communication units 11, 21, and 31 may be configured using the communication interface 93. For example, the memory units 12, 28, and 32 may be configured using the auxiliary memory device 94. Furthermore, the control units 13, 27, and 33 may be configured using the processor 91 and the main memory device 92.

[0103] (Modification) In the present embodiment, the determination device 30 and the manager terminal device 40 are configured as separate devices, but they may also be configured as an integrated device. FIG. 16 is a diagram showing a modification of the determination device 30 configured in this manner. The determination device 30 shown in FIG. 16 includes an input unit 34 and an output unit 35 in addition to the configuration shown in FIG. 8. The input unit 34 and the output unit 35 of the determination device 30 shown in FIG. 16 function in the same manner as the input unit 42 and the output unit 43 of the manager terminal device 40, respectively. The control unit 33 operates in response to an operation on the input unit 34 and outputs information on the estimation result using the output unit 35.

[0104] In this embodiment, the learning device 10 and the determination device 30 are configured as separate devices, but they may also be configured as an integrated device. FIG. 17 is a diagram showing a modified example of the determination device 30 configured in this manner. The memory unit 32 of the determination device 30 shown in FIG. 17 also functions as a teacher data memory unit 326 and a preprocessed teacher data memory unit 327. The control unit 33 of the determination device 30 shown in FIG. 17 also functions as a learning control unit 334. The determination model memory unit 321, the teacher data memory unit 326, and the preprocessed teacher data memory unit 327 function in the same way as the learned model memory unit 123, the teacher data memory unit 121, and the preprocessed teacher data memory unit 122 of the learning device 10, respectively. The preprocessing control unit 332 and the learning control unit 334 function in the same way as the preprocessing control unit 132 and the learning control unit 133 of the learning device 10.

[0105] In this embodiment, the measurement terminal device 20 and the determination device 30 are configured as separate devices, but they may also be configured as an integrated device. FIG. 18 is a diagram showing a modified example of the determination device 30 configured in this manner. The storage unit 32 of the determination device 30 shown in FIG. 18 also functions as a measurement information storage unit 328. The measurement information storage unit 328 functions in the same way as the storage unit 28 of the measurement terminal device 20. The control unit 33 of the determination device 30 shown in FIG. 18 also functions as a measurement control unit 335. The measurement control unit 335 functions in the same way as the control unit 27 of the measurement terminal device 20. The input unit 34, display unit 36, audio output unit 37, imaging unit 38, and sound collection unit 39 of the determination device 30 shown in FIG. 18 function in the same way as the input unit 22, display unit 23, audio output unit 24, imaging unit 25, and sound collection unit 26 of the measurement terminal device 20, respectively.

[0106] The learning device 10 may be implemented using multiple information processing devices. For example, the learning device 10 may be implemented using a device such as a cloud. For example, in the learning device 10, the memory unit 12 and the control unit 13 may each be implemented in different information processing devices. For example, the memory unit 12 of the learning device 10 may be distributed and implemented across multiple information processing devices. The determination device 30 may be implemented using multiple information processing devices. For example, the determination device 30 may be implemented using a device such as a cloud. For example, in the determination device 30, the memory unit 32 and the control unit 33 may each be implemented in different information processing devices. For example, the memory unit 32 of the determination device 30 may be distributed and implemented across multiple information processing devices.

[0107] The application running on the manager terminal device 40 may not be intended to estimate information related to labor productivity indexes, but may instead perform processing using the results of such estimations to provide the user with information obtained. Such an application may, for example, perform processing using an API (Application Programming Interface) provided by the determination device 30. In this case, instead of information indicating the information itself (estimation result information) obtained from the determination device 30, the output unit 43 may output other information obtained by processing using the estimation result information to the manager terminal device 40.

[0108] The above-described assessment system evaluates the subject based on responses in simple, short dialogues, enabling long-term, continuous evaluation of daily labor productivity indicators. Specifically, by using three devices—a screen, a microphone, and a camera capable of capturing a face—flexible measurement is possible without limitations on location or time. Furthermore, this embodiment can achieve highly accurate estimation by incorporating labor productivity evaluation criteria developed by experienced industrial physicians. Furthermore, this embodiment can promote understanding of the importance of mental health management and contribute to solving social issues.

[0109] According to the above-described embodiment, the determination system includes a learning unit and a determination unit. The learning unit may be included in a learning device, and the determination unit may be included in the determination device. The learning unit uses one or more input information from among objective information obtained by measuring the user's biological activity while the user is responding to a question, subjective information obtained from the user's subjective selection, attribute information representing the user's attributes, information on changes in the objective information over time, information on changes in the subjective information over time, and information on changes in the attribute information over time to learn a determination model that calculates labor productivity-related information indicating an index of the user's labor productivity by supervised learning using training data including the input information and the correct labor productivity-related information, or by reinforcement learning. The determination unit obtains an estimation result of the labor productivity-related information of the target person by inputting the input information obtained about the target person into the determination model.

[0110] The input information may include at least objective information, which may include at least biological information obtained by a non-contact sensor.

[0111] The labor productivity related information may include at least presenteeism, absenteeism, and performance from the user's presenteeism, absenteeism, health status, psychological state, performance which expresses an evaluation of labor productivity using multiple levels of values, engagement, work performance, sales performance, and personnel evaluation.

[0112] The objective information may include one or more of the user's facial expression, voice features, posture, sleep time, sleep quality, working hours, and workload. The subjective information may include one or more of the user's emotional state and speech content. The attribute information may include one or more of the user's age, gender, personality traits, occupation, and annual income. The labor productivity related information may include one or more of the user's presenteeism, absenteeism, health status, psychological state, performance, engagement, work performance, sales performance, and personnel evaluation. Note that when the subjective information does not include the emotional state, the labor productivity related information may include the psychological state.

[0113] The objective and subjective information of the user may be acquired when the user answers questions from a psychiatrist. Action unit information may be used as facial expression information. Reaction time from the end of a question to the start of speech, intonation, voice volume, sound pressure level, acoustic spectrum, and speech duration may be used as speech features.

[0114] The user from whom the training data was obtained and the person to be estimated may be different users or may be the same user.

[0115] The question may be one that requires metacognition about the user's state.

[0116] The embodiments of this disclosure have been described in detail above with reference to the drawings, but the specific configuration is not limited to this embodiment, and includes designs within the scope that do not deviate from the gist of this disclosure.

[0117] The above-described embodiments can be used to develop mental health applications for smartphones, tablet devices, and computers intended for use by companies and individuals. Furthermore, by introducing the assessment system of the above-described embodiments into health checkups, it can be applied to the prevention, early detection, and intervention of mental illnesses. Furthermore, in addition to supporting the management of employee productivity, by introducing the assessment system of the above-described embodiments into an in-vehicle camera to check the driver's emotions, it can be used to prevent serious negligence such as road rage and drunk driving.

[0118] REFERENCE SIGNS LIST 100... Determination system, 10... Learning device, 11... Communication unit, 12... Memory unit, 121... Teacher data memory unit, 122... Preprocessed teacher data memory unit, 123... Learned model memory unit, 13... Control unit, 131... Information control unit, 132... Preprocessing control unit, 133... Learning control unit, 20... Measurement terminal device, 21... Communication unit, 22... Input unit, 23... Display unit, 24... Audio output unit, 25... Imaging unit, 26... Sound collection unit, 27... Control unit, 28... Memory unit, 30... Determination device, 31... Communication unit, 32... Memory unit, 321... Determination model memory unit, 322... Attribute information memory unit, 323... Measurement information memory unit, 324... Preprocessed data memory unit, 325... History information memory unit, 326... Teacher data memory unit, 327...Preprocessed teacher data storage unit, 328...Measurement information storage unit, 33...Control unit, 331...Information control unit, 332...Preprocessing control unit, 333...Determination unit, 334...Learning control unit, 335...Measurement control unit, 34...Input unit, 35...Output unit, 36...Display unit, 37...Audio output unit, 38...Imaging unit, 39...Sound collection unit, 40...Administrator terminal device, 41...Communication unit, 42...Input unit, 43...Output unit, 44...Memory unit, 45...Control unit, 70...Network, 90...Information processing device, 91...Processor, 92...Main memory device, 93...Communication interface, 94...Auxiliary memory device, 95...Input / output interface, 96...Internal bus

Claims

1. A determination device comprising: a determination unit that obtains an estimation result of labor productivity related information of an estimation subject, who is a user to be estimated, by inputting the input information obtained about the estimation subject, who is the user to be estimated, into a determination model that calculates labor productivity related information that indicates an index related to the labor productivity of the user, using one or more input information from among objective information obtained by measuring the user's biological activity while the user is responding to a question, subjective information obtained from the user's subjective selection, attribute information that represents the user's attributes, information on changes over time in the objective information, information on changes over time in the subjective information, and information on changes over time in the attribute information.

2. The determination device according to claim 1, wherein the input information includes at least the objective information, and the objective information includes at least information about the living body obtained by a non-contact sensor.

3. The determination device of claim 1, wherein the labor productivity related information includes at least one of the user's presenteeism, absenteeism, health status, psychological state, performance that expresses the evaluation of the labor productivity in multiple levels of values, engagement, work performance, sales performance, and personnel evaluation, including the presenteeism, absenteeism, and performance.

4. The determination device of claim 1, wherein the objective information includes one or more of the user's facial expression, voice features, posture, sleep time, sleep quality, working hours, and workload; the subjective information includes one or more of the user's emotional state and speech content; the attribute information includes one or more of the user's age, gender, personality tendencies, occupation, and annual income; and the labor productivity related information includes one or more of the user's presenteeism, absenteeism, health condition, psychological state, performance expressing an evaluation of the labor productivity using multiple values, engagement, work performance, sales performance, and personnel evaluation; and when the emotional state is not included in the subjective information, the labor productivity related information may include the psychological state.

5. The determination device according to claim 1, wherein the question is a question that requires metacognition regarding the user's state.

6. A learning device comprising: a learning unit that learns, by supervised learning using training data including the input information and correct labor productivity-related information, or by reinforcement learning, a determination model that calculates labor productivity-related information that indicates an index of the user's labor productivity, using one or more input information from among objective information obtained by measuring the user's biological activity while the user is responding to a question, subjective information obtained from the user's subjective selection, attribute information that indicates the user's attributes, information on changes over time in the objective information, information on changes over time in the subjective information, and information on changes over time in the attribute information.

7. A determination system comprising: a learning unit that learns, by supervised learning using training data including input information and correct labor productivity-related information or by reinforcement learning, a determination model that calculates labor productivity-related information that indicates an index of the user's labor productivity using one or more input information selected from objective information obtained by measuring the user's biological activity while the user is responding to a question, subjective information obtained from the user's subjective selection, attribute information that indicates the user's attributes, information on changes over time in the objective information, information on changes over time in the subjective information, and information on changes over time in the attribute information; and a determination unit that obtains an estimation result of the labor productivity-related information of the estimation target person, who is the user to be estimated, by inputting the input information obtained about the estimation target person into the determination model.

8. The determination system according to claim 7, wherein the user from whom the training data was obtained and the estimated subject are different users or the same user.

9. A determination method comprising: an estimation step of obtaining an estimation result of labor productivity related information of an estimation subject, who is a user to be estimated, by inputting the input information obtained about the estimation subject, who is the user to be estimated, into a determination model that calculates labor productivity related information that indicates an index related to the labor productivity of the user, using one or more input information from among objective information obtained by measuring the user's biological activity while the user is responding to a question, subjective information obtained from the user's subjective selection, attribute information that represents the user's attributes, information on changes over time in the objective information, information on changes over time in the subjective information, and information on changes over time in the attribute information.

10. A learning method comprising: a learning step of learning, by supervised learning using training data including the input information and correct labor productivity-related information, or by reinforcement learning, a determination model that calculates labor productivity-related information that indicates an index of the user's labor productivity, using input information of one or more of objective information obtained by measuring the user's biological activity while the user is responding to a question, subjective information obtained from the user's subjective selection, attribute information that indicates the user's attributes, information on changes over time in the objective information, information on changes over time in the subjective information, and information on changes over time in the attribute information.

11. A computer program for causing a computer to function as the determination device according to claim 1.

12. A computer program for causing a computer to function as the learning device according to claim 6.

Images