Judgment device, learning device, judgment system, judgment method, learning method, and computer program

Abstract

This reduces the burden of measurement, enabling accurate estimation of labor productivity on a daily basis. [Solution] The learning unit learns a judgment model, either through supervised learning or reinforcement learning, to calculate labor productivity-related information that indicates an indicator of the user's labor productivity, using one or more input information from among objective information obtained by measuring the user's biological data when the user responds to a question, subjective information obtained through the user's subjective selection, the user's attribute information, time-series change information of objective information, time-series change information of subjective information, and time-series change information of attribute information. The judgment unit obtains an estimation result of the labor productivity-related information of the estimated person by inputting the input information obtained about the estimated person into the learned judgment model.

Description

Technical Field

[0001] The present invention relates to a determination device, a learning device, a determination system, a determination method, a learning method, and a computer program.

Background Art

[0002] Conventionally, there are technologies for measuring and managing physical and mental health based on subjective health status of users and biometric activity data which is the required time for objective simple tasks (for example, see Patent Document 1), and technologies for estimating mental states from the expressions, voices, and biological signals of subjects (for example, see Patent Document 2). Also, there are technologies for estimating presenteeism and calculating economic losses by having users individually answer questionnaire items prepared on a server (for example, see Patent Document 3), and technologies for calculating economic losses such as leave risk using vital data related to the number of steps and speech, and also taking measures such as sending an email to the administrator (for example, see Patent Document 4).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Summary of the Invention

Problems to be Solved by the Invention

[0004] Improving labor productivity while maintaining the physical and mental health of workers benefits both workers and employers such as companies. Therefore, it is conceivable to estimate labor productivity indicators such as presenteeism and absenteeism in order to detect potential mental health problems among workers at an early stage. However, the technologies in Patent Documents 1 and 2 do not calculate economic losses such as labor productivity. The technology in Patent Document 3 calculates economic losses by estimating presenteeism, but it has the drawback of relying on the subjective evaluation of respondents rather than aiming to measure the changing mental state of workers in their daily work. Furthermore, the technology in Patent Document 4 calculates economic losses using vital data, which requires workers to constantly wear measuring devices.

[0005] In view of the above circumstances, the present invention provides a technology that enables accurate estimation of labor productivity on a daily basis while reducing the burden of measurement. [Means for solving the problem]

[0006] One aspect of the present invention is a determination device comprising a determination unit that obtains an estimation result of labor productivity-related information for an estimated person, which is a user to be estimated, by inputting the input information obtained for the estimated person, which is a user to be estimated, into a determination model that calculates labor productivity-related information indicating an indicator of the user's labor productivity, using one or more input information from among objective information obtained by measuring the user's biological body when the user is responding to a question, subjective information obtained by the user's subjective choice, attribute information representing the user's attributes, time-dependent change information of the objective information, time-dependent change information of the subjective information, and time-dependent change information of the attribute information.

[0007] One aspect of the present invention is the determination device described above, wherein the input information includes at least the objective information, and the objective information includes at least biological information obtained by a non-contact sensor.

[0008] One aspect of the present invention is the determination device described above, wherein the labor productivity-related information includes at least one of the following: presenteeism, absenteeism, and performance, from the user's presenteeism, absenteeism, health status, psychological state, performance which represents the evaluation of labor productivity as a multi-level value, engagement, work performance, sales performance, and personnel evaluation.

[0009] One aspect of the present invention is the judgment device described above, wherein the objective information includes one or more pieces of information from the user's facial expression, voice characteristics, posture, sleep duration, sleep quality, working hours, and workload; the subjective information includes one or more pieces of information from the user's emotional state and spoken content; the attribute information includes one or more pieces of information from the user's age, gender, personality traits, occupation, and annual income; the labor productivity-related information includes one or more pieces of information from the user's presenteeism, absenteeism, health status, psychological state, performance (expressed as a multi-level value for evaluating labor productivity), engagement, work performance, sales performance, and personnel evaluation; and if the subjective information does not include the emotional state, the labor productivity-related information may include the psychological state.

[0010] One aspect of the present invention is the determination device described above, wherein the question is a question that requires metacognition regarding the user's state.

[0011] One aspect of the present invention is a learning device comprising a learning unit that learns a determination model, which uses one or more input information from among objective information obtained by measuring the user's biological body when the user responds to a question, subjective information obtained by the user's subjective selection, attribute information representing the user's attributes, time-dependent change information of the objective information, time-dependent change information of the subjective information, and time-dependent change information of the attribute information, to calculate labor productivity-related information indicating an indicator of the user's labor productivity, either through supervised learning using training data including input information and correct labor productivity-related information, or through reinforcement learning.

[0012] One aspect of the present invention is a determination system comprising: a learning unit that learns a determination model, which uses one or more input information from among objective information obtained by measuring the user's biological characteristics when the user responds to a question, subjective information obtained through the user's subjective selection, attribute information representing the user's attributes, time-dependent change information of the objective information, time-dependent change information of the subjective information, and time-dependent change information of the attribute information, to calculate labor productivity-related information indicating an indicator of the user's labor productivity, either through supervised learning using training data including input information and correct labor productivity-related information, or through reinforcement learning; and a determination unit that inputs the input information obtained for the target person, who is the user to be estimated, to the determination model to obtain an estimation result of the target person's labor productivity-related information.

[0013] One aspect of the present invention is the above-described determination system, wherein the user from whom the training data was obtained and the estimated target person are different users or the same user.

[0014] One aspect of the present invention is a determination method comprising an estimation step of obtaining an estimation result of labor productivity-related information for a target person, which is a user to be estimated, by inputting the input information obtained for the target person, which is a user to be estimated, into a determination model that calculates labor productivity-related information indicating an indicator of the user's labor productivity, using one or more input information from among objective information obtained by measuring the user's biological body when the user is responding to a question, subjective information obtained by the user's subjective choice, attribute information representing the user's attributes, time-dependent change information of the objective information, time-dependent change information of the subjective information, and time-dependent change information of the attribute information.

[0015] One aspect of the present invention is a learning method having a learning step of learning, by supervised learning using teacher data including input information and correct labor productivity-related information, or by reinforcement learning, a determination model that calculates labor productivity-related information indicating an index related to the labor productivity of the user using one or more of the following: objective information obtained by measuring the living body of the user when the user is reacting to a question, subjective information obtained by the subjective selection of the user, attribute information representing the attributes of the user, time-series change information of the objective information, time-series change information of the subjective information, and time-series change information of the attribute information.

[0016] One aspect of the present invention is a computer program for causing a computer to function as the above-described determination device.

[0017] One aspect of the present invention is a computer program for causing a computer to function as the above-described learning device.

Advantages of the Invention

[0018] According to the present invention, it is possible to accurately estimate labor productivity daily while suppressing the load associated with measurement.

Brief Description of the Drawings

[0019] [Figure 1] It is a schematic block diagram showing the system configuration of a determination system according to an embodiment of the present invention. [Figure 2] It is a diagram showing a state of measurement of an estimation target person using the determination system according to the same embodiment. [Figure 3] It is a diagram showing an example of the flow of conversation with a dialogue agent using a measurement terminal device according to the same embodiment. [Figure 4] It is a schematic block diagram showing an example of the functional configuration of a learning device according to the same embodiment. [Figure 5] It is a diagram showing an example of teacher data according to the same embodiment. [Figure 6] It is a diagram showing an example of preprocessed teacher data according to the same embodiment. [Figure 7] It is a schematic block diagram showing an example of the functional configuration of the measurement terminal device according to the same embodiment. [Figure 8] It is a schematic block diagram showing an example of the functional configuration of the determination device according to the same embodiment. [Figure 9] It is a schematic block diagram showing an example of the functional configuration of the administrator terminal device according to the same embodiment. [Figure 10] It is a flowchart showing an example of the processing of the learning device according to the same embodiment. [Figure 11] It is a flowchart showing an example of the processing of the determination device according to the same embodiment. [Figure 12] It is a diagram showing an example of the information on the emotional state used in the proof-of-concept experiment of the determination system according to the same embodiment. <000010​​​​​​​​​​​​​​​​​​​​​​​​An embodiment of the present invention will be described below with reference to the drawings. In this embodiment, the judgment system uses a worker as the target of estimation. Based on objective indicators that do not rely on the subjective information of the target of estimation, the judgment system estimates not only a simple emotional state at the time of indicator measurement, but also labor productivity. Estimating labor productivity makes it possible to quantitatively evaluate economic losses and the improvement effects of interventions. Furthermore, by utilizing the knowledge of, for example, doctors and psychologists as criteria for estimation, it becomes possible to achieve objective and highly accurate estimation of labor productivity and appropriate guidance for those who require intervention.

[0021] Figure 1 is a schematic block diagram showing the system configuration of a judgment system 100 according to one embodiment of the present invention. The judgment system 100 includes a learning device 10, a measurement terminal device 20, a judgment device 30, and an administrator terminal device 40. In Figure 1, one measurement terminal device 20 and one administrator terminal device 40 are shown, but the judgment system 100 may have multiple measurement terminal devices 20 and administrator terminal devices 40. The learning device 10, the measurement terminal device 20, the judgment device 30, and the administrator terminal device 40 may be connected to each other via a network 70 so as to be communicative. Also, the judgment device 30, the measurement terminal device 20, and the administrator terminal device 40 are connected to each other via a network 70 so as to be communicative. The network 70 may be a wireless communication network or a wired communication network. The network 70 may be configured using a public network such as the Internet, or it may be configured using a private network such as a local area network (LAN). The network 70 may be configured by combining multiple networks.

[0022] Figure 2 shows the measurement process of a person to be estimated using the judgment system 100. The measurement terminal device 20 is an example of a device that incorporates sensors capable of measuring the vital information of the person to be estimated. Non-contact sensors such as cameras and microphones are used as these sensors. The person to be estimated responds to questions from the dialogue agent, which are output as images and sounds using the display and speaker of the measurement terminal device 20. The measurement terminal device 20 may display the image of the dialogue agent and not output the dialogue agent's voice, or it may not display the image of the dialogue agent and may output the dialogue agent's voice. The measurement terminal device 20 acquires audiovisual information, including video and audio data, while the person to be estimated is interacting with the dialogue agent in response to its questions, and transmits it to the judgment device 30.

[0023] The determination device 30 extracts response responses such as response time, gaze, facial expression, and tone of voice of the person to be estimated as features based on audiovisual information received from the measurement terminal device 20, and uses the extracted features to estimate the labor productivity of the person to be estimated 80. Since these response responses are vital information, they are objective information that is not subjective to the subject of the person to be estimated. The response responses used for estimation are the combination of response responses that showed the highest estimation accuracy in the empirical experiment. In addition to response responses, or instead, subjective information such as the content of the person's utterances during responses, or attribute information of the person to be estimated may also be used for estimation.

[0024] Figure 3 shows an example of the flow of dialogue with a conversational agent using the measurement terminal device 20. First, the measurement terminal device 20 outputs Question 1 from the conversational agent. The subject answers Question 1 (Answer 1). Next, the measurement terminal device 20 outputs Question 2 from the conversational agent. The subject answers Question 2 (Answer 2). Furthermore, the measurement terminal device 20 outputs Question 3 from the conversational agent. The subject answers Question 3 (Answer 3). The measurement terminal device 20 ends the measurement. The questions are intended to observe the user's response to some kind of stimulus. The number of questions is arbitrary; for example, there can be just one. The content of the questions is also arbitrary, but it is desirable to include questions that ask about metacognition regarding one's own state, etc., rather than questions that can be answered immediately based on knowledge (such as questions asking for one's name), such as "How are you feeling today?". Examples of such questions include "If you were to describe your mood with a color, what would it be?" and "If you were to describe your mood with the weather, what would it be?".

[0025] Let t1 be the start time and t2 the end time for Question 1, t3 be the start time and t4 the end time for Answer 1, t5 be the start time and t6 the end time for Question 2, t7 be the start time and t8 the end time for Answer 2, t9 be the start time and t10 the end time for Question 3, t11 be the start time and t12 the end time for Answer 3. The measurement terminal device 20 transmits audiovisual information for all or part of the interval from time t1 to t12 to the judgment device 30. The audiovisual information for part of the interval may consist of audiovisual information from multiple discontinuous intervals, and the intervals for video data and audio data may be different. The judgment device 30 extracts features from the video data and audio data contained in the received audiovisual information. For example, the judgment device 30 obtains facial expression information from the video data from time t1 to t4, and response time and prosodic information such as pitch and tone representing voice color from the audio data from time t2 to t4 as features. The determination device 30 may acquire other features as well. Note that even if the information is of the same type, if the interval of the video data or audio data from which the information was obtained is different, it will be treated as a different feature. For example, facial expression information acquired from video data at times t1 to t4 and facial expression information acquired from video data at times t5 to t8 are different features. The determination device 30 uses the acquired features to estimate labor productivity.

[0026] The assessment system 100 can monitor mental health by measuring the response of the estimated subject and estimating their labor productivity, for example, on a daily basis, and observing changes in the estimation results over a long period. Furthermore, the assessment system 100 can also provide diagnostic support by prompting the estimated subject to consult an industrial physician based on the daily labor productivity estimation results and changes in the estimation results over a long period.

[0027] Figure 4 is a schematic block diagram showing a specific example of the functional configuration of the learning device 10. The learning device 10 is configured using information processing equipment such as a personal computer or a server. The learning device 10 includes a communication unit 11, a storage unit 12, and a control unit 13.

[0028] The communication unit 11 is a communication device. The communication unit 11 may be configured, for example, as a network interface. The communication unit 11 communicates data with other devices via the network 70 in accordance with the control of the control unit 13. The communication unit 11 may be a wireless communication device or a wired communication device.

[0029] The storage unit 12 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 12 stores data used by the control unit 13. The storage unit 12 may function, for example, as a training data storage unit 121, a pre-processed training data storage unit 122, and a trained model storage unit 123.

[0030] The training data storage unit 121 stores training data used in the learning process executed in the learning device 10. The training data stored in the training data storage unit 121 includes video data and audio data of the subject during their interaction with the dialogue agent, and correct labor productivity-related information of the subject. The subject is a worker and may be the same user as the estimated subject or a different user. The training data may further include one or more of the following: objective information of the subject that cannot be obtained through interaction with the dialogue agent (e.g., sleep duration, sleep quality, working hours, workload) and changes in that objective information over time; subjective information answered by the subject separately from the dialogue (e.g., emotional state) and changes in that subjective information over time; and attribute information of the subject and changes in that attribute information over time. The labor productivity-related information is information about labor productivity and includes one or more of the following: presenteeism value, absenteeism value, and performance value. In addition to these values, labor productivity-related information may include, or substitute for, one or more of the following: information representing psychological state, information representing health status, engagement information, work performance information, sales performance information, and personnel evaluation information. Performance is information that expresses evaluations of labor productivity, such as presenteeism, absenteeism, psychological state, health status, engagement, work performance, sales performance, and personnel evaluation, on a multi-level scale. Note that since the emotional state in subjective information and the psychological state in labor productivity-related information are the same type of information, if the emotional state is included in the subjective information, the psychological state will not be included in the labor productivity-related information.

[0031] The pre-processed training data storage unit 122 stores pre-processed training data. Pre-processed training data is training data that includes information obtained by performing pre-processing on training data. For example, pre-processed training data may be training data to which predetermined information obtained from the training data by pre-processing has been added, or it may be training data in which some or all of the values ​​included in the training data have been replaced with other values ​​by pre-processing. The predetermined information obtained by pre-processing is a feature quantity used for estimating labor productivity-related information. The predetermined information obtained by pre-processing may include, for example, information such as facial expressions and posture obtained from video data, or information such as reaction time, intonation, volume, sound pressure level, acoustic spectrum, and speech time obtained from audio data, and furthermore, changes in such information over time.

[0032] The trained model storage unit 123 stores the trained model obtained by a training process using the pre-processed training data stored in the pre-processed training data storage unit 122.

[0033] The control unit 13 is composed of a processor such as a CPU (Central Processing Unit) and memory (main memory). The control unit 13 functions as an information control unit 131, a pre-processing control unit 132, and a learning control unit 133 when the processor executes a program. Note that all or part of the functions of the control unit 13 may be implemented using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The above program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and semiconductor memory devices (e.g., SSDs: Solid State Drives), as well as storage devices such as hard disks and semiconductor memory devices built into computer systems. The above program may be transmitted via a telecommunications line.

[0034] The information control unit 131 controls the input and output of information. For example, the information control unit 131 acquires training data from other devices (information processing devices or storage media) and records it in the training data storage unit 121. The information control unit 131 may receive audiovisual information, including video and audio data of the subject being measured during a conversation with a dialogue agent, from the measurement terminal device 20, and generate training data by adding objective information of the subject being measured and its changes over time, subjective information of the subject being measured and its changes over time, attribute information of the subject being measured and its changes over time, and correct labor productivity-related information of the subject being measured to the received audiovisual information. The information to be added may be received from other devices such as the administrator terminal device 40, read from a recording medium, or input by an input unit (not shown) provided in the learning device 10. The video data, audio data, objective information and its changes over time, subjective information and its changes over time, and attribute information and its changes over time are used as explanatory variables for estimating labor productivity evaluation index-related information. Furthermore, the information control unit 131 transmits the learned model stored in the learned model storage unit 123 to another device (for example, the determination device 30).

[0035] The preprocessing control unit 132 generates preprocessed training data by performing predetermined preprocessing on the training data. For example, the preprocessing control unit 132 acquires response feature quantities from each video data and audio data included in the training data, and generates preprocessed training data by adding the acquired feature quantities to the training data or by replacing the video data and audio data of the training data with them. The feature quantities obtained from the video data are objective information such as facial expressions and posture, and the feature quantities obtained from the audio data are objective information such as reaction time, intonation, volume, sound pressure level, acoustic spectrum, and speech time, as well as subjective information such as the content of speech. These feature quantities can be acquired by any existing technology. Alternatively, the changes in these feature quantities over time may also be used as feature quantities.

[0036] The learning control unit 133 executes a learning process using the pre-processed training data stored in the pre-processed training data storage unit 122. Specific examples of such learning processes include supervised learning methods such as neural networks, decision trees, and SVMs (support vector machines). By performing supervised learning using the pre-processed training data, the learning control unit 133 uses objective information obtained by measuring the biological characteristics of the user during interaction with the dialogue agent, subjective information obtained through the user's subjective choices, user attribute information, and information on the temporal changes of these characteristics as input features to generate a trained model for outputting labor productivity-related information, including one or more of the user's presenteeism, absenteeism, performance, psychological state, health state, engagement, work performance, sales performance, and personnel evaluation. The learning control unit 133 records the generated trained model in the trained model storage unit 123. The trained model obtained by the learning control unit 133 may be transmitted to the judgment device 30 and recorded in the judgment model storage unit 321 of the judgment device 30, as described in Figure 8 later. Such a trained model can use some information obtained from objective information, subjective information, attribute information, and information on the changes in these pieces of information over time as features to calculate labor productivity-related information, including one or more of the following for the estimated person: presenteeism, absenteeism, performance, psychological state, health status, engagement, work performance, sales performance, and personnel evaluation.

[0037] Furthermore, if the learning device 10 acquires information similar to pre-processed training data as training data, it does not need to have a pre-processed training data storage unit 122 and a pre-processing control unit 132. The learning control unit 133 executes the learning process using the training data stored in the training data storage unit 121.

[0038] Figure 5 shows an example of training data. The training data includes video and audio data of the subject being measured during interaction with a dialogue agent, and correct labor productivity-related information. The video and audio data are obtained, for example, by a measurement terminal device 20. The training data may include only one of the video and audio data. The training data may further include some or all of the objective information, subjective information, attribute information, information on changes in objective information over time, information on changes in subjective information over time, and information on changes in attribute information over time.

[0039] Objective information includes, for example, one or more of the following: sleep duration, sleep quality, working hours, and workload. Sleep quality information can be measured using any existing device. If the user's perceived sleep quality is used, this information is included in subjective information. Subjective information includes, for example, emotional states representing the emotions selected by the person being measured. Emotional states are represented by emotional line segments or grids, such as plotting them on Russell's annular model. Attribute information includes, for example, one or more of the following: age, gender, personality traits, occupation, and annual income.

[0040] Labor productivity-related information is, for example, information obtained during a consultation with a psychiatrist. Labor productivity-related information includes, but may include, some of, information on presenteeism, absenteeism, psychological state, health status, performance, engagement, work performance, sales performance, and personnel evaluation. For example, labor productivity-related information may include at least one of the presenteeism, absenteeism, and performance values.

[0041] The presenteeism value can be obtained using the one-item version of the presenteeism scale, but other indicators can also be used. The absenteeism value can be obtained using the number of days absent due to illness over a specified period, such as one year. Psychological state is represented by plotting on Russell's cyclic model. Health status can be represented using any indicator that expresses health status using quantitative values. Examples of health status indicators include depression level and stress level. Examples of depression level indicators include, but are not limited to, the Quick Indicator of Depressive Symptoms (QIDS-J) and the Self-Rating Depression Scale (SDS). Stress level can be represented using values ​​from stress checks, etc. Performance values ​​are determined by the findings of a psychiatrist or psychologist, or are estimates of the findings by a psychiatrist or psychologist. Performance is shown using any number of levels. For example, performance may be represented by two-level label data (dangerous / normal) or by three-level label data (blue (normal), yellow (caution), red (dangerous)). For example, a psychiatrist might determine performance values ​​by referring to video and audio data of the subject during their interaction with a dialogue agent, subjective information such as emotional state, attribute information, and the subject's mental workload and free-response data that are not included in the training data. For engagement, an engagement score that quantifies the relationship between employees and the company can be used. For work performance, sales performance, and personnel evaluations, values ​​of any indicator used by the company can be used.

[0042] Note that emotional state is used when the output data of the learning model is a presenteeism value, and is not included in the training data if psychological state is included in the labor productivity-related information. For example, emotional state is used when estimating the presenteeism value.

[0043] Figure 6 shows an example of preprocessed training data. The preprocessed training data shown in Figure 6 is data to which various features obtained by preprocessing have been added to the training data shown in Figure 5. Specifically, at least some of the information obtained from video data, such as facial expressions, posture, and emotional state, and the audio features obtained from audio data, such as reaction time, intonation, volume, sound pressure level, acoustic spectrum, speech duration, and speech content, are acquired as features and added to the training data. Facial expressions, posture, and audio features are objective information, while emotional state and speech content are subjective information. Furthermore, one or more of the time-series changes in objective information and time-series changes in subjective information may be added to the preprocessed training data.

[0044] Facial expression information is represented using action units, which are a series of movements of each facial feature. Specific examples include eyebrow movement (up and down), frown lines, eye opening and closing, corners of the mouth rising and falling, and mouth opening and closing. Posture information is represented using values ​​that show the tilt of the head along three axes: front, back, left, right, up, and down. The facial expression and posture information may be obtained from all or part of the following video data: for example, from the start time t1 of Question 1 to the end time t4 of Answer 1; from the start time t5 of Question 2 to the end time t8 of Answer 2; from the start time t9 of Question 3 to the end time t12 of Answer 3; from the start time t3 to the end time t4 of Answer 1; from the start time t7 to the end time t8 of Answer 2; and from the start time t11 to the end time t12 of Answer 3.

[0045] The reaction time is the time from when the dialogue agent finishes asking a question until the subject begins speaking. For example, in Figure 3, this is the time from time t2 to time t3, from time t6 to time t7, and from time t10 to time t11. All of these times may be used as features, some of them may be used as features, or the result of a predetermined calculation such as the average of all these times may be used as a feature.

[0046] Furthermore, intonation is the time-series change in pitch obtained from the audio data. Volume is the tone obtained from the audio data. Speech content is obtained by speech recognition of the audio data. The intonation, volume, sound pressure level, acoustic spectrum, and speech content may be obtained from all or part of the audio data from the start time t3 to end time t4 of response 1, the audio data from the start time t7 to end time t8 of response 2, and the audio data from the start time t11 to end time t12 of response 3.

[0047] Figure 7 is a schematic block diagram showing a specific example of the functional configuration of the measurement terminal device 20. The measurement terminal device 20 is, for example, an information processing device such as a smartphone, tablet, personal computer, or dedicated device of the person being estimated. Alternatively, the measurement terminal device 20 may be an information processing device such as a tablet terminal, personal computer, or dedicated device installed in a facility used by the person being estimated. The measurement terminal device 20 comprises a communication unit 21, an input unit 22, a display unit 23, an audio output unit 24, an imaging unit 25, a sound collection unit 26, a control unit 27, and a storage unit 28.

[0048] The communication unit 21 is a communication device. The communication unit 21 may be configured, for example, as a network interface. The communication unit 21 communicates data with other devices via the network 70 in accordance with the control of the control unit 27. The communication unit 21 may be a wireless communication device or a wired communication device.

[0049] The input unit 22 includes a keyboard, mouse, buttons, touch panel, etc., and receives information input through user operation. Various types of information are input through the input unit 22.

[0050] The display unit 23 is, for example, an image display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display. The display unit 23 may also be an interface for connecting the image display device to the measurement terminal device 20. In this case, the display unit 23 generates a video signal for displaying image data and outputs the video signal to the image display device connected to it. The display unit 23 may be configured as a touch panel integrated with the input unit 22.

[0051] The audio output unit 24 is a device that outputs sound, such as a speaker. The audio output unit 24 may also be an interface for connecting an audio output device such as a speaker or headphones to the measurement terminal device 20. In this case, the audio output unit 24 generates an audio signal for playing back audio data and outputs the audio signal to the audio output device connected to it.

[0052] The imaging unit 25 is a camera that captures images. The imaging unit 25 captures images of the user. The sound collection unit 26 is a microphone that converts the user's voice into audio data.

[0053] The control unit 27 is composed of a processor such as a CPU and memory. The control unit 27 functions when the processor executes a program. Note that all or part of the functions of the control unit 27 may be implemented using hardware such as an ASIC, PLD, or FPGA. The above program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and semiconductor storage devices (e.g., SSDs), as well as storage devices such as hard disks and semiconductor storage devices built into computer systems. The above program may be transmitted via a telecommunications line.

[0054] The control unit 27 may, for example, execute an application installed on its own device. A specific example of such an application is an application provided to the measurement terminal device 20 as a dedicated application for the judgment system 100. Such an application may be pre-installed on the measurement terminal device 20, or it may be downloaded each time the estimation of labor productivity-related information of the person to be estimated is performed. For example, if it is implemented as a web browser application, the measurement terminal device 20 may download and execute the application from a device specified by the web server (for example, the web server itself or another server) when the measurement terminal device 20 connects to a specific web server. The specific web server may be the judgment device 30. The control unit 27 operates according to the program of the application that is currently running.

[0055] The control unit 27 displays video data of the conversational agent asking predetermined questions on the display unit 23, and outputs audio data of the questions corresponding to the video data via the audio output unit 24. While the conversational agent is outputting questions and while the person to be estimated is answering those questions, the control unit 27 acquires video data captured by the imaging unit 25 and audio data acquired by the sound acquisition unit 26. The control unit 27 transmits audiovisual information, which includes the acquired video data and audio data plus the identification information of the person to be estimated, to the determination device 30. The control unit 27 may also include information such as sleep duration, sleep quality, working hours, workload, emotional state, and attribute information entered by the person to be estimated via the input unit 22 in the audiovisual information. The identification information and attribute information of the person to be estimated may be stored in advance in the storage unit 28.

[0056] The memory unit 28 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. The memory unit 28 stores various types of information used by the control unit 27. For example, the memory unit 28 stores a dialogue application in which the dialogue agent asks questions to the estimated target. The memory unit 28 also stores identification information and attribute information of the estimated target.

[0057] Figure 8 is a schematic block diagram showing a specific example of the functional configuration of the determination device 30. The determination device 30 is configured using information processing equipment such as a personal computer or a server device. The determination device 30 includes a communication unit 31, a storage unit 32, and a control unit 33.

[0058] The communication unit 31 is a communication device. The communication unit 31 may be configured, for example, as a network interface. The communication unit 31 communicates data with other devices via the network 70 in accordance with the control of the control unit 33. The communication unit 31 may be a device that performs wireless communication or a device that performs wired communication.

[0059] The storage unit 32 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 32 stores data used by the control unit 33. The storage unit 32 may function as, for example, a determination model storage unit 321, an attribute information storage unit 322, a measurement information storage unit 323, a pre-processed data storage unit 324, and a history information storage unit 325.

[0060] The judgment model storage unit 321 stores the judgment model used by the judgment unit 333 when performing judgment processing. The judgment model may be configured using information from a pre-trained model generated by a learning process. Such a learning process may be performed by another device (e.g., a learning device 10) or by the device itself (the judgment device 30). The judgment model does not necessarily have to be generated by a learning process.

[0061] The attribute information storage unit 322 stores the identification information of the estimated subject, the attribute information of the estimated subject, and the information on the changes in attribute information over time in association with each other. The measurement information storage unit 323 stores the audiovisual information received from the measurement terminal device 20. The audiovisual information may include objective information such as the estimated subject's sleep time, sleep quality, working hours, and workload, as well as information on their changes over time; subjective information such as the estimated subject's emotional state, as well as information on its changes over time; and attribute information of the estimated subject and information on its changes over time.

[0062] The preprocessed data storage unit 324 stores preprocessed data including audiovisual information of the estimated subject, objective information and its temporal changes, subjective information and its temporal changes, attribute information and its temporal changes, and feature quantities obtained from the audiovisual information. The feature quantities obtained from the audiovisual information include objective information such as facial expressions, reaction time, intonation, volume, sound pressure level, acoustic spectrum, speech time, and posture, as well as their temporal changes, and subjective information such as the content of speech.

[0063] The history information storage unit 325 stores history information showing the history of the estimation results of labor productivity-related information of the person being estimated. The history information is information that associates the measurement date and time with the labor productivity-related information estimated using the audiovisual information of the person being estimated obtained at that measurement date and time.

[0064] The control unit 33 is configured using a processor such as a CPU and memory. The control unit 33 functions as an information control unit 331, a pre-processing control unit 332, and a determination unit 333 when the processor executes a program. Note that all or part of the functions of the control unit 33 may be implemented using hardware such as an ASIC, PLD, or FPGA. The above program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and semiconductor storage devices (e.g., SSDs), as well as storage devices such as hard disks and semiconductor storage devices built into computer systems. The above program may be transmitted via a telecommunications line.

[0065] The information control unit 331 acquires audiovisual information from the measurement terminal device 20, which includes video and audio data of the estimated target person during their interaction with the dialogue agent, and the estimated target person's identification information. The audiovisual information may further include one or more of the following: objective information such as sleep duration, sleep quality, working hours, and workload, and their changes over time; subjective information such as emotional state, and its changes over time; and attribute information, and its changes over time. The information control unit 331 may extract the estimated target person's identification information from the audiovisual information, and read attribute information and its changes over time identified by the extracted identification information from the attribute information storage unit 322 and add it to the audiovisual information. The information control unit 331 also transmits information indicating the estimation results of labor productivity-related information obtained by the determination unit 333, and information based on the estimation results, to other devices such as the measurement terminal device 20 and the administrator terminal device 40. Such information exchange between the information control unit 331 and other devices may be performed, for example, by communication via the communication unit 31.

[0066] The preprocessing control unit 332 obtains features by performing predetermined preprocessing on the video data and audio data included in the audiovisual information acquired by the information control unit 331, and adds the acquired features to the audiovisual information to generate preprocessed data. If features other than video data and audio data are included in the audiovisual information, those features are set as they are in the preprocessed data. The preprocessing performed by the preprocessing control unit 332 is the same as the preprocessing performed when generating a trained model used by the determination unit 333. In other words, the processing that the preprocessing control unit 332 performs on the video data and audio data received from the audiovisual information is the same as the processing that the preprocessing control unit 132 performs on the video data and audio data of the training data. The preprocessing control unit 332 stores the generated preprocessed data in the preprocessed data storage unit 324.

[0067] The determination unit 333 performs a determination process using the determination model stored in the determination model storage unit 321 and the features obtained from the preprocessed data stored in the preprocessed data storage unit 324. The determination process provides an estimation result of labor productivity-related information for the person to be estimated. The determination unit 333 adds the estimation result to the history information stored in the history information storage unit 325.

[0068] Figure 9 is a schematic block diagram showing a specific example of the functional configuration of the administrator terminal device 40. The administrator terminal device 40 is configured using, for example, a smartphone, tablet, or personal computer. The administrator terminal device 40 includes a communication unit 41, an input unit 42, an output unit 43, a storage unit 44, and a control unit 45.

[0069] The communication unit 41 is a communication device. The communication unit 41 may be configured, for example, as a network interface. The communication unit 41 communicates data with other devices via the network 70 in accordance with the control of the control unit 45. The communication unit 41 may be a device that performs wireless communication or a device that performs wired communication.

[0070] The input unit 42 includes a keyboard, mouse, buttons, touch panel, etc., and receives information input through user operation. The activity type is entered via the input unit 42.

[0071] The output unit 43 outputs information in a format that the user can recognize. The output unit 43 may be an image display device such as a liquid crystal display or an organic EL display. The output unit 43 may also be an interface for connecting an image display device to the administrator terminal device 40. In this case, the output unit 43 generates a video signal for displaying image data and outputs the video signal to the image display device connected to it. The output unit 43 may also be a device that outputs sound, such as a speaker. The output unit 43 may also be an interface for connecting an audio output device such as a speaker or headphones to the administrator terminal device 40. In this case, the output unit 43 generates an audio signal for playing audio data and outputs the audio signal to the audio output device connected to it. The output unit 43 may also be configured as a touch panel integrated with the input unit 42.

[0072] The storage unit 44 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 44 stores data used by the control unit 45. The storage unit 44 stores data necessary when the control unit 45 performs processing.

[0073] The control unit 45 is composed of a processor such as a CPU and memory. The control unit 45 functions when the processor executes a program. Note that all or part of the functions of the control unit 45 may be implemented using hardware such as an ASIC, PLD, or FPGA. The above program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and semiconductor storage devices (e.g., SSDs), as well as storage devices such as hard disks and semiconductor storage devices built into computer systems. The above program may be transmitted via a telecommunications line.

[0074] The control unit 45 may, for example, execute an application installed on its own device (administrator terminal device 40). A specific example of such an application is an application provided to the administrator terminal device 40 as a dedicated application for the judgment system 100. Another specific example of such an application is a web browser application. Such an application may be pre-installed on the administrator terminal device 40, or it may be downloaded each time a judgment process is executed. For example, if it is implemented as a web browser application, the administrator terminal device 40 may download and execute the application from a device specified by the web server (for example, the web server itself or another server) when the administrator terminal device 40 connects to a specific web server. The specific web server may be the judgment device 30. The control unit 45 operates according to the program of the application being executed.

[0075] The control unit 45 controls the administrator terminal device 40 according to the operator's actions and information received from the judgment device 30. For example, when the control unit 45 receives information transmitted from the judgment device 30 via the network 70 at the communication unit 41, it generates screen data based on the received information and displays the screen data on the output unit 43. Such screen data includes images and characters that represent the information transmitted from the judgment device 30. For example, when the control unit 45 receives information transmitted from the judgment device 30 via the network 70 at the communication unit 41, it generates audio data based on the received information and outputs the audio data from the output unit 43.

[0076] Next, the operation of the judgment system 100 will be explained. First, training data is prepared in order to perform the learning process by the learning device 10. For example, the subjects of measurement are white-collar office workers. The subjects of measurement answer questions about sleep duration and quality, working hours, workload, attribute information, emotional information (plotted using Russell's cyclic model), and psychological workload. In addition, the subjects of measurement engage in a dialogue with a conversational agent, including questions about their physical condition such as "How are you feeling today?" as shown in Figure 3, and their responses are recorded on video. An industrial physician specializing in psychiatry reviews the video and audio of the recorded video, and also reviews the answers to questions about psychological workload, labeling the performance in multiple stages such as two stages (dangerous, normal) or three stages (dangerous, caution, normal). Based on the information obtained from the subjects of measurement and the labeled information, training data is generated.

[0077] Figure 10 is a flowchart showing a specific example of the processing of the learning device 10. First, the information control unit 131 acquires training data and writes it to the training data storage unit 121 (step S101). The training data may be input by a user, acquired by communication from other information devices, or acquired from a recording medium connected to the learning device 10. The information control unit 131 may also receive audiovisual information, including video data and audio data during interaction with the dialogue agent, from the measurement terminal device 20, and generate training data by adding information received from other devices such as the administrator terminal device 40 to the received audiovisual information. The information to be added includes objective information such as sleep time, sleep quality, working hours, and workload; information on changes in objective information; subjective information such as emotional state; information on changes in subjective information over time; attribute information; information on changes in attribute information over time; and information related to labor productivity.

[0078] The preprocessing control unit 132 performs predetermined preprocessing on each of the multiple training data to generate preprocessed training data (step S102). Specifically, the preprocessing control unit 132 acquires objective information such as facial expressions and posture from one or more predetermined sections of video data. Furthermore, the preprocessing control unit 132 acquires objective information such as reaction time, intonation, volume, sound pressure level, acoustic spectrum, and speech duration, as well as subjective information such as speech content, from one or more predetermined sections of audio data. The preprocessing control unit 132 adds the acquired information to the training data and writes the generated preprocessed training data to the preprocessed training data storage unit 122.

[0079] The learning control unit 133 performs a learning process using multiple pre-processed training data stored in the pre-processed training data storage unit 122 (step S103). For example, the learning control unit 133 learns a trained model using multiple sets of features and labor productivity-related information read from the pre-processed training data. The trained model is a model that outputs labor productivity-related information including one or more of the user's presenteeism value, absenteeism value, performance value, psychological state, health state, engagement, work performance, sales performance, and personnel evaluation, by providing at least some of the following information as features: information obtained by measuring the biological state of the user during interaction with the dialogue agent, objective information such as sleep duration, sleep quality, working hours, and workload, subjective information obtained through the user's subjective choices, user attribute information, and information on the temporal changes of that information. The output performance value is an estimated value of the findings by a psychiatrist. The learning control unit 133 records the generated trained model in the trained model storage unit 123.

[0080] The types of features to be used for training are assumed to be predetermined. The training data acquired in step S102 only needs to contain at least the information necessary to obtain the features to be used for training. Furthermore, the preprocessing control unit 132 only needs to generate at least the features to be used for training in the preprocessing in step S102, and does not need to generate features that will not be used for training.

[0081] The learning device 10 may perform the processing shown in Figure 10 by changing the combination of feature types used. The learning control unit 133 of the learning device 10 compares the estimation result obtained by inputting features obtained from pre-processed training data not used for training into the trained model generated by the processing shown in Figure 10 using each different combination of feature types, with the correct answer set in the labor productivity-related information of the pre-processed training data. The learning control unit 133 of the learning device 10 may calculate the accuracy rate of the estimation result for each trained model using multiple pre-processed training data sets, and select the trained model with the best accuracy rate, or an accuracy rate better than a predetermined rate, as the trained model to output to the judgment device 30. In this case, the learning control unit 133 may add information about the types of features used in the trained model to the selected trained model. The information control unit 131 reads the selected trained model from the trained model storage unit 123 and outputs it to the judgment device 30. The information control unit 331 of the determination device 30 writes the trained model acquired from the learning device 10 to the determination model storage unit 321 as a determination model.

[0082] Next, the operation of the labor productivity-related information estimation process by the judgment system 100 will be explained. The person to be estimated inputs the start of a dialogue via the input unit 22 of the measurement terminal device 20. Furthermore, the person to be estimated may input their own identification information, objective information, information on changes in objective information over time, subjective information, changes in subjective information over time, attribute information, information on changes in attribute information over time, etc., before or after the dialogue. For example, objective information is information on sleep duration, sleep quality, working hours, and workload; information on changes in objective information over time is information on changes in sleep duration, sleep quality, working hours, and workload over time; subjective information is information on emotional state; and information on changes in subjective information over time is information on changes in emotional state over time. The control unit 27 executes the dialogue application and outputs video data and audio data of the dialogue agent via the display unit 23 and audio output unit 24, respectively. The control unit 27 acquires video data and audio data from the start of the first question asked by the dialogue agent to the person to be estimated until the person to be estimated answers the last question, via the imaging unit 25 and sound collection unit 26, respectively. The control unit 27 adds the input identification information of the estimated subject, objective information, information on the change in objective information over time, subjective information, information on the change in subjective information over time, attribute information, information on the change in attribute information over time, and the measurement time to the audiovisual information, including the acquired video data and audio data, and transmits it to the judgment device 30. Some of the information added to the audiovisual information may be information that has been previously stored in the storage unit 28. In addition, the control unit 27 may read information on working hours, workload, and their changes over time from a company's attendance management system or the like.

[0083] Figure 11 is a flowchart illustrating a specific example of the processing of the determination device 30. The information control unit 331 of the determination device 30 receives audiovisual information from the measurement terminal device 20 (step S201). The preprocessing control unit 332 may obtain identification information of the person being measured from the audiovisual information, and may read attribute information and information on the change in attribute information over time from the attribute information storage unit 322 and add it to the audiovisual information. The information control unit 331 writes the audiovisual information to the measurement information storage unit 323.

[0084] The preprocessing control unit 332 performs the same preprocessing on the audiovisual information as the preprocessing control unit 132 to generate preprocessed data (step S202). Specifically, the preprocessing control unit 332 acquires objective information such as facial expressions and posture from the video data, and objective information such as reaction time, intonation, volume, sound pressure level, acoustic spectrum, and speech duration, as well as subjective information such as speech content, from the audio data. The preprocessing control unit 332 may also acquire the types of features to be generated from the information attached to the judgment model. The preprocessing control unit 332 writes the preprocessed data generated by adding the acquired information to the audiovisual information to the preprocessed data storage unit 324.

[0085] The determination unit 333 inputs the features obtained from the preprocessed data generated in step S202 into the determination model to obtain the estimation result of labor productivity-related information of the person to be estimated (step S203). The features are information obtained by measuring the biological state of the person to be estimated while they are interacting with the dialogue agent, objective information such as sleep duration, sleep quality, working hours, and workload, subjective information obtained through the subjective choices of the person to be estimated, attribute information of the person to be estimated, and information on the temporal changes of these information, at least some of these. The determination unit 333 may also obtain the types of features to be input into the determination model from information attached to the determination model. The determination unit 333 adds the identification information of the person to be estimated, the estimation result, and the measurement time to the history information stored in the history information storage unit 325, associating them.

[0086] The information control unit 331 outputs the estimation result to either or both of the measurement terminal device 20 and the administrator terminal device 40 (step S204). For example, the information control unit 331 determines whether or not a warning is necessary based on the estimation result. In order to determine whether or not a warning is necessary, the information control unit 331 reads past estimation results stored in association with the identification information of the person to be estimated from the history information storage unit 325. The information control unit 331 determines whether or not the current estimation result estimated in step S203 meets predetermined conditions indicating low labor productivity, and whether or not the change in labor productivity-related information from past estimation results to the current estimation result meets predetermined conditions indicating a deterioration in labor productivity. The predetermined conditions indicating low labor productivity may include, for example, that at least one of the presenteeism, engagement, work performance, sales performance, or performance evaluation values ​​in the current estimation is lower than the respective predetermined standard; that the absenteeism value in the current estimation is higher than the predetermined standard; that the psychological state in the current estimation falls within a predetermined region in Russell's cyclic model; that the health state in the current estimation matches a state of caution or poor health; or that the performance in the current estimation is red. The predetermined conditions indicating deterioration in labor productivity may include that the rate of decrease in at least one of the presenteeism, engagement, work performance, sales performance, or performance evaluation values ​​is above the respective predetermined standard; that the rate of increase in the absenteeism value is above a predetermined standard; that the psychological state is approaching a predetermined region in Russell's cyclic model by a predetermined amount of change or more; that the health state has remained in a state of caution for a predetermined number of times or more; or that the percentage of performance in the most recent few estimation results is above a predetermined standard. Note that these are examples, and the predetermined conditions can be set arbitrarily. The information control unit 331 determines that a warning is necessary if any of the conditions are met.

[0087] If the information control unit 331 determines that there is no need for a warning, it sends result notification information to the measurement terminal device 20, including a message indicating that the results are normal. If it determines that there is a need for a warning, it sends result notification information to the measurement terminal device 20, including a message recommending a consultation with a doctor or occupational health staff. The control unit 27 of the measurement terminal device 20 outputs the result notification information received by the communication unit 21 from the judgment device 30 via the display unit 23 and the audio output unit 24. The information control unit 331 may also send information about persons requiring attention to the administrator's terminal device 40 of the administrator if it determines that there is a need for a warning. The information about persons requiring attention includes, for example, the name of the person to be warned, the estimated result, the measurement date and time, and the history of the estimated result. The control unit 45 of the administrator terminal device 40 outputs the information about persons requiring attention received by the communication unit 41 from the judgment device 30 to the output unit 43.

[0088] The audiovisual information received in step S201 only needs to include information necessary to obtain the features used in the judgment process in step S203. In addition, in step S202, the preprocessing control unit 332 only needs to generate the features that will be input to the judgment model in step S203, and does not need to generate features that will not be input to the judgment model.

[0089] The determination unit 333 of the determination device 30 may perform estimation using a trained model that has been trained using the past data of the person to be estimated as training data, or it may perform estimation using a trained model that has been trained using training data of a measurement subject different from the person to be estimated as the determination model. Furthermore, the determination unit 333 of the determination device 30 may combine the estimation result obtained using the trained model that has been trained using the past data of the person to be estimated as training data as the determination model with the estimation result obtained using the trained model that has been trained using training data of a measurement subject different from the person to be estimated as the determination model to obtain the final estimation result. Examples of combinations include calculating the mean and weighted addition.

[0090] Furthermore, the determination device 30 may have a learning control unit 133 of the learning device 10. The learning control unit 133 of the determination device 30 receives correct labor productivity-related information input via the input unit 22 of the measurement terminal device 20 or via the input unit 42 of the administrator terminal device 40, and uses the features input to the determination model in step S203 and the received correct labor productivity-related information as training data to learn (update) the determination model.

[0091] Next, we will describe the demonstration experiment using the judgment system 100 described above. The participants were approximately 100 healthy adult workers. Participants answered questions displayed on the display unit 23 (monitor) of the measurement terminal device 20 according to a prescribed schedule, and their responses were recorded as video data including video and audio data using the imaging unit 25 and sound collection unit 26. The study period was two weeks. As survey items, at baseline, we obtained each participant's basic attributes (age, gender, job title, annual income, education level), commuting time, working hours, sleep time, Athens Insomnia Scale, personality trait scale (Japanese version Ten Item Personality Inventory), and psychological distress (K6). As daily survey items, we obtained each participant's emotional state (more than twice a day), mental workload (NASA-TLX) (once a day), presenteeism scale (single item version), working hours, sleep tracking (10 days), free-response text, and video and audio data from conversations with a dialogue agent using a PC (personal computer) built-in camera or external webcam.

[0092] Figure 12 shows an example of emotional state information. The emotional state information was obtained by plotting the position on Russell's annular model in response to the question, "Please indicate your current feelings on the graph." Emotional states were plotted at least twice a day.

[0093] In the dialogue with the conversational agent, questions 1-3 were asked, similar to Figure 3, and the responses to each question were recorded. Question 1 was, "How are you feeling today?" Video data was then used to capture the participants' facial expressions, hand and foot movements, and posture.

[0094] Furthermore, based on recorded video data, mental workload, working hours, and free-response comments, an industrial psychiatrist assessed participants' performance on a three-point scale (blue, yellow, red), and these results were used as training data. A pre-trained model was created that took as input information (facial expressions, limb movements, and posture information obtained from video data at times t1 to t4, and speech features (reaction time, intonation, volume, and speech duration) obtained from audio data at times t2 to t4, as well as the time-series changes in these facial expressions, postures, and speech features, and working hours, and outputted presenteeism and performance values. Approximately 1500 video data points were recorded. Data from some participants was used for training, and the presenteeism and performance values ​​were estimated using data from participants not used in training as estimation targets. The degree of decline in work performance was used as the performance value.

[0095] Figures 13 and 14 show the experimental results. Figure 13 shows the relationship between the estimated presenteeism result estimated by the judgment device 30 and the true value of presenteeism. Figure 14 shows the relationship between the performance value estimated by the judgment device 30 and the true value of the performance judgment by the psychiatrist. Low is represented in blue, Middle in yellow, and High in red. These figures show that the judgment system 100 of this embodiment achieves high estimation accuracy.

[0096] The above-described embodiment can be used, for example, by using a worker's smartphone terminal as the measurement terminal device 20, conducting measurements at any time and place, and transmitting the results obtained by the judgment device 30 to a manager's terminal device 40, such as a supervisor, for sharing. The manager can then provide labor management and rest guidance to the worker based on the estimation results.

[0097] Furthermore, in environments where the target person's terminal cannot be used, it is possible to use a smart mirror or other terminal installed in a shared space as a measurement terminal device 20 to estimate the target person's physical condition and labor productivity.

[0098] Furthermore, by using in-vehicle cameras and microphones, measurements can be taken inside the vehicle and used for assessing driving risks based on that data.

[0099] Furthermore, in the embodiment described above, the learning device 10 has learned the pre-trained model through supervised learning, but it may also learn the pre-trained model through reinforcement learning based on feedback from, for example, a doctor or psychologist.

[0100] Furthermore, the questions used in the dialogue between the conversational agent may be pre-prepared or generated by AI (artificial intelligence). For example, the generating AI may be instructed to generate questions that require metacognition regarding the user's state, and the questions obtained as answers may be used in the dialogue with the person being measured or the person being estimated. In addition, the information control unit 131 of the learning device 10 may acquire information on the questions used when the training data was obtained, in addition to the training data, and the learning control unit 133 may add this question information to the trained model. The control unit 27 of the measurement terminal device 20 may acquire questions attached to the trained model used as the judgment model from the judgment device 30, and control the conversational agent to present the acquired questions to the user. In addition, the control unit 27 of the measurement terminal device 20 may instruct the generating AI to generate questions that require metacognition regarding the user's state, and control the conversational agent to present the questions obtained as answers to the user.

[0101] As mentioned above, the advantage of estimating labor productivity-related information using features such as facial expressions obtained from video data and features obtained from audio data is that it can be implemented simply and objectively by the user. Conventional stress checks conducted annually by companies and other organizations have many questions, take a long time to answer, and because they are self-administered questionnaires, there is a risk that users may write false information and obtain results that do not reflect their actual mental health status. In this embodiment, users only need to respond to questions from the system in a short time, such as a few seconds to a dozen seconds, and since estimation is performed using objective information, the problems of conventional stress checks do not occur.

[0102] Figure 15 is a schematic diagram of an example hardware configuration of an information processing device 90 applied to this embodiment. The information processing device 90 comprises a processor 91, main memory 92, communication interface 93, auxiliary storage device 94, input / output interface 95, and internal bus 96. The processor 91, main memory 92, communication interface 93, auxiliary storage device 94, and input / output interface 95 are connected to each other via the internal bus 96 so as to be able to communicate with each other. The information processing device 90 may be applied to, for example, a learning device 10, a measurement terminal device 20, and a determination device 30. In this case, for example, the communication unit 11, communication unit 21, and communication unit 31 may be configured using the communication interface 93. For example, the storage unit 12, storage unit 28, and storage unit 32 may be configured using the auxiliary storage device 94. Also, the control unit 13, control unit 27, and control unit 33 may be configured using the processor 91 and main memory 92.

[0103] (modified version) In this embodiment, the determination device 30 and the administrator terminal device 40 are configured as separate devices, but they may be configured as a single device. Figure 16 shows a modified example of the determination device 30 configured in this way. The determination device 30 shown in Figure 16 includes an input unit 34 and an output unit 35 in addition to the configuration shown in Figure 8. The input unit 34 and output unit 35 of the determination device 30 shown in Figure 16 function similarly to the input unit 42 and output unit 43 of the administrator terminal device 40, respectively. The control unit 33 operates in response to operations on the input unit 34 and outputs information of the estimation result using the output unit 35.

[0104] In this embodiment, the learning device 10 and the determination device 30 are configured as separate devices, but they may be configured as a single integrated device. Figure 17 shows a modified example of the determination device 30 configured in this way. The storage unit 32 of the determination device 30 shown in Figure 17 also functions as a teacher data storage unit 326 and a pre-processed teacher data storage unit 327. The control unit 33 of the determination device 30 shown in Figure 17 also functions as a learning control unit 334. The determination model storage unit 321, the teacher data storage unit 326, and the pre-processed teacher data storage unit 327 function similarly to the learned model storage unit 123, the teacher data storage unit 121, and the pre-processed teacher data storage unit 122 of the learning device 10, respectively. The pre-processing control unit 332 and the learning control unit 334 function similarly to the pre-processing control unit 132 and the learning control unit 133 of the learning device 10.

[0105] In this embodiment, the measurement terminal device 20 and the determination device 30 are configured as separate devices, but they may be configured as an integrated device. Figure 18 shows a modified example of the determination device 30 configured in this way. The storage unit 32 of the determination device 30 shown in Figure 18 also functions as a measurement information storage unit 328. The measurement information storage unit 328 functions similarly to the storage unit 28 of the measurement terminal device 20. The control unit 33 of the determination device 30 shown in Figure 18 also functions as a measurement control unit 335. The measurement control unit 335 functions similarly to the control unit 27 of the measurement terminal device 20. The input unit 34, display unit 36, audio output unit 37, imaging unit 38, and sound collection unit 39 of the determination device 30 shown in Figure 18 function similarly to the input unit 22, display unit 23, audio output unit 24, imaging unit 25, and sound collection unit 26 of the measurement terminal device 20, respectively.

[0106] The learning device 10 may be implemented using multiple information processing devices. For example, the learning device 10 may be implemented using a cloud or other device. For example, in the learning device 10, the storage unit 12 and the control unit 13 may be implemented on different information processing devices. For example, the storage unit 12 of the learning device 10 may be distributed and implemented across multiple information processing devices. The determination device 30 may be implemented using multiple information processing devices. For example, the determination device 30 may be implemented using a cloud or other device. For example, in the determination device 30, the storage unit 32 and the control unit 33 may be implemented on different information processing devices. For example, the storage unit 32 of the determination device 30 may be distributed and implemented across multiple information processing devices.

[0107] The application running on the administrator terminal device 40 may not be intended to estimate information on labor productivity indicators itself, but may be an application that provides the user with information obtained by processing those estimation results. Such an application may be an application that processes using an API (Application Programming Interface) provided by the determination device 30, for example. In this case, the administrator terminal device 40 may output other information obtained by processing using the estimation result information, instead of information indicating the information itself (estimation result information) obtained from the determination device 30, via the output unit 43.

[0108] The aforementioned assessment system enables long-term, continuous evaluation of daily labor productivity indicators by evaluating the subject based on their responses during simple, short conversations. Specifically, by using a screen, microphone, and a camera capable of capturing the face, flexible measurement is possible without limitations on location or time. Furthermore, this embodiment can achieve highly accurate estimations by introducing labor productivity evaluation criteria established by experienced industrial physicians. In addition, this embodiment can promote understanding of the importance of mental health management and contribute to solving social issues.

[0109] According to the embodiments described above, the judgment system comprises a learning unit and a judgment unit. The learning unit may be provided in a learning device, and the judgment unit may be provided in a judgment device. The learning unit learns a judgment model that calculates labor productivity-related information indicating an indicator of the user's labor productivity, using one or more input information from among objective information obtained by measuring the user's biological characteristics when the user responds to a question, subjective information obtained through the user's subjective selection, attribute information representing the user's attributes, time-series change information of objective information, time-series change information of subjective information, and time-series change information of attribute information, either through supervised learning using training data including input information and correct labor productivity-related information, or through reinforcement learning. The judgment unit obtains an estimation result of the labor productivity-related information of the estimated person by inputting the input information obtained about the estimated person into the judgment model.

[0110] The input information may include at least objective information. The objective information may include at least biological information obtained by non-contact sensors.

[0111] Labor productivity-related information may include at least one of the following: presenteeism, absenteeism, and performance, from the user's presenteeism, absenteeism, health status, psychological state, performance (expressed as a multi-level value for labor productivity evaluation), engagement, work performance, sales performance, and personnel evaluation.

[0112] Objective information may include one or more of the following: the user's facial expressions, vocal features, posture, sleep duration, sleep quality, working hours, and workload. Subjective information may include one or more of the following: the user's emotional state and the content of their speech. Attribute information may include one or more of the following: the user's age, gender, personality traits, occupation, and annual income. Labor productivity-related information may include one or more of the following: the user's presenteeism, absenteeism, health status, psychological state, performance, engagement, work performance, sales performance, and personnel evaluation. Note that if emotional state is not included in subjective information, psychological state may be included in labor productivity-related information.

[0113] Objective and subjective user information may be obtained when the user responds to a questionnaire administered by a psychiatrist. Information from the action unit may be used as facial expression information. As speech features, the response time from the end of the question to the start of speech, intonation, volume, sound pressure level, acoustic spectrum, and speech duration may be used.

[0114] The users from whom training data is obtained and the target individuals for estimation may be different users, or they may be the same user.

[0115] The question may also require metacognition regarding the user's state.

[0116] While embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention. [Industrial applicability]

[0117] The embodiments described above can be used to develop mental health applications for smartphones, tablets, and computers, intended for use by companies and individuals. Furthermore, by introducing the judgment system of the embodiments described above into health checkups, it can be applied to the prevention, early detection, and intervention of mental illnesses. In addition to supporting the management of employee productivity, by introducing the judgment system of the embodiments described above into in-vehicle cameras to check the driver's emotions, it can be used to prevent serious negligence such as aggressive driving and drunk driving. [Explanation of symbols]

[0118] 100…Judgment system, 10…Learning device, 11…Communication unit, 12…Storage unit, 121…Teacher data storage unit, 122…Preprocessed teacher data storage unit, 123…Trained model storage unit, 13…Control unit, 131…Information control unit, 132…Preprocessing control unit, 133…Learning control unit, 20…Measurement terminal device, 21…Communication unit, 22…Input unit, 23…Display unit, 24…Audio output unit, 25…Imaging unit, 26…Sound collection unit, 27…Control unit, 28…Storage unit, 30…Judgment device, 31…Communication unit, 32…Storage unit, 321…Judgment model storage unit, 322…Attribute information storage unit, 323…Measurement information storage unit, 324…Preprocessed data storage unit, 325…History information storage unit, 326…Teacher data storage unit, 327…Pre-processed training data storage unit, 328…Measurement information storage unit, 33…Control unit, 331…Information control unit, 332…Pre-processing control unit, 333…Determination unit, 334…Learning control unit, 335…Measurement control unit, 34…Input unit, 35…Output unit, 36…Display unit, 37…Audio output unit, 38…Imaging unit, 39…Sound collection unit, 40…Administrator terminal device, 41…Communication unit, 42…Input unit, 43…Output unit, 44…Storage unit, 45…Control unit, 70…Network, 90…Information processing unit, 91…Processor, 92…Main memory, 93…Communication interface, 94…Auxiliary memory, 95…Input / output interface, 96…Internal bus

Claims

1. A determination unit obtains an estimation result of labor productivity-related information for a target person, by inputting the input information obtained for the target person, who is the user to be estimated, into a determination model that calculates labor productivity-related information indicating an indicator of the user's labor productivity, using one or more input information from among objective information obtained by measuring the user's biological characteristics when the user responds to a question, subjective information obtained through the user's subjective choices, attribute information representing the user's attributes, time-series change information of the objective information, time-series change information of the subjective information, and time-series change information of the attribute information. A determination device equipped with the following features.

2. The aforementioned input information includes at least the aforementioned objective information, The aforementioned objective information includes at least biological information obtained by non-contact sensors. The determination device according to claim 1.

3. The aforementioned labor productivity-related information includes, at least, presenteeism, absenteeism, and performance from the user's presenteeism, absenteeism, health status, psychological state, performance (expressed as a multi-level value for labor productivity evaluation), engagement, work performance, sales performance, and personnel evaluation. The determination device according to claim 1.

4. The aforementioned objective information includes one or more of the following: the user's facial expression, voice characteristics, posture, sleep duration, sleep quality, working hours, and workload. The subjective information includes one or more pieces of information from the user's emotional state and the content of their speech. The aforementioned attribute information includes one or more of the following: the user's age, gender, personality traits, occupation, and annual income. The aforementioned labor productivity-related information includes one or more of the following: the user's presenteeism, absenteeism, health status, psychological state, performance (expressed on a multi-level scale as an evaluation of labor productivity), engagement, work performance, sales performance, and personnel evaluation. If the subjective information does not include the emotional state, the labor productivity-related information may include the psychological state. The determination device according to claim 1.

5. The aforementioned question is a question that requires metacognition regarding the user's state. The determination device according to claim 1.

6. A learning unit learns a judgment model that calculates labor productivity-related information indicating an indicator of the user's labor productivity, using one or more input information from among objective information obtained by measuring the user's biological data when the user responds to a question, subjective information obtained through the user's subjective selection, attribute information representing the user's attributes, time-series change information of the objective information, time-series change information of the subjective information, and time-series change information of the attribute information, either through supervised learning using training data including input information and correct labor productivity-related information, or through reinforcement learning. A learning device equipped with the following features.

7. A learning unit learns a judgment model that calculates labor productivity-related information indicating an indicator of the user's labor productivity, using one or more input information from among objective information obtained by measuring the user's biological data when the user responds to a question, subjective information obtained through the user's subjective selection, attribute information representing the user's attributes, time-series change information of the objective information, time-series change information of the subjective information, and time-series change information of the attribute information, either through supervised learning using training data including input information and correct labor productivity-related information, or through reinforcement learning. The determination model includes a determination unit that obtains an estimation result of labor productivity-related information for the estimated person by inputting the input information obtained for the estimated person, who is the user to be estimated, A judgment system equipped with the following features.

8. The user from whom the training data was obtained and the target person for estimation are different users or the same user. The determination system according to claim 7.

9. Estimation step: To obtain an estimation result of labor productivity-related information for a target person, which is a user to be estimated, by inputting the input information obtained for the target person, which is a user to be estimated, into a determination model that calculates labor productivity-related information indicating an indicator of the user's labor productivity, using one or more input information from among objective information obtained by measuring the user's biological data when the user responds to a question, subjective information obtained through the user's subjective selection, attribute information representing the user's attributes, time-series change information of the objective information, time-series change information of the subjective information, and time-series change information of the attribute information. A method for determining the thumbnail.

10. A learning step involves learning a judgment model that calculates labor productivity-related information indicating an indicator of the user's labor productivity, using one or more input information from among objective information obtained by measuring the user's biological data when the user responds to a question, subjective information obtained through the user's subjective selection, attribute information representing the user's attributes, time-series change information of the objective information, time-series change information of the subjective information, and time-series change information of the attribute information, either through supervised learning using training data including input information and correct labor productivity-related information, or through reinforcement learning. A learning method that has [something].

11. Computers A computer program for causing the determination device described in claim 1 to function.

12. Computers A computer program for causing a learning device to function as described in claim 6.

Images