Information processing device, information processing method, and information processing program

Abstract

To provide an information processing device which determines and presents the type of a fatigue.SOLUTION: An information processing device 1 according to the present invention comprises: an acquisition unit which acquires information including biological information of a user; previously acquired biological information and a learned model which have learned the type of a mental fatigue; and display control means which causes a display unit to display the type of the mental fatigue determined based on the acquired information and the learned model.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Patent Document 1 discloses a mental and physical state estimation system that includes a heart rate information acquisition unit that acquires heart rate information, which is information related to the heart rate of a subject, a heart rate variability calculation unit that calculates the heart rate variability of very low frequency components (VLF) from the acquired heart rate information, and a mental and physical state estimation unit that estimates the subject's concentration and effort state from the calculated value of the heart rate variability. However, since fatigue is only perceived on a single scale, such as strong or weak, the subject is not convinced. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-175921 A Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide an information processing device that determines and displays the type of fatigue. [Means for solving the problem]

[0005] The information processing device of the present invention is characterized by an acquisition unit that acquires information including a user's biometric information, a trained model that has learned the previously acquired biometric information and the type of mental fatigue, and a display control means that displays on a display unit the type of mental fatigue determined based on the acquired information and the trained model. Effect of the Invention

[0006] It is possible to provide an information processing device that determines and displays the type of fatigue. [Brief description of the drawings]

[0007] [Figure 1] 1 is a diagram illustrating an example of an overall configuration of an information processing apparatus according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information analyzing device and a terminal device according to an embodiment. [Diagram 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a detector according to an embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a functional configuration of an information processing device according to an embodiment. [Figure 5A] FIG. 4 is an example of a sequence diagram illustrating each process of the first embodiment. [Figure 5B] FIG. 4 is an example of a sequence diagram illustrating each process of the first embodiment. [Figure 6] 6 is an example of a screen displayed on a display unit of the terminal device according to the embodiment. [Figure 7] FIG. 2 is a diagram showing a respiratory waveform according to an embodiment. [Figure 8] 1 is an example of a learning dataset used for estimation according to an embodiment. [Figure 9] FIG. 1 is a diagram showing the relationship between explanatory variables, intermediate characteristics, and objective variables according to an embodiment. [Figure 10] FIG. 11 is a diagram showing a relationship between fatigue types and respiration information according to the embodiment. [Figure 11] 1A to 1C are diagrams illustrating a seating determination method according to an embodiment. [Figure 12] 1A to 1C are diagrams illustrating a conversation determination method according to an embodiment. [Figure 13] 1 is an example of visualizing the concentration level on a time axis according to an embodiment. [Figure 14] 13 is an example of visualizing a fatigue level on a weekly basis according to an embodiment. [Figure 15] 13 is an example of visualization of remaining energy according to the embodiment. [Figure 16] 13 is an example of visualizing the fatigue level and the concentration level on a weekly basis according to an embodiment. [Figure 17]FIG. 11 is an example of a sequence diagram illustrating each process of the second embodiment. [Figure 18] 13 is an example of a schedule table according to the embodiment in which a concentration degree is superimposed and displayed. [Figure 19] FIG. 13 is an example of a sequence diagram illustrating each process of the third embodiment. [Figure 20] 13 is an example of a flowchart illustrating each process of the third embodiment. [Figure 21] 11 is a diagram illustrating an example of a change in concentration level depending on a task according to the embodiment; [Figure 22] 1 is an example of a table of tasks and lumped time constants according to an embodiment. [Figure 23] 13 is an example of a table of types of fatigue and corresponding recovery actions according to the embodiment. [Figure 24] 13 is an example of a display screen showing a recovery action on the terminal device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the description of the drawings, the same elements are given the same reference numerals, and duplicated description will be omitted.

[0009] <First embodiment> [Overall configuration of information processing device] Fig. 1 is a diagram showing an example of the overall configuration of an information processing device 1. As shown in Fig. 1, the information processing device 1 has various devices and terminals including an information analysis device 2, a terminal device 3, and a detector 4.

[0010] The communication network 100 is a communication network in which an unspecified number of communications are carried out, and is constructed by the Internet, an intranet, a LAN (Local Area Network), etc. The communication network 100 may include not only wired communication but also communication networks using wireless communication such as 4G (4th Generation), 5G (5th Generation), WiMAX (Worldwide Interoperability for Microwave Access), and LTE (Long Term Evolution).

[0011] The information analysis device 2 is responsible for information analysis and storage in the information processing device 1. It has an estimation means for estimating the user's fatigue state, concentration state, etc., a schedule optimization calculation means for calculating an optimal schedule for the user, and an action recommendation calculation unit for calculating action recommendations that encourage the user to take appropriate actions. The calculation results are stored in the storage unit. The information analysis device 2 may be a commonly used PC (Personal Computer) or a portable notebook PC.

[0012] The terminal device 3 is a communication terminal that is used by a user who uses the information processing device 1 and can acquire information related to the user. It is responsible for inputting information required for this system. The input means is input from a keyboard on a computer terminal or a signal from a sensor worn by the user. The terminal device 3 also serves as a means for providing information to the user, and is composed of a visualization means for outputting the results of the calculation unit and information accumulated in the storage unit, and displays the calculation results and accumulated information to the user on the user's terminal. The terminal device 3 is also realized by an information processing device (computer system) for communication equipped with a general OS, etc., and is one of the components that configure the information processing device 1.

[0013] The terminal device 3 may be a commonly used PC (Personal Computer), a portable notebook PC, a mobile phone, a smartphone, a tablet terminal, a communication terminal, etc. The terminal device 3 may further be a communication device or a communication terminal capable of operating browser software and various application software.

[0014] The detector 4 is a means for acquiring biometric information of the user, and continuously senses the biometric information of the user while the user is working. For example, a contact-type respiration sensor that measures the user's respiration signal is one example. The respiration sensor has a wireless communication function such as Bluetooth (registered trademark) for wireless communication with a smartphone or a PC used by the user while working. It constantly detects the user's respiration signal and transmits the detected respiration waveform to the smartphone or PC via wireless communication. [Hardware configuration] Next, the hardware configuration of the communication terminal or device constituting the information processing device 1 according to the embodiment will be described with reference to Figures 2 and 3. Note that components may be added or deleted from the hardware configuration of the communication terminal or device shown in Figures 2 and 3 as necessary.

[0015] <Hardware configuration of information analysis device 2 and terminal device 3> 2 is a diagram showing an example of the hardware configuration of the information analysis device 2 and the terminal device 3. The information analysis device 2 and the terminal device 3 are formed with roughly the same configuration, so the following description will focus on the terminal device 3, and the information analysis device 2 follows suit. As shown in FIG. 2, the information analysis device 2 and the terminal device 3 are constructed, for example, by a computer, and include a CPU 301 (201), a ROM 302 (202), a RAM 303 (203), a display 308 (208), and a network I / F 309 (209). In addition, they include a short-distance I / F 316 (216), a media 306 (206), a media I / F 307 (207), a keyboard 311 (211), a mouse 312 (212), a HD (hard disk) 304 (204), a HDD hard disk controller 305 (205), and a bus line 310 (210).

[0016] Of these, the CPU 301 (201) controls the overall operation of the information analyzing device 2 and the terminal device 3. The ROM 302 (202) stores programs and the like used to drive the CPU 301 (201). The RAM 303 (203) is used as a work area for the CPU 301 (201). The display 308 (208) displays various information such as a cursor, a menu, a window, characters, or an image. In this embodiment, the display 308 (208) functions as an example of a display means. The short-distance I / F 316 (216) is a communication circuit for performing data communication with a communication device or a communication terminal or the like having a wireless communication interface such as NFC (Near Field Communication), Bluetooth (registered trademark, omitted below), or Wi-Fi (registered trademark, omitted below).

[0017] Here, the HD 304 (204) stores various data such as programs. The HDD controller 305 (205) controls reading and writing of various data from and to the HD 304 (204) under the control of the CPU 301 (201). Here, the terminal device 3 and the information analyzing device 2 may have a hardware configuration equipped with an SSD (Solid State Drive), a CD-RW drive 314 (214), or a CD 315 (215) instead of the HD 304 (204) and the HDD controller 305 (205).

[0018] The network I / F 309 (209) is an interface for data communication using the communication network 100. The keyboard 311 (211) and the mouse 312 (212) are a type of input means for operating the information analyzing device 2 and the terminal device 3 by a user pressing, clicking, tapping, or the like on a predetermined button, icon, or the like arranged on the display 308 (208). The media I / F 307 (207) controls reading or writing (storing) data from or to a recording medium 306 (206) such as a flash memory. The bus line 310 (210) is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 301 (201).

[0019] The above program may be recorded in a computer-readable recording medium as an installable or executable file, or may be distributed by downloading via a network. Examples of the recording medium include a CD-R (Compact Disc Recordable), a DVD (Digital Versatile Disk), a Blu-ray Disc (Blu-ray is a registered trademark; omitted below), an SD card, and a USB memory. The recording medium may be provided domestically or internationally as a program product. For example, the information analysis device 2 realizes the information analysis method according to the present invention by executing the program according to the present invention.

[0020] <Hardware configuration of detector 4> Fig. 3 is a diagram showing an example of a hardware configuration of the detector 4. As shown in Fig. 3, the detector 4 is constructed by, for example, a microcomputer, and includes an MPU 401, a RAM 402, an I / F 404, a sensor 403, and a switch 405.

[0021] The sensor 403 is composed of an inertial measurement sensor unit, a microphone, a temperature detection sensor, etc. The inertial measurement sensor unit is a sensor that obtains three-dimensional inertial motion, and obtains translational motion and rotational motion in three orthogonal axial directions. An acceleration sensor and a gyro sensor are used. By combining with a geomagnetic sensor, the sensor attitude and orientation can be indicated as Euler angles.

[0022] The microphone is constructed using a MEMS microphone (Micro Electro Mechanical Systems: MEMS) or an electret condenser microphone, and can capture voice information, such as whether the user is talking, the user's breathing sounds, and the environmental sounds in which the user is placed.

[0023] The control unit of the MPU 401 is realized by an integrated circuit chip including a microcomputer, a logic device such as an FPGA (Field Programmable Gate Array), or a combination of an integrated circuit chip and a logic device. The control unit also processes data obtained by the sensor unit and issues commands for data communication to the system. A switch 405 allows external operations such as turning the power on and off and resetting the system.

[0024] The I / F (interface) 404 is an interface for communication between the terminal device 3 and the detector 4, and communicates data via wireless communication such as Wi-Fi or Bluetooth connection or wired connection. When the detector 4 is attached to a user and operates wirelessly, it may be provided on the user side as a wireless communication slave unit, and a wireless communication master unit may be provided separately in a wearable device such as a smartphone or smartwatch. Communication with the terminal device 3 is made possible via the smartphone or the like.

[0025] The detector 4, which is a means for acquiring user biometric information, is attached, for example, to the user's chest or waist. Although examples of attachment methods are not shown, the attachment may be anywhere as long as it can acquire body movements due to breathing, such as a method of attaching the detector to clothing, including a belt, by using a clip-like object to hook it, a method of attaching the sensor to a belt-like object and wrapping the belt around the chest and waist, or a method of hanging the detector from the neck with a neck strap. [Function block configuration]

[0026] 4 shows a functional block diagram of this embodiment, which is made up of an information analysis device 2, a terminal device 3, and a detector 4. Each functional block will be described in turn.

[0027] <Functional configuration of information analysis device 2> First, the functional configuration of the information analyzing device 2 will be described. As shown in Fig. 4, the information analyzing device 2 has a storage / reading unit 21, a display control unit 22, a setting unit 23, a determination unit 24, an acquisition unit 25, and a transmission / reception unit 26. Each of these functional units is a function or means realized by operating any of the hardware resources shown in Fig. 2 by an instruction from the CPU 201 according to a program for the information analyzing device 2 that is expanded from at least one of the ROM 202, the HD 204, and the recording medium 206 to the RAM 203. The information analyzing device 2 also has a storage unit 2000 constructed by at least one of the ROM 202, the HD 204, and the recording medium 206 shown in Fig. 2. Furthermore, a communication program (communication application), a browser application, etc. for communicating with the terminal device 3 via the communication network 100 are installed and stored in the storage unit 2000.

[0028] Next, a detailed description will be given of each functional configuration of the information analyzing device 2. The transmitting / receiving unit 26 of the information analyzing device 2 shown in Fig. 4 is mainly realized by the processing of the CPU 201 for the network I / F 209 and the short-range communication I / F 216, and transmits and receives various data (or information) to and from the terminal device 3 via the communication network 100.

[0029] The setting unit 23 is realized mainly by the processing of the CPU 201, and performs processing for setting setting information input by the user. In this embodiment, the setting unit 23 functions as an example of a setting means.

[0030] The determination unit 24 is realized mainly by the processing of the CPU 201, and performs various determination processes in the entire information analyzing device 2. In this embodiment, the determination unit 24 functions as an example of a determination means.

[0031] The acquisition unit 25 is mainly realized by the processing of the CPU 201, and acquires biometric information and the like obtained from the detection unit 4. The acquisition unit also acquires user information input from the keyboard 311, mouse 312, and the like of the terminal device 3. In this embodiment, the acquisition unit 25 functions as an example of an acquisition means.

[0032] The storage and reading unit 21 is mainly realized by processing of the CPU 201 on at least one of the ROM 202, the HD 204, and the recording medium 206, and stores various data (or information) in the storage unit 2000 and reads various data (or information) from the storage unit 2000. In this embodiment, the storage and reading unit 21 functions as an example of a storage and reading means.

[0033] Next, a description will be given of the database included in the information processing device 2. A set value information DB (database) 2001 accumulates threshold value information and the like that is set in advance when used by the estimation means of the calculation unit.

[0034] The input information DB 2002 accumulates user information obtained from an input unit and biometric information of the user obtained by a biometric information means. For example, the information includes information input by the user using the keyboard 311 of the terminal device 3, information selected by the user from information displayed on the display 308 by operating the mouse 312, and information obtained from another application used by the user using an API (Application Programming Interface). The user information here includes user context information, task information, attribute information, etc.

[0035] The calculation result DB 2004 accumulates the schedule optimization calculation results obtained from the calculation unit, and the results of estimating the user's fatigue state, concentration state, and the like.

[0036] The trained model DB2003 is constructed using a learning data set and a learning algorithm that indicates the relationship between the target biometric information and the corresponding fatigue type, fatigue level, and concentration level.

[0037] Task DB2005 records the time constant of fatigue specific to each task. The time constant of fatigue is an index that quantifies the rate of increase in fatigue, which differs depending on the task. A wide range of tasks, such as meetings, research work, programming work, and writing papers, and their time constants are recorded in a table here.

[0038] The recommendation information DB 2006 stores information required for recommending concentration actions and recovery actions based on the calculation results of the fatigue type and fatigue state. This information records appropriate recovery actions for each of the five fatigue types. In addition, multiple types of recovery actions are recorded so that the recovery action corresponds to the level of each fatigue type.

[0039] In the above information processing device, the functions of the calculation unit and the storage unit may be realized on a cloud server, where the cloud server is a server that provides resources for cloud computing.

[0040] <Functional configuration of terminal device 3> Next, a description will be given of the functional configuration of the terminal device 3. As shown in Fig. 4, the terminal device 3 has a communication unit 31, a transmission / reception unit 32, a storage / readout unit 33, a storage unit 34, an input unit 35, a reception unit 36, and a display control unit 37. Each of these functional units is a function or means realized by operating any of the hardware resources shown in Fig. 2 in response to an instruction from the CPU 301 in accordance with a program for the terminal device 3 that is loaded from at least one of the ROM 302, the HD 305, and the recording medium 306 into the RAM 303.

[0041] The terminal device 3 may be constructed by a single computer such as a commonly used PC (Personal Computer) or a portable notebook PC, or may be constructed by multiple computers to which each section (function or means) such as storage is divided and arbitrarily assigned. All or part of the functions of the terminal device 3 may be a server computer existing in a cloud environment or a server computer existing in an on-premise environment. The terminal device 3 may further be a communication device or a communication terminal capable of operating software such as browser software.

[0042] The display control unit 37 is mainly realized by the processing of the CPU 301 on the display 308 of the terminal device 3, and controls the display of various screens and information (data) on the terminal device 3. The display control unit 37 also uses, for example, a browser to display a display screen generated by HTML or the like on the display 308 of the terminal device 3. In this embodiment, the display control unit 37 functions as an example of a display control means.

[0043] <Functional configuration of detector 4> Next, a functional configuration of the detector 4 will be described. As shown in Fig. 4, the detector 4 has a transmitting / receiving unit 42, a processing unit 43, a detecting unit 44, a memory / reading unit 41, and a memory unit 45. Each of these functional units is a function or means realized by operating any of the hardware resources shown in Fig. 3 by an instruction from the MPU 401 according to the program for the detector 4 expanded in the RAM 402.

[0044] The detector 4 is a detection means for continuously sensing the user's biological information while the user is working. One example is a contact-type respiratory sensor that measures the user's respiratory signal. Here, the respiratory signal is a signal generated with breathing movements such as exhalation and inhalation. The movement of the lungs causes the chest to contract, and when the lungs take in oxygen from the air, the respiratory muscles such as the diaphragm and intercostal muscles near the abdomen move. Contact-type respiratory sensors include a band type that is attached to the user's chest or abdomen.

[0045] In addition to the movement of the chest and abdomen, there is a mask type that senses the temperature change around the mouth accompanying breathing. Also, by using a small measuring device such as an inertial measurement unit as a breathing sensor, the movement of the chest and abdomen accompanying breathing can be acquired with a small wearable device. In addition, it is possible to acquire breathing motion without contact by using radio waves. The I / F 404 of the detector 4 has a wireless communication function such as Bluetooth (registered trademark) for wireless communication with a smartphone or a PC used by the user during work. It constantly detects the user's breathing signal and transmits the detected breathing waveform or extracted feature information to the smartphone or terminal device 3 by wireless communication as appropriate.

[0046] <Sequence> Next, the overall flow of this system will be described with reference to the sequence diagrams shown in Figures 5A and 5B. The sequence is composed of an information analysis device 2, a terminal device 3, and a detector 4. First, a user inputs user information into the terminal device 3 using a keyboard (step S11).

[0047] User information here refers to the acquisition of user context information, task information, and attribute information. Context information is information that indicates the situation surrounding the user, such as the current office location, time, and work environment (temperature, humidity, air pressure, and illuminance). Task information is information contained in the user's scheduler.

[0048] Specifically, this information is the user's own schedule for the day or week, information about work to be done, and information about work that has been done. Information about work that has been done includes the degree of task completion, the actual time spent working on the task, and the name of the task that was actually performed. It also includes time information such as the user's planned start time, planned finish time, actual start time, and actual finish time. Attribute information is information such as gender and age. It also includes schedule information of others related to the work the user is engaged in.

[0049] As a result, the input information is displayed as a user setting value (step S12). The input user information is transmitted to the information analyzing device 2 (step S13). The information analyzing device 2 records the received user information in the input information DB 2002.

[0050] The detector 4 is placed near the chest or abdomen of the user, and starts measurement (step S14). Information detected by the detector 4 is processed by a processing unit 43 carried out by the MPU 401 inside the detector. Pre-processing such as noise removal and feature extraction, which will be described later, are performed (step S15). The setting information input by the user includes the timing and frequency at which the detector 4 transmits the feature, and the timing is measured by a timer in the MPU 401 of the detector 4. The extracted feature is appropriately transmitted to the terminal device 3 via the transmitting / receiving unit 42 (step S16). The feature received by the terminal device 3 is conveniently transmitted to the information analysis device 2 (step S17).

[0051] The information analysis device 2 receives the feature amount transmitted from the detector 4. The information analysis device 2 estimates the type of fatigue from the received feature amount and information obtained from the trained model DB2003 and the user input information DB2002 (step S18). The information analysis device 2 also estimates the degree of fatigue (step S19) and the degree of concentration (step S20), and records the obtained calculation results in the calculation result DB2004.

[0052] A continuation of Fig. 5A is shown in Fig. 5B. When a user wishes to display a measurement record, a display request is made from the terminal device 3 (step S21). The display request made by the user includes display setting values ​​such as the type of display and information on the label to be displayed, and the information is transmitted to the information analyzing device 2 (step S22).

[0053] The information analyzing device 2 calculates the information in the calculation result DB 2004 in accordance with the user's request and generates a display. At that time, it calculates the concentration score history (step S23) and the fatigue score history (step S24). Depending on the display content requested, it also calculates the remaining energy (step S25) and the label value (step S26). It performs a graph generation process of the calculated histories (step S27). It generates a display screen in which the generated graph is appropriately laid out (step S28).

[0054] These requested display screens are transmitted to the terminal device (step S29), and the terminal device 3 displays the received display screens (step S30).

[0055] [Estimation method] Next, a method for estimating the type of fatigue, the degree of fatigue, and the degree of concentration will be described. First, an example of the results estimated and visualized in this embodiment is shown in FIG. 6. There are three components: a concentration score 601, a fatigue score 602, and a radar chart 603 expressing the type of fatigue. The concentration score 601 and the fatigue score 602 are set to maximum values ​​such as 100 to 10, and the level of the user's state is expressed relatively and quantitatively with respect to the maximum value. The laser chart 603 of the type of fatigue indicates five types: "sleepiness," "discomfort," "unstable," "sluggishness," and "blurredness." Each type is expressed in five stages, and by showing them in a radar chart, it is easy to understand at a glance which items are relatively high. Each will be described in detail below.

[0056] <Expression of fatigue type> Next, a method of expressing the type of fatigue will be described. This function is suitable for the information processing device 2, but may be performed in the terminal device 3 or the detector 4. The type of fatigue is classified and estimated using information on the feature amount and a trained model provided in the memory unit (trained model DB2003). In this embodiment, the fatigue types refer to five types: "sleepiness," "discomfort," "unstable," "tiredness," and "blurred feeling." The fatigue types indicate how the user subjectively feels about each of these five types of fatigue. In addition, each of these five types of fatigue has its own degree, and by expressing each type as a numerical index, it is possible to accurately express the user's current state.

[0057] It was found that by dividing the fatigue types into five types, the user's state can be accurately expressed, and the user can feel a high degree of satisfaction with this index. A high degree of satisfaction by the user means nothing other than an increase in consistency with the user's subjective evaluation. With one or two indexes up to now, the consistency with the subjective evaluation was low and sufficient results could not be obtained. This was obtained from academic psychology findings. In psychological research, there are countless options for how to express the five types of fatigue, and whether to limit the types to five or four, but narrowing them down to these five improves the sense of satisfaction. The feature of this embodiment is that fatigue is divided into five types and the consistency with the subjective evaluation is increased.

[0058] These five types of fatigue are evaluated using five indices, as explained below. Specifically, the sleepiness scale is evaluated using five items: feeling sleepy, wanting to lie down, yawning, lack of motivation, and general fatigue.

[0059] The discomfort scale is rated using five items: headache, heavy head, feeling unwell, foggy head, and dizziness.

[0060] The insecurity scale is assessed using five items: feeling anxious, feeling depressed, feeling restless, being irritable, and having trouble organizing my thoughts.

[0061] The scale of fatigue is assessed using five items: fatigue in the arms, pain in the lower back, pain in the hands and fingers, fatigue in the legs, and stiff shoulders.

[0062] The scale of blurriness is assessed using five items: watery eyes, tired eyes, sore eyes, dry eyes, and blurry eyes.

[0063] The five fatigue types are unevenly distributed depending on the user's work style and personality. By repeatedly accumulating fatigue type classification estimation results in the memory unit and calculating the occurrence frequency of the estimation results within a certain period of time, it is possible to know the results specific to the user, improving the sense of satisfaction and estimation accuracy. It is also possible to calculate the contribution rate of fatigue types and identify the type of fatigue with the highest frequency.

[0064] These five types of fatigue change depending on the user's condition. Correlating the user's work content with the type of fatigue can help improve the work. It is also valuable to inform the user at the optimal time for the user. It is also possible to inform the user based on a threshold value previously set in the input information DB 2002.

[0065] <Expression of fatigue and concentration> Fatigue level and concentration level are quantitatively evaluated. Fatigue level is a comprehensive indicator of the fatigue types shown above, and is very easy for users to use. In addition, quantifying concentration level also makes it easier to understand the influence of the environment and work content. In addition, when calculating the remaining energy level, which will be described later, the five types of fatigue are also used to fine-tune, but it is the indicators of fatigue level and concentration level that contribute most.

[0066] <Feature extraction method> The detector 4 extracts features from detected information such as the user's breathing, and quantifies each feature item. Breathing information will be described as an example of biometric information. Features extracted by the detector 4 are not limited to breathing, but include various other features such as body temperature and pulse rate. Features in the case of breathing are obtained from a breathing waveform, and include, but are not limited to, breathing rate, breathing intensity, breathing waveform shape, and breathing variation. These features are appropriately acquired, and the acquired features are stored in the storage unit 45. This is repeatedly performed at a set timing and frequency.

[0067] An example of a respiratory waveform is shown in Figure 7. The respiratory waveform is a waveform that shows the time series change of the respiratory signal. The detector 4 has a built-in acceleration sensor, and it is possible to generate a respiratory waveform from the fluctuation of the chest. The respiratory waveform is a periodic waveform, and can be roughly divided into three periods: an exhalation period, an inhalation period, and a period during which breathing is stopped.

[0068] There are many features related to breathing. Some of them are explained below. First, one of the features is the time required for breathing. The time required for one breath is expressed as the sum of the time for exhalation, the time for inhalation, and the time for which breathing is stopped.

[0069] The respiration rate indicates the interval between the peaks of the respiration waveform, and the shorter the interval, the faster the respiration rate. The respiration strength indicates the height of the peaks of the respiration waveform, and the higher the peak, the stronger the respiration strength. The arrow 701 in Figure 7 indicates the width of the waveform, and indicates the respiration strength.

[0070] The shape of the respiratory waveform indicates whether the respiratory state is inhalation-dominant or exhalation-dominant. This balance between exhalation and inhalation is called the exhalation ratio, and is also useful as a feature. The exhalation ratio can be quantified by integrating the amount of variation from the average respiratory waveform on the exhalation side and the inhalation side separately, and looking at the balance.

[0071] Respiratory variation refers to the temporal variation that occurs in the peak intervals of the respiratory waveform. Features obtained from respiratory variation can be obtained by performing time domain analysis and frequency domain analysis on the respiratory variation information. For example, time domain analysis includes statistical analysis and geometric analysis such as Poincaré plot. In addition, frequency domain analysis may use the fast Fourier transform (FFT) method or maximum entropy method (MEM method).

[0072] The above items are examples of the feature of breathing, and are not limited thereto. For example, the ratio between the arrow 704 and the arrow 703 in FIG. 7 can also be used as a feature of the breathing waveform. The processing unit 43 calculates these feature amounts from the breathing waveform accumulated for a set period, and stores the numerical values ​​in the storage unit 45. The transmission / reception unit 42 transmits the time series information of the recorded feature amounts to the terminal device 3 and the information analysis device 2 at set time intervals. In this embodiment, the MPU 401 of the detector 4 performs the calculation and stores it in the RAM 402, but the breathing waveform may be transmitted and the feature amount may be extracted using the CPU 301 of the terminal device 3 or the CPU 201 of the information analysis device 2.

[0073] <Estimation method> Next, a method for estimating the type of fatigue, the level of fatigue, and the level of concentration will be described. FIG. 8 is an example of a data set used to create an estimation model. Sample numbers 1 to 3 are shown as individual rows. The first column lists the sample number 801. The next three columns list the feature amounts of biological information such as the respiratory information described above as explanatory variables 802. The estimation model is an example of information stored in the learning model 2003.

[0074] The biological information contained in the learning dataset includes, for example, respiratory information regarding a person's breathing in response to changes in fatigue state or concentration state, and refers to the respiratory features described above.

[0075] The seven columns from the fourth column to the tenth column are five columns of fatigue types, which are the objective variable 803, and two columns of fatigue level and concentration level. One example of fatigue level and concentration level included in the trained model is information indicating the concentration level and feeling of fatigue, which are subjective indices of the user obtained through a questionnaire or the like. Information on the subjective indices is collected using the Numerical Rating Scale (NRS) or a Likert scale.

[0076] NRS is an evaluation method used in clinical settings for the evaluation of "pain," etc. Specifically, it asks, "How concentrated are you now, with 10 being the most concentrated experience you have ever had?" If you cannot concentrate at all, 0 is assigned, 1-3 is low concentration, 4-6 is medium concentration, and 7-10 is high concentration.

[0077] The information processing device 1 according to the present embodiment is characterized by a trained model 2003 that has learned biometric information and a subjective index of mental fatigue or concentration based on a subjective index of mental fatigue or concentration input in advance, an acquisition unit 25 that acquires information including the biometric information of the user, and a display control means 37 that causes the display unit 308 to display the acquired and input information and the subjective index of mental fatigue or concentration determined based on the trained model 2003. The subjective index is the objective variable 803 shown in FIG. 8. The objective variable 803 is a quantification of "sleepiness," "discomfort," "unstable," "tiredness," "blurred," "fatigue," and "concentration" as subjective indexes of the user. As a result, the mental fatigue index and concentration index based on the user's subjectivity are estimated and output, so that the user is more satisfied, and the discomfort of the output result is reduced. By visualizing the concentration level, it is possible to take actions to increase the concentration level. By reducing the discomfort, the stress of use is reduced and it is possible to continue using the device.

[0078] An estimation model was created using the learning dataset created in this way. Although Figure 8 shows three samples, in reality, a larger number of samples are used. Also, although Figure 8 shows three columns of feature amounts, more feature amounts may be used. Having a large amount of information may increase the estimation accuracy, but conversely, it may become noise and decrease the estimation accuracy. It is important to select an appropriate number and appropriate items.

[0079] The learning algorithm may use statistical causal inference. Alternatively, known algorithms such as supervised learning, unsupervised learning, and reinforcement learning may be used. Statistical causal inference is a technique for estimating causal structures and causal parameters between explanatory variables and target variables from data. Supervised learning is a method for learning the relationship between input data and output data by providing a combination of input data and corresponding correct output data as teaching data.

[0080] The machine learning methods are shown in Figure 9. Each method is labeled Pattern 1 to Pattern 4. Each method has a different way of selecting the objective variable and explanatory variables.

[0081] Unsupervised learning is a method of learning the characteristics of input data without being given supervised data. Pattern 1 is shown as a representative example. In pattern 1, fatigue level and concentration level are set as objective variables, and the type of fatigue is set as the intermediate characteristic of fatigue level, making it possible to grasp the type of fatigue that is the main cause of fatigue level and recommend effective recovery actions. In addition, as in pattern 2, fatigue level and concentration level are set as objective variables, and the type of fatigue is set as the intermediate characteristic, making it possible to grasp the cause of concentration inhibition. In addition, as in pattern 3, fatigue level may be set as the intermediate characteristic when the objective variable is concentration level. In addition, fatigue level, concentration level, and type of fatigue may be set as objective variables as in pattern 4.

[0082] Fig. 10 shows the contribution rate of respiratory features to fatigue type estimation. The horizontal axis shows the contribution rate of five fatigue types to the respiratory waveform features of over 40. Fatigue types refer to the five types mentioned above: "sleepiness," "discomfort," "unstable," "tiredness," and "blurred vision." As shown in this figure, there are features that have a high correlation between fatigue types and respiratory information that represents a person's breathing.

[0083] For example, the fourth item from the left shows a relatively large contribution rate, and in particular, the contribution rate to "tiredness" is the largest. Such correlations are known, and many studies have been conducted. For example, as the feeling of drowsiness increases, breathing disturbance increases and breathing rate decreases. In contrast, discomfort is a feature that shows the most significant correlation with breathing disturbance. Different features have high correlations for different types of fatigue. Due to this difference, the contribution rate of the features differs for each type of fatigue. Taking this into consideration, an estimation model is constructed. By utilizing this difference, it is possible to estimate the type of fatigue from the features obtained from biometric information. The estimation model is constructed using the machine learning described above, and for items with multiple features with high contribution rates such as those shown here, estimation can be performed with a high probability.

[0084] The fatigue intensity is estimated using the repeatedly obtained feature analysis results and the trained model 2003 provided in the storage unit. For example, the trained model 2003 is a machine-learned model that has previously learned changes in the feature amounts of biological information according to changes in fatigue scores. The repeatedly obtained estimation results are accumulated in the storage unit, and the average value of the estimation results within a certain period is calculated to estimate the fatigue level of two or more classes. In addition, the concentration level of two or more classes is estimated using a similar method.

[0085] Next, a method for measuring a high-quality respiration waveform will be described. Figure 11 is a diagram showing seating determination. It was found that a lot of noise is introduced into the respiration waveform when the user's body movements are large or the user is away from the seat. Figure 11 shows how the body movements of the user are detected according to the output strength of an acceleration sensor included in the inertial measurement unit, and the state in which the user is seated and working can be estimated without decreasing the estimation accuracy of the user's fatigue state, concentration state, etc.

[0086] A threshold is set for the output signal level of the acceleration sensor, and if the signal strength continues to exceed the threshold, it is determined that the body movement is large. The respiratory waveform signal is used as the effective measurement time only when it is determined that the person is working. In this way, by determining whether the person is working while seated and at rest and measuring only those parts, the respiratory waveform signal can be obtained correctly, and the estimation accuracy can be prevented from decreasing.

[0087] Fig. 12 is a diagram showing a conversation determination method. It was found that noise is introduced into the respiration waveform when the user is talking. The purpose is to detect the user's conversation state according to the output strength of the microphone's voice signal, estimate the state in which the user is talking, and not to reduce the estimation accuracy of the user's fatigue state, concentration state, etc.

[0088] A threshold is set for the output signal level of the microphone's voice signal, and it is determined that the person is talking when the signal strength continues to exceed the threshold. The respiratory waveform signal is used as the effective measurement time only when it is determined that the person is not talking. In this way, by determining whether the person is working quietly without talking to others and measuring only that part, the respiratory waveform signal can be obtained correctly, and the estimation accuracy can be prevented from decreasing.

[0089] [Visualization method] <Visualization of concentration level, fatigue level, and type of fatigue> Next, the contents displayed for visualization will be described. Fig. 6 shows an example of visualization. Concentration level 601, fatigue level 602, and a laser chart 603 of fatigue type are displayed on a computer display screen 600. The fatigue type may be a laser chart as shown in the figure, or a visually appealing method such as color may be selected. The advantage of the radar chart is that it displays the results of fatigue type against fatigue level, and allows the balance and characteristics to be seen in the composition ratio while grasping the magnitude of the amount.

[0090] As shown in Fig. 6, the estimated results of fatigue and concentration obtained by the calculation unit may be quantified and displayed as relative scores. By displaying the scores so that they can be understood at a glance, the user can grasp his / her own condition in real time. In addition, by displaying up to five fatigue factors, the discrepancy with the user's own sense is reduced, and the user's satisfaction with the visualization results is increased.

[0091] The information processing device 1 according to this embodiment is characterized by an acquisition unit 25 that acquires information including biometric information of the user (step S14), a trained model 2003 that has previously learned the biometric information and the type of mental fatigue, and a display control means 37 that causes the display unit 308 to display (step S30) the type of mental fatigue that is determined based on the input information and the trained model 2003 (step S18). The displayed screen shows the types of fatigue in a radar chart as shown in FIG. 6. This reduces the sense of incongruity of the output result, since the type of mental fatigue of the user is estimated and output. By visualizing the type of fatigue, it is possible to take action to promote recovery.

[0092] As shown in Figure 13, the estimated results of fatigue and concentration obtained by the calculation unit may be graphed in chronological order and displayed on the output unit. By displaying the results with time on the horizontal axis, the user can grasp at a glance the changes in their condition throughout the day. Linking the changes throughout the day to their own behavior provides an opportunity for the user to reflect on their own work style.

[0093] As shown in Fig. 14, the daily average scores of fatigue and concentration obtained by the calculation unit may be displayed in a bar graph. By displaying the scores before and after the start of work in a week in a list, the user himself can understand the changes in his condition over the week. Since the user himself can grasp the trend of changes in his condition over the week at a glance, it provides an opportunity for the user to objectively match the trends of changes over the week with his own behavior and to subjectively reflect on his own work style.

[0094] As shown in Fig. 15, the estimated value of the fatigue level may be converted into an index called remaining energy level and displayed in order to make it easier for the user to understand. The remaining energy level is a concept created in this embodiment, and is known to have a strong appeal. The remaining energy level is an expression that represents the finite energy that a person possesses, imitating the expression used for mobile phone batteries, etc. The expression and calculation method do not have to be exactly as described, as long as there is a method that is easier for the user to understand.

[0095] The estimated level of fatigue due to work, meetings, etc. increases over time, and by utilizing this, the remaining energy level is calculated as the reciprocal of the estimated level of fatigue. The remaining energy level is calculated based on a reference value of 90% of the highest estimated level of fatigue in a reference period, and 30% of the lowest estimated level. The reference period may be the first week of using the information processing device 1, or the past week to month.

[0096] The standard value may be determined based on the user's estimated value, or may be a standard value prepared by the system administrator. When the user becomes fatigued, the estimated fatigue level increases and the remaining energy level decreases, and when fatigue is recovered by a recovery action, the estimated fatigue level decreases and the remaining energy level increases. By displaying the user's remaining energy level as a percentage in this way, the user can link their actions to fatigue at a glance.

[0097] It has been found that if the remaining energy is used up to the point of 0%, recovery becomes significantly worse. For this reason, it is advisable to leave a margin of, say, 20% before going on a break, such as the weekend. However, it is difficult for a user to determine how much energy they need based on their own subjective feelings alone. Even if they feel subjectively, they may not always be able to make an appropriate decision at the right time. It is important that an external information processing device judges what the user will likely be feeling subjectively, and provides a message based on that quantitative judgment. This allows the user to reaffirm their subjective feelings and take appropriate action.

[0098] The information processing device 1 according to this embodiment is characterized in that it estimates the user's remaining energy level based on the subjective index of mental fatigue and the type of mental fatigue (step S25). This makes it easier to take the next action, such as taking a break or concentrating on one's work, by visualizing the remaining energy level. By looking back from multiple angles, such as comparing the state at the start of work for the day with the state at the start of work every day, it is easy to find the cause of one's own illness, the cause of fatigue, and one's own unique tendencies. This makes it easier to find recovery actions that are suitable for oneself, and allows one to maintain health and performance. In addition, one can optimize the pace of work for one week by oneself.

[0099] Figure 16 shows an example of fatigue and concentration displayed on a weekly basis. The average score for each day of the week is displayed. This is an example calculated from accumulated data over several months, but it can also be a result over a long period such as a year. The average score for daily concentration and fatigue and the ratio of work performance can be displayed in a graph at once. The user can grasp the trend of changes in work performance and his / her own condition at a glance, and the trend of changes over the week and his / her own behavior are linked to give the user an opportunity to reflect on his / her own work style. At this time, one of the types of fatigue can be selected as the label A ratio to see the time series changes of that type of fatigue. The label is the five types of fatigue shown above, for example, "drowsiness". When "drowsiness" is displayed on a weekly basis, it can show a clear periodicity. For example, the difference between the beginning of the week and the weekend. This information can be used by the user to take measures and manage their schedule.

[0100] <Second embodiment>

[0101] Next, a second embodiment will be described. The feature of this embodiment is that the individual tendency of the user's concentration level (fatigue level, fatigue type) is superimposed on the schedule table, allowing efficient and optimal planning. The schedule table is an example of schedule information. As shown in FIG. 16, this utilizes the fact that long-term data accumulation allows cyclical fluctuations on a weekly basis to appear as the user's tendency and be visualized. The user's tendency can also be said to be the unique personality of the user. FIG. 17 is a sequence diagram of this embodiment that supports schedule optimization.

[0102] Here, we will mainly explain the level of concentration, but it is also worth superimposing the level of fatigue and type of fatigue. In particular, the type of fatigue is likely to show the individuality of the user, so schedule management tailored to each individual user is effective. For example, a "feeling of fatigue" is influenced by blood pressure fluctuations due to the body clock, and is higher in the morning for people with anemia. Also, a "feeling of sleepiness" is higher after a meal for big eaters, which can lead to a decrease in postprandial blood sugar levels and cerebral blood flow, which are influenced by the amount of insulin. Such reactions may depend on the user's physical characteristics and eating habits.

[0103] First, the user inputs a schedule, etc. This includes information that the user directly inputs into the input unit (such as planned start time, planned finish time, and desired working hours), information that the user inputs using a keyboard on a PC screen, information that the user selects from displayed information using a mouse, and information obtained from another application used by the user using an API (Application Programming Interface) (such as schedule information from a calendar application) (step S101). This input information is transmitted to the information processing device 2 and recorded in the input information DB 2002 (step S102).

[0104] The user starts inputting biometric information (step S103). The input biometric information is analyzed for features (step S104) and appropriately transmitted to the terminal device 3 (step S105). This information is simultaneously transmitted to the information processing device 2 (step S106).

[0105] The information in the trained model DB2003 and the previously transmitted feature amount are analyzed to estimate the fatigue level, concentration level, and type of fatigue (step S107). The estimation results are appropriately recorded in the calculation result DB2004.

[0106] A display request is made by the user from the terminal device 3 (step S108). This display request includes a request for superimposed display of the schedule table and the concentration level (fatigue level, fatigue type). In addition, an optimal schedule proposal may be requested. Information including the type of this request is transmitted to the information processing device 2 (step S109).

[0107] The information processing device 2 that has received the request reads out past history from the calculation result DB 2004, and calculates the performance from the past history (step S110). This calculation produces a result that reflects the user's tendencies, etc. A display that visualizes the user's tendencies is generated. A display screen is generated that displays a diagram showing the concentration level (fatigue level, fatigue type) trend superimposed on the schedule (step S111). Although the concentration level is mainly explained here, the user's tendencies may also be expressed by the fatigue level or type of fatigue. In particular, the type of fatigue has the advantage that it is easy to reflect the user's individuality and increases the sense of satisfaction.

[0108] If requested, an optimal schedule is proposed (step S112). For example, if there is a tendency for concentration levels (fatigue levels and types) to be higher in the mornings of the first half of the week, a task that is difficult and produces results during that time period is proposed. The generated display screen is transmitted to the terminal device 3 and displayed by the display unit 308 of the terminal device 3 (step S114).

[0109] <How to overlay schedule and concentration level (fatigue level / type of fatigue)> Next, a method for superimposing and displaying a schedule and a concentration level (fatigue level and type of fatigue) will be described. A schedule table is created using input information (step S101) entered by the user. Tasks and their time periods are input as input information. Tasks include time information such as free time, meeting time, working time, break time, starting time, and finishing time.

[0110] Free time is time during which a user can freely assign their own work content. Meeting time is time set regardless of whether it is online or offline, or time for meetings or listening set by others. Work time is time during which a user performs work. Break time is time during which a user can eat, take a break, and refresh themselves. Planned work content is a task set by the user themselves, such as creating materials or programming work. It is desirable to manage each time in 15-minute intervals, but it is also possible to set it in 1-minute increments.

[0111] Next, the history of concentration level (fatigue level, fatigue type) is calculated. The history of concentration level (fatigue level, fatigue type) can be calculated over various time axes, as shown in Figures 13, 14, and 15. From this concentration level history, the user's unique tendencies can be seen. This user's unique tendency of concentration level is displayed on the schedule.

[0112] FIG. 18 is an example of a schedule table 710 with a concentration level 712 superimposed thereon. In the schedule table, tasks such as the meeting 713 described above are displayed according to the input time. The form shown in FIG. 18 is a one-week schedule table. By displaying the concentration level (fatigue level, fatigue type) as a score in this form, one week can be viewed at a glance. In particular, by including a holiday, the tendency of the concentration level (fatigue level, fatigue type) before and after the holiday reflects the inherent tendency of the user. The concentration level score 711 is expressed in three stages. The three stages may be expressed in a visually easy-to-recognize manner such as by color. It may also be expressed as a numerical value to make it easy to understand. In this way, when the user displays the output result on his / her own terminal, the tendency of the change in the state of the concentration level (fatigue level, fatigue type) linked to the actual performance can be seen at a glance. The display unit may be one hour or 15 minutes.

[0113] The schedule allows users to check their free time. Within that free time, it is possible to visualize and clearly understand time periods when concentration levels (fatigue levels, fatigue types) tend to be high. This allows users to efficiently consider schedules such as what time periods to handle newly received tasks. Also, because the tendency for concentration levels (fatigue levels, fatigue types) is known, it becomes possible to allocate appropriate tasks to appropriate time periods.

[0114] <How to propose a schedule> In this embodiment, when a new task is scheduled, it is possible to propose an appropriate time period and an appropriate schedule. Next, a method for proposing a schedule will be described.

[0115] In the user information input step (step S101) in the sequence diagram of FIG. 17, past results when processing a task are input.

[0116] The user's performance refers to meetings, breaks, work, and their contents that proceeded as scheduled, and meetings, breaks, work, and their contents that did not proceed as scheduled. Using the information from the input section, the time that proceeded as scheduled and the time that did not proceed as scheduled are calculated.

[0117] As performance information, the schedule for the past day or past week is listed, and the user inputs performance information for each schedule information. The performance information is classified into four types of information (labels A, B, C, and D) and information on implementation items such as the degree of achievement.

[0118] The user also inputs whether the schedule was completed and the level of achievement as performance information. If the contents of the schedule information are completed on time, label A is assigned. If the contents of the schedule information are completed beyond the scheduled end time, label B is assigned. If the contents of the schedule information cannot be completed, label C is assigned. If a different schedule was carried out, label D is assigned. If label D is assigned, information on the reason for incompletion and the actual items carried out are assigned as performance information and stored in the input information DB2002 of the memory unit.

[0119] In addition, to show the level of execution, a numerical indicator is recorded as the achievement level. Labels A and B are recorded as 100% achievement. Label C is recorded as achievement. Label D is recorded as 0% achievement.

[0120] The results for "user schedule" are calculated for each label. For example, the following processing is performed for each piece of result information with labels A, B, C, and D. Working hours, meeting hours, and break times are calculated from the result information between the start time and end time of work, and these are stored in the memory unit. The total working hours, meeting hours, and break times for one week, and the total working hours for each day are calculated, and these are stored in the input information DB2002 in the memory unit. This allows the user to easily understand how they are spending their time.

[0121] Next, the user's performance is analyzed using the input information (step S101). The difference in performance for the "user's schedule" is calculated for each label and for each schedule. The difference between the performance time and the scheduled time between the start time and the end time of work is calculated and stored in the input information DB 2002 in the storage unit. The change in the difference time with the schedule for one week is stored in the storage unit. This allows the user to easily understand whether he is good or bad at each schedule.

[0122] Next, a schedule optimization calculation is performed. An optimized schedule is calculated using the planned information, the actual information, and the information input by the user. Work time and break time may be automatically allocated to free time in the user's plan so that the desired work time set by the user is exceeded.

[0123] Also, the calculation process may include a method of allocating work time by utilizing the performance information stored in the storage unit. For example, if a task equivalent to a task that did not proceed as planned by the user is registered in the schedule, the time required for the task can be increased and allocated based on the performance information stored in the input information DB2002. The schedule calculation results are stored in the calculation result DB2004 in the storage unit, and a schedule is proposed to the user on the display screen shown in FIG. 18. At this time, the schedule proposal may be made taking into consideration the time period when the concentration level (fatigue level, fatigue type) is high and the contents of the task. At this time, the time constant determined by the task, which will be described later, is taken into account to calculate the possibility of performing the task, and the schedule is proposed for a time period with appropriate possibility of performing.

[0124] The user can approve the proposed schedule or may modify the proposed schedule himself.

[0125] A modified example of a schedule table using the functions of another application (an application having an existing calendar function) will be described. Information obtained from another application used by the user using an API has a wide range of useful values. For example, there are other applications that have a function that allows easy setting of "focus time" by automatic setting or repeat setting. The focus time entered in this application may have a function that allows adjustment and suggestion of the start and end times and length of the focus time according to the concentration level (fatigue level and type of fatigue) 712 obtained by this information processing device. As a result, the schedule table easily inputted in another application can be fine-tuned and optimized by this information processing device, thereby improving the user's work efficiency.

[0126] Also, for example, the conference time 713 and the like may be fine-tuned by the conference planner, such as the date and time and the length of the conference, based on information on the concentration level 712 of the participating users. In the user information input (step S101), the users participating in the conference are identified, and data from the calculation result DB 2004 of those users is collected. By performing performance calculations (step S110) for those multiple users, the optimum conditions for the greatest number of users are calculated. A schedule proposal (step S112) that satisfies these conditions is made. At this time, the users participating in the conference may be identified, and information on the schedules of the users participating in the conference may also be obtained from other applications via an API. Also, the results of the schedule proposal (step S112) may be provided to users in a push-type manner based on the other application via an API.

[0127] It may also be linked to a notification function in the function of the second other application. For example, it is effective to change the schedule according to the concentration level (fatigue level, fatigue type) at the timing of the notification. It may also have a function to automatically adjust schedule information according to the high, medium, or low score of the concentration level (fatigue level, fatigue type) at this timing.

[0128] The information processing device 1 according to the present embodiment is characterized in that the acquisition unit acquires schedule information including the tasks of the user, and the display control means 37 superimposes the schedule information showing the tasks for a certain period of time and the display result of the subjective index of the type of mental fatigue or the mental fatigue or the concentration level on the schedule information and displays them on the display unit 308. This allows the user's tendency and schedule to be visualized at the same time, allowing the user to select an appropriate time period when planning the next task, and allows the user to perform the task efficiently. Since the time period when it is easy to concentrate is known in advance, scheduling after understanding the tendency allows the user to naturally perform work in a well-balanced manner, eliminating unnecessary stress. By visualizing the fatigue level and concentration level linked to the past work content, the user can look back on the tendency, and when the user is unable to secure the time they want, this can serve as an opportunity to reduce wasted work and wasted time in meetings.

[0129] <Third embodiment> This embodiment is characterized in that recovery actions are notified at appropriate timing based on the obtained fatigue type and remaining energy. Figure 19 shows a sequence diagram.

[0130] Information related to the user's task is input from the terminal device 3 (step S201). The information is transmitted to the information analysis device 2 (step S202) and recorded in the input information DB2002. The detector 4 detects the user's biometric information (step S204) and analyzes the features (step S204). The analysis results are transmitted as appropriate to the terminal device 3 (step S205). At the same time, the analysis results are transmitted to the information analysis device 2 (step S206). From the feature information, the type of fatigue is estimated (step S207) and the concentration and fatigue levels are estimated (step S208). The results are recorded in the calculation result DB2004.

[0131] The task load is calculated from the currently executed task information recorded in the input information DB 2002 (step S209). The load is calculated based on the time constant information of the task recorded in the task DB 2005. This calculation estimates the expected change in the remaining energy amount in the future. This estimated change in the remaining energy amount and appropriately input biological information (step S206) are added to the calculation. The calculation of the change in the remaining energy level is repeated as appropriate to determine whether it exceeds a set threshold. When the remaining energy level exceeds the set threshold, a calculation is performed to recommend a recovery action (step S210). An appropriate recovery action is selected based on the type of fatigue and remaining energy level estimated earlier. For this selection, a table recorded in the recommendation information DB 2006 is used.

[0132] A display screen is generated (step S211) and transmitted to the terminal device 3 (step S212). The terminal device 3 displays the display screen (step S213).

[0133] Next, the timing for displaying recovery actions will be described with reference to the flowchart in Fig. 20. Generally, many users take recovery actions after the remaining energy reaches 0%, and in this case, it takes a long time before recovery. By objectively quantifying this remaining energy and taking recovery actions when only a few percent of the remaining energy remains, very efficient recovery can be achieved. The present embodiment is characterized in that the terminal device 3 notifies the user of the appropriate timing.

[0134] Task information of the task currently being worked on by the user is obtained from the input information DB 2002 (step S301). Based on this task information, a corresponding task is extracted from the information in the task DB 2005. The concentration time constant of the extracted task is calculated from the table in the task information DB (step S302).

[0135] The detector 4 inputs the biometric information and extracts the feature amount (step S303). This information is transmitted to the information analysis device 2. From this information, the type of fatigue, the fatigue level, the concentration level, and the remaining energy level are calculated (step S304). At this time, the set value information DB 2001 contains the information input in step S201, and the threshold values ​​for the concentration level and the remaining energy level are set based on this information. It is determined whether these threshold values ​​are exceeded (step S305). If not exceeded, the input of the biometric information continues (step S304). Furthermore, the detector 4 continues to transmit the feature amount to the information analysis device 2 as appropriate.

[0136] When the concentration level or remaining energy exceeds a threshold (step S307), a recovery action is selected according to the type of fatigue (step S308). This selection is made by referring to the table in the recommendation information DB 2006.

[0137] The information analyzing device 2 generates an image of a message recommending the selected recovery action, transmits this image to the terminal device 3, and causes the display unit of the terminal device 3 to display the image (step S309).

[0138] Next, a method for calculating the task load will be described. FIG. 21 is a graph showing the relationship between the time spent concentrating on a task and the level of concentration. Concentration cannot be sustained permanently, and it is recommended that users take a rest at regular intervals. In general, the maximum time a person can concentrate on something is said to be 90 minutes, as shown by the curve f1, and tasks that require concentration are often divided into time periods. For example, tasks that require very high levels of concentration, such as simultaneous interpretation, are limited to 15 to 20 minutes. Moreover, general tasks such as attending classes are often 45 to 50 minutes long, and university classes are often 90 minutes long.

[0139] By dividing tasks and relaxing concentration at appropriate times before concentration starts to decline, overall work performance can be improved. However, the duration of concentration is likely to vary depending on individual differences and physical condition, and there are cases where concentration lasts longer, as shown in concentration curve f2, and cases where concentration declines quickly, as shown in curve f3. Therefore, by grasping the user's condition and tendency in real time using biometric information, it becomes possible to display recommended recovery actions that divide tasks at appropriate times.

[0140] A recommended example of an action to improve daily work performance is to set a timer and repeatedly perform work and short breaks. A predicted curve of the user's concentration from the start time to the end time of a certain day is calculated from the past change tendency of the user's concentration level stored in the memory. If it is determined that the concentration level falls below a preset limit before the completion of the pending work, the user is prompted to take a break. The length of the prompted break time may be set by the system or the user in advance. A certain threshold may be set as shown by the dashed line 721 in FIG. 21. The effect of the break is calculated from the change in the concentration level or remaining energy level before and after the break and recorded in the memory. A concentration curve with a break and a work curve without a break may be compared and output to the user.

[0141] This takes into account individual differences and changes in the duration of concentration due to physical condition, and recommends actions such as dividing tasks at appropriate times so that the user can maintain a high level of concentration, thereby improving overall performance throughout the day.

[0142] By making it possible to appropriately ease concentration based on not only individual differences and changes in concentration duration due to physical condition, but also the type of task, it becomes possible to support recovery according to the user's work style. By including the user's context information when recommending recovery actions, it is possible to notify the user of the most effective recovery action that the user can currently take. This will naturally improve the overall performance of a day's work by supporting appropriate recovery according to the user's work style.

[0143] 22 is a table showing the types of tasks and their concentration time constants. This table is calculated based on the user's history so far, and may also include history information such as average work time.

[0144] 23 is a correspondence table showing appropriate recovery actions 912 for fatigue types 911. A recovery message screen is generated based on this table.

[0145] FIG. 24 shows an example of presenting a recommendation of an appropriate recovery action based on the results obtained by the estimation means. This shows an example of a notification to encourage the recommended recovery action depending on the type of fatigue of the user. For example, if the user's remaining energy exceeds a threshold and it is estimated that the user is feeling a strong "feeling of fatigue," the table in FIG. 23 is referenced and the user is recommended to take a break by stretching or taking a walk. The correspondence table of recommended actions and fatigue types in FIG. 23 is included in the recommendation information DB 2006.

[0146] As described above, one embodiment of the present invention is as follows.

[0147] <1> The information processing device 1 according to this embodiment is characterized by an acquisition unit 25 that acquires information including biometric information of the user (step S14), a trained model 2003 that has learned the biometric information acquired in advance and the type of mental fatigue, and a display control means 37 that displays on the display unit 308 (step S30) the type of mental fatigue that is determined based on the acquired information and the trained model 2003 (step S18). As a result, the type of mental fatigue of the user is estimated and output, which gives the user a high sense of satisfaction, thereby reducing the sense of incongruity of the output result. By visualizing the type of fatigue, it is possible to take action to promote recovery.

[0148] <2> The information processing device 1 according to the present embodiment is characterized by a trained model 2003 that has trained bioinformation and a subjective index of mental fatigue or concentration based on a subjective index of mental fatigue or concentration acquired in advance, an acquisition unit 25 that acquires information including the bioinformation of the user (step S14), and a display control means 37 that causes the display unit 308 to display the subjective index of mental fatigue or concentration determined based on the acquired and input information and the trained model 2003 (step S19) (step S20). The subjective index is the objective variable 803 shown in FIG. 8. The objective variable 803 is a quantification of "sleepiness," "discomfort," "unstable," "tiredness," "blurred," "fatigue," and "concentration" as subjective indexes of the user. As a result, the mental fatigue index and concentration index based on the user's subjectivity are estimated and output, which gives the user a high sense of satisfaction, thereby reducing the sense of discomfort in the output result. By visualizing the concentration level, it is possible to take actions to increase the concentration level. By reducing the sense of discomfort, the stress of use is reduced and it is possible to continue using the device.

[0149] <3> The information processing device 1 according to the present embodiment is characterized in that it determines the remaining energy of the user based on the subjective index of mental fatigue and the type of mental fatigue (step S25), and controls the display control unit to display the remaining energy on the display unit (step S30). <1> ~ <2> This makes it easier to take the next action, such as taking a break or concentrating, by visualizing the remaining energy level. By looking back from multiple angles, such as comparing the state at the start of a day's work with the state at the start of each day's work, it is easier to find the cause of one's illness or fatigue, as well as one's own unique tendencies. This makes it easier to find recovery actions that are suitable for oneself, and helps maintain one's health and performance. In addition, one can optimize the pace of work for one week.

[0150] <4> In the information processing device 1 according to the present embodiment, the acquisition unit acquires schedule information including tasks of the user, and adds schedule information indicating tasks for a certain period of time to the schedule information. <1> The type of mental fatigue described in <2> The display control means 37 is characterized in that it causes the display unit 308 to display the display result of the subjective index of the mental fatigue or the concentration level described in the above to be superimposed. <1> ~ <3> This information processing device is described in. By visualizing the user's tendency and schedule at the same time, an appropriate time period can be selected when planning the next task, and the task can be performed efficiently. Since the time periods when it is easy to concentrate can be known in advance, scheduling based on an understanding of the tendency allows the work to be performed naturally and with a good balance, eliminating unnecessary stress. By visualizing the degree of fatigue and concentration linked to the content of past work, the user can look back on the tendency, and when the desired time cannot be secured, this can serve as an opportunity to reduce wasted work and wasted time in meetings.

[0151] <5> The information processing device 1 according to this embodiment is characterized by a display control means 37 that displays on the display unit 308 a message recommending a recovery action selected according to the type of fatigue. <1> ~ <4> This makes it easier to obtain the effect of the recovery action.

[0152] <6> The information processing device 1 according to the present embodiment is characterized in that the acquisition unit acquires schedule information including the user's tasks, and the display control means 37 causes the display unit to display the message at a timing based on the tasks. <1> ~ <5> This makes it possible to optimize the timing of notifying the user of a message for recovery actions according to the user's condition. This allows the user to take recovery actions before unconsciously becoming too fatigued to recover from it.

[0153] <7> The information processing device 1 according to the present embodiment is characterized in that it acquires biometric information using an acceleration sensor. <1> ~ <8> This makes it possible to sense respiration data with high accuracy because it is a contact type, and because it is a small sensor, it can be used continuously with little stress even when it is in contact with the body.

[0154] <8> The information processing device 1 according to the present embodiment is characterized in that it acquires biometric information using a microphone. <1> ~ <8> This makes it possible to easily obtain respiratory data at low cost. In addition, because it is a small sensor and is not easily affected by body movements, it can be used continuously even if the microphone is worn closely, with little stress on the user. A single microphone can obtain multimodal data such as heart rate, body movements, and the presence or absence of conversation.

[0155] <9> The information processing device 1 according to the present embodiment is characterized in that it estimates whether the user is seated or not. <1> ~ <9> This makes it possible to determine whether the person is working quietly while seated and to measure only that part, thereby correctly acquiring a respiratory waveform signal and improving estimation accuracy.

[0156] <10> The information processing device 1 according to the present embodiment is characterized in that it estimates whether or not a user is talking. <1> ~ <10> This information processing device determines whether the person is working quietly without talking to others and measures only that part, making it possible to correctly acquire a respiratory waveform signal and thus improve estimation accuracy.

[0157] <11> The information processing device 1 according to the present embodiment is characterized in that it acquires features of respiration and heart rate. <1> ~ <11> This makes it possible to improve the accuracy of estimating fatigue and concentration levels by performing multimodal measurements. It is possible to remove features closely related to respiration and heart rate, such as respiratory sinus arrhythmia (RSA).

[0158] <12> The trained model 2003 of the information processing device 1 according to the present embodiment is characterized in that the trained model 2003 is further enhanced by information from an acquisition unit 25 that acquires information including the biometric information of the user. <1> ~ <11> The information processing device described in the above. The acquisition unit 25 acquires user information. The information reflects the user's unique characteristics. By further inputting this user-specific information into a learning model, the information becomes extremely valuable when estimating the user's unique characteristics. By using data in a form that identifies the user so as to weight such added information, a learning model customized for a specific user is created. This dramatically improves the estimation accuracy.

[0159] <13> The information processing method according to this embodiment is characterized by the steps of: acquiring information including the user's biometric information (step S14); creating a trained model 2003 that learns the biometric information acquired in advance and the type of mental fatigue; determining the type of mental fatigue based on the acquired information and the trained model (step S18); and displaying the determined type of mental fatigue on a display unit (step S30). As a result, the type of mental fatigue of the user is estimated and output, which gives the user a high sense of satisfaction, thereby reducing the sense of incongruity of the output result. By visualizing the type of fatigue, it is possible to take action to promote recovery.

[0160] <14> The information processing program according to the present embodiment is characterized in that it causes a computer to execute the steps of acquiring information including the user's biometric information (step S14), creating a trained model 2003 that learns the biometric information and the type of mental fatigue acquired in advance, determining the type of mental fatigue based on the acquired information and the trained model (step S18), and displaying the determined type of mental fatigue on a display unit (step S30). As a result, the type of mental fatigue of the user is estimated and output, which gives the user a high sense of satisfaction, thereby reducing the sense of incongruity of the output result. By visualizing the type of fatigue, it is possible to take action to promote recovery. [Explanation of symbols]

[0161] 1. Information processing device 2 Information analysis device 3 Terminal Equipment 4. Detector 22 Display control unit of information analysis device (an example of a display control means) 25 Acquisition unit (an example of an acquisition means) 37 Display control unit of terminal device (an example of a display control means) 208, 308 Display (an example of a display unit or display means) 2003 trained model

Claims

1. An acquisition unit that acquires information including biometric information of a user; A pre-trained model that has learned the biological information and types of mental fatigue acquired in advance, An information processing device characterized by a display control unit that displays on a display unit the type of mental fatigue determined based on the acquired information and the trained model.

2. An acquisition unit that acquires information including biometric information of a user; A trained model that has learned previously acquired biometric information and a subjective index of mental fatigue or concentration level; An information processing device characterized by a display control unit that displays on a display unit the subjective index of mental fatigue or concentration determined based on the acquired information and the trained model.

3. Based on the subjective index of mental fatigue and the type of mental fatigue, 3. The information processing device according to claim 1, further comprising a display control unit that determines a remaining energy amount of the user and causes a display unit to display the remaining energy amount.

4. The acquisition unit acquires schedule information including tasks of the user, The schedule information indicating tasks for a certain period of time; A display result of the type of mental fatigue according to claim 1 or the subjective index of the mental fatigue or the concentration level according to claim 2 is superimposed on the schedule information, 3. The information processing apparatus according to claim 2, further comprising display control means for causing the display unit to display the information.

5. a message recommending a recovery action selected based on the fatigue type; 2. The information processing apparatus according to claim 1, further comprising a display control means for causing the display unit to display the information.

6. The acquisition unit acquires the schedule information including tasks of the user, The recommended message is displayed at a timing based on the task.

6. The information processing apparatus according to claim 5, further comprising display control means for causing the display unit to display the information.

7. An acquisition step of acquiring information including biometric information of a user; A step of creating a trained model that has learned information including previously acquired biological information and types of mental fatigue; determining a type of mental fatigue based on information including the acquired biological information and the trained model; A step of displaying the determined type of mental fatigue on a display unit; An information processing method comprising:

8. acquiring information including biometric information of a user; A step of creating a trained model that has learned information including previously acquired biological information and types of mental fatigue; determining a type of mental fatigue based on information including the acquired biological information and the trained model; A step of displaying the determined type of mental fatigue on a display unit; An information processing program that enables a computer to

Images