Fatigue degree estimation system, fatigue degree estimation device, fatigue degree estimation method, and program

Abstract

To provide a fatigue degree estimation system or the like, capable of estimating a fatigue degree with higher accuracy.SOLUTION: A fatigue degree estimation system 100 is configured to estimate the fatigue degree of a user 99 on the basis of the image of the user 99. The fatigue degree estimation system 100 includes a PC 97 which has a second imaging device 20b for capturing the image and a first display part which is the working screen of the user 99, an external display 98 which has a second display part which is the working screen of the user 99 and has an area larger than that of the first display part and is arranged adjacent to the PC 97, and an estimation part 12 which estimates the fatigue degree of the user 99 on the basis of the image. The second imaging device 20b is arranged to be biased to the side of the external display 98 with respect to the center of the first display part.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a fatigue level estimation system, a fatigue level estimation device, a fatigue level estimation method, and a program. [Background technology]

[0002] Conventionally, there is a need to estimate a person's degree of fatigue. For example, if a user working can know his / her own degree of fatigue, it is useful as information for dealing with the degree of fatigue, such as taking a rest at an appropriate time. As a technology for estimating the degree of fatigue, for example, Patent Document 1 describes an estimation of the degree of fatigue from an image of a user captured by a terminal device such as a smartphone. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2018 / 074371 [Patent Document 2] JP 2000-194450 A [Patent Document 3] Japanese Patent Application Publication No. 6-70314 Summary of the Invention [Problem to be solved by the invention]

[0004] However, there are cases where the accuracy of estimating a user's fatigue level using an image is insufficient. In view of this, the present disclosure provides a fatigue level estimation system and the like that can estimate a user's fatigue level with higher accuracy. [Means for solving the problem]

[0005] In order to solve the above problems, a fatigue estimation system according to one embodiment of the present disclosure is a fatigue estimation system that estimates a user's fatigue level based on an image of the user, and includes: a PC having a second imaging device that captures the image and a first display unit that is a work screen for the user; an external display that has a second display unit that is a work screen for the user and has a larger area than the first display unit and is positioned next to the PC; and an estimation unit that estimates the user's fatigue level based on the image, wherein the second imaging device is positioned biased toward the external display with respect to the center of the first display unit.

[0006] Furthermore, a fatigue level estimation system according to another embodiment of the present disclosure is a fatigue level estimation system that estimates a user's fatigue level based on an image of the user, the fatigue level estimation system comprising: a PC having a first display unit that is a work screen for the user; an external display having a second display unit that is a work screen for the user and has a larger area than the first display unit and is positioned next to the PC; two imaging devices that capture the image, at least one of the PC and the external display being owned by the PC; and an estimation unit that estimates the user's fatigue level based on the image, wherein the estimation unit estimates the user's fatigue level based on the image captured by one of the two imaging devices if the position of a mouse cursor displayed at any position on the first display unit and the second display unit, as operated by the user, is within the first display unit, and estimates the user's fatigue level based on the image captured by the other of the two imaging devices if the position of the mouse cursor is within the second display unit.

[0007] Furthermore, a fatigue level estimation device according to one embodiment of the present disclosure is a fatigue level estimation device that estimates a user's fatigue level based on an image of the user, and includes a PC having a first display unit that is a work screen for the user and an external display having a second display unit that is a work screen for the user and has a larger area than the first display unit, and the PC includes a second imaging device that is positioned toward the external display side with respect to a center of the first display unit of the PC, and an acquisition unit that acquires the image captured by the second imaging device, and an estimation unit that estimates the fatigue level of the user based on the acquired image.

[0008] Furthermore, a fatigue level estimation method according to one embodiment of the present disclosure is a fatigue level estimation method using a fatigue level estimation system that estimates a user's fatigue level based on an image of the user, the fatigue level estimation system including: a PC having a first display unit that is a work screen for the user; an external display that is arranged next to the PC and has a second display unit that is a work screen for the user and has a larger area than the first display unit; and a second imaging device included in the PC that captures the image, the second imaging device being positioned biased toward the external display side with respect to a center of the first display unit that is the work screen for the user of the PC, and the fatigue level estimation method includes obtaining the image captured by the second imaging device from the PC, and estimating the user's fatigue level based on the obtained image.

[0009] Furthermore, a fatigue level estimation method according to another embodiment of the present disclosure is a fatigue level estimation method using a fatigue level estimation system that estimates a user's fatigue level based on an image of the user, the fatigue level estimation system including a PC having a first display unit that is a work screen for the user, an external display that is disposed next to the PC and has a second display unit that is a work screen for the user and has a larger area than the first display unit, and two imaging devices that capture the images, of which the PC includes at least one of the PC and the external display, and the fatigue level estimation method includes obtaining a position of a mouse cursor that is displayed at any position on the first display unit and the second display unit by operation of the user, obtaining the image captured by one of the imaging devices if the position of the mouse cursor is within the first display unit, and obtaining the image captured by the other of the imaging devices if the position of the mouse cursor is within the second display unit, and estimating the user's fatigue level based on the obtained images.

[0010] Moreover, one aspect of the present disclosure may be realized as a program for causing a computer to execute the fatigue level estimation method, or as a computer-readable recording medium storing the program. Effect of the Invention

[0011] According to the present disclosure, fatigue level can be estimated with higher accuracy. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of use of a fatigue level estimation system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating a functional configuration of the fatigue level estimation system according to the embodiment. [Diagram 3] FIG. 3 is a diagram for explaining the effect of fatigue level estimation according to the embodiment. [Figure 4] FIG. 4 is a diagram for explaining the effect of fatigue level estimation according to the embodiment. [Diagram 5] FIG. 5 is a diagram for explaining the effect of fatigue level estimation according to the embodiment. [Figure 6] FIG. 6 is a diagram for explaining the effect of the optical sheet according to the embodiment. [Figure 7] FIG. 7 is a flowchart showing a fatigue level estimating method according to the embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of the configuration of a fatigue level estimating system according to a modified example of the embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of the configuration of a fatigue level estimating system according to a modified example of the embodiment. [Figure 10A] FIG. 10A is a diagram for explaining estimation timing in the embodiment. [Figure 10B] FIG. 10B is a diagram for explaining the estimation timing in the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] (Knowledge that led to this disclosure) As shown in Patent Document 1, there has been a need to estimate a person's fatigue level, and various technological developments have been made. In this context, there are many technologies that have reached the practical stage of capturing an image of a user and estimating the user's fatigue level from the captured image, since it is expected to be realized with a simple configuration. In particular, in recent years, there are many situations in which tasks such as business and learning are performed on terminal devices such as PCs, and such PCs can be used as information processing resources to estimate fatigue levels.

[0014] Some PCs are equipped with an imaging device (a so-called in-camera) that captures an image of the user. Using such an imaging device is preferable because it allows the fatigue level estimation system to be realized using only the PC.

[0015] On the other hand, with the recent rapid miniaturization of PCs due to improvements in information storage density and information processing performance, laptop PCs are often used from the viewpoint of storage and portability. Such laptop PCs are equipped with a relatively small display (hereinafter sometimes referred to as an internal display) with priority given to improving storage and portability. For example, many laptop PCs are equipped with an internal display of 17 inches or less. On the other hand, when working with a PC, it is more efficient to have a large screen for working, and there are situations where a user wishes to use a display that is larger in screen area than the laptop's internal display (it only needs to be larger in area, and at least one of the vertical and horizontal directions is longer than the internal display). In such cases, a so-called external display may be installed next to the PC and used.

[0016] Generally, when an external display is available, people often work mainly using the external display between the internal display and the external display. For example, people may work by frequently looking at the external display while displaying information that should be checked relatively infrequently on the internal display, or may work only using the external display without using the internal display at all.

[0017] In such a case, the user's gaze is basically directed toward the external display. Furthermore, when the gaze is directed toward the external display, in many cases, the direction of the face, and even the direction of the body, is also directed toward the external display. If the user's fatigue level is estimated using the in-camera of the PC, and an external display installed next to the user is used, the in-camera will capture the face of the user facing the adjacent external display as described above. Depending on the arrangement of the external display, in such a case, the in-camera of the PC may only be able to capture one side of the user's face. In particular, the in-camera of the PC is usually provided on the center line in the left-right direction of the internal display in order to capture the user looking at the internal display of the PC from the front, which makes it noticeable that the face of the user facing the external display can only be captured on one side.

[0018] When estimating a user's fatigue level from an image, particularly if an image of the user's face is used, it is necessary to capture facial feature points (e.g., eyes, cheeks, open mouth, etc.) for the estimation. However, some of these feature points may not be obtained from one side of the user's face.

[0019] For example, by using a machine learning model that has been trained by inputting a large number of images of one side of a user's face and a face from the front (including both sides of the face), it is possible to input one side of the user's face and output the face from the front, thereby obtaining an estimated front view of the user's face even in a situation where only one side of the user's face is available, and to use the estimated face for estimating the user's fatigue level. In this way, it becomes possible to estimate the user's fatigue level regardless of the shooting angle of the user's face. However, when using such machine learning, it is necessary to prepare a large amount of face data for learning, and the inference process also requires a large amount of processing resources, which is cumbersome.

[0020] Here, there is known a PC in which an imaging device is arranged at an offset position, deviating from the center line in the left-right direction of the internal display, as shown in Patent Document 2 or Patent Document 3. In this disclosure, a fatigue level estimation system is described for estimating a fatigue level with higher accuracy from an image of a user captured by an imaging device included in a PC, using an imaging device arranged at an offset position, even when working using an external display installed next to the PC as described above.

[0021] This disclosure also mentions a fatigue level estimation system for estimating fatigue levels with greater accuracy by combining an imaging device included in a PC and an imaging device included in the external display when working using a PC and an external display placed next to the PC.

[0022] Hereinafter, the embodiments of the present disclosure will be described with reference to the drawings. Note that the embodiments described below all show comprehensive or specific examples of the present disclosure. Therefore, the numerical values, components, the arrangement and connection of the components, as well as the steps and the order of the steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Therefore, among the components in the following embodiments, components that are not described in the independent claims of the present disclosure will be described as optional components.

[0023] In addition, each figure is a schematic diagram and is not necessarily precisely illustrated. Therefore, the scales and the like are not necessarily the same in each figure. In each figure, the same reference numerals are given to substantially the same configurations, and duplicated explanations are omitted or simplified.

[0024] In the following description, the device will be described as operating in a so-called screen expansion mode, in which the amount of information that can be displayed as a whole is expanded by displaying different areas on the PC display and the external display. In the screen expansion mode, unlike the so-called mirroring mode, in which the same information is displayed on both the PC display and the external display, it is necessary to view both the PC display and the external display in order to view all the displayed information. Note that the disclosure of the present application described below is not particularly limited to such an operation mode between the PC display and the external display, since there are situations in which the above-mentioned mirroring mode is effective.

[0025] (Embodiment) [Configuration of fatigue level estimation system] First, a fatigue level estimation system according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram for explaining an example of use of the fatigue level estimation system according to an embodiment. Note that Fig. 1 shows the position of the second imaging device 20b through the rear side of the housing, but since the second imaging device 20b is an in-camera, it is disposed on the side of the internal display surface that images the user 99.

[0026] As shown in Fig. 1, a fatigue level estimation system 100 (see Fig. 2 described later) in this embodiment is realized, for example, by being built into a computer (an example of a PC 97) used by a user 99. By realizing the fatigue level estimation system 100 in a form built into a computer used by a user 99, the fatigue level estimation system 100 can be realized without using an external information processing server, by using a second imaging device 20b, which is an in-camera mounted on the computer, and a display, etc., as part of the configuration of the fatigue level estimation system 100. In this embodiment, a sub-display (an example of an external display 98) used by the user 99 is used.

[0027] In this way, the input to the fatigue level estimation system 100 can be obtained by capturing an image using the second imaging device 20b, and the output from the fatigue level estimation system 100 can be presented using a display. Since the work performed by the user 99 also requires the use of a computer, the fatigue level of the user 99 can be estimated in parallel with the work using the computer used for the work. Note that some of the processing functions and information storage functions of the fatigue level estimation system 100 may be realized by a cloud server or the like.

[0028] The fatigue level estimation system 100 in the present disclosure is a system that estimates the fatigue level of a user 99 in an image by using an image of the user 99. Therefore, as long as the fatigue level estimation system 100 can acquire an image of the user 99, it can estimate the fatigue level with high accuracy even if the user 99 is gazing at an external display 98.

[0029] Next, the functional configuration of the fatigue level estimation system 100 according to the present embodiment will be described in detail with reference to Fig. 2. Fig. 2 is a block diagram showing the functional configuration of the fatigue level estimation system according to the embodiment.

[0030] 2, the fatigue level estimation system 100 in this embodiment includes a fatigue level estimation device 10, a PC 97, and an external display 98. The devices constituting the fatigue level estimation system 100 may be integrated into one housing or the like, or may be realized as multiple separate devices by connecting the devices to each other via a communication line.

[0031] The fatigue level estimation device 10 includes an acquisition unit 11, an estimation unit 12, and an output unit 13. The fatigue level estimation device 10 is realized by a processor, a memory, and a program executed by using these. The fatigue level estimation device 10 is installed in, for example, a PC 97 as one of its functions.

[0032] The acquisition unit 11 is a functional unit that acquires an image of the user 99. As an example, the acquisition unit 11 acquires an image of the user 99 captured by the second imaging device 20b. The acquisition unit 11 may be integrated with the second imaging device 20b. In one example of the present embodiment, the fatigue level of the user 99 is calculated in real time from the image acquired by the acquisition unit 11. Here, "in real time" includes a delay of about several milliseconds to several seconds, taking into account the time required for calculation processing, data transfer, and the like.

[0033] The acquisition unit 11 may be realized in any manner as long as it can acquire an image. For example, the acquisition unit 11 may acquire an image previously stored in a storage device (not shown). In this manner, the fatigue level estimation system 100 does not need to estimate the fatigue level instantaneously. The acquisition unit 11 transmits the acquired image to the estimation unit 12.

[0034] The estimation unit 12 is a functional unit that estimates the fatigue level of the user 99 based on the acquired image. The estimation unit 12 is realized, for example, by a machine learning model that learns based on the faces of many users 99 and the fatigue levels of the users 99 at that time. In this embodiment, since the focus is on obtaining a more appropriate image for more accurate fatigue level estimation, the function of the estimation unit 12 that estimates the fatigue level of the user 99 from the acquired image, that is, any function may be used. As the function of the estimation unit 12, any existing function may be applied, and a function that estimates the fatigue level of the user 99 appearing in the image from the image. An example of a function of estimating the fatigue level of the user 99 by the estimation unit 12 is shown below. The estimation unit 12 may determine the fatigue level of the user 99 from the left-right difference of the mouth of the user 99, for example. Usually, when the fatigue level of the user 99 increases, the difference between the top and bottom of the left and right ends of the mouth of the user 99 increases. In addition, the estimation unit 12 may determine the fatigue level of the user 99 from the degree of opening of the mouth. Typically, as the user 99's fatigue level increases, the user 99's mouth opens wider. The estimation unit 12 may also determine the user 99's fatigue level from the facial color of a specific location, such as the forehead. Typically, as the user 99's fatigue level increases, blood circulation deteriorates (blood flow rate decreases), causing a decrease in red components in the facial color. Of course, the user 99's fatigue level may be estimated from the user 99's facial image using methods other than these, and the method is not limited here.

[0035] The output unit 13 is a functional unit that outputs the estimation result by the estimation unit 12. The output unit 13 outputs the fatigue level of the user 99 by transmitting the fatigue level of the user 99, which is the estimation result, to a terminal device owned by the user 99, the PC 97, an administrator who manages the work of the user 99, or the like. The fatigue level output by the output unit 13 is, for example, accumulated in a memory unit or the like and displayed while being updated as a change over time, or is instantly displayed as an instantaneous value at that time.

[0036] The PC 97 is a so-called laptop type PC. The PC 97 includes an internal display (not shown) constituting a first display unit that is a work screen used when the PC 97 is used alone, and a second imaging device 20b. The PC 97 also includes various functional units required for information processing for work. Here, Figs. 3 to 5 are diagrams for explaining the effect of fatigue level estimation according to the embodiment.

[0037] Fig. 3 shows a user 99 working, a PC 97x (particularly only the display portion), and an external display 98 in a fatigue level estimation system 100 according to a comparative example. The PC 97x is similar to the PC 97 except that it is provided with a first imaging device 20a instead of the second imaging device 20b. In contrast, Figs. 4 and 5 show a user 99 working, a PC 97, and an external display 98 in fatigue level estimation systems 100 according to two embodiments.

[0038] As shown in Figs. 3 to 5, a user 99 gazing at an external display 98 is imaged by the first imaging device 20a of the PC 97x or the second imaging device 20b of the PC 97. The fatigue level of the user 99 is estimated by the estimation unit 12 based on the image thus captured. As shown in Figs. 4 and 5, the second imaging device 20b is disposed biased toward the external display 98 in the left-right direction in which the second imaging device 20b is aligned with the external display 98 relative to the center of the display of the PC 97 (the display constituting the first display unit). For example, the external display 98 is disposed on the right side of the PC 97x or the PC 97. Therefore, the second imaging device 20b is disposed biased toward the right side relative to the center of the display of the PC 97.

[0039] On the other hand, unlike the second imaging device 20b, the first imaging device 20a is disposed at the center of the display of the PC 97. Then, as shown by the angle of view 97xa, the image of the user 99 captured by the first imaging device 20a is only on the left side of the user 99 as long as the user 99 is gazing at the external display 98 as shown by the line of sight 99a.

[0040] In contrast, as shown in Fig. 4, for example, when the second imaging device 20b is disposed at a position 1 / 3 of the length of the display of the PC 97 in the left-right direction from the right end of the display of the PC 97, the captured image of the user 99 includes parts of the left and right sides of the user 99 as shown in the angle of view 97a, even if the user 99 is gazing at the external display 98 in the same manner. Therefore, the image captured by the second imaging device 20b at the position shown in Fig. 4 is more suitable for estimating the fatigue level with higher accuracy than the comparative example. In other words, the fatigue level estimation system 100 including the second imaging device 20b at the position shown in Fig. 4 is capable of estimating the fatigue level with higher accuracy.

[0041] 5, for example, when the second imaging device 20b is disposed at the right end of the display of the PC 97, the captured image of the user 99 includes most of the left and right sides of the user 99 as shown in the angle of view 97a, even if the user 99 is gazing at the external display 98 in the same manner. Therefore, the image captured by the second imaging device 20b at the position shown in FIG. 5 is more suitable for estimating the fatigue level with higher accuracy than the comparative example. In other words, the fatigue level estimation system 100 including the second imaging device 20b at the position shown in FIG. 5 is capable of estimating the fatigue level with higher accuracy.

[0042] In this way, the second imaging device 20b included in the PC 97 is arranged biased toward the external display 98 with respect to the center of the display of the PC 97, so that an image of the user 99 that allows for relatively accurate estimation can be captured even if the user is gazing at the external display 98. In particular, the second imaging device 20b is arranged to the right of a position that is 1 / 3 of the length of the display of the PC 97 in the left-right direction from the right end of the display of the PC 97, so that an image of the user 99 that allows for more accurate estimation can be captured. Although the position of the second imaging device 20b in the up-down direction is not shown in the figure, in terms of improving the accuracy of the estimation of the fatigue level, the image of the user 99 that is captured by looking up from below is better. Therefore, the second imaging device 20b may be arranged biased downward with respect to the center of the display of the PC 97.

[0043] Returning to FIG. 2, the external display 98 is a display constituting a second display unit for expanding the first display unit of the PC 97. The external display 98 is placed next to the PC 97 when used. Generally, the external display 98 is often placed on the right side of the PC 97, but it may be placed on the left side or upper side of the PC 97, or on the lower side of the PC 97. The external display 98 is used for the purpose of expanding the display of the PC 97, so it is preferable that the length in at least one of the vertical or horizontal directions is larger than the display of the PC 97, and that the area is larger than the display of the PC 97.

[0044] Generally, when a person accumulates fatigue (when the level of fatigue increases), the person tries to continue working in a manner that reduces fatigue. In doing so, the person tends to focus on the second display unit having a larger area, that is, the display unit configured by the external display 98. Therefore, in a situation where fatigue accumulation has progressed and estimation of the level of fatigue is more important, the user 99 tends to work mainly using the second display unit configured by the external display 98. Thus, the fatigue level estimation system 100, which is capable of estimating the level of fatigue with a higher degree of accuracy even when the user is gazing at the external display 98, can effectively function.

[0045] In addition, by attaching an optical sheet or the like that makes the external display 98 and the display of the PC 97 have a narrow viewing angle, the line of sight 99a of the user 99 when gazing at the external display 98 and the line of sight 99a of the user 99 when gazing at the display of the PC 97 can be kept within a predetermined range (based on the degree of narrow viewing angle), and the arrangement position of the second imaging device 20b can be easily determined. In other words, an optical sheet for narrow viewing angle may be attached to the display of the external display 98 and the display of the PC 97 so that an appropriate image is easily captured according to the arrangement position of the second imaging device 20b. An example will be described with reference to FIG. 6. FIG. 6 is a diagram for explaining the effect of the optical sheet in the embodiment. In FIG. 6, (a) shows an area where the face of the user 99 is likely to be located when an optical sheet that makes the external display 98 have a narrow viewing angle is attached, and (b) shows an area where the face of the user 99 is likely to be located when such an optical sheet is not attached. As shown in (a) of FIG. 6, when the optical sheet is attached, the external display 98 can be normally viewed only from a direction within 30° on one side when the direction perpendicular to the external display 98 is set to 0°. Then, the user 99 (the head) can only move within a narrow range (predetermined range) according to such a viewing angle in order to gaze at the external display 98, and the arrangement position of the second imaging device 20b can be determined so that the image can be captured within such a range. On the other hand, as shown in (b) of FIG. 6, when the optical sheet is not attached, the external display 98 can be normally viewed from a direction within 80° on one side when the direction perpendicular to the external display 98 is set to 0°. Then, the user 99 (the head) can gaze at the external display 98 from any position with a wide viewing angle. Therefore, the user 99 (the head) can move within a wide range, and it is difficult to determine the arrangement position of the second imaging device 20b so that the image can be captured within such a range. Although the viewing angles of 30° and 80° have been used as examples here, this angle is not limited thereto, and as long as measures are taken to narrow the viewing angle, the method is not limited to the use of optical sheets, and other means may be used.

[0046] [Fatigue level estimation system operation] Next, the operation of the fatigue level estimation system 100 will be described with reference to FIG. 7. FIG. 7 is a flowchart showing a fatigue level estimation method according to an embodiment. As shown in FIG. 7, when the operation of the fatigue level estimation system 100 is started, an image of the user 99 is captured by the second imaging device 20b. Even if the user 99 is gazing at the external display 98, since the second imaging device 20b is disposed biased toward the external display 98 side from the center of the display of the PC 97, an image showing the entire face of the user 99 is likely to be obtained. As a result, the captured image is likely to include an image showing the entire face of the user 99.

[0047] The captured image is acquired by the acquisition unit 11 (step S101). Then, based on the acquired image, the estimation unit 12 estimates the fatigue level of the user 99 (step S102). The estimated fatigue level of the user 99 is output by the output unit 13 (step S103) and is displayed to the user 99, a manager, etc.

[0048] Since the image used for the estimation is one that is likely to include an image showing the entire face of the user 99, it is possible to estimate the fatigue level with higher accuracy.

[0049] [Variation 1] A first modified example of the embodiment will be described below with reference to Fig. 8. In the first modified example described below, differences from the above embodiment will be mainly described, and descriptions of substantially the same points may be omitted. Fig. 8 is a diagram for explaining a configuration example of a fatigue level estimation system according to a modified example of the embodiment.

[0050] FIG. 8 shows a display of a PC 97 and an external display 98 according to the modified example as viewed from a user 99. For example, even if the user 99 mainly uses the external display 98, the user 99 may also use the display of the PC 97. In such a case, it is appropriate to use the first imaging device 20a. In this modified example, the fatigue level estimation system includes the first imaging device 20a in addition to the second imaging device 20b, so that the first imaging device 20a and the second imaging device 20b can be switched for use. Here, as shown in FIG. 8, a mouse cursor 96 is displayed at any position on the first display unit and the second display unit by the user's operation on the display (first display unit) of the PC 97 and the external display 98 (second display unit). The position of the mouse cursor 96 is also the position where the user 99 is gazing, so that it is appropriately switched to use the image captured by the first imaging device 20a or the image captured by the second imaging device 20b based on the position of the mouse cursor 96.

[0051] The position of the mouse cursor 96 corresponds to a user interface device such as a mouse operated by the user 99. In this modification, the acquisition unit 11 receives an input from a user interface device such as a mouse and acquires the position of the mouse cursor 96. If the acquired position of the mouse cursor 96 is within the first display unit, the fatigue level of the user 99 can be estimated based on the image captured by the first imaging device 20a, and therefore the acquisition unit 11 acquires the image captured by the first imaging device 20a. On the other hand, if the acquired position of the mouse cursor 96 is within the second display unit, the fatigue level of the user 99 can be estimated based on the image captured by the second imaging device 20b, and therefore the acquisition unit 11 acquires the image captured by the second imaging device 20b.

[0052] This allows for more accurate estimation of the fatigue level even when the display of the PC 97 is used occasionally. The first imaging device 20a is not limited to being placed at the center of the display of the PC 97, but may be placed on the opposite side of the external display 98 side from the second imaging device 20b. For example, the first imaging device 20a may be placed on the opposite side of the PC 97 and the external display 98 from a position that is 1 / 3 of the length of the PC in the arrangement direction from the end of the PC 97 on the opposite side to the external display 98 in the arrangement direction. In particular, if the first imaging device 20a and the second imaging device 20b are placed line-symmetrically with respect to the center of the display of the PC 97, the same process can be performed on the external display 98 placed to the left just by replacing the first imaging device 20a and the second imaging device 20b.

[0053] [Variation 2] The second modification of the embodiment will be described below with reference to Fig. 9. In the second modification described below, differences from the modification will be mainly described, and descriptions of substantially the same points may be omitted. Fig. 9 is a diagram for explaining a configuration example of a fatigue level estimation system according to the modification of the embodiment.

[0054] FIG. 9 shows a display of a PC 97 and an external display 98a according to the modification as viewed from a user 99. This modification is different from the modification 1 in that a third imaging device 20c is provided on the external display 98 instead of the second imaging device 20b of the PC 97. The third imaging device 20c operates in the same manner as the second imaging device 20b in the modification 1. That is, the first imaging device 20a and the third imaging device 20c are switched and used. As in the modification 1, the position of the mouse cursor 96 is used when switching between the first imaging device 20a and the third imaging device 20c.

[0055] In this modification, the acquisition unit 11 also receives an input from a user interface device such as a mouse and acquires the position of the mouse cursor 96. If the acquired position of the mouse cursor 96 is within the first display unit, the fatigue level of the user 99 can be estimated based on the image captured by the first imaging device 20a, and therefore the acquisition unit 11 acquires the image captured by the first imaging device 20a. On the other hand, if the acquired position of the mouse cursor 96 is within the second display unit, the fatigue level of the user 99 can be estimated based on the image captured by the third imaging device 20c, and therefore the acquisition unit 11 acquires the image captured by the third imaging device 20c.

[0056] [Effects, etc.] As described above, the fatigue level estimation system 100 according to the first embodiment of the present invention is a fatigue level estimation system 100 that estimates the fatigue level of a user 99 based on an image of the user 99, and includes a PC 97 having a second imaging device 20b that captures an image and a first display unit that is a work screen for the user 99, an external display 98 that has a second display unit that is a work screen for the user 99 and has a larger area than the first display unit and is positioned next to the PC 97, and an estimation unit 12 that estimates the fatigue level of the user 99 based on the image, where the second imaging device 20b is positioned biased toward the external display 98 with respect to the center of the first display unit.

[0057] According to such a fatigue level estimation system 100, even if the user 99 is gazing at the external display 98, an image in a direction close to the front of the user 99 can be captured by the second imaging device 20b included in the PC 97. Therefore, an image more suitable for estimating a fatigue level can be used, and therefore a fatigue level can be estimated more accurately.

[0058] Furthermore, the fatigue level estimation system 100 according to the second embodiment of the present invention is the fatigue level estimation system 100 according to the first embodiment, in which the external display 98 is disposed to the right of the PC 97 when viewed by a user 99 in a posture for using the PC 97 and the external display 98, and the second imaging device 20b is disposed offset to the right of the center of the first display unit.

[0059] This allows an image to be captured in a direction close to directly in front of the user 99 even when the user is gazing at an external display 98 disposed to the right of the PC 97 .

[0060] Furthermore, the fatigue level estimation system 100 according to the third embodiment is the fatigue level estimation system 100 according to the first or second embodiment, and the second imaging device 20b is positioned biased toward the external display 98 in the arrangement direction of the PC 97 and the external display 98 from a position that is 1 / 3 of the length of the PC 97 in the arrangement direction from the end of the PC 97 on the external display 98 side.

[0061] According to this, even if the user 99 is gazing at the external display 98, an image in a direction closer to the front of the user 99 can be captured by the second imaging device 20b included in the PC 97.

[0062] A fatigue level estimation system 100 according to a fourth embodiment of the present invention is the fatigue level estimation system 100 according to any one of the first to third embodiments, in which the second imaging device 20b is positioned offset downward with respect to the center of the first display unit.

[0063] According to this, an image taken from below the user 99 can be obtained as an image more suitable for estimating the degree of fatigue, so that the degree of fatigue can be estimated with higher accuracy.

[0064] Moreover, the fatigue level estimation system 100 according to a fifth embodiment of the present invention is the fatigue level estimation system 100 according to any one of the first to fourth embodiments, and further includes a first imaging device 20a that is arranged on the opposite side to the external display 98 side than the second imaging device 20b.

[0065] This allows the first imaging device 20a and the second imaging device 20b to be used by switching between them.

[0066] Furthermore, a fatigue level estimation system 100 according to a sixth embodiment of the present invention is the fatigue level estimation system 100 according to the fifth embodiment, and the estimation unit 12 estimates the fatigue level of the user 99 based on an image captured by the first imaging device 20a if the position of a mouse cursor 96, which is displayed at any position on the first display unit and the second display unit when operated by the user 99, is within the first display unit, and estimates the fatigue level of the user 99 based on an image captured by the second imaging device 20b if the position of the mouse cursor 96 is within the second display unit.

[0067] This allows switching between the first imaging device 20a and the second imaging device 20b based on the position of the mouse cursor 96.

[0068] A fatigue level estimation system 100 according to a seventh embodiment of the present invention is a fatigue level estimation system 100 that estimates a fatigue level of a user 99 based on an image of the user 99, and includes a PC 97 having a first display unit that is a work screen for the user 99, an external display 98 that has a second display unit that is a work screen for the user 99 and has a larger area than the first display unit and is disposed next to the PC 97, and two imaging devices that capture images, at least one of which is the PC 97 and the external display 98, and a PC 97 that estimates a fatigue level of the user 99 based on the image. The estimation unit 23 estimates the fatigue level of the user 99 based on an image captured by one of the two imaging devices (the first imaging device 20a) if the position of a mouse cursor 96, which is displayed at any position on the first display unit and the second display unit when operated by the user 99, is within the first display unit, and estimates the fatigue level of the user 99 based on an image captured by the other of the two imaging devices (the second imaging device 20b or the third imaging device 20c) if the position of the mouse cursor 96 is within the second display unit.

[0069] According to this, even if the user 99 is gazing at the external display 98, an image in a direction close to the front of the user 99 can be captured by either the first imaging device 20a or the third imaging device 20c. Therefore, an image more suitable for estimating the degree of fatigue can be used, and therefore the degree of fatigue can be estimated more accurately.

[0070] Furthermore, the fatigue level estimation system 100 according to the eighth embodiment is the fatigue level estimation system 100 according to the seventh embodiment, in which one of the two imaging devices is a first imaging device 20a possessed by a PC 97, and the other of the two imaging devices is a third imaging device 20c possessed by an external display 98.

[0071] With this, even if the user 99 is gazing at the external display 98, an image in a direction close to the front of the user 99 can be captured by either the first imaging device 20a possessed by the PC 97 or the third imaging device 20c possessed by the external display 98a.

[0072] Furthermore, the fatigue level estimation device 10 according to the ninth embodiment of the present invention is a fatigue level estimation device 10 that estimates the fatigue level of a user 99 based on an image of the user 99, and is equipped with an acquisition unit 11 that acquires an image captured by a second imaging device 20b from a PC 97 that has a first display unit that is a work screen for the user 99 and is adjacent to an external display 98 that has a second display unit that is a work screen for the user 99 and has a larger area than the first display unit, and the PC 97 includes a second imaging device 20b that is positioned biased toward the external display 98 with respect to the center of the first display unit of the PC 97, and an estimation unit 12 that estimates the fatigue level of the user 99 based on the acquired image.

[0073] By using this in the fatigue level estimation system 100, the above-described fatigue level estimation system 100 can be realized.

[0074] Furthermore, the fatigue level estimation method according to the tenth embodiment of the present invention is a fatigue level estimation method using a fatigue level estimation system 100 that estimates the fatigue level of a user 99 based on an image of the user 99. The fatigue level estimation system 100 includes a PC 97 having a first display unit which is a work screen for the user 99, an external display 98 which is arranged next to the PC 97 and has a second display unit which is a work screen for the user 99 and has a larger area than the first display unit, and a second imaging device 20b which is included in the PC 97 and captures images, the second imaging device 20b being arranged biased toward the external display 98 with respect to the center of the first display unit. In the fatigue level estimation method, an image captured by the second imaging device 20b is obtained from the PC 97, and the fatigue level of the user 99 is estimated based on the obtained image.

[0075] According to this, it is possible to achieve the same effects as those of the fatigue level estimation system 100 described above.

[0076] Furthermore, a fatigue level estimation method according to an eleventh embodiment of the present invention is a fatigue level estimation method using a fatigue level estimation system 100 that estimates the fatigue level of a user 99 based on an image of the user 99. The fatigue level estimation system 100 includes a PC 97 having a first display unit that is a work screen for the user 99, an external display 98 that is arranged next to the PC 97 and has a second display unit that is a work screen for the user 99 and has a larger area than the first display unit, and two imaging devices that capture images, of which the PC 97 includes at least one of the PC 97 and the external display 98. The fatigue level estimation method obtains the position of a mouse cursor 96 that is displayed at any position on the first display unit and the second display unit by operation by the user 99, and obtains an image captured by one of the imaging devices (the first imaging device 20a) if the position of the mouse cursor 96 is within the first display unit, and obtains an image captured by the other imaging device (the second imaging device 20b or the third imaging device 20c) if the position of the mouse cursor 96 is within the second display unit, and estimates the fatigue level of the user 99 based on the obtained image.

[0077] According to this, it is possible to achieve the same effects as those of the fatigue level estimation system 100 described above.

[0078] A program according to a twelfth aspect of the present embodiment is a program for causing a computer to execute the fatigue level estimation method according to the tenth or eleventh aspect.

[0079] According to this, it is possible to achieve the same effects as those of the above-described fatigue level estimation system 100 using a computer.

[0080] (Other embodiments) The fatigue level estimation system, fatigue level estimation device, fatigue level estimation method, and program according to the present disclosure have been described above based on the above-mentioned embodiments, but the present disclosure is not limited to the above-mentioned embodiments. For example, the present disclosure also includes forms obtained by applying various modifications conceived by a person skilled in the art to each embodiment, and forms realized by arbitrarily combining the components and functions of each embodiment within the scope of the present disclosure.

[0081] For example, in all the above-mentioned embodiments, the configuration in which the external display 98 is arranged on the right side of the PC 97 has been described under the assumption that the external display 98 is often arranged on the right side of the PC 97. This has the effect of enabling highly accurate estimation of the fatigue level in most cases, but the external display 98 may be arranged on the left side or upper side of the PC 97, or on the lower side of the PC 97. In that case, similarly, the second imaging device 20b may be arranged in a position biased toward the external display 98 arranged on the left side, upper side, or lower side of the PC 97. For example, when the second imaging device 20b is arranged in a position biased toward the external display 98 arranged on the left side of the PC 97, the second imaging device 20b in the above-mentioned embodiment may be arranged in mirror symmetry with respect to the center in the left-right direction of the display as an axis.

[0082] For example, although the fatigue level estimation system has been described as estimating the fatigue level in real time, the estimation may be performed when an image particularly suitable for estimating the fatigue level is obtained. For example, when about 10 minutes of images are collected in one hour, 10 minutes more suitable for estimation may be selected. The 10 minutes suitable for estimation may be 10 minutes spent gazing at an external display, or 10 minutes spent gazing at the display of a PC97. In this way, it is possible to perform a more suitable fatigue level estimation using only the first imaging device (without providing a second imaging device).

[0083] The 10 minutes suitable for estimation will be further described with reference to FIG. 10A and FIG. 10B. FIG. 10A and FIG. 10B are diagrams for explaining the timing of estimation in the embodiment. The 10 minutes suitable for estimation may be, for example, 10 minutes when the angle of the line of sight 99a of the user 99 with respect to the display of the PC 97 satisfies a condition. FIG. 10A shows the angle of the line of sight 99a of the user 99 when viewed from the same viewpoint as FIG. 4. FIG. 10B shows the change over time of the angle of the line of sight 99a of the user 99 shown in FIG. 10A as a magnitude in the vertical axis direction. The time during which the display of the PC 97 is used and the angle θ1 of the line of sight 99a of the user 99 when the angle perpendicular to the display of the PC 97 is set to 0° is the time when the user gazes at the PC 97, and the angle θ2 is set to a threshold angle θ3. t For example, if the 10 minutes suitable for estimation are the 10 minutes during which the user is gazing at the display of a PC97, then θ1<θ t In other words, the condition for the angle of the line of sight 99a of the user 99 with respect to the display of the PC 97 is θ1<θ tIt is to satisfy. For example, if the 10 minutes suitable for estimation are the 10 minutes in which the difference between the line of sight 99a and the angle perpendicular to the display is the smallest (i.e., the line of sight 99a is viewed from the front without any angle) among the time during which the user gazes at the display of the PC 97, the 10 minutes in which the integral value of θ1 for the 10 minutes is the smallest is selected as shown in FIG. 10B. In other words, the condition for the angle of the line of sight 99a of the user 99 with respect to the display of the PC 97 is that the integral value of θ1 for the 10 minutes is the smallest. By doing so, it is possible to estimate the fatigue level with higher accuracy. Note that the 10 minutes here is an example, and it is sufficient if it is longer than the minimum period in which the estimation unit 12 can estimate the fatigue level of the user 99.

[0084] Furthermore, for example, the present disclosure can be realized not only as a fatigue level estimation system and a fatigue level estimation device, but also as a program including, as steps, processes performed by each component of the fatigue level estimation system, and a computer-readable recording medium on which the program is recorded. The program may be pre-recorded on the recording medium, or may be supplied to the recording medium via a wide area communication network including the Internet.

[0085] In other words, the above-mentioned comprehensive or specific aspects may be realized by a system, an apparatus, an integrated circuit, a computer program, or a computer-readable recording medium, or may be realized by any combination of a system, an apparatus, an integrated circuit, a computer program, and a recording medium. [Explanation of symbols]

[0086] 10. Fatigue Level Estimation Device 11 Acquisition Department 12 Estimation part 13 Output section 20a First imaging device 20b Second imaging device 20c Third imaging device 96 Mouse Cursor 97a, 97xa Angle of view 98, 98a External displays 99 users 99a line of sight 100 Fatigue Level Estimation System

Claims

1. 1. A fatigue level estimation system that estimates a fatigue level of a user based on an image of the user, a PC having a second imaging device that captures the image and a first display unit that is a work screen for the user; an external display having a second display section that is a working screen for the user and has a larger area than the first display section, the external display being disposed next to the PC; an estimation unit that estimates a fatigue level of the user based on the image, the second imaging device is disposed offset toward the external display with respect to the center of the first display unit; Fatigue level estimation system.

2. the external display is disposed on the right side of the PC when the user is in a position to use the PC and the external display; the second imaging device is disposed offset to the right side with respect to the center of the first display unit, The fatigue level estimation system according to claim 1 .

3. the second imaging device is disposed, in an arrangement direction of the PC and the external display, shifted toward the external display from a position that is 1 / 3 of a length of the PC in the arrangement direction from an end of the PC on the external display side. The fatigue level estimation system according to claim 1 .

4. The second imaging device is disposed downwardly with respect to the center of the first display unit. The fatigue level estimation system according to claim 1 .

5. Further, the imaging device includes a first imaging device disposed on an opposite side to the external display side than the second imaging device. The fatigue level estimation system according to any one of claims 1 to 4.

6. The estimation unit is if a position of a mouse cursor, which is displayed at any position on the first display unit and the second display unit by an operation by the user, is within the first display unit, estimating a fatigue level of the user based on the image captured by the first imaging device; if the position of the mouse cursor is within the second display unit, estimating a fatigue level of the user based on the image captured by the second imaging device; The fatigue level estimation system according to claim 5 .

7. 1. A fatigue level estimation system that estimates a fatigue level of a user based on an image of the user, a PC having a first display unit which is a work screen for the user; an external display having a second display section that is a working screen for the user and has a larger area than the first display section, the external display being disposed next to the PC; Two imaging devices for capturing the image, the PC having at least one of the PC and the external display; an estimation unit that estimates a fatigue level of the user based on the image, The estimation unit is if a position of a mouse cursor, which is displayed at any position on the first display unit and the second display unit by an operation by the user, is within the first display unit, estimating a fatigue level of the user based on the image captured by one of the two imaging devices; if the position of the mouse cursor is within the second display unit, estimating a fatigue level of the user based on the image captured by the other of the two imaging devices; Fatigue level estimation system.

8. the one of the two imaging devices is a first imaging device included in the PC, The other of the two imaging devices is a third imaging device included in the external display. The fatigue level estimation system according to claim 7 .

9. 1. A fatigue level estimation device that estimates a fatigue level of a user based on an image of the user, an acquisition unit that acquires the image captured by a second imaging device from a PC having a first display unit that is a work screen for the user and an external display having a second display unit that is a work screen for the user and has a larger area than the first display unit, the PC including a second imaging device that is biased toward the external display with respect to a center of the first display unit of the PC; and an estimation unit that estimates a fatigue level of the user based on the acquired image. Fatigue level estimation device.

10. 1. A fatigue level estimation method using a fatigue level estimation system that estimates a fatigue level of a user based on an image of the user, comprising: The fatigue level estimation system includes: a PC having a first display section which is a working screen for the user; and an external display arranged next to the PC and having a second display section which is a working screen for the user and has a larger area than the first display section; a second imaging device included in the PC for capturing the image, the second imaging device being disposed offset toward the external display with respect to a center of the first display unit; In the fatigue level estimation method, Acquire the image captured by the second imaging device from the PC; estimating a fatigue level of the user based on the acquired image; Fatigue estimation method.

11. 1. A fatigue level estimation method using a fatigue level estimation system that estimates a fatigue level of a user based on an image of the user, comprising: The fatigue level estimation system includes: a PC having a first display section which is a working screen for the user; and an external display arranged next to the PC and having a second display section which is a working screen for the user and has a larger area than the first display section; and two imaging devices for capturing the image, the PC including at least one of the PC and the external display; The fatigue level estimation method includes: acquiring a position of a mouse cursor that is displayed at any position on the first display unit and the second display unit by an operation by the user; If the position of the mouse cursor is within the first display unit, the image captured by the one of the imaging devices is acquired; If the position of the mouse cursor is within the second display unit, the image captured by the other of the imaging devices is acquired; estimating a fatigue level of the user based on the acquired image; Fatigue estimation method.

12. A method for causing a computer to execute the fatigue level estimation method according to claim 10 or 11, program.

Images