Mirror device, estimation device, and control device

Abstract

The present invention provides a mirror device that reduces the burden on the subject and estimates their biometric information from information obtained non-contactually from the subject at the appropriate time. [Solution] The mirror device 1 has a display device 20 for displaying what is in front of it, and estimates the biological information of a subject from information obtained non-contact from a subject positioned in front of the display device. The mirror device comprises a camera 15 positioned with the area in front of the display device as its shooting range, a temperature sensor 16 which is an example of a sensor for detecting the presence of an object, positioned with the area overlapping the shooting range as its detection range, and a control unit 10 that controls the on / off switching of the camera. During the first period, if no face is detected from the captured image obtained from the camera, the control unit switches the camera from on to off, and when the camera is off, if it detects the presence of an object based on a sensor signal from the sensor, it switches the camera from off to on.

Description

Technical Field

[0004]

[0001] The present disclosure relates to a mirror device, an estimation device, and a control device.

Background Art

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

[0004] When performing control using biological information, such as in the system of Patent Document 1, biological information at an appropriate timing is required. Therefore, it is desirable to reduce the burden on the subject and obtain biological information at an appropriate timing.

[0005] According to one embodiment, the mirror device has a display device for displaying what is in front of it, and estimates the biological information of a subject from information obtained non-contact from a subject positioned in front of the display device, and comprises a camera positioned with the front of the display device as its shooting range, a sensor positioned with an overlapping range with the shooting range as its detection range for detecting the presence of an object, and a control unit that controls the on / off switching of the camera. In a first period, if no face is detected from the captured image obtained from the camera, the control unit switches the camera from on to off, and when the camera is off, if it detects the presence of an object based on a sensor signal from the sensor, it switches the camera from off to on.

[0006] According to one embodiment, the estimation device is an estimation device that estimates the biological information of a subject from information obtained from the subject in a non-contact manner, and comprises a camera for photographing the subject, a sensor for detecting the presence of an object, which is positioned as a detection range overlapping the shooting range of the camera, and a control unit that controls the on / off switching of the camera. In the first period, if no face is detected from the image obtained from the camera, the control unit switches the camera from on to off, and when the camera is off, if the presence of an object is detected based on a sensor signal from the sensor, the control unit switches the camera from off to on.

[0007] According to one embodiment, the control device is a control device for a mirror device, the mirror device having a display device for displaying what is in front of it, and estimating the biological information of a subject from information obtained non-contact from a subject positioned in front of the display device. The mirror device has a camera positioned with the area in front of the display device as its shooting range, and a sensor positioned with the area overlapping the shooting range as its detection range for detecting the presence of an object. During a first period, if the control device does not detect a face from the image captured by the camera, it switches the camera from on to off, and if it detects the presence of an object based on a sensor signal from the sensor while the camera is off, it switches the camera from off to on.

[0008] Further details will be described in the embodiments below. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a schematic diagram of the system configuration according to the embodiment. [Figure 2] Figure 2 is a diagram illustrating the overview of the system's operation. [Figure 3] Figure 3 is a schematic diagram of an example of a mirror device. [Figure 4] Figure 4 is a schematic diagram of an example of a mirror device. [Figure 5] Figure 5 is a schematic diagram of an example of a mirror device. [Figure 6] Figure 6 is a schematic side view showing an example of the configuration of a mirror device. [Figure 7] Figure 7 is a schematic diagram illustrating an example of the relationship between the camera's shooting range and the temperature sensor's detection range in a mirror device. [Figure 8] Figure 8 shows the camera's shooting range, the temperature sensor's detection range, and the user's position relative to them. [Figure 9] Figure 9 shows the camera's shooting range, the temperature sensor's detection range, and the user's position relative to them. [Figure 10] Figure 10 is a schematic diagram of the mirror device configuration. [Figure 11] Figure 11 is a flowchart illustrating an example of the biometric information estimation process in a mirror device. [Figure 12] Figure 12 is a flowchart illustrating an example of the switching process in a mirror device. [Figure 13] Figure 13 is a flowchart illustrating another example of the switching process in a mirror device. [Figure 14] Figure 14 is a schematic diagram of the configuration of the equipment control server. [Figure 15] Figure 15 shows an example of control information corresponding to biological information. [Figure 16] Figure 16 is a flowchart illustrating an example of the flow of device control processing on the device control server. [Figure 17]Figure 17 illustrates another example of device control processing on a device control server. [Modes for carrying out the invention]

[0010] <1. Overview of the mirroring device, estimation device, and control device> (1) The mirror device according to the embodiment has a display device for displaying what is in front of it, and estimates the biological information of a subject from information obtained non-contact from a subject positioned in front of the display device, and comprises a camera positioned with the front of the display device as its shooting range, a sensor positioned with an overlapping range with the shooting range as its detection range for detecting the presence of an object, and a control unit that controls the on / off switching of the camera. During the first period, if no face is detected from the captured image obtained from the camera, the control unit switches the camera from on to off, and when the camera is off, if it detects the presence of an object based on a sensor signal from the sensor, it switches the camera from off to on.

[0011] If the control unit does not detect a face in the captured image obtained from the camera during the first period, it switches the camera from on to off, thereby suppressing heat generation and reducing the thermal impact on the camera.

[0012] Furthermore, when the control unit detects the presence of an object based on sensor signals from the sensor while the camera is off, it switches the camera from off to on. This means that the camera automatically activates, for example, when a subject comes in front of a mirror device to check their face and figure in the mirror. As a result, the camera automatically activates and begins operations to acquire information when the subject is in a position where information used to estimate biometric data can be obtained. This reduces the burden on the subject of having to operate the camera themselves, and the burden of having to wait for the camera to turn on after coming to a position where information used to estimate biometric data can be obtained.

[0013] (2) The mirror device of (1), preferably, the control unit further switches the camera from on to off during a first period when no face is detected in the captured image, and then during a second period, if no object is detected based on the sensor signal from the sensor, switches the camera from off to on. As a result, even when no object is detected in front of the mirror device based on the sensor signal from the sensor, the camera automatically switches from off to on after the second period elapses. Therefore, for example, even if the presence is not detected due to a sensor failure or the like, information can be obtained for estimating biometric information when the subject was present within the shooting range of the camera of the mirror device.

[0014] (3) The mirror device of (1)(2), preferably, the control unit further switches the camera from on to off during a first period when no face is detected in the captured image, and then, if an object is detected based on the sensor signal from the sensor and the camera is switched from off to on, during a third period shorter than the first period, switches the camera from on to off if no face is detected in the captured image obtained from the camera. Even when the sensor reacts while the camera is off, it may be due to a temporary change in the surrounding environment or the presence of an object other than a person such as a pet within the detection range, and there may be no subject (person) present. By switching the camera from on to off when no face is detected in the captured image obtained from the camera during the third period, the camera turns off again when no actual person is detected. That is, thereby, the presence of a person within the shooting range of the camera of the mirror device can be accurately detected, and heat generation can be further suppressed.

[0015] (4) The mirror device according to (1) to (3), preferably, the camera and the sensor are arranged in a positional relationship where at least a part of the detection range of the sensor overlaps with the imaging range. More preferably, the camera and the sensor are arranged in a positional relationship where the detection range of the sensor includes at least the center of the imaging range. Thereby, it is detected by the sensor that the subject exists in front of the mirror device. At this time, since the camera automatically switches from off to on, the camera starts up earlier than when an operation to turn on the camera is performed after the subject reaches the front of the mirror device.

[0016] (5) The mirror device according to (4), preferably, the camera and the sensor are arranged in a positional relationship that further includes a point closer to the edge than the center of the imaging range in the detection range of the sensor. Thereby, even when the subject exists at a position farther from the front of the mirror device, if it is within the detection range of the sensor, the presence of the subject is detected. At this time, since the camera automatically switches from off to on, the camera turns on before the subject reaches the front of the mirror device. Thereby, quick startup can be realized in the mirror device.

[0017] (6) The mirror device according to (1) to (5), preferably, the sensor is a temperature sensor used for non-contact measurement of the temperature of the subject. The control unit determines that the presence of a person is detected when the difference between the temperature obtained based on the sensor signal from the temperature sensor when the camera is in the off state and the temperature obtained based on the sensor signal from the temperature sensor when the camera is switched from on to off state is greater than or equal to a threshold value. Thereby, the presence of the subject can be detected using a sensor other than the camera, which is a sensor for obtaining information used for estimating biological information.

[0018] (7) The mirror device according to (1) to (6), preferably, the display device includes a half mirror arranged so that a semi-transmissive reflective surface faces forward, and a display arranged opposite to the rear surface of the half mirror. Thereby, the front of the display device is reflected by the half mirror and displayed on the half mirror, and the display of the display behind the half mirror is also visible from the front of the half mirror.

[0019] (8) The estimation device according to the embodiment is an estimation device that estimates the biological information of a subject from information obtained from the subject in a non-contact manner, and comprises a camera for photographing the subject, a sensor for detecting the presence of an object, which is positioned as a detection range overlapping the shooting range of the camera, and a control unit that controls the switching of the camera on and off. During the first period, if no face is detected from the image obtained from the camera, the control unit switches the camera from on to off, and when the camera is off, if the presence of an object is detected based on a sensor signal from the sensor, the control unit switches the camera from off to on.

[0020] If the control unit does not detect a face in the captured image obtained from the camera during the first period, it switches the camera from on to off, thereby suppressing heat generation and reducing the thermal impact on the camera. Furthermore, if the control unit detects the presence of an object based on a sensor signal from the sensor while the camera is off, it switches the camera from off to on, so that the camera automatically activates, for example, when the estimation device is pointed at the subject or when the subject stands in front of the estimation device. As a result, the camera automatically activates when the subject is in a position where information for estimating biometric information can be obtained, enabling the acquisition of information. This reduces the burden on the subject of having to operate the camera and the burden of having to wait for the camera to turn on after coming to a position where information used for estimating biometric information can be obtained.

[0021] (9) The control device according to the embodiment is a control device for a mirror device, the mirror device having a display device for displaying what is in front of it, and estimating the biological information of a subject from information obtained non-contact from a subject positioned in front of the display device. The mirror device has a camera positioned with the area in front of the display device as its shooting range, and a sensor positioned with the area overlapping the shooting range as its detection range for detecting the presence of an object. During the first period, if the control device does not detect a face from the image captured by the camera, it switches the camera from on to off, and when the camera is off, it detects the presence of an object based on a sensor signal from the sensor, it switches the camera from off to on. This reduces the thermal impact of the mirror device on the camera. In addition, for example, the camera is automatically activated when a subject comes in front of the mirror device and checks their face and figure in the mirror. Therefore, the camera is automatically activated at the timing when information used to estimate biological information can be obtained, enabling operations to acquire information. This reduces the burden on the subject of having to operate the camera and the burden on the subject of having to wait from the position where information used to estimate biological information can be obtained until the camera is turned on.

[0022] <2. Examples of estimation devices, mirroring devices, and control devices> (System Overview) Figure 1 is a schematic diagram of the system configuration according to the embodiment. Figure 2 is a diagram showing an overview of the operation of this system. This system includes a mirror device 1 having a camera 15 and a temperature sensor 16. The mirror device 1 is an example of a biological information estimation device. In the embodiment, the estimation device realized by the mirror device 1 estimates the biological information of a subject from information obtained from the subject in a non-contact manner.

[0023] The mirror device 1 has a display device 20 for displaying what is in front. The display device 20, as an example, has a half mirror 18 and a display 14 positioned opposite the back surface of the half mirror 18A. The side of the half mirror 18 with the semi-transparent reflective surface 18A is the front of the mirror device 1. The camera 15 is positioned so that its shooting range is in front of the mirror device 1, and the temperature sensor 16 is positioned so that its detection range is in front of the mirror device 1. Preferably, the mirror device 1 is provided with an LED 17 which serves as a light source to illuminate the front. The mirror device 1 has a control device 10. Under control by the control device 10, the mirror device 1 communicates with the face recognition server 3, the estimation result server 5, and the equipment control server 9.

[0024] The facial recognition server 3 has a facial information database (DB) 31, in which information about the facial images of registered users (hereinafter referred to as registered users) is registered in association with the user's identification information (hereinafter referred to as user ID). The estimation result server 5 stores the estimated biometric information for each registered user.

[0025] The device control server 9 has a data storage device 94 and stores information about user devices 7A, 7B, 7C, ... (hereinafter referred to as "device information") associated with user IDs. The device control server 9 controls user devices 7A, 7B, 7C, ... (hereinafter referred to as "user devices 7"). User devices 7 can be any device that can communicate with the device control server 9 and can be controlled by the device control server 9. Examples of user devices 7 include air purifiers, air conditioners, electric cookers, refrigerators, washing machines, televisions, solar power generators, lighting equipment, bathroom equipment, electric locks, automobiles, smartphones, personal computers, cameras, audio equipment, robotic vacuum cleaners, robotic transporters, game consoles, and unmanned aerial vehicles (drones).

[0026] The mirror device 1 and user device 7 are installed, for example, in the user's home. The mirror device 1 and user device 7 may communicate with each other and be associated by means of short-range wireless communication or the like. The facial recognition server 3, estimation result server 5, and device control server 9 may be located in different locations from the mirror device 1 and user device 7. In that case, the mirror device 1 and user device 7 and the facial recognition server 3, estimation result server 5, and device control server 9 communicate via a communication network such as the Internet.

[0027] When user H (the subject) stands in front of mirror device 1, the presence of the person is detected by temperature sensor 16. Temperature sensor 16 is an example of a sensor that detects the presence of an object within its detection range. User H, who is in front of mirror device 1, is photographed by camera 15 of mirror device 1, and facial information is detected from the captured image (facial detection). The detected facial information is passed from mirror device 1 to facial recognition server 3.

[0028] The facial recognition server 3 uses the facial information registered in the facial information database 31 to authenticate whether the facial information received from the mirror device 1 belongs to a registered user. If authentication is successful, the facial recognition server 3 passes the user ID of user H to the mirror device 1.

[0029] If authentication is successful, the mirror device 1 estimates the user H's biometric information from the captured image. Figure 2(A) shows the process of estimating biometric information in the mirror device 1. Preferably, the user H is informed that the process of estimating biometric information is underway by the display 41 of the mirror device 1 (for example, "Measuring A").

[0030] In this embodiment, "biological information" refers to information obtained from the human body, including information representing a person's life activities. Information representing a person's life activities includes information representing the state of the body and information representing the load on the body. Information representing life activities includes, for example, information representing the state of the subject's blood vessels, which quantifies the load on the blood vessels (hereinafter referred to as "vascular information"), and information related to the subject's heart rate, which quantifies the speed of blood flow from the heart (hereinafter referred to as "heart rate information"). "Estimation" refers to obtaining biological information by processing information obtained non-contactually from the body, and as an example, it refers to obtaining a value by processing image data obtained by shooting with camera 15. The obtained value is an estimated value representing the state of the subject's blood vessels, or an estimated value representing the state of the heart rate.

[0031] In addition to biometric information estimated from image data captured by camera 15, the estimated results from the output of a sensor that detects the presence of an object can also be used as biometric information. For example, the mirror device 1 may estimate information related to the user H's temperature (hereinafter referred to as temperature information) based on the surface temperature of user H obtained from the sensor signal of temperature sensor 16, and use this as biometric information.

[0032] Figure 2(B) shows the state in which the Mirror Device 1 has obtained the biometric information estimation result. The obtained biometric information estimation result is shown to user H on the display 42 of the Mirror Device 1 and is also passed to the estimation result server 5 along with the user ID. Alternatively, the obtained biometric information estimation result may be notified to user H by voice. The estimation result server 5 stores the biometric information estimation result for user H.

[0033] The estimated biometric information obtained is further passed to the device control server 9 along with the user ID. The device control server 9 controls other devices according to the estimated biometric information (device control). The control content is shown to user H by the display 43 of the mirror device 1 (for example, "Air conditioner 'high'"). Alternatively, user H may be notified of the control content by voice, such as "The air conditioner has been set to 'high'."

[0034] Figure 2(C) shows the state after user H has left the front of the mirror device 1. When the temperature sensor 16 no longer detects the presence of a person in front, the mirror device 1 switches off the camera 15 (switching control).

[0035] (Explanation of the mirror device) Figures 3 to 5 are schematic diagrams of an example of the mirror device 1. As shown in Figure 3, the mirror device 1 is, for example, a mirror that is installed on a washbasin with the front facing the washbasin. When user H uses the washbasin, he naturally positions his face in front of the mirror device 1. As another example, the mirror device 1 may be a portable mirror, as shown in Figure 4, that is used, for example, by placing it on a tabletop. As yet another example, the mirror device 1 may be a full-length mirror installed in an entrance hall, corridor, or bedroom, or mounted on a wall, as shown in Figure 5. Even in such cases, user H naturally positions his face in front of the mirror device 1 when checking his face or clothing.

[0036] Figure 6 is a schematic side view showing an example of the configuration of the mirror device 1. The mirror device 1 has a display device 20 for displaying the view in front. The display device 20, as an example, has a half mirror 18 and a display 14 positioned opposite the back surface of the half mirror 18A. The display 14 displays, for example, an estimation result screen (estimation result image) that displays the estimation result of user H's biometric information. When the display 14 is off, the view in front is reflected in the half mirror 18. When the display 14 is lit, the estimation result screen of the display 14 can be viewed from in front of the half mirror 18 through the half mirror 18. This realizes the display 41 in Figure 2(A) and the displays 42 and 43 in Figure 2(B).

[0037] The temperature sensor 16 measures the temperature within the detection range. If a user H is present within the detection range, the temperature sensor 16 measures the surface temperature of user H. The temperature sensor 16 may be, for example, an infrared sensor. The temperature sensor 16 is, for example, an 8x8 IR detection element array.

[0038] The camera 15 and temperature sensor 16 are positioned, for example, behind the mirror device 1. Preferably, the camera 15 and temperature sensor 16 are positioned on the back side of the semi-transparent reflective surface 18A near the center of the half mirror 18. In this case, holes 15A and 16A are provided at positions on the half mirror 18 corresponding to the camera 15 and temperature sensor 16, respectively. This allows the camera 15 to capture a frontal image of the user H's face through the hole 15A when the user H is positioned in front of the mirror device 1, as shown in Figure 2(A). Also, when the user H is positioned in front of the mirror device 1, as shown in Figure 2(A), the temperature of the user H's body surface can be easily measured by the temperature sensor 16 through the hole 16A. Note that positioning the camera 15 and temperature sensor 16 behind the mirror device 1 is just one example; they may be positioned in other locations, such as on the surface of the half mirror 18 or above the half mirror 18.

[0039] Figure 7 is a schematic diagram illustrating an example of the relationship between the shooting range A1 of camera 15 and the detection ranges A2 and A3 of temperature sensor 16. Figures 8 and 9 show the relationship between the shooting range A1 of camera 15, the detection ranges A2 and A3 of temperature sensor 16, and the user's position. The temperature sensor 16 is positioned so that its detection ranges A2 and A3 overlap with the shooting range A1 of camera 15. In the example in Figure 7, the detection ranges A2 and A3 of temperature sensor 16 are all included within the shooting range A1 of camera 15.

[0040] Preferably, the detection range A2 of the temperature sensor 16 is in the center or approximately in the center of the shooting range A1 of the camera 15. This allows the temperature sensor 16 to detect temperature changes caused by a user H1 who is in a close position in front of the mirror device 1, as shown in Figure 8.

[0041] Preferably, the detection range A3 of the temperature sensor 16 is located closer to the edge than the center of the camera's shooting range A1. By positioning the temperature sensor 16 so that its detection range is further defined by range A3 in addition to the camera's shooting range A1, the temperature sensor 16 can also detect temperature changes caused by a user H2 located in front of the mirror device 1 and far from the mirror device 1, as shown in Figure 9.

[0042] Note that the locations of detection ranges A2 and A3 are examples only and are not limited to these. For example, all elements of the IR detection element array may be monitored. However, increasing the number of monitored objects increases the processing load, and depending on the hardware configuration, real-time detection may become difficult. Therefore, the number of monitored objects should be considered in accordance with the hardware configuration and other factors.

[0043] Figure 10 is a schematic diagram of the configuration of the mirror device 1. The mirror device 1 has a control device 10 that is connected to and controls a display 14, a camera 15, a temperature sensor 16, and an LED 17. The control device 10 has a control unit 11, such as a CPU (Central Processing Unit), a memory 12, such as a ROM (Read Only Memory) or RAM (Random Access Memory), and a communication unit 13, such as a communication module for communicating with other devices. The memory 12 stores a program 121 executed by the control unit 11. The control device 10 may be mounted on the mirror device 1, or it may be a different device such as a computer or terminal device connected to the mirror device 1.

[0044] The control unit 11 executes the first estimation process 111 by executing the program 121. The first estimation process 111 is a process that estimates biological information of the subject, user H, such as vascular information and heart rate information, by analyzing the images captured by the camera 15. Known techniques can be used to analyze the images of user H and estimate vascular information and heart rate information, for example, by statistical processing.

[0045] The control unit 11 executes the detection process 112 by running the program 121. The detection process 112 is the process of detecting the face of the target user H by analyzing the image captured by the camera 15. Face detection refers to obtaining face information from the captured image.

[0046] Face information is, for example, a face image. In this case, the control unit 11 may, as an example, binarize the captured image to extract edges and detect the portion where circular edges representing the contour of the face exist as a face image. The control unit 11 may further detect the portion within the circular edges where black pixels representing both pupils and the nose exist as a face image. Face information may also be, as another example, facial features. The control unit 11 may extract, for example, eyes, nose, mouth, and facial contour as feature points from the captured image and detect their positions as features. The face authentication server 3 considers that the information about face images registered in the face information DB 31 may be, as an example, the face image itself, or it may be facial features.

[0047] Preferably, the control unit 11 executes the second estimation process 113 by executing the program 121. The second estimation process 113 is a process that estimates the temperature information of the target user H from the temperature obtained based on the sensor signal of the temperature sensor 16. The process of estimating the temperature information of user H from the temperature obtained based on the sensor signal can employ known techniques for estimating temperature information from the surface temperature of the target.

[0048] The control unit 11 executes a display process 114 to display the screen on the display 14 by executing program 121. The control unit 11 also executes a switching process 115 by executing program 121. The switching process 115 will be described later.

[0049] (Biometric information estimation processing) Figure 11 is a flowchart illustrating an example of the biological information estimation process in the mirror device 1. The control unit 11 of the control device 10 uses the temperature obtained based on the sensor signal of the temperature sensor 16 to determine whether or not a person is present within the detection range of the temperature sensor 16 (temperature monitoring). For example, the control unit 11 determines that a person is present if the temperature obtained based on the sensor signal of the temperature sensor 16 is above a threshold value stored in advance for determining the presence of a person. The control unit 11 performs temperature monitoring at predetermined timings. These predetermined timings are, for example, at regular time intervals, such as every second.

[0050] If the presence of a person is detected within the detection range of the temperature sensor 16 (YES in step S101), the control unit 11 causes the camera 15 to take a picture (step S103). The control unit 11 performs detection processing 112 on the captured image. If a face is detected from the captured image (YES in step S105), the control unit 11 transmits the detected face information to the face recognition server 3 (step S107).

[0051] When the facial recognition server 3 receives facial information from the mirror device 1 (step S201), it performs facial recognition using the facial information registered in the facial information DB 31 (step S203). If facial recognition is successful, the facial recognition server 3 sends the user ID of the authenticated registered user to the mirror device 1 (step S205).

[0052] When the control unit 11 receives a user ID from the face recognition server 3 (YES in step S109), it executes a first estimation process 111 to estimate biometric information from the captured image (step S111). Preferably, while the first estimation process 111 is being executed in step S111, the control unit 11 executes a display process 114 and displays on the display 14 an indication that estimation is in progress 41 (Figure 2(A)). If the control unit 11 does not receive a user ID from the face recognition server 3 after a predetermined period has elapsed since sending face information to the face recognition server 3 in step S107 (NO in step S109), it does not perform any further processing and terminates the process.

[0053] When the control unit 11 obtains the estimation result of biological information through the first estimation process 111, it sends it to the estimation result server 5 along with the user ID received in step S109 (step S113), and terminates the process. Preferably, in step S113, the control unit 11 executes the display process 114 and displays the estimation result 42 on the display 14 (Figure 2(B)).

[0054] The estimation result server 5 receives the estimated biometric information and user ID from the mirror device 1 (step S301), and records the estimated biometric information in association with the user ID (step S303).

[0055] As the control unit 11 of the mirror device 1 performs the above processing, as shown in Figures 2(A) and (B), the user H's biometric information is automatically estimated simply by the user H standing in front of the mirror. Furthermore, by enabling the mirror device 1 to function as a biometric information estimation device, biometric information is estimated during everyday activities such as when using the bathroom in the example in Figure 3, when checking one's face etc. on a table in the example in Figure 4, and when checking one's clothing etc. in a full-length mirror in the example in Figure 5. Therefore, the mirror device 1 can reduce the burden on the user, such as the need to perform operations to acquire information used for biometric information estimation, or to take actions such as adopting a posture for information acquisition, or to take time to acquire information.

[0056] In the biometric information estimation process according to this embodiment, user authentication is performed using captured images, so biometric information of users other than registered users is not estimated. Furthermore, users do not need to perform any user authentication operations when acquiring the information used to estimate biometric information. Therefore, the mirror device 1 can reduce the burden on the subject when estimating biometric information.

[0057] (Switching process) In the control device 10 of the mirror device 1, the control unit 11 executes a switching process 115 by executing a program 121 (Figure 10). The switching process 115 is a process that controls the switching of the camera 15 on and off. When the camera 15 is off, it is a state in which the operation of at least the elements that can be heat sources within the camera 15 has been stopped. Elements that can be heat sources within the camera 15 include, for example, an image sensor (not shown), a lens motor driver, and a main processor. When the camera 15 is on, it is a state in which the camera 15 is powered on and capable of shooting. Switching the camera 15 on and off is, as an example, a switch between supplying power to at least these elements and cutting off the power supply. Preferably, switching the camera 15 on and off is a switch between supplying power to the entire camera 15 and cutting off the power supply.

[0058] The control unit 11 switches the camera 15 from on to off if no face is detected in the image captured by the camera 15 for a predetermined period of time (first period). The first period is, for example, about 30 seconds.

[0059] Furthermore, the control unit 11 uses the temperature sensor 16 as a sensor for detecting the presence of an object, and detects the presence of a person in front of the mirror device 1 within the detection range of the temperature sensor 16. For example, when the camera 15 is switched from on to off, the control unit 11 temporarily stores the temperature detected by the temperature sensor 16 as the initial switching temperature. When the camera 15 is off, the control unit 11 compares the temperature detected by the temperature sensor 16 (ambient temperature) with the initial switching temperature. If the difference between the ambient temperature and the initial switching temperature exceeds a threshold, the control unit 11 detects the presence of a person in front of the mirror device 1. In other words, if a temperature change of more than a threshold occurs from the time the camera 15 is switched from on to off, the control unit 11 detects the presence of a person in front of the mirror device 1. The threshold is a temperature change threshold for determination that is stored in advance as a criterion for determining the presence of a person, and for example, it is about 3°C. The temperature difference may be such that the ambient temperature is higher than the initial switching temperature, or the ambient temperature is lower than the initial switching temperature. As another example, the control unit 11 may detect the presence of a person in front of the mirror device 1 when the ambient temperature detected by the temperature sensor 16 while the camera 15 is off exceeds a pre-stored threshold.

[0060] Figure 12 is a flowchart illustrating an example of the switching process 115. After turning on the camera 15 (step S401), the control unit 11 of the control device 10 starts timing the first period (first timer) (step S403).

[0061] If face information is detected from the image captured by camera 15 before the first period is reached (YES in step S405), the control unit 11 resets the timing of the first timer and then restarts the timing of the first period (step S403).

[0062] If no face information is detected from the images captured by camera 15 during the first period (NO in step S405, YES in S407), the control unit 11 switches camera 15 from on to off (step S409). Preferably, in step S409, the control unit 11 also switches off the display 14 and LED 17. This further reduces heat generation.

[0063] In step S409, the control unit 11 may not turn off the display 14, but instead display on the screen that the camera 15 is stopped. Depending on the hardware of the camera 15, it may take some time to start up, and shooting may not start immediately even when the user comes in front of the mirror device 1. In such cases, displaying that the camera 15 is stopped until it has finished starting up makes the operating status easier to understand, thus improving user usability. The message that the camera 15 is stopped may be displayed, for example, until the system returns to S401.

[0064] The control unit 11 obtains the temperature based on the sensor signal obtained from the temperature sensor 16 when the camera 15 is switched from on to off in step S409, and temporarily stores it as the initial switching temperature (step S411). During the period when the camera 15 is off, the control unit 11 detects the ambient temperature from the sensor signal from the temperature sensor 16 at predetermined timings. The predetermined timing is, for example, a constant time interval, such as every second. The control unit 11 compares the detected ambient temperature with the initial switching temperature (step S415).

[0065] If the difference between the ambient temperature and the initial switching temperature is greater than or equal to the threshold used for the determination (YES in step S417), the control unit 11 detects the presence of a person in front of the mirror device 1. In this case, the control unit 11 returns to step S401 and repeats the series of operations. That is, after turning on the camera 15 (step S401), the control unit 11 starts timing the first period (step S403). If the display 14 and LED 17 were also turned off in step S409, the control unit 11 turns on the camera 15, the display 14, and the LED 17 in step S401.

[0066] When the control unit 11 of the mirror device 1 performs the above switching process 115, the camera 15 is automatically turned off when the user leaves the front of the mirror device 1 for a predetermined period (first period) (Figure 2(C)). Preferably, the display 14 and LED 17 are also turned off. This reduces heat generation compared to when the camera 15 is always on, and minimizes the thermal impact on the camera 15.

[0067] If camera 15 operates for an extended period, it may overheat and malfunction. Therefore, there is a desire to keep camera 15 stopped during periods when shooting is not required. However, if camera 15 is stopped and then started only when it is time to photograph a person's face, the timing of the shot may be missed, or the waiting time between capturing a face and starting to shoot becomes long, which is inconvenient.

[0068] In contrast, in the switching process 115 described above, when the camera 15 is off and the presence of a person is detected in front of the mirror device 1, the camera 15 switches from off to on. Therefore, as shown in Figure 2(A), when user H stands in front of the mirror device 1, the camera 15 automatically turns on and takes pictures quickly. In the mirror device 1, the camera 15 is automatically activated at the timing when it is possible to acquire information to estimate user H's biometric information, thus shortening the time until activation compared to having to operate the camera 15 after it becomes possible to acquire information to estimate biometric information.

[0069] In the mirror device 1, the temperature sensor 16 is positioned as detection ranges A2 and A3, overlapping with the shooting range A1 of the camera 15 (Figure 7). Because the detection range A2 of the temperature sensor 16 is at least in the center or approximately in the center of the shooting range A1 of the camera 15, the presence of user H in front of the mirror device 1 is detected by the temperature change from the ambient temperature, as shown in Figure 8. At this time, the camera 15 is automatically switched from off to on by the above switching process, so the camera 15 starts up and becomes ready to shoot faster than if user H were to reach in front of the mirror device 1 and then operate the camera 15 to turn it on.

[0070] Furthermore, since the detection range A3 of the temperature sensor 16 is located closer to the edge than the center of the camera 15's shooting range A1, as shown in Figure 9, even if user H is located far from the front of the mirror device 1, the presence of user H is detected if the temperature change from the ambient temperature exceeds a threshold. At this time, the camera 15 is automatically switched from off to on by the above switching process, so the camera 15 turns on before user H reaches the front of the mirror device 1. As a result, the mirror device 1 can achieve quick startup while suppressing the thermal effects on the camera 15.

[0071] Preferably, the control unit 11 starts timing the second period (second timer) when the camera 15 is switched from on to off in step S409 (step S413). The second period is a second determination period used for determining the timing of switching the camera 15 from off to on, and is preferably longer than the first period. The second period is, for example, about 60 seconds. If no person is detected in front of the mirror device 1 during the second period (NO in step S417, YES in S419), the control unit 11 ends timing the second period and returns to step S401 to repeat the series of operations.

[0072] If the first period is extended, the camera 15 will be running for a longer time, which can lead to overheating. On the other hand, if the second period is extended, the camera 15 will be stopped for a longer time, which can lead to heat dissipation. How to set these lengths can be adjusted depending on the hardware of the camera 15 and the expected usage scenario. Depending on the hardware of the camera 15, it may take some time from powering on until it can start shooting, resulting in a waiting time before face detection can begin. In such cases, it is desirable to keep the camera 15 running as much as possible. However, if the running time is made too long, the effect of preventing adverse effects from overheating will be diminished, so it is necessary to decide on the distribution of each period according to the implementation.

[0073] As a result, even if the sensor signal from the temperature sensor 16 does not detect the presence of a person in front of the mirror device 1, the camera 15 will automatically switch from off to on after the second period has elapsed. Therefore, even if the presence of a person is not detected due to, for example, a malfunction of the temperature sensor 16, it becomes possible to acquire information used to estimate biometric information when user H was standing in front of the mirror device 1.

[0074] (Variation of the switching process) Figure 13 is a flowchart illustrating another example of the switching process 115. The flowchart in Figure 13 differs from that in Figure 12 in that it further includes steps S423 to S427. In this modified example, when the control unit 11 detects the presence of a person in front of the mirror device 1 in step S417, it turns on the camera 15 (step S421) and then starts timing the third period (third timer) (step S423). If no face information is detected from the image captured by the camera 15 during the third period (NO in step S425, YES in S427), the control unit 11 ends timing the third period and returns to step S409 to repeat the subsequent operations. That is, the control unit 11 switches the camera 15 from on to off (step S409).

[0075] The third period is a third determination period used to determine whether to switch the camera 15 from off to on and then switch it back off based on the results of human detection. The third period can be a short time, about the time required to detect a face image from the captured image through image analysis, etc., and is shorter than the first period.

[0076] Even if the temperature sensor 16 detects a temperature change exceeding a threshold while the camera 15 is off, this may be due to a temporary change in ambient temperature or the presence of a pet within the detection range. When the switching process 115 related to the modified version is performed, if no face is detected within the third period, the camera 15 switches from on to off, so if the temperature change is due to the presence of a person, the camera 15 turns off again. In other words, when the switching process 115 related to the modified version is performed, the presence of a person in front of the mirror device 1 can be detected with greater accuracy, and heat generation can be further suppressed.

[0077] (Equipment control) Figure 14 is a schematic diagram of the configuration of the device control server 9. The device control server 9 is, for example, a computer having a control unit 91 which is a CPU and memory 92 which is a ROM, RAM, etc., or a single computer and its peripheral devices. The device control server 9 may be implemented by multiple computers working together.

[0078] Memory 92 stores the program 921 that is executed by the control unit 91. The program 921 may be provided by recording it on a recording medium or by using a communication line such as the Internet.

[0079] The device control server 9 further includes a data storage device 94 for storing device information of user devices 7A, 7B, 7C, ... in association with user IDs. Device information is information about devices owned or available to the user. The data storage device 94 is, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The device control server 9 also includes a communication unit 93, such as a communication module for communicating with other devices.

[0080] The control unit 91 executes the device control process 911 by executing the program 921. The device control process 911 is a process that controls other devices according to the estimated results of biological information obtained from the mirror device 1. For example, the program 921 defines control information corresponding to the estimated results of biological information. The control information is, for example, the parameters of the user device to be controlled (hereinafter also referred to as the target device) corresponding to the estimated results of biological information.

[0081] Figure 15 shows an example of control information corresponding to the estimated results of biological information. In Figure 15, control information T1 for the air conditioner (cooling), control information T2 for the air purifier, and control information T3 for the electric cooker are shown. Control information T1 specifies that the air conditioner (cooling) should be turned on when the estimated temperature information value is 35.7°C or higher, and further specifies the parameters as follows: fan speed "low" and temperature "27°C" when the estimated temperature information value is between 35.7°C and 36.2°C, fan speed "medium" and temperature "26°C" when it is between 36.3°C and 36.5°C, and fan speed "high" and temperature "24°C" when it is 36.6°C or higher. Control information T2 specifies that the air purifier should be turned on when estimated heart rate information is obtained, and further specifies the parameters as follows: fan speed "medium" when the estimated heart rate information value is 100 or less, and fan speed "high" when it is 101 or higher. Control information T3 specifies that if the estimated vascular information value is 140 or higher, the electronic cooker should suggest a menu, and further specifies "X1," "X2," and "X3" as the suggested menus.

[0082] Figure 16 is a flowchart showing an example of the flow of the device control processing 911 in the device control server 9. When the control unit 91 receives the estimated result of biological information from the mirror device 1 (YES in step S501), it determines the target device and parameters in the target device based on the control information corresponding to the estimated result of biological information (step S503), and outputs a control signal to the target device (step S505).

[0083] In step S503, the control unit 91, as an example, refers to control information T1 to T3 to determine the target device and its parameters. If the estimated biological information from the mirror device 1 is, for example, shown in display 42 of Figure 2(B), that the estimated temperature information value is 36.5°C, the estimated blood vessel information value is 75 to 110, and the estimated heart rate information value is 90, the control unit 91 decides from control information T1 to turn on the air conditioner (cooling) and set the parameters to "strong" for airflow and "24°C" for temperature. The control unit 91 also decides from control information T2 to turn on the air purifier and set the parameter to "medium" for airflow. The control unit 91 also decides from control information T3 not to suggest a menu using the electronic cooker. In step S505, the control unit 91 outputs a control signal to the air conditioner to turn on the cooling and set the parameters to "strong" for airflow and "24°C". Furthermore, the control unit 91 outputs a control signal to the air purifier that turns it on and sets the airflow to "medium".

[0084] The parameters may also be determined in, for example, the mirror device 1 or the estimation result server 5. In this case, the parameters are transmitted from the mirror device 1 or the estimation result server 5 to the device control server 9. The device control server 9 converts the received parameters into control signals and transmits them to the target device.

[0085] When controlling multiple devices using a user terminal such as a smartphone, user actions such as operation are required, which places a burden on the user. Similarly, when measuring biometric information and controlling devices based on the measurement results, the user must measure their biometric information before controlling the devices, which also places a burden on the user.

[0086] In response to this, the above control allows the device control server 9 to automatically adjust the state of the user's device to correspond to the user's estimated biometric information using the biometric information estimation results. Furthermore, by using the estimation results from the mirror device 1, the user can simply stand in front of the mirror device 1 without performing any operations or actions, and the state of the user's device will automatically correspond to the user's estimated biometric information.

[0087] Note that the control information in Figure 15 is just one example. The control information may define the target device and parameters by combining the estimated results of multiple types of biological information, such as estimated temperature information and estimated blood vessel information. The control information may also define the off state of the user device. Furthermore, the control information may define the target device and parameters by combining other elements, such as the date and time when the information used to estimate the biological information was acquired. In addition, the control information may define parameters as information read from other devices, such as menus suggested by an electric cooker, programs suggested by a television, music suggested by an audio system, or ingredients in a refrigerator.

[0088] (A modified example of equipment control) Program 921 may define control information corresponding to the relationship between the statistical values ​​of the estimation results read from the estimation result server 5 and the estimated results of the obtained biological information. The relationship between the statistical values ​​of the estimation results read from the estimation result server 5 and the estimated results of the obtained biological information may, for example, be the difference between the average value of the estimation results read from the estimation result server 5 and the estimated results of the obtained biological information, or it may be the proportion of the estimated results of the obtained biological information within the range of the estimation results read from the estimation result server 5.

[0089] Figure 17 shows another example of the device control process 911. The flowchart in Figure 17 differs from that in Figure 16 in that it further includes step S502. In the modified example, the control unit 91 uses the user ID received from the mirror device 1 along with the estimated biological information to read the estimation results stored in the estimation result server 5 in association with the user ID (step S502). In the device control process 911 according to the modified example, in step S503, the control unit 91 uses the relationship between the statistical values ​​of the estimation results read from the estimation result server 5 and the received estimated biological information to determine the parameters to be set on the target device. For example, if the estimated temperature information of user H is higher than the normal estimated temperature information of user H, the device control server 9 outputs a control signal to the air conditioner to turn on the cooling and set the fan speed to "high" and the temperature to "24℃".

[0090] By performing the above control, the device control server 9 can control the user device according to the result of comparing the obtained estimate of user H with the normal estimate of user H. This makes it possible to set the state of the user device to a state that is more appropriate to the estimate of the user's biometric information.

[0091] Furthermore, program 921 may specify control information such as turning on the device, shortening the operating time, or outputting a warning when the estimation result is outside the range of estimation results read from the estimation result server 5. In this case, the device control server 9 can turn on monitoring user devices such as cameras, or restrict the use of cars, game consoles, etc., depending on the estimation result of user H.

[0092] The parameter determination may also be performed by the mirror device 1 or the estimation result server 5, as described above. In that case, the device control server 9 may also obtain control signals to be transmitted to the target device from other devices, such as another server corresponding to the target device, and then transmit the control signals to the target device.

[0093] (Variation 1) The above describes an example in which the mirror device 1 functions as a biometric information estimation device, but the biometric information estimation device is not limited to the mirror device 1. Preferably, the estimation device is implemented by a device that the user interacts with in daily life, without the user being aware of acquiring the information used to estimate biometric information. For example, it may be a part of the room's equipment such as walls, ceilings, doors, or windows, an operation panel of a user device such as a car, or an operating device such as a smartphone or personal computer. As a result, the information used to estimate biometric information is automatically acquired and the biometric information is estimated without the user having to perform any actions or operations to acquire the information used to estimate biometric information. Furthermore, the results of the biometric information estimation can be used to switch the device on or off or control the user device.

[0094] (Modification 2) The sensor used to detect the presence of an object is not limited to the temperature sensor 16; any sensor different from the camera 15 may be used. For example, a so-called human presence sensor that detects when a person enters the detection range, or a motion detection sensor that detects when a moving object (human, animal, or other inanimate moving object) enters the range, can be used. The presence of an object can be detected using, in addition to temperature, infrared light, visible light, laser beam, ultrasound, sound, static electricity, vibration, or a combination thereof. For example, the mirror device 1 may be equipped with additional sensors other than the temperature sensor mentioned above, in addition to the temperature sensor 16.

[0095] (Variation 3) In the above explanation, it is assumed that biological information is estimated from image data obtained by the camera 15. The camera 15 is an example of a sensor that acquires information used to estimate biological information from a subject in a non-contact manner. Instead of the camera 15, the temperature sensor 16 may be used as a sensor to acquire information used to estimate biological information, or, as described above, both the camera 15 and the temperature sensor 16 may be used as sensors to acquire information used to estimate biological information. In addition, sensors other than the camera 15 and the temperature sensor 16 that detect objects, or separately provided sensors, may be used as sensors to acquire information used to estimate biological information.

[0096] (Modification 4) The half-mirror 18 and the display 14 are examples of the display device 20. The display device 20 is not limited to a device having a half-mirror. The display device 20 can be any display device that shows what is in front of it, for example, the entire surface may be a display. In this case, the display device 20 should display the image of what is in front of it captured by the camera 15.

[0097] <3. Addendum> The present invention is not limited to the above embodiments, and various modifications are possible. For example, in the above embodiment, the biometric information described was based on data obtained non-contactually and represented information indicating the state of a person's life activities. However, the present invention can also be applied to devices that acquire personally identifiable information as biometric information, such as facial features, gait, vein patterns, and iris patterns. This estimation device can be used as a personal authentication device for purposes such as access control. [Explanation of Symbols]

[0098] 1: Mirror device (estimation device), 10: Control device, 15: Camera, 16: Temperature sensor (sensor for detecting the presence of an object), 18: Half mirror, 18A: Semi-transparent reflective surface, A1: Shooting range, A2, A3: Detection range

Claims

1. A mirror device having a display device for displaying what is in front of it, and estimating the biological information of a subject from information obtained non-contact from a subject located in front of the display device, A camera positioned to capture the area in front of the aforementioned display device, A sensor is positioned to detect the presence of an object, with the detection range being an area overlapping the aforementioned shooting range. The camera comprises a control unit that controls the on / off switching of the camera, The control unit, During the first period, if no face is detected in the captured image obtained from the camera, the camera is switched from on to off. When the camera is off and the presence of an object is detected based on the sensor signal from the sensor, the camera is switched from off to on. Mirror device.

2. The control unit further switches the camera from off to on if, during the first period, no face is detected in the captured image and the camera is switched from on to off, and then during the second period, if the presence of an object is not detected based on the sensor signal from the sensor, the control unit switches the camera from off to on. The mirror device according to claim 1.

3. The control unit further switches the camera from on to off after failing to detect a face in the captured image during the first period, then switches the camera from off to on after detecting the presence of an object based on the sensor signal from the sensor, and then switches the camera from on to off during a third period shorter than the first period if no face is detected in the captured image obtained from the camera. The mirror device according to claim 1.

4. The camera and the sensor are positioned such that the detection range includes at least the center of the shooting range. The mirror device according to claim 1.

5. The camera and the sensor are positioned such that the detection range further includes points closer to the edges than the center of the shooting range. The mirror device according to claim 4.

6. The sensor is a temperature sensor used to measure the temperature of the subject in a non-contact manner, The control unit determines that it has detected the presence of a person when the difference between the temperature obtained based on the sensor signal from the temperature sensor when the camera is in the off state and the temperature obtained based on the sensor signal from the temperature sensor when the camera is switched from on to the off state is greater than or equal to a threshold. The mirror device according to claim 1.

7. The display device includes a half-mirror positioned so that its semi-transparent reflective surface faces forward, and a display positioned opposite the rear surface of the half-mirror. The mirror device according to claim 1.

8. An estimation device that estimates the biological information of a subject from information obtained from the subject in a non-contact manner, A camera for photographing the aforementioned subject, A sensor for detecting the presence of an object is positioned with a detection range that overlaps with the shooting range of the aforementioned camera, The camera comprises a control unit that controls the on / off switching of the camera, The control unit, During the first period, if no face is detected in the captured image obtained from the camera, the camera is switched from on to off. When the camera is off and the presence of an object is detected based on the sensor signal from the sensor, the camera is switched from off to on. Estimation device.

9. A control device for a mirror device, The mirror device has a display device for displaying what is in front of it, and estimates the biological information of a subject from information obtained non-contact from a subject located in front of the display device, A camera positioned to capture the area in front of the aforementioned display device, A sensor for detecting the presence of an object is positioned with a detection range that overlaps with the aforementioned shooting range, It has, During the first period, if no face is detected in the captured image obtained from the camera, the camera is switched from on to off. When the camera is off and the presence of an object is detected based on the sensor signal from the sensor, the camera is switched from off to on. Control device.

Images