User interface device, control method, and program

Abstract

To suppress influence caused by shaking of the imaging unit that can occur during input.SOLUTION: A user interface device 1 includes: an imaging unit C1; an input unit 5 that receives inputs; a functional unit F1; and a control unit 31. The functional unit F1 has operation modes including a first state not prompting to input to the input unit 5 and a second state prompting to input. The control unit 31 adjusts exposure time T1 of the imaging unit C1. The control unit 31 is configured so as to, when the functional unit F1 is in the first state, adjust the exposure time T1 to a first length T11, and when the functional unit F1 is in the second state, adjust the exposure time T1 to a second length T12 shorter than the first length T11.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure generally relates to a user interface device, a control method, and a program, and more particularly, to a user interface device including an imaging unit and an input unit, and a control method and a program for the user interface device. [Background technology]

[0002] Patent Document 1 discloses an information processing device that determines whether a learner needs instruction. The information processing device is implemented in a terminal device (for example, a smartphone, a tablet terminal, a personal computer, a television receiver, etc.) that can be connected to a local network or a global network.

[0003] The information processing device includes a face information acquisition unit that acquires face information of the learner from an image acquired from an imaging unit such as a camera. The information processing device further includes a state detection unit that detects the state of the learner based on the face information extracted by the face information acquisition unit. The state detection unit evaluates the concentration level of the learner by detecting at least one of the following: the learner's line of sight, the state of the pupils, the number of blinks, the movement of the eyebrows, the movement of the eyelids, the movement of the cheeks, the movement of the nose, the movement of the lips, and the movement of the jaw. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-126214 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, when a learner (user) answers (inputs) a question into a terminal device (user interface device), the imaging unit itself may shake depending on the operating conditions, causing so-called camera shake in the video (image) acquired from the imaging unit. Such shake may undermine the reliability of the assessment of the learner's concentration level.

[0006] The present disclosure has been made in consideration of the above-mentioned circumstances, and aims to provide a user interface device, a control method, and a program that are less susceptible to the effects of shaking of the imaging unit that may occur during input. [Means for solving the problem]

[0007] A user interface device according to an aspect of the present disclosure includes an imaging unit, an input unit that accepts an input, a function unit, and a control unit. An estimation unit; The functional unit has, as operation states, a first state in which the input to the input unit is not prompted, and a second state in which the input is prompted. The control unit adjusts an exposure time of the imaging unit. The estimation unit estimates a state of a subject based on an image captured by the imaging unit. The control unit adjusts the exposure time to a first length when the functional unit is in the first state, and adjusts the exposure time to a second length shorter than the first length when the functional unit is in the second state. A user interface device according to another aspect of the present disclosure includes an imaging unit, an input unit that accepts an input, a functional unit, a control unit, and a concentration level estimation unit. The functional unit has, as operation states, a first state in which the input to the input unit is not prompted, and a second state in which the input is prompted. The control unit adjusts the exposure time of the imaging unit. The concentration level estimation unit estimates the concentration level of a subject included in an image captured by the imaging unit based on the image. The control unit adjusts the exposure time to a first length when the functional unit is in the first state, and adjusts the exposure time to a second length shorter than the first length when the functional unit is in the second state.

[0009] A control method according to one aspect of the present disclosure is a control method for a user interface device including an imaging unit and an input unit. The control method includes a function step, an adjustment step, An estimation step; The functioning step has, as operation states, a first state in which no input to the input unit is prompted, and a second state in which the input is prompted. The adjustment step adjusts an exposure time of the imaging unit. In the estimating step, a state of a subject is estimated based on an image captured by the imaging section. In the adjusting step, the exposure time is adjusted to a first length when the functional step is in the first state, and the exposure time is adjusted to a second length shorter than the first length when the functional step is in the second state. A control method according to another aspect of the present disclosure is a control method for a user interface device including an imaging unit and an input unit. The control method includes a function step, an adjustment step, and a concentration level estimation step. The function step has, as operation states, a first state in which input to the input unit is not prompted, and a second state in which the input is prompted. The adjustment step adjusts the exposure time of the imaging unit. The concentration level estimation step estimates the concentration level of a subject included in an image based on an image captured by the imaging unit. The adjustment step adjusts the exposure time to a first length when the function step is in the first state, and adjusts the exposure time to a second length shorter than the first length when the function step is in the second state.

[0010] A program according to an embodiment of the present disclosure includes the steps of: Either This is a program for executing the control method. Effect of the Invention

[0011] The present disclosure has the advantage of being less susceptible to the effects of shaking of the imaging unit that may occur during input. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram of a learning management system including a user interface device according to an embodiment. [Diagram 2] FIG. 2 is a conceptual diagram for explaining the operation of the user interface device. [Diagram 3] 3A and 3B are conceptual diagrams for explaining the positional relationship between an input unit and an imaging unit in the user interface device. [Figure 4] 4A and 4B are conceptual diagrams for explaining the positional relationship between an input unit and an imaging unit in the user interface device. [Diagram 5] FIG. 5 is a diagram for explaining adjustment of an exposure time of an imaging unit in the user interface device. [Figure 6] FIG. 6 is a diagram for explaining camera shake occurring in an image captured by an imaging section in the comparative example. [Figure 7] FIG. 7 is a flow chart for explaining the operation of the user interface device. [Figure 8] FIG. 8 is a diagram for explaining a first modified example (excluding a specific image) of the user interface device of the above embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] (1) Overview Each drawing described in the following embodiments is a schematic drawing, and the size and thickness ratios of each component in each drawing do not necessarily reflect the actual dimensional ratios.

[0014] The user interface device 1 (see FIG. 1) according to one aspect of the present embodiment is configured to support the learning and training of a subject M1 (see FIG. 1: user), for example, in academics (learning support). Therefore, the subject M1 corresponds to a "learner" such as an elementary school student, a junior high school student, a high school student, a vocational school student, a cram school student, or a preparatory school student, but there is no particular limit to the age, and the subject M1 may correspond to a university student taking a national examination or a working person. The subject of the subject is also not particularly limited, and if the subject M1 is, for example, a junior high school student, the subject may be Japanese language, mathematics, English, science, social studies, art, or music. It is assumed that the subject M1 performs online learning using the user interface device 1, which is a "learning terminal." The subject of the user interface device 1 to support is not limited to "academics," but may also be "work" such as office work or monitoring and management work.

[0015] The "learning terminal" of this embodiment may include the learning terminal 2A shown in Figures 3A and 3B and the learning terminal 2B shown in Figures 4A and 4B in addition to the learning terminal 2 shown in Figure 2. In the following, unless otherwise specified, the description will be mainly focused on the learning terminal 2.

[0016] The user interface device 1 (learning terminal 2, 2A, 2B) is, for example, a tablet terminal equipped with a touch panel display (screen H1). The user interface device 1 is not limited to a tablet terminal, and may be a smartphone, a notebook computer, or a desktop personal computer. The learning terminal 2 presents learning materials on the screen H1. The learning materials are teaching materials including video information of an instructor providing learning instruction, and text information related to exercise problem Q1 (hereinafter simply referred to as "problem Q1"), etc.

[0017] In addition, the user interface device 1 has a function (estimation unit 33) of estimating the concentration level of the subject M1 from images captured during learning in order to monitor and manage the subject M1, who is the learner, and to evaluate the suitability of the difficulty level of the learning materials, in other words, to improve the efficiency of learning.

[0018] Here, the user interface device 1 includes an imaging section C1, an input section 5, a functional section F1, and a control section 31, as shown in FIG.

[0019] The imaging unit C1 captures an image of the subject M1. The input unit 5 accepts (external) input. Here, the input unit 5 includes an icon G1 (image area) having an input function, which is displayed on the screen H1 of the display unit 20, as shown in FIG. 2. The input unit 5 may include an entry field (screen area) that accepts input of any character.

[0020] The functional unit F1 has, as its operating states, a first state in which it does not prompt input to the input unit 5, and a second state in which it prompts input. Here, the functional unit F1 includes a display unit 20. The second state is a state in which predetermined information prompting input is displayed on the screen. The first state is a state in which the predetermined information is not displayed. The control unit 31 adjusts the exposure time T1 of the imaging unit C1.

[0021] The predetermined information includes, for example, an icon G1 (a choice that is an answer to question Q1) having an input function as the input unit 5. The predetermined information may further include information related to question Q1. In other words, the second state is a state in which a prompt is presented to urge input to the input unit 5.

[0022] Incidentally, in the user interface device 1, the imaging unit C1 and the display unit 20 that presents the input unit 5 (screen area) are integrally provided. That is, the housing 10 of the user interface device 1 holds the imaging unit C1 and the display unit 20. When the subject M1 (user) answers (inputs) an answer to the user interface device 1 (here, a tap operation on the screen H1), the imaging unit C1 itself may shake in conjunction with the shaking of the housing 10 depending on the operation situation.

[0023] Therefore, in this embodiment, the control unit 31 adjusts the exposure time T1 to a first length T11 when the functional part F1 is in a first state, and adjusts the exposure time T1 to a second length T12 shorter than the first length T11 when the functional part F1 is in a second state.

[0024] According to this configuration, in the second state in which the functional unit F1 is prompting input, the exposure time T1 is adjusted to a second length T12 shorter than the first length T11. Therefore, when the subject M1 (user) makes an input to the user interface device 1, even if the imaging unit C1 itself shakes, the image IM1 is less likely to be blurred (camera shake). As a result, there is an advantage in that the image IM1 is less susceptible to the influence of shaking of the imaging unit C1 that may occur during input.

[0025] In particular, since the user interface device 1 of this embodiment estimates the concentration level of the subject M1 from the image captured by the imaging unit C1 during learning, by adjusting the exposure time T1 described above, the estimated concentration level is less susceptible to the effects of shaking of the imaging unit C1 that may occur during input.

[0026] In this embodiment, the imaging unit C1 is built into the housing 10 in such a manner that the lens of the imaging element is exposed. However, the imaging unit C1 may be attached to the housing 10 later.

[0027] The control method according to the present embodiment is a control method for a user interface device 1 including an imaging unit C1 and an input unit 5. The control method includes a function step and an adjustment step. The function step has, as operation states, a first state in which input to the input unit 5 is not prompted, and a second state in which input is prompted. The adjustment step adjusts the exposure time T1 of the imaging unit C1. In the adjustment step, when the function step is in the first state, the exposure time T1 is adjusted to a first length T11, and when the function step is in the second state, the exposure time T1 is adjusted to a second length T12 shorter than the first length T11. This configuration can provide a control method that can reduce the influence caused by shaking of the imaging unit C1 that may occur during input. This control method is used on a computer system (user interface device 1). That is, this control method can also be embodied as a program. The program according to the present embodiment is a program for causing one or more processors to execute this control method.

[0028] (2)Details Hereinafter, the user interface device 1 according to this embodiment and the entire system (learning management system 100) including its peripheral configuration will be described in detail with reference to Fig. 1. Note that at least a part of the peripheral configuration may be included in the configuration of the user interface device 1.

[0029] (2.1) Overall Structure As shown in Fig. 1, the learning management system 100 comprises one or more user interface devices 1, that is, learning terminals 2 (only one is shown in Fig. 1), and a server 200. The "server" here is assumed to be composed of one server device. However, the "server" may be composed of multiple server devices. Furthermore, multiple server devices may, for example, construct a cloud (cloud computing).

[0030] As described above, the learning terminal 2 is a tablet terminal equipped with a touch panel display (screen H1). The learning terminal 2 is a terminal used by the learner, subject M1 (user), for online learning. In the example of FIG. 1, the subject M1 uses the learning terminal 2 in a management area within a facility 300. For example, the facility 300 is the home of the learner, subject M1, and the management area is subject M1's room. However, the facility 300 is not limited to the home of subject M1, and may be a school or cram school attended by subject M1, in which case the management area may be a classroom.

[0031] The subject M1 engages with the "teaching material" by performing touch operations (tapping) or swiping operations on the screen H1 using his / her fingertip or a pen-shaped operating member (so-called touch pen). Here, the teaching material includes information on a teaching video in which a lecturer's learning instruction is filmed. The teaching video is, for example, recorded in advance, and shows a lecturer of each subject using a blackboard or the like to provide learning instruction on a predetermined theme (for example, "quadratic functions" in mathematics) according to a predetermined curriculum. The teaching material further includes text information and graphic information such as graphs on one or more problems Q1, and it is preferable that the content of the problem Q1 is set with the content of the teaching video. For example, it is preferable that the teaching material is presented from the learning terminal 2 so that the subject M1 solves the quadratic function problem Q1 after watching a teaching video of learning instruction on quadratic functions.

[0032] If the answer to question Q1 is a selection method in which one or more answers are selected from a plurality of options prepared in advance, the subject M1 selects the answer he / she thinks is correct from the plurality of options by tapping. If the answer to question Q1 is a method in which the answer is directly entered in a field, the subject M1 inputs the answer into the field displayed on the screen H1 using his / her fingertip or a touch pen. In other words, the user interface device 1 includes an input unit 5 that accepts the answer (input) to question Q1 by the subject M1, and in this embodiment, the touch panel type display (screen H1 of the display unit 20) of the learning terminal 2 has not only a function of presenting teaching materials (screen display) but also a function related to input.

[0033] The server 200 is installed outside the facility 300 (the home of the subject M1), for example, at a facility of an individual, organization, or company that manages the learning situation of one or more subjects M1. If the facility 300 is, for example, a school or a cram school, the server 200 may be installed inside the facility 300.

[0034] The learning terminal 2 is connected to an external wide area network NT1 including the Internet via a router or the like installed in the facility 300. A server 200 is also connected to the wide area network NT1. Dedicated application software for communicating with the server 200 is pre-installed in the learning terminal 2. The learning terminal 2 and the server 200 are capable of bidirectional communication via the wide area network NT1. The detailed configurations of the learning terminal 2 and the server 200 will be described later.

[0035] (2.2) Learning terminal (user interface device) 1, the learning terminal 2 includes a processing unit 3, a functional unit F1, an input unit 5, and an imaging unit C1. The learning terminal 2 further includes a communication interface for communicating with the server 200 and the like, and a storage unit 21 including a rewritable non-volatile memory such as an EEPROM (Electrically Erasable Programmable Read-Only Memory). The storage unit 21 may be the memory of the processing unit 3. The learning terminal 2 may further include a mouse, a keyboard, a pointing device, and the like.

[0036] The input unit 5 accepts an operation input from the outside (the subject M1). As described above, the icon G1 (image area) displayed on the display unit 20 corresponds to the input unit 5.

[0037] The functional unit F1 has, as its operating states (functional steps), a first state in which it does not prompt input to the input unit 5, and a second state in which it prompts input. Here, the functional unit F1 includes a display unit 20. The display unit 20 includes a touch panel display (screen H1) and has both a function of presenting teaching materials (screen display) and a function of accepting input operations such as touch operations. The display is, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display.

[0038] The second state is a state in which predetermined information is displayed on the screen to prompt the user to input an answer to the question Q1. The predetermined information includes an icon G1 having an input function as the input unit 5. Here, the predetermined information includes an icon G1 corresponding to two options, "Y" and "N," which simply represent "YES" and "NO," respectively. There is no particular limit to the number of icons G1 (options) displayed on one screen. Here, the predetermined information also includes the question Q1 itself. That is, the display unit 20 outputs the question Q1 and the icon G1 on one screen at the same time. However, the question Q1 and the icon G1 may be displayed on different screens at different times, for example, the icon G1 may be displayed on the next screen after transitioning from the screen displaying the question Q1.

[0039] The first state is a state in which predetermined information is not displayed. Here, the first state is, for example, a state in which information related to explanation P1 such as theorems, formulas, calculation methods, and solutions is displayed, or a state in which a teaching video of a learning instruction is displayed. When the question Q1 and the icon G1 are displayed on different screens at different times, the first state may be a state in which the question Q1 is displayed (the icon G1 is not displayed).

[0040] Hereinafter, the period during which the functional unit F1 is in the first state may be referred to as the first state period W1, and the period during which the functional unit F1 is in the second state may be referred to as the second state period W2 (see FIG. 5). The first state period W1 is, so to speak, a period during which the subject M1 is in a passive state. On the other hand, the second state period W2 is, so to speak, a period during which the subject M1 is in an interactive state.

[0041] The imaging unit C1 has an imaging element and is a camera for capturing an image of a subject. In this embodiment, an RGB camera is used as the imaging unit C1. The imaging element is, for example, a two-dimensional image sensor such as a CCD (Charge Coupled Devices) image sensor or a CMOS (Complementary Metal-Oxide Semiconductor) image sensor. The imaging unit C1 further has an optical system such as a lens, and captures an image by forming an image of light from a subject (here, a subject M1) on an imaging surface (light receiving surface) of the imaging element through the optical system, and converting the light from the subject into an electrical signal by the imaging element.

[0042] As shown in FIG. 2, the imaging unit C1 is attached, for example, at the center in the left-right direction above the screen H1 on the front (the surface on which the screen H1 is located) of the learning terminal 2. As a result, while the subject M1 is studying using the learning terminal 2, at least the face of the subject M1 is included in the angle of view of the imaging unit C1. While the subject M1 is studying, the imaging unit C1 captures the subject M1 under the control of the processing unit 3 described later. Although a detailed description will be given later, depending on the type of learning terminal, the imaging unit C1 may be disposed near the right end in the left-right direction above the screen H1 (see FIGS. 3A and 3B), or may be disposed near the left end (see FIGS. 4A and 4B).

[0043] The processing unit 3 can be realized by a computer system including one or more processors (microprocessors) and one or more memories. In other words, the one or more processors execute one or more programs (applications) stored in one or more memories to function as the processing unit 3. Here, the programs are pre-recorded in the memory of the processing unit 3, but they may be provided via a telecommunications line such as the Internet or recorded on a non-transitory recording medium such as a memory card.

[0044] The processing unit 3 has a presentation unit 30, a control unit 31, an image processing unit 32, and an estimation unit 33 (concentration degree estimation unit). That is, the processing unit 3 has a function as the control unit 31, a function as the image processing unit 32, and a function as the estimation unit 33. The processing unit 3 is also configured to execute control processing related to communication with the server 200.

[0045] The presentation unit 30 is configured to present learning materials used by the subject M1 in learning. The presentation unit 30 causes the display unit 20 to display learning materials corresponding to a predetermined learning process on the screen. The processing unit 3 receives learning material data transmitted from the server 200 and stores (stores) it in the storage unit 21. The learning material data is managed in association with a process ID assigned to the learning process. The processing unit 3 may receive all of the learning material data at once, or may receive data related to the explanation P1, the question Q1, and the answer options (icons G1) each time they are presented.

[0046] The control unit 31 executes control related to the imaging element of the imaging unit C1. In this embodiment, the control unit 31 is configured to adjust the exposure time T1 (the period during which light is accumulated in the imaging element) of the imaging unit C1. The control unit 31 also has a function of adjusting the frame rate.

[0047] The processing unit 3 also acquires image data from the imaging unit C1 that images the subject M1 who is handling the teaching material. For example, the control unit 31 controls the imaging unit C1 to start imaging the subject M1 together with the start of the learning process. Specifically, the learning process is started when the subject M1 taps an operation area (image area) of "Start" displayed on the screen H1. In conjunction with the start of the learning process, the processing unit 3 presents the corresponding teaching material and starts imaging the subject M1 by the imaging unit C1. The concentration level of the subject M1 is estimated based on the image data.

[0048] The learning process ends when the subject M1 answers the final question Q1 in the learning material and, for example, taps an operation area (image area) of "End" displayed on the screen H1. In conjunction with the end of the learning process, the processing unit 3 ends the presentation of the corresponding learning material and ends the imaging of the subject M1 by the imaging unit C1.

[0049] The processing unit 3 associates the image data captured in the learning process with the process ID of the learning process and stores (stores) it in the storage unit 21. It is assumed that the image data is moving image data. In the present disclosure, an "image IM1" is, for example, data equivalent to one frame, and the moving image data is made up of a large number of images IM1 arranged in a time series direction B1 (see FIG. 1). The moving image data also includes data in which the interval between one frame and another is relatively long (frame-by-frame advance). The image data may be one frame, i.e., one still image data, of the subject M1 at a certain moment in the first state period W1 or the second state period W2.

[0050] In this embodiment, the control unit 31 adjusts the exposure time T1 to a first length T11 when the functional unit F1 is in the first state (see FIGS. 2 and 5) (adjustment step). Also, the control unit 31 adjusts the exposure time T1 to a second length T12 shorter than the first length T11 when the functional unit F1 is in the second state (see FIGS. 2 and 5) (adjustment step). That is, the control unit 31 sets the exposure time T1 to the first length T11 in the first state period W1 in which text information or graphic information related to the commentary P1 or a teaching video of the learning instruction is displayed. On the other hand, the control unit 31 sets the exposure time T1 to the second length T12 in the second state period W2 in which the question Q1 and the icon G1 are displayed.

[0051] 2 and 5 show a vertical synchronization signal V1 that is a reference for the timing for driving the imaging element of the imaging section C1. The control section 31 generates the vertical synchronization signal V1 and transmits the vertical synchronization signal V1 to a drive circuit of the imaging section C1, so that the imaging element performs imaging at an exposure time T1 according to the vertical synchronization signal V1. Note that the imaging interval A1 shown in FIG. 5 corresponds to an interval for one frame. In this embodiment, the imaging interval A1 is constant in the first state period W1 and the second state period W2, and the imaging element is exposed when the vertical synchronization signal V1 is at a high level.

[0052] The control unit 31 monitors whether the current operating state of the functional unit F1 is the first state or the second state, and changes the exposure time T1 when the operating state switches from the first state to the second state or from the second state to the first state. Here, when the display of the icon G1 starts, the control unit 31 sets the exposure time T1 to the second length T12, and when the display of the icon G1 ends, the control unit 31 sets the exposure time T1 to the first length T11. At the start of imaging, the control unit 31 sets the exposure time T1 to the first length T11.

[0053] The first length T11 is, for example, a fixed length set in advance. The second length T12 is assumed to be shorter than the first length T11 and to be changed depending on the situation. The second length T12 may be a fixed length set in advance, similar to the first length T11. The processing unit 3 stores information on the first length T11 and the second length T12 in the storage unit 21.

[0054] In this embodiment, the control unit 31 determines the second length T12 based on the positional relationship between the input unit 5 and the imaging unit C1. The determination of the second length T12 will be described in detail later.

[0055] The image processing unit 32 is configured to perform image processing (image processing such as enlargement, reduction, rotation, inversion, interpolation, edge processing, etc.) on the image IM1 captured by the imaging unit C1. The image processing unit 32 outputs data that has been subjected to the image processing to the estimation unit 33.

[0056] Here, in this embodiment, the image processing unit 32 is configured to align a plurality of images IM1 arranged in a time series direction B1 (see FIG. 1) captured by the imaging unit C1 in the second state, and to synthesize the plurality of images IM1 together (synthesis process). In the example of FIG. 1, the image processing unit 32 generates a synthetic image IM2 using a plurality of images IM1 (three images in FIG. 1 as an example) captured during the second state period W2. Specifically, the image processing unit 32 extracts image regions including a face, fingers, etc. from each of the plurality of images IM1, aligns pixel positions of the image regions, and generates a synthetic image IM2 by averaging pixel values ​​(luminance values) of the pixels. For the second state period W2, the image processing unit 32 outputs the synthetic image IM2, which is the result of the synthesis process, to the estimation unit 33. Note that the image processing unit 32 may output the image IM1 captured during the first state period W1 to the estimation unit 33 without performing the above synthesis process.

[0057] In other words, the image IM1 captured during the second state period W2, in which the exposure time T1 is shorter than the first state period W1, may have a deteriorated S / N ratio, and may be less reliable in the subsequent process (concentration degree estimation process). By performing the synthesis process, the effect of the deterioration of the S / N ratio caused by the shorter exposure time T1 can be suppressed.

[0058] The estimation unit 33 is configured to estimate the state of the subject M1 included in the image IM1 based on the image data (image IM1) captured by the imaging unit C1. Here, the estimation unit 33 estimates (measures) the concentration level of the subject M1 who is handling the teaching material, using the image data (including the composite image IM2) that has been subjected to image processing and that is output from the image processing unit 32.

[0059] The estimation unit 33 performs image analysis on the image data, for example, to detect the position and size of the face, fingers, etc. of the subject M1. Then, the estimation unit 33 extracts biometric information on the facial expression (including blinking and eye movement), facial direction, posture, and movement (gestures of fingers, etc.) of the subject M1 based on the detection results on the face, fingers, etc. of the subject M1, and estimates the concentration level of the subject M1. The biometric information of the subject M1 may include not only the facial expression, facial direction, posture, and movement at a certain moment, but also information on changes in the facial expression, facial direction, and movement over time. The "certain moment" is, for example, the moment when the question Q1 is presented on the screen H1. If the learning terminal 2 is equipped with a microphone, the voice data of the subject M1 inputted through the microphone may also be used as one of the biometric information to estimate the concentration level.

[0060] The concentration level is expressed, for example, as a percentage. The closer the concentration level is to 100%, the more the subject M1 is concentrating on working on the learning material. The estimation unit 33 may calculate the concentration level using a predetermined arithmetic formula such that the greater the amount of change per unit time in facial expression, face direction, posture, and movement, the lower the concentration level. The estimation unit 33 may calculate the concentration level in units of each explanation P1 and each question Q1. An ID is assigned to each of the explanation P1 and the question Q1, and the estimation unit 33 may link the concentration level to the ID of the corresponding explanation P1 or question Q1. The estimation unit 33 may measure the transition of the concentration level during the implementation period of the learning process and calculate the average value (or the maximum or minimum value).

[0061] The estimation unit 33 may estimate the concentration level using a trained model (classifier) ​​for which machine learning has been completed, with the image data as input data. That is, the classifier outputs the concentration level as a classification result based on features related to the facial expression, face direction, posture, and movement of the subject M1 in the image data.

[0062] The processing unit 3 associates the estimation result (concentration data) of the estimation unit 33 with the process ID, the ID of the explanation P1, the ID of the question Q1, and the terminal ID of the learning terminal 2 (which may be the user ID assigned to the subject M1), and transmits it to the server 200.

[0063] The timing of transmitting the estimation result (concentration data) of the estimation unit 33 to the server 200 is not particularly limited. It may be transmitted when the learning process is completed, or may be transmitted each time one explanation P1 or question Q1 is completed, or may be transmitted periodically, or may be transmitted when the estimation result (concentration data) of the estimation unit 33 falls below a certain value.

[0064] (2.3) Server 1, the server 200 includes a processing unit 4 and a storage unit 43. The server 200 further includes a communication interface for communicating with one or more learning terminals 2 and the like.

[0065] The storage unit 43 includes a rewritable non-volatile memory such as an EEPROM. The storage unit 43 may be a memory of the processing unit 4. The storage unit 43 stores a question database that manages a large amount of data related to explanations P1, questions Q1, answer options, and answers (correct answers) corresponding to various subjects. The question database can be updated when new explanations P1 and questions Q1 are created.

[0066] The processing unit 4 can be realized by a computer system including one or more processors (microprocessors) and one or more memories. In other words, the one or more processors execute one or more programs (applications) stored in one or more memories to function as the processing unit 4. Here, the programs are pre-recorded in the memory of the processing unit 4, but they may be provided via a telecommunications line such as the Internet or by being recorded on a non-transitory recording medium such as a memory card.

[0067] The processing unit 4 is configured to execute control processing related to communication with each learning terminal 2. The processing unit 4 extracts questions that are most suitable for the subject M1 from the question database in the storage unit 43, generates data of learning materials, and transmits them to the corresponding learning terminal 2.

[0068] In this embodiment, the processing unit 4 has functions for executing an acquisition process, a generation process, and a correctness determination process, and includes an acquisition unit 40, a generation unit 41, and a correctness determination unit 42, as shown in FIG.

[0069] The acquisition unit 40 is configured to acquire data (answer data) related to the learning process from the learning terminal 2 through communication with the learning terminal 2. The answer data includes data related to the terminal ID (which may be a user ID) of the learning terminal 2, the process ID, the ID of the explanation P1, the ID of the question Q1, the answer of the subject M1, the concentration level, and the like.

[0070] The generating unit 41 generates information on, for example, the next learning process based on the concentration level. Specifically, the generating unit 41 generates information on the learning materials to be used in the next learning process based on the estimated concentration level and the correctness of the answer, which will be described later. That is, the generating unit 41 judges whether the difficulty level of the learning materials used in the learning process was appropriate for the subject M1 based on the answer data obtained in the most recent learning process. The server 200 selects learning materials with a more appropriate difficulty level for the subject M1, transmits information on the learning materials to the learning terminal 2, and causes the learning terminal 2 to display the learning materials on the screen H1 when the next learning process is carried out.

[0071] The correctness judgment unit 42 is configured to judge whether the acquired answer is correct or not. Based on the ID of the question Q1 of interest, the correctness judgment unit 42 searches the question database stored in the storage unit 43 and extracts a reference answer associated with the ID. The correctness judgment unit 42 compares the extracted reference answer with the answer input by the subject M1 to judge whether it is a "correct answer" or an "incorrect answer". For example, if the concentration level is high and the proportion of incorrect answers is high, the generation unit 41 judges that the difficulty level of the teaching material is too high for the subject M1 and selects teaching material with a lower difficulty level than the previous teaching material. On the other hand, for example, if the concentration level is low and the proportion of correct answers is high, the generation unit 41 judges that the difficulty level of the teaching material is too easy for the subject M1 and selects teaching material with a higher difficulty level than the previous teaching material.

[0072] The estimated concentration level is not limited to be used in the process of selecting learning materials. For example, the server 200 may create notification information regarding the target person M1's efforts with the learning materials based on the concentration level and the correctness of the answer, and transmit the notification information to the learning terminal 2 or an information terminal carried by the target person M1's parent, teacher, etc. The server 200 may also control equipment such as air conditioners installed in the facility 300 based on the estimated concentration level to promote an increase in the concentration level.

[0073] [advantage] Here, advantages of the user interface device 1 in this embodiment will be described with reference to FIGS.

[0074] FIG. 6 shows a state of a subject M1 who is studying online using a user interface device 1X (study terminal 2X) of a comparative example. Unlike the study terminal 2, the study terminal 2X of the comparative example does not have an adjustment function for changing the exposure time T1 between a first state period W1 in the first state and a second state period W2 in the second state. Therefore, when the subject M1 taps on the screen of the study terminal 2X to answer a question, the housing of the study terminal 2X shakes and the imaging unit itself shakes, and the image IM1 captured by the imaging unit at that moment may be blurred as shown in FIG. 6. If the concentration level of the subject M1 is evaluated based on the image IM1 in a blurred state, there is a possibility that an appropriate evaluation will not be performed. In other words, the concentration level estimation unit of the study terminal 2X may erroneously recognize that the subject M1 himself has moved his face due to a break in concentration during study, for example, and as a result, may evaluate the concentration level of the subject M1 low. Alternatively, there may be a possibility that a large blur occurs so that the facial expression cannot be identified, in which case it may be impossible to evaluate the concentration level.

[0075] In contrast, in the user interface device 1 (learning terminal 2) of this embodiment, in the second state period W2, the exposure time T1 is adjusted to a second length T12 that is shorter than the first length T11 in the first state period W1 (see FIGS. 2 and 5). Therefore, when the subject M1 (user) inputs to the user interface device 1, even if the imaging unit C1 itself shakes, the image IM1 is less likely to be blurred. As a result, there is an advantage that the image IM1 is less susceptible to the influence of shaking of the imaging unit C1 that may occur during input.

[0076] In particular, since the user interface device 1 of this embodiment estimates the concentration level of the subject M1 from the image captured by the imaging unit C1 during learning, by adjusting the exposure time T1 described above, the estimated concentration level is less susceptible to the effects of shaking of the imaging unit C1 that may occur during input.

[0077] (2.4) Determining the length of exposure time Hereinafter, the process of determining the second length T12 performed by the control unit 31 in the user interface device 1 (the learning terminals 2, 2A, 2B) will be described along with Usage Example 1 and Usage Example 2. In this embodiment, as described above, the control unit 31 determines the second length T12 based on the positional relationship between the input unit 5 and the imaging unit C1.

[0078] In this embodiment, the control unit 31 determines the second length T12 based on the distance L1 between the input unit 5 and the imaging unit C1. Since the icon G1 has a function as the input unit 5, the control unit 31 determines the second length T12 based on the positional relationship between the display position of the icon G1 and the imaging unit C1. Hereinafter, a description will be given with reference to Figs. 3A to 4B. In Figs. 3A to 4B, as an example, the position of the center of gravity X1 (pixel position) in the entire image area of ​​the two icons G1 is set as the position of the input unit 5. The control unit 31 specifies the center of gravity X1 by the XY coordinates of the pixel of the screen H1. The storage unit 21 may store data indicating the correspondence between the center of gravity X1 of the icon G1 for each question Q1 and the distance L1 in advance. The control unit 31 specifies the distance L1 from the center of gravity X1 of the icon G1 to be displayed by referring to the correspondence data.

[0079] [Usage example 1] 3A and 3B show a tablet terminal used with the housing 10 in a landscape orientation, like the learning terminal 2, but the position of the imaging unit C1 differs from that of the learning terminal 2. In the learning terminal 2A of FIGS. 3A and 3B, the imaging unit C1 is disposed near the right end in the left-right direction at the top of the screen H1.

[0080] In a tablet terminal in which the user interface device 1 is used with the housing 10 in a landscape orientation, such as the learning terminals 2 and 2A, if the subject M1 is right-handed, the position (holding position) in which the learning terminal 2 is held with the left hand may be around the lower left. When the subject M1 taps the input unit 5 (icon G1 in this case) with his / her right hand, the degree of shaking of the imaging unit C1 may change depending on the difference in the distance L1 between the icon G1 and the imaging unit C1.

[0081] 3A, when both the two icons G1 and the imaging unit C1 are located in the upper right, the distance L1 is relatively small. When the subject M1 taps one of the icons G1 in the upper right with his / her right hand, the housing 10 of the learning terminal 2 may swing relatively widely around the holding position. As a result, the imaging unit C1 located in the upper right may also swing relatively widely.

[0082] As shown in Fig. 3B, when two icons G1 are displayed at the bottom left of the screen H1 and the imaging unit C1 is located at the top right, the distance L1 is relatively large. When the subject M1 taps one of the icons G1 at the bottom left with his / her right hand, the housing 10 of the learning terminal 2 shakes around the holding position as a fulcrum, but the shaking may be sufficiently small compared to the case of Fig. 3A. As a result, the imaging unit C1 may shake less compared to the case of Fig. 3A.

[0083] In short, in the user interface device 1, the display position of the input unit 5 (icon G1 in this case) on the screen H1 may change depending on the type of question Q1, etc. As a result, the distance L1 between the icon G1 and the imaging unit C1 may change depending on the presentation status of the teaching material.

[0084] Therefore, the control unit 31 is configured to adjust the second length T12 so that the shorter the distance L1 between the input unit 5 and the imaging unit C1, the shorter the second length T12 (hereinafter, the process related to this adjustment will be referred to as a "first adjustment process"). In other words, the control unit 31 is configured to adjust the second length T12 so that the longer the distance L1 between the input unit 5 and the imaging unit C1, the longer the second length T12. The second length T12 may be calculated by substituting the distance L1 into a predetermined arithmetic expression, or the storage unit 21 may store in advance data (e.g., a data table) that indicates the correspondence relationship between the distance L1 and the second length T12.

[0085] By performing the first adjustment process by the control unit 31, when the icon G1 is displayed in a position close to the imaging unit C1 as in Fig. 3A, the exposure time T1 is shortened in preparation for a relatively large shake of the imaging unit C1. On the other hand, when the icon G1 is displayed in a position far from the imaging unit C1 as in Fig. 3B, the imaging unit C1 is unlikely to shake significantly, so the exposure time T1 is lengthened (however, it is not adjusted to be longer than the first length T11). Therefore, when the fulcrum (holding position) that supports the user interface device 1 is far from the imaging unit C1, the above adjustment makes it more difficult for blurring to occur in the image IM1.

[0086] There are also cases where the user interface device 1 is used in a state where it is not held by a person but is held on a fixed base (stand) for placing it in an upright state on a piece of furniture such as a desk. In this case, it is preferable that the holding position is, for example, the lower part of the housing 10 and the first adjustment process is executed.

[0087] [Usage example 2] 4A and 4B, unlike the learning terminals 2 and 2A, the learning terminal 2B is a smartphone used with the housing 10 in a vertically elongated state. In addition, in the learning terminal 2B, the imaging unit C1 is disposed near the left end in the left-right direction at the top of the screen H1.

[0088] In a smartphone in which the user interface device 1 is used with the housing 10 in a vertically elongated state like the learning terminal 2B, if the subject M1 is right-handed, the position (holding position) in which the learning terminal 2 is held with the left hand may be the center left or the upper left. When the subject M1 taps the input unit 5 (icon G1 in this case) with the right hand, the degree of shaking of the imaging unit C1 may change depending on the difference in the distance L1 between the icon G1 and the imaging unit C1.

[0089] 4A, when two icons G1 are displayed at the bottom left of the screen H1 and the imaging unit C1 is located at the top left, the distance L1 is relatively large. When the subject M1 taps one of the icons G1 at the bottom left with his / her right hand, the housing 10 of the learning terminal 2 may swing relatively widely around the holding position. As a result, the imaging unit C1 located at the top left may also swing relatively widely.

[0090] As shown in Fig. 4B, when two icons G1 are displayed in the upper right corner of the screen H1 and the imaging unit C1 is located in the upper left corner, the distance L1 is relatively small. When the subject M1 taps one of the icons G1 in the upper right corner with his / her right hand, the housing 10 of the learning terminal 2 shakes around the holding position as a fulcrum, but the shaking may be sufficiently small compared to the case of Fig. 4A. As a result, the imaging unit C1 may shake less compared to the case of Fig. 4A.

[0091] Therefore, when the user interface device 1 is, for example, a smartphone (learning terminal 2B), the control unit 31 may adjust the second length T12 so that the longer the distance L1 between the input unit 5 and the imaging unit C1, the shorter the second length T12 (hereinafter, the process related to this adjustment is referred to as the "second adjustment process"). In other words, the control unit 31 may adjust the second length T12 so that the shorter the distance L1 between the input unit 5 and the imaging unit C1, the longer the second length T12. The second length T12 may be calculated by substituting the distance L1 into a predetermined calculation formula, or the storage unit 21 may store in advance a data table showing the correspondence relationship between the distance L1 and the second length T12.

[0092] By performing the second adjustment process by the control unit 31, when the icon G1 is displayed at a position far from the imaging unit C1 as in Fig. 4A, the exposure time T1 is shortened in preparation for a relatively large shake of the imaging unit C1 (however, it is not adjusted to be longer than the first length T11). On the other hand, when the icon G1 is displayed at a position close to the imaging unit C1 as in Fig. 4B, the exposure time T1 is longer since the imaging unit C1 is less likely to shake significantly. Therefore, when the fulcrum (holding position) that supports the user interface device 1 is close to the imaging unit C1, the above adjustment makes it more difficult for blurring to occur in the image IM1.

[0093] [Determining holding position] In the above-mentioned use example 1 and use example 2, it is assumed that the function of either the first adjustment process or the second adjustment process is implemented depending on the type of the learning terminal 2, 2A, 2B. In other words, it is assumed that the function of the first adjustment process is implemented for tablet terminals (learning terminals 2, 2A), and the function of the second adjustment process is implemented for smartphones (learning terminal 2B).

[0094] However, for example, both the first adjustment process and the second adjustment process may be implemented in one learning terminal as long as it has a function of automatically determining the holding position. For example, the learning terminal 2 may be equipped with a gyro sensor, and the holding position may be estimated from the detection result (angle change of the housing 10) detected by the gyro sensor. Based on this estimation result, the learning terminal 2 decides whether to execute the first adjustment process or the second adjustment process. If there is no angle change for a predetermined time or more, it may be assumed that the learning terminal 2 is placed on the fixed stand, and the first adjustment process may be executed.

[0095] Furthermore, the learning terminal 2 may estimate not only the holding position but also the orientation of the housing 10 (whether the housing 10 is used in a landscape orientation or a portrait orientation) from the detection result detected by the gyro sensor. The holding position may be determined by a pressure sensor or the like other than the gyro sensor.

[0096] (2.5) Operation Hereinafter, the operation of the user interface device 1 will be described with reference to Fig. 7. The order of operations in the following operation example is merely an example, and is not particularly limited.

[0097] The subject M1 starts online learning using the user interface device 1 (start of the learning process: S1). In response to the start of the learning process, the user interface device 1 starts capturing an image of the subject M1 with the imaging unit C1 (S2). At the start of the imaging, the imaging unit C1 captures an image with an exposure time T1 of a first length T11.

[0098] During the imaging, the user interface device 1 monitors whether the current operating state of the functional unit F1 is the first state or the second state (S3), and estimates the concentration level of the subject M1 based on the image data (S4). The user interface device 1 transmits data on the estimation result of the concentration level to the server 200 at predetermined intervals.

[0099] If the functional unit F1 is in the second state where input is prompted, i.e., the icon G1 is being displayed (S5: Yes), the user interface device 1 sets the exposure time T1 to the second length T12 (S6). If the functional unit F1 is in the first state where input is not prompted, i.e., the commentary P1 or the instruction video is being displayed (S5: No), the user interface device 1 sets the exposure time T1 to the first length T11 (S7).

[0100] When the learning process ends (S8: Yes), the user interface device 1 ends the image capture (S9). As long as the learning process is ongoing (S8: No), the user interface device 1 continues to capture images, monitor the operating state of the functional unit F1, and estimate the concentration level (to S3).

[0101] (3) Variations The above embodiment is merely one of various embodiments of the present disclosure. The above embodiment can be modified in various ways depending on the design, etc., as long as the object of the present disclosure can be achieved. In addition, the same function as the user interface device 1 according to the above embodiment may be embodied in a control method, a computer program, or a non-transitory recording medium on which a computer program is recorded, etc.

[0102] Modifications of the above embodiment are listed below. The modifications described below can be combined as appropriate and applied. Hereinafter, the above embodiment may be referred to as a "basic example."

[0103] The user interface device 1 in the present disclosure includes a computer system. The computer system is mainly composed of a processor and a memory as hardware. The function of the user interface device 1 in the present disclosure is realized by the processor executing a program recorded in the memory of the computer system. The program may be pre-recorded in the memory of the computer system, may be provided through an electric communication line, or may be provided by being recorded in a non-transitory recording medium such as a memory card, an optical disk, or a hard disk drive that can be read by the computer system. The processor of the computer system is composed of one or more electronic circuits including a semiconductor integrated circuit (IC) or a large scale integrated circuit (LSI). The integrated circuits such as IC or LSI referred to here are called by different names depending on the degree of integration, and include integrated circuits called system LSI, VLSI (Very Large Scale Integration), or ULSI (Ultra Large Scale Integration). Furthermore, a field-programmable gate array (FPGA) that is programmed after the manufacture of an LSI, or a logic device that can reconfigure the connection relationship inside the LSI or the circuit partition inside the LSI, can also be adopted as a processor. The electronic circuits may be integrated in one chip or distributed among multiple chips. The chips may be integrated in one device or distributed among multiple devices. The computer system referred to here includes a microcontroller having one or more processors and one or more memories. Thus, the microcontroller is also composed of one or more electronic circuits including a semiconductor integrated circuit or a large-scale integrated circuit.

[0104] Furthermore, it is not essential that the multiple functions of the user interface device 1 are concentrated in one housing. For example, the components of the user interface device 1 may be distributed across multiple housings.

[0105] Conversely, multiple functions of the user interface device 1 may be integrated in one housing. Furthermore, at least some of the functions of the user interface device 1, for example, some of the functions of the user interface device 1 may be realized by the cloud (cloud computing) or the like.

[0106] (3.1) First Modification The user interface device 1 of the first modified example will be described below with reference to some drawings of the basic example and Fig. 8. In the following, components that are substantially the same as those of the user interface device 1 of the basic example will be given the same reference numerals and their description may be omitted as appropriate. Fig. 8 shows a vertical synchronization signal V1 transmitted from the control unit 31 to the drive circuit of the imaging unit C1.

[0107] The user interface device 1 according to this modification includes an imaging unit C1, an input unit 5 that accepts input, a functional unit F1, and an estimation unit 33 (see FIG. 1). The functional unit F1 has, as its operating states, a first state in which no input to the input unit 5 is prompted, and a second state in which input is prompted. The estimation unit 33 estimates the state of a subject M1 included in an image IM1 based on an image IM1 captured by the imaging unit C1.

[0108] The user interface device 1 according to this modified example further includes a control unit 31. However, unlike the basic example, the control unit 31 fixes the exposure time T1 to a first length T11 whether the functional unit F1 is in the first state or the second state (see FIG. 8). In other words, the exposure time of the first state period W1 is the same as the exposure time of the second state period W2. Note that, in this modified example as well, the imaging interval A1 is constant in the first state period W1 and the second state period W2.

[0109] The estimation unit 33 according to this modification also differs from the estimation unit 33 of the basic example in that, when a specific condition is satisfied, the estimation unit 33 estimates the state of the subject M1 by excluding a specific image IM1 captured by the imaging unit C1 when the functional unit F1 is in the second state. In other words, when a specific condition is satisfied, the estimation unit 33 estimates the concentration level using an image IM1 (image) captured in the first state period W1, without using a specific image IM1 (image) captured in the second state period W2. When the estimation unit 33 executes the concentration level estimation process in real time during shooting, the estimation unit 33 may, for example, temporarily stop the estimation process if a specific condition is satisfied in the second state period W2.

[0110] Here, the "specific condition" is a condition regarding the positional relationship between the input unit 5 and the imaging unit C1 at the time when the specific image IM1 is captured. As an example, in this modified example, similar to the basic example, the predetermined information includes an icon G1 having an input function as the input unit 5. Therefore, the specific condition is a condition regarding the positional relationship between the icon G1 and the imaging unit C1 at the time when the specific image IM1 is captured.

[0111] In the case of "Use Example 1" described in the basic example, the specific condition may be that the distance L1 between the input unit 5 and the imaging unit C1 is shorter than a predetermined distance at the time of capturing the specific image IM1 (see FIG. 3A). In this case, for example, when a fulcrum (holding position) supporting the user interface device 1 is far from the imaging unit C1, it is possible to suppress a decrease in the reliability of the estimation result of the estimation unit 33. Note that if the distance L1 between the input unit 5 and the imaging unit C1 is equal to or greater than a predetermined distance (see FIG. 3B, for example), the estimation unit 33 estimates the concentration level using the image IM1 (video) even if it is captured in the second state period W2.

[0112] Furthermore, in the case of "Use Example 2" described in the basic example, the specific condition may be that the distance L1 between the input unit 5 and the imaging unit C1 is longer than a predetermined distance at the time of capturing the specific image IM1 (see FIG. 4A). In this case, for example, when the fulcrum (holding position) supporting the user interface device 1 is close to the imaging unit C1, it is possible to suppress a decrease in the reliability of the estimation result of the estimation unit 33. Note that if the distance L1 between the input unit 5 and the imaging unit C1 is equal to or less than a predetermined distance (see FIG. 4B, for example), the estimation unit 33 estimates the concentration level using the image IM1 (video) even if it is captured in the second state period W2.

[0113] According to the configuration of this modified example, the state of the subject M1 is estimated by excluding a specific image IM1 captured in the second state where the functional unit F1 is encouraging input (if a specific condition is satisfied). Therefore, even if the imaging unit C1 itself shakes and causes camera shake in the specific image IM1 when the user makes an input to the user interface device 1, it is possible to suppress a decrease in the reliability of the estimation result of the estimation unit 33 due to the specific image IM1. As a result, the state is less susceptible to the influence caused by the shaking of the imaging unit C1 that may occur during input.

[0114] The predetermined distance may be zero or a distance close to zero. In this case, regardless of the distance L1, the estimation unit 33 estimates the concentration level using only the image IM1 captured in the first state period W1, without using a specific image IM1 captured in the second state period W2.

[0115] (3.2) Other Modifications Other modifications are listed below.

[0116] In the basic example, the functional unit F1 includes a display unit 20, the second state is a state in which predetermined information prompting input is displayed on the screen, and the first state is a state in which the predetermined information is not displayed. The functional unit F1 may include at least one of an audio output unit (e.g., a speaker), an optical output unit (e.g., an LED), and a vibration output unit (vibration function) instead of (or in addition to) the display unit 20. However, it is easier for the subject M1 (user) to recognize the first state and the second state when predetermined information prompting input is displayed on the screen.

[0117] In addition, when the functional unit F1 includes a voice output unit, the second state is, for example, a state in which synthetic voice data that prompts input is output, and the first state is a state in which the synthetic voice data is not output. When the functional unit F1 includes a light output unit, the second state is, for example, a light-on state that prompts input, and the first state is an off state. When the functional unit F1 includes a vibration output unit, the second state is, for example, a state in which vibration that prompts input is output (a state in which the housing 10 is vibrated), and the first state is a state in which the vibration is not output.

[0118] In the basic example, the imaging interval A1 is constant in the first state period W1 and the second state period W2. The control unit 31 may adjust the imaging interval A1 in the imaging unit C1 to be shorter when the functional unit F1 is in the second state. That is, the control unit 31 may increase the frame rate in the second state period W2 to be higher than the frame rate in the first state period W1. The control unit 31 transmits the vertical synchronization signal V1 with the increased frame rate in the second state period W2 to the drive circuit of the imaging unit C1. In this way, by increasing the frame rate when the functional unit F1 is in the second state, it is possible to suppress the influence of the deterioration of the S / N ratio caused by the shortening of the exposure time T1.

[0119] In the basic example, the number of imaging units C1 is one. However, the user interface device 1 may include a plurality of imaging units C1. For example, the plurality of imaging units C1 may include an imaging unit C1 built into the housing 10 with the lens exposed as in the basic example, and an imaging unit C1 that is attached to the periphery of the housing 10 afterwards.

[0120] When there are multiple imaging units C1, the user interface device 1 does not necessarily need to constantly adjust the exposure time T1 to the second length T12 for all imaging units C1 in the second state. For example, if the subject M1 moves during learning and the image processing unit 32 detects that the face of the subject M1 no longer exists in the image data captured by a certain imaging unit C1 among the multiple imaging units C1, the control unit 31 may set the exposure time T1 to the first length T11 for that imaging unit C1.

[0121] In the basic example, the input unit 5 is an icon G1 (screen area) having an input function that is displayed on the screen H1. However, instead of (or in addition to) such a screen area, the input unit 5 may include a mechanical operation portion such as an operation button that is arranged on the outer periphery of the housing 10 near the screen H1.

[0122] (4) Summary As described above, the user interface device (1) according to the first aspect includes an imaging unit (C1), an input unit (5) that accepts input, a functional unit (F1), and a control unit (31). The functional unit (F1) has, as operating states, a first state in which input to the input unit (5) is not prompted, and a second state in which input is prompted. The control unit (31) adjusts the exposure time (T1) of the imaging unit (C1). When the functional unit (F1) is in the first state, the control unit (31) adjusts the exposure time (T1) to a first length (T11), and when the functional unit (F1) is in the second state, the control unit (31) adjusts the exposure time (T1) to a second length (T12) shorter than the first length (T11).

[0123] According to this aspect, in the second state in which the functional unit (F1) is prompting input, the exposure time (T1) is adjusted to a second length (T12) shorter than the first length (T11). Therefore, when the user inputs to the user interface device (1), even if the imaging unit (C1) itself shakes, the image (IM1) is less likely to be blurred. As a result, the image is less susceptible to the influence of shaking of the imaging unit (C1) that may occur during input.

[0124] Regarding the user interface device (1) according to the second aspect, in the first aspect, the control unit (31) determines the second length (T12) based on the positional relationship between the input unit (5) and the imaging unit (C1).

[0125] According to this embodiment, blurring of the image (IM1) becomes less likely to occur.

[0126] With respect to the user interface device (1) according to the third aspect, in the first or second aspect, the control unit (31) adjusts the second length (T12) so that the shorter the distance (L1) between the input unit (5) and the imaging unit (C1), the shorter the second length (T12).

[0127] According to this aspect, for example, when the fulcrum that supports the user interface device (1) is separated from the imaging section (C1), blurring of the image (IM1) is less likely to occur.

[0128] With respect to the user interface device (1) according to the fourth aspect, in the first or second aspect, the control unit (31) adjusts the second length (T12) so that the longer the distance (L1) between the input unit (5) and the imaging unit (C1), the shorter the second length (T12).

[0129] According to this aspect, for example, when the fulcrum that supports the user interface device (1) is close to the imaging section (C1), blurring of the image (IM1) becomes less likely to occur.

[0130] With respect to the user interface device (1) according to the fifth aspect, in any one of the first to fourth aspects, the control unit (31) further adjusts the imaging interval (A1) in the imaging unit (C1) to be shorter when the functional unit (F1) is in the second state.

[0131] According to this aspect, when the functional unit (F1) is in the second state, the imaging interval (A1) is shortened, i.e., the frame rate is increased, thereby suppressing the effects of degradation of the S / N ratio caused by a shortened exposure time (T1).

[0132] The user interface device (1) according to the sixth aspect, in any one of the first to fifth aspects, further includes an image processing unit (32) that aligns a plurality of images (IM1) arranged in a time series direction (B1) captured by the imaging unit (C1) in the second state, and combines the plurality of images (IM1) together.

[0133] According to this embodiment, it is possible to suppress the influence of deterioration of the S / N ratio caused by a shortened exposure time (T1).

[0134] Regarding the user interface device (1) according to the seventh aspect, in any one of the first to sixth aspects, the functional unit (F1) includes a display unit (20). The second state is a state in which predetermined information prompting an input is displayed on the screen. The first state is a state in which the predetermined information is not displayed.

[0135] According to this embodiment, the predetermined information prompting the user to input is displayed on the screen, so that the user can easily distinguish between the first state and the second state.

[0136] In the seventh aspect of the user interface device (1) according to the eighth aspect, the predetermined information includes an icon (G1) having an input function as the input unit (5). The control unit (31) determines the second length (T12) based on the positional relationship between the display position of the icon (G1) and the imaging unit (C1).

[0137] According to this embodiment, blurring of the image (IM1) becomes less likely to occur.

[0138] A user interface device (1) according to a ninth aspect includes an imaging unit (C1), an input unit (5) that accepts an input, a functional unit (F1), and an estimation unit (33). The functional unit (F1) has, as operation states, a first state in which input to the input unit (5) is not prompted, and a second state in which input is prompted. The estimation unit (33) estimates the state of a subject (M1) included in an image (IM1) based on an image (IM1) captured by the imaging unit (C1). When a specific condition is satisfied, the estimation unit (33) estimates the state of the subject (M1) by excluding a specific image (IM1) captured by the imaging unit (C1) when the functional unit (F1) is in the second state.

[0139] According to this aspect, the state of the subject (M1) is estimated excluding the specific image (IM1) captured in the second state in which the functional unit (F1) is encouraging input. Therefore, even if the imaging unit (C1) itself shakes and the specific image (IM1) becomes blurred when the user inputs to the user interface device (1), it is possible to suppress a decrease in the reliability of the estimation result of the estimation unit (33) due to the specific image (IM1). As a result, the state is less susceptible to the influence caused by the shaking of the imaging unit (C1) that may occur during input.

[0140] Regarding the user interface device (1) according to the tenth aspect, in the ninth aspect, the specific condition is a condition related to the positional relationship between the input section (5) and the imaging section (C1) at the time of capturing the specific image (IM1).

[0141] According to this embodiment, it is possible to further prevent the reliability of the estimation result of the estimating unit (33) from decreasing.

[0142] With respect to the user interface device (1) of the eleventh aspect, in the ninth or tenth aspect, the specific condition is that the distance (L1) between the input section (5) and the imaging section (C1) is shorter than a predetermined distance at the time when the specific image (IM1) is captured.

[0143] According to this aspect, for example, when the fulcrum that supports the user interface device (1) is distant from the imaging section (C1), it is possible to further suppress a decrease in the reliability of the estimation result of the estimating section (33).

[0144] With respect to the user interface device (1) of the twelfth aspect, in the ninth or tenth aspect, the specific condition is that the distance (L1) between the input section (5) and the imaging section (C1) is longer than a predetermined distance at the time when the specific image (IM1) is captured.

[0145] According to this aspect, for example, when the fulcrum that supports the user interface device (1) is close to the imaging section (C1), it is possible to further suppress a decrease in the reliability of the estimation result of the estimating section (33).

[0146] Regarding the user interface device (1) according to the thirteenth aspect, in any one of the ninth to twelfth aspects, the functional unit (F1) includes a display unit. The second state is a state in which predetermined information for prompting input is displayed on a screen. The first state is a state in which the predetermined information is not displayed.

[0147] According to this embodiment, the predetermined information prompting the user to input is displayed on the screen, so that the user can easily distinguish between the first state and the second state.

[0148] Regarding the user interface device (1) according to the fourteenth aspect, in the thirteenth aspect, the predetermined information includes an icon (G1) having an input function as the input unit (5). The specific condition is a condition regarding the positional relationship between the icon (G1) and the imaging unit (C1) at the time of capturing the specific image (IM1).

[0149] According to this embodiment, blurring of the image (IM1) becomes less likely to occur.

[0150] The user interface device (1) according to the 15th aspect, in any one of the first to fourteenth aspects, further includes a concentration level estimation unit (estimation unit 33) that estimates the concentration level of a subject (M1) included in an image (IM1) captured by an imaging unit (C1) based on the image (IM1).

[0151] According to this aspect, the estimation result of the concentration level is less susceptible to the influence caused by shaking of the imaging section (C1) that may occur during input.

[0152] A control method according to a sixteenth aspect is a control method for a user interface device (1) including an imaging unit (C1) and an input unit (5). The control method includes a functioning step and an adjustment step. The functioning step has, as operation states, a first state in which input to the input unit (5) is not prompted, and a second state in which input is prompted. The adjustment step adjusts the exposure time (T1) of the imaging unit (C1). In the adjustment step, when the functioning step is in the first state, the exposure time (T1) is adjusted to a first length (T11), and when the functioning step is in the second state, the exposure time (T1) is adjusted to a second length (T12) shorter than the first length (T11).

[0153] According to this aspect, it is possible to provide a control method capable of reducing the influence of shaking of the imaging section (C1) that may occur during input.

[0154] A program according to a seventeenth aspect is a program for causing one or more processors to execute the control method according to the sixteenth aspect.

[0155] According to this aspect, it is possible to provide a function that can reduce the influence of shaking of the imaging section (C1) that may occur during input.

[0156] The configurations according to the second to eighth and fifteenth aspects are not essential for the user interface device (1) according to the first aspect and may be omitted as appropriate. The configurations according to the tenth to fifteenth aspects are not essential for the user interface device (1) according to the ninth aspect and may be omitted as appropriate. [Explanation of symbols]

[0157] 1. User Interface Device 20 Display section 31 Control Unit 32 Image processing section 33 Estimation part (concentration degree estimation part) 5 Input section A1 Imaging Interval B1 Time series direction C1 Imaging unit F1 Functional section G1 Icon IM1 Images L1 distance M1 Target T1 Exposure time T11 First length T12 Second length

Claims

1. An imaging unit; An input unit that accepts input; a functional unit having, as operation states, a first state in which the input to the input unit is not prompted, and a second state in which the input is prompted; A control unit that adjusts an exposure time of the imaging unit; an estimation unit that estimates a state of a subject based on an image captured by the imaging unit; Equipped with the control unit adjusts the exposure time to a first length when the functional unit is in the first state, and adjusts the exposure time to a second length shorter than the first length when the functional unit is in the second state. User interface device.

2. The estimation unit detects a position or size of at least a face or fingers of the subject that is the subject included in the image, and extracts biometric information including information regarding at least one of facial expression, facial direction, posture, movement, and change in movement of the subject based on the detection result. The user interface device according to claim 1 .

3. An imaging unit; An input unit that accepts input; a functional unit having, as operation states, a first state in which the input to the input unit is not prompted, and a second state in which the input is prompted; A control unit that adjusts an exposure time of the imaging unit; A concentration level estimation unit that estimates a concentration level of a subject included in an image captured by the imaging unit based on the image; Equipped with the control unit adjusts the exposure time to a first length when the functional unit is in the first state, and adjusts the exposure time to a second length shorter than the first length when the functional unit is in the second state. User interface device.

4. The control unit determines the second length based on a positional relationship between the input unit and the imaging unit. The user interface device according to any one of claims 1 to 3.

5. The control unit adjusts the second length so that the shorter the distance between the input unit and the imaging unit is, the shorter the second length becomes. The user interface device according to any one of claims 1 to 4.

6. The control unit adjusts the second length so that the second length becomes shorter as the distance between the input unit and the imaging unit becomes longer. The user interface device according to any one of claims 1 to 4.

7. Furthermore, the control unit adjusts the imaging interval of the imaging unit to be shorter when the functional unit is in the second state. The user interface device according to any one of claims 1 to 6.

8. and an image processing unit that performs position alignment on a plurality of images captured by the imaging unit in the second state and arranged in a time series direction, and combines the plurality of images together. The user interface device according to any one of claims 1 to 7.

9. The functional unit includes a display unit, The second state is a state in which predetermined information prompting the input is displayed on a screen, The first state is a state in which the predetermined information is not displayed. The user interface device according to any one of claims 1 to 8.

10. the predetermined information includes an icon having an input function as the input unit, The control unit determines the second length based on a positional relationship between a display position of the icon and the imaging unit. The user interface device according to claim 9.

11. A control method for a user interface device having an imaging unit and an input unit, comprising: a functional step having, as operation states, a first state in which no input to the input unit is prompted, and a second state in which the input is prompted; an adjustment step of adjusting an exposure time of the imaging unit; an estimating step of estimating a state of a subject based on an image captured by the imaging unit; Including, In the adjustment step, the exposure time is adjusted to a first length when the functional step is in the first state, and the exposure time is adjusted to a second length shorter than the first length when the functional step is in the second state. Control methods.

12. The estimation step includes detecting the position or size of at least the face or fingers of the subject that is the subject included in the image, and extracting biometric information including information regarding at least one of facial expression, facial direction, posture, movement, and changes in movement of the subject based on the detection result. The control method according to claim 11.

13. A control method for a user interface device having an imaging unit and an input unit, comprising: a functional step having, as operation states, a first state in which no input to the input unit is prompted, and a second state in which the input is prompted; an adjustment step of adjusting an exposure time of the imaging unit; A concentration level estimation step of estimating a concentration level of a subject included in an image captured by the imaging unit based on the image; Including, In the adjustment step, the exposure time is adjusted to a first length when the functional step is in the first state, and the exposure time is adjusted to a second length shorter than the first length when the functional step is in the second state. Control methods.

14. A program for causing one or more processors to execute the control method described in any one of claims 11 to 13.

Images