Information processing apparatus, information processing method, and program

Abstract

To provide an information processor for improving the determination accuracy of the concentration degree of a user, and for suppressing the increase of the workload of the information processor.SOLUTION: The information processing apparatus 10 includes the storage 21 that stores the acquired video of the user's face, and the controller 22 that determines the user's degree of concentration based on the user's blinking motion detected from α image frames among the image frames included in the video stored in the storage 21 and the change in facial expression of the user's face detected from β image frames, β being smaller than α.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program for determining a user's concentration level. [Background technology]

[0002] With the recent progress of the aging society, there is an increasing need for elderly people to maintain and restore their health through training. However, if such training is performed in a non-concentrated state, it cannot be expected to be effective. Therefore, it is desirable to be able to determine whether or not a person undergoing training is concentrating.

[0003] In response to this, configurations have been proposed that determine a user's concentration level based on an image of the user's face, as described in Patent Documents 1 and 2, for example. Patent Document 1 discloses a configuration that determines a user's concentration level based on changes in the user's facial expression detected from an image of the face. Patent Document 2 discloses a configuration that determines a user's concentration level based on the user's blinking behavior detected from an image of the face. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-64363 [Patent Document 2] Japanese Patent Publication No. 2022-175709 Summary of the Invention [Problem to be solved by the invention]

[0005] When determining the concentration level of a user, it is possible to combine the configurations of Patent Document 1 and Patent Document 2 to determine the concentration level based on the blinking behavior of the user and changes in facial expression. This can improve the accuracy of determining the concentration level of the user compared to a configuration that determines the concentration level based on only one of the above.

[0006] However, when an information processing device analyzes both blinking behavior and changes in facial expression from an image of a user's face, the workload of the information processing device increases compared to a configuration in which only one of the analysis processes is performed, which may result in a decrease in the speed at which the information processing device determines the concentration level or cause the information processing device to freeze.

[0007] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an information processing device that can improve the accuracy of determining the degree of concentration of a user and can suppress an increase in the workload of the information processing device. [Means for solving the problem]

[0008] A typical configuration of an information processing device according to the present invention for solving the above problem is characterized by comprising a storage means for storing acquired images of the user's face, and a judgment means for judging the user's level of concentration based on the user's blinking action detected from α image frames among the image frames contained in the image stored in the storage means, and changes in the user's facial expression detected from β image frames that are less than α.

[0009] According to the present invention, in an information processing device, it is possible to improve the accuracy of determining the concentration level of a user and to suppress an increase in the workload of the information processing device. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a configuration of an information processing device according to an embodiment of the present invention; [Figure 2] 1 is a list of action units. [Figure 3] 10 is a flowchart of a concentration level determination sequence executed by the information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0011] An information processing device 10 according to an embodiment of the present invention will be described below with reference to the drawings. The information processing device 10 is a device that determines a user's concentration level based on an image of the user's face, and is, for example, a computer such as a personal computer or a smartphone.

[0012] Any number of people can be users, but in this embodiment, we assume elderly people who are trying to return to a healthy state from frailty. Frailty refers to a state in which physical and mental vitality (motor function, cognitive function, etc.) declines with age, and, due to the coexistence of multiple chronic diseases, daily living functions are impaired and physical and mental fragility appears, but at the same time, it is possible to maintain and improve daily living functions with appropriate intervention and support.

[0013] Fig. 1 is a block diagram showing the configuration of an information processing device 10. As shown in Fig. 1, the information processing device 10 is configured to include a data input unit 20 for inputting data, a storage unit 21 such as a memory for storing data and computer programs, a control unit 22 such as a CPU for controlling the information processing device 10 in accordance with the computer programs stored in the storage unit 21, a monitor 23 for outputting image data, and a communication unit 24 for transmitting and receiving information to and from external devices. The data input unit 20, storage unit 21, control unit 22, monitor 23, and communication unit 24 are configured to be able to transmit and receive data to and from each other via a system bus 25.

[0014] The control unit 22 (determination means) controls each component constituting the information processing device 10, reads and writes data stored in the memory unit 21, determines the user's concentration level, etc. The control unit 22 also controls external devices by sending and receiving signals via the communication unit 24. The data input unit 20 is a keyboard, a mouse, etc. when the information processing device 10 is a personal computer. When the information processing device 10 is a smartphone, the monitor 23 is generally configured as a touch panel display, so the monitor 23 also serves as the data input unit 20.

[0015] Various data are stored in the storage unit 21 (storage means), including an execution program for a concentration level determination sequence (to be described later) executed by the control unit 22 to determine the user's concentration level, and an application program used by the operator who operates the information processing device 10 when this program is executed.

[0016] Furthermore, two AI engines, a blink determination engine 21a and a facial expression determination engine 21b, which are used to determine the user's concentration level, are stored in the storage unit 21. The blink determination engine 21a and the facial expression determination engine 21b are each trained models that have been trained by machine learning (deep learning) using as training data a combination of video data of the faces of multiple subjects and self-reported data on the subjects' levels of concentration while the video was being captured.

[0017] The blink determination engine 21a detects blinking from multiple image frames included in the input facial video data, and then determines the level of concentration based on the detected blinking into three levels: "non-concentrated," "normal," and "concentrated."

[0018] The facial expression determination engine 21b detects a facial expression from one image frame included in the input facial video data. Then, based on the detected facial expression, it determines the concentration level in three levels: "unfocused," "normal," and "concentrated." When detecting facial expressions, the facial expression determination engine 21b is configured to evaluate all or some of the action units AU1 to AU46 shown in FIG. 2, which are defined by the FACS (Facial Action Coding System). In this embodiment, the open source Py-Feat is used to detect faces, landmarks, and action units from image frames, and the detected data is converted into visualized data. This visualized data becomes data on the action units to be evaluated. FACS is a system of movements of specific parts of the face that appear when expressing facial expressions, and is widely used to describe and identify facial expressions. The magnitude relationship of the above-mentioned concentration levels is "unfocused" < "normal" < "concentrated."

[0019] Furthermore, a camera 1 as an external device is electrically connected to the information processing device 10 via a wired cable. The camera 1 captures an image of the user under the control of the control unit 22 and transmits the captured image data to the information processing device 10. If the information processing device 10 has a function for capturing an image, this function may be used instead of the camera 1. Furthermore, the camera 1 may be connected to the information processing device 10 via a communication network such as the Internet.

[0020] Next, a concentration level determination sequence for determining the user's concentration level, which is executed by the control unit 22, will be described with reference to the flowchart shown in Fig. 3. The following describes, as an example, how the user's concentration level is determined during a 93-second training session for recovering from frailty to a healthy state.

[0021] 3, first, the control unit 22 starts an application program for determining the user's concentration level in response to an operation of the data input unit 20 by the operator of the information processing device 10 (S1). As a result, an operation screen for the application program is displayed on the monitor 23 of the information processing device 10. This operation screen includes a start button for starting the determination of the concentration level and setting buttons for performing various settings related to the determination of the concentration level.

[0022] Next, control unit 22 controls camera 1 in response to the operator's selection of the start button, causing camera 1 to start capturing an image of the user's face (S2). Control unit 22 also switches the display screen on monitor 23 to a screen indicating that concentration level determination is in progress (S3). This screen includes an end button that can be selected to end the determination of the concentration level.

[0023] The operator selects the end button when the 93 seconds of training is completed. In response to the selection of the end button, the control unit 22 controls the camera 1 to stop capturing video of the user's face (S4). In this embodiment, the camera 1 captures video of the user's face during the 93 seconds of training at a frame rate of 30 fps.

[0024] Next, the control unit 22 controls the camera 1 to transmit the captured image data of the user's face from the camera 1 to the information processing device 10. The control unit 22 also controls the information processing device 10 to store the received image data in the storage unit 21 (S5).

[0025] Next, control unit 22 inputs the video data of the user's face stored in storage unit 21 to blink determination engine 21a also stored in storage unit 21. Blink determination engine 21a detects the user's blinking behavior from α image frames included in the input video data, and determines the user's concentration level into three levels: "non-concentrated," "normal," and "concentrated" based on the detected blinking behavior. In this embodiment, blink determination engine 21a detects the user's blinking behavior based on 2,790 image frames, which are all image frames included in the 93 seconds of video data.

[0026] That is, in the above step, the control unit 22 uses the blink determination engine 21a to determine the user's concentration level based on the user's blinking action detected from α image frames included in the video data of the user's face stored in the storage unit 21 (S6). If the control unit 22 determines that the user's concentration level is "non-concentration," it displays the determination result on the monitor 23 and ends the concentration level determination sequence (S7, S9).

[0027] On the other hand, if the control unit 22 determines that the user's concentration level is "normal" or "concentrated," it inputs the video data of the user's face stored in the storage unit 21 to the facial expression determination engine 21b also stored in the storage unit 21. The facial expression determination engine 21b detects the user's facial expression for each of 93 image frames corresponding to image frames every 1.0 second out of 2,790 image frames included in the input video data, and determines the user's concentration level in three levels: "non-concentrated," "normal," or "concentrated" based on the detected facial expression. In other words, the facial expression determination engine 21b outputs 93 determination results as concentration level determination results.

[0028] If there is a time series transition of "concentrated" → "normal" → "non-concentrated" among the 93 concentration level determination results output from the facial expression determination engine 21b, the control unit 22 determines the user's concentration level to be "non-concentrated." On the other hand, if there is no such time series transition, the control unit 22 compares the number of "normal" and "concentrated" among the 93 determination results. If there are more "concentrated" than "normal," the control unit 22 determines the user's concentration level to be "concentrated," and if there are more "normal" than "concentrated," the control unit 22 determines the user's concentration level to be "normal." Note that if the number of "concentrated" and "normal" is the same, the control unit 22 determines the user's concentration level to be "concentrated."

[0029] That is, in the above step, the control unit 22 uses the facial expression determination engine 21b to determine the concentration level of the user based on changes in the user's facial expression detected from β image frames, which are fewer than the α frames, included in the video data of the user's face stored in the storage unit 21 (S8). Thereafter, the control unit 22 displays the determination result on the monitor 23, and ends the concentration level determination sequence (S9).

[0030] As described above, the control unit 22 executes the concentration level determination sequence and determines the user's concentration level based on the user's blinking and facial expression changes. Therefore, compared to a configuration in which the concentration level is determined based on only either the blinking or facial expression changes, the accuracy of determining the user's concentration level can be improved.

[0031] Furthermore, the control unit 22 determines the user's concentration level based on the user's blinking detected from α image frames among the image frames included in the video of the user's face stored in the storage unit 21 and on changes in the user's facial expression detected from β image frames, which is less than α. This allows for fewer image frames to be used for the detection compared to a configuration in which the user's concentration level is determined based on the user's blinking and changes in facial expression detected from the same number of image frames, thereby reducing the workload of the information processing device 10. Note that facial expressions generally do not change instantaneously like blinking. Therefore, reducing the number of image frames used to detect changes in facial expression as described above has little effect on the accuracy of the concentration level determination.

[0032] Therefore, the information processing device 10 according to the present embodiment can improve the accuracy of determining the concentration level of the user and suppress an increase in the workload of the information processing device 10. As a result, it is possible to suppress a decrease in the speed at which the information processing device 10 determines the concentration level, freezing of the information processing device 10, and the like.

[0033] Furthermore, when the user's concentration level determined based on the user's blinking is "non-concentrated," the control unit 22 does not determine the concentration level based on changes in the user's facial expression, and causes the monitor 23 to display the determination result. On the other hand, when the user's concentration level determined based on the user's blinking is "normal" or "concentrated," the control unit 22 determines the concentration level based on changes in the user's facial expression, and causes the monitor 23 to display the determination result. That is, when the concentration level determined based on the user's blinking is less than a predetermined level, the control unit 22 outputs a signal for displaying the determination result without determining the concentration level based on changes in the user's facial expression, and when the concentration level is equal to or greater than the predetermined level, the control unit 22 outputs a signal for displaying the determination result. This provides the following effects.

[0034] That is, when determining the user's concentration level in order to increase the training effect, it is important to detect when the user's concentration level is "non-concentrated" and determine the timing for taking a break or changing the training menu. Also, as described above, when determining the concentration level based on facial expression, the facial expression determination engine 21b is used to evaluate the action units AU1 to AU46 shown in Fig. 2, which imposes a greater workload on the information processing device 10 than when only the blink action is evaluated using the blink determination engine 21a.

[0035] On the other hand, when the concentration level determined based on the blinking action of the user is less than a predetermined level, the control unit 22 does not determine the concentration level based on the change in the user's facial expression, but instead displays the determination result on the monitor 23, thereby notifying the user that the concentration level is "non-concentrated" without determining the concentration level based on the change in the facial expression that indicates a heavy workload. Therefore, an increase in the workload of the information processing device 10 can be further suppressed.

[0036] Although the present embodiment has been described with respect to a configuration in which the determination result of the user's concentration level is displayed on the monitor 23 of the information processing device 10, the present invention is not limited to this, and the following configuration may also be used. That is, video data of the user's face transmitted to the information processing device 10 from an external device such as a personal computer or a smartphone via a communication network such as the Internet is stored in the storage unit 21. The control unit 22 then executes the concentration level determination sequence using the video data stored in the storage unit 21, and transmits a signal to the external device to display the determination result, which can also provide the same effect as described above. [Explanation of symbols]

[0037] 10...information processing device, 21...storage unit (storage means), 22...control unit (determination means)

Claims

1. a storage means for storing the acquired image of the user's face; a determination means for determining a concentration level of the user based on blinking of the user detected from α image frames among the image frames included in the video stored in the storage means and changes in facial expression of the user detected from β image frames, which are less than α; An information processing device comprising:

2. The information processing device described in claim 1, characterized in that the judgment means outputs a signal to display the judgment result without judging the concentration level based on changes in the user's facial expression when the user's concentration level judged based on the user's blinking behavior is less than a predetermined value, and outputs a signal to judge the concentration level based on changes in the user's facial expression and display the judgment result when the user's concentration level judged based on the user's blinking behavior is greater than or equal to the predetermined value.

3. an acquisition step in which a computer acquires an image of a user's face; a determination step in which the computer determines the concentration level of the user based on blinking behavior of the user detected from α image frames among the image frames included in the video acquired in the acquisition step and changes in facial expression of the user detected from β image frames, which are less than α; An information processing method comprising:

4. The information processing method described in claim 3, characterized in that in the judgment process, if the user's concentration level judged based on the user's blinking behavior is less than a predetermined value, a signal is output to display the judgment result without judging the concentration level based on changes in the user's facial expression, and if the concentration level is greater than or equal to the predetermined value, a signal is output to judge the concentration level based on changes in the user's facial expression and display the judgment result.

5. an acquisition step of acquiring an image of a user's face; a determination step of determining a concentration level of the user based on blinking of the user detected from α image frames among the image frames included in the video acquired by the acquisition step, and changes in facial expression of the user detected from β image frames that are less than α; A program characterized by causing a computer to execute the above.

6. The program described in claim 5, characterized in that in the judgment process, the computer is operated so that if the user's concentration level judged based on the user's blinking behavior is less than a predetermined value, a signal is output to display the judgment result without judging the concentration level based on changes in the user's facial expression, and if the user's concentration level is greater than or equal to the predetermined value, a signal is output to judge the concentration level based on changes in the user's facial expression and display the judgment result.

Images