Information processing device, information processing method, and program

Abstract

To provide a technique for improving movement control of an index on a display unit according to the facial orientation of a user.SOLUTION: An information processing apparatus is provided with a display unit for displaying a movable index, a detection unit for detecting the facial orientation of a user, and a movement control unit for suppressing movement of the index on the display unit when the facial orientation of the user detected by the detection unit is oriented within a predetermined angular range from a frontal orientation.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program for an operation using face detection. [Background technology]

[0002] 2. Description of the Related Art Various techniques have been proposed for operating an information processing device such as a personal computer based on the results of detecting a person's face or gestures, etc., captured by an imaging device such as a camera.

[0003] Patent Document 1 proposes a technology that makes it easier for users to operate using gestures by detecting specific gestures. Patent Document 2 also proposes a technology that provides a mouse alternative that enables people with disabilities who have difficulty using both hands to operate programs via a GUI (Graphical User Interface) without the need for attachments. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2017-004553 [Patent Document 2] Japanese Patent Application Laid-Open No. 2007-310914 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when using the above-mentioned conventional technology to move an indicator such as a pointer displayed on a display according to the direction of the user's face, if the user unintentionally turns their face in a direction different from the direction in which they want the indicator to move, the indicator will also move in a direction that the user does not want in accordance with the direction of the user's face, which may make the technology difficult for the user to use.

[0006] Therefore, the present invention has been made in consideration of the above circumstances, and provides a technique for improving the control of the movement of an index on a display unit in accordance with the orientation of the user's face. [Means for solving the problem]

[0007] In order to achieve the above object, the present invention employs the following configuration.

[0008] According to one aspect of the present invention, there is provided an information processing device comprising: a display unit that displays a movable index; a detection unit that detects the orientation of a user's face; and a movement control unit that suppresses movement of the index on the display unit when the orientation of the user's face detected by the detection unit is within a predetermined angle range from a frontal orientation. As a result, even if a user unintentionally turns their face in a direction different from a direction in which they want the index on the display unit to move, the index on the display unit will not move if the angle of the orientation of the face with respect to the frontal orientation is small, thereby suppressing a phenomenon in which the user unintentionally moves their face to move the index.

[0009] The information processing device may further include a notification unit that notifies the user that movement of the index is suppressed when the orientation of the user's face detected by the detection unit is within the predetermined angle range from a frontal orientation. This allows the user to understand whether or not the index displayed on the display unit can be moved with the current orientation of the user's face. The movement control unit may also change the predetermined angle range depending on the distance between the detection unit and the user. This optimizes the sensitivity of the start of movement of the index when the user changes the orientation of the user's face by increasing the threshold angle of the predetermined angle range when the user is closer to the detection unit and decreasing the threshold angle when the user is farther from the detection unit. It can be made into

[0010] The present invention can also be understood as an information processing method including at least a part of the above-described processing, a program for causing a computer to execute such a method, or a computer-readable recording medium on which such a program is non-temporarily recorded. The above-described configurations and processing can be combined with each other to constitute the present invention as long as no technical contradiction occurs. [Effects of the Invention]

[0011] According to the present invention, it is possible to improve the control of the movement of the indicator on the display unit in accordance with the direction of the user's face. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing a schematic configuration example of a PC to which the present invention is applied. [Figure 2] FIG. 2 is a block diagram illustrating an example of a PC according to one embodiment. [Figure 3] FIG. 3 is a flowchart showing an example of a processing flow of a PC according to an embodiment. [Figure 4] FIG. 4 is a flowchart showing an example of a processing flow of a PC according to an embodiment. [Figure 5] 5A and 5B are examples of a display on the display unit and an image of a user output from a camera in one embodiment. [Figure 6] 6A to 6D are examples of the display on the display unit and an image of the user output from the camera in one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] <Application example> An application example of the present invention will be described below. In conventional technology, when an indicator such as a pointer displayed on a display is moved in accordance with the direction of the user's face, if the user unintentionally turns their face in a direction different from the direction in which they want the indicator to move, the indicator also moves in a direction that the user does not want, in accordance with the direction of the user's face, which can make the device difficult for the user to use.

[0014] Fig. 1 is a diagram schematically illustrating an example configuration of a PC 100 to which the present invention is applied. In the example configuration shown in Fig. 1, the PC 100 is equipped with a camera 110. The camera 110 has a function of capturing an image of the face of a user 200 of the PC 100, and the PC 100 processes the image of the face of the user 200 captured by the camera 110 to identify the orientation of the face of the user 200, and controls the movement of an indicator such as a pointer displayed on a display unit such as a display of the PC 100 in accordance with the orientation of the face of the user 200. Specific content of the processing executed by the PC 100 will be described later.

[0015] PC 100 performs face detection processing on the image of user 200's face captured by camera 110, and if the orientation of the identified user 200's face is within a predetermined angle range from the front orientation, it suppresses movement of an index on the display unit. As a result, even if user 200 unintentionally turns his face in a direction different from the direction in which he wants to move the index on the display unit of PC 100, the index will not move if the angle of the face orientation with respect to the front orientation is small, so it is possible to suppress a phenomenon in which user 200 accidentally moves his face and moves the index.

[0016] <Description of the embodiment> An embodiment of the technology disclosed herein will be described. An information processing device according to this embodiment is assumed to be a PC 100 equipped with a camera 110 shown in FIG. 1 as an example. FIG. 2 is a block diagram showing an example of the configuration of the PC 100 according to this embodiment. As shown in FIG. 2, the PC 100 The device has a camera 110, a control unit 120, a storage unit 130, a communication unit 140, an input unit 150, and a display unit 160.

[0017] The camera 110 functions as a detection unit that detects the orientation of the face of the user 200, captures an image of the face of the user 200 using the PC 100, and outputs the captured image to the control unit 120. The control unit 120 includes a central processing unit (CPU), random access memory (RAM), read-only memory (ROM), and the like, and controls each unit in the PC 100 and performs various information processing. The control unit 120 has an image processing unit 121, an identification unit 122, and a display control unit 123. The image processing unit 121 processes the image output from the camera 110 and performs face detection processing of the user 200. The identification unit 122 identifies the orientation of the face of the user 200 based on the result of the face detection processing by the image processing unit 121, and performs determination processing on whether to suppress movement of an index displayed on the display unit 160 based on the identified face orientation. The display control unit 123 is a movement control unit that executes movement control processing of an index displayed on the display unit 160 based on the result of the determination processing by the identification unit 122. The display control unit 123 also functions as a notification unit that notifies the user that movement of an index displayed on the display unit 160 will be suppressed. In the following description, suppression of movement of an index is assumed to be stopping the movement of the index, but suppression of movement of an index can also be achieved by movement control such as slowing down the movement speed instead of stopping the movement of the index.

[0018] The storage unit 130 stores programs executed by the control unit 120 and various data used in the processes executed by the control unit 120. For example, the storage unit 130 is an auxiliary storage device such as a hard disk drive or a solid state drive.

[0019] The communication unit 140 communicates with various external devices (not shown) of the PC 100. The input unit 150 can be operated by the user of the PC 100, and has a function of inputting instructions related to the processing to be executed by the control unit 120. Examples of the input unit 150 include a keyboard and a pointing device. The display unit 160 displays information related to the processing results of the control unit 120. Note that the information related to the processing results by the control unit 120 may be stored in the storage unit 130 and displayed on the display unit 160 at any timing.

[0020] 3 and 4 are flowcharts showing an example of a processing flow of PC 100. As an example, assume that user 200 of PC 100 operates input unit 150 of PC 100 to move a pointer to a keyboard displayed on display unit 160 to perform input. PC 100 executes the processing of the flowchart shown in FIG. 3 and the processing of the flowchart shown in FIG. 4 in parallel. First, with reference to FIG. 3, the processing executed by control unit 120 will be described. As an example, control unit 120 executes the processing of the flowchart shown in FIG. 3 for each image output from camera 110. However, control unit 120 does not need to execute the processing of the flowchart shown in FIG. 3 for all images output from camera 110. Control unit 120 may execute the processing intermittently, for example, every time a predetermined number of images are output from camera 110, or for images output from camera 110 at predetermined time intervals.

[0021] In step S101, the display control unit 123 of the control unit 120 displays an input screen on the display unit 160. At this time, the display control unit 123 also displays a pointer, which is an indicator that can be moved by a user operation, on the display unit 160. Display examples of the input screen and the pointer on the display unit 160 will be described later.

[0022] In step S102, control unit 120 acquires from camera 110 an image captured by camera 110. In this case, since the face of user 200 is within the imaging range of camera 110, camera 110 captures an image of user 200's face and outputs the captured image to control unit 120.

[0023] Next, in step S103, the image processing unit 121 of the control unit 120 executes face detection processing on the image received from the camera 110 in step S102. Since the image captured by the camera 110 includes the face of the user 200, the face of the user 200 is detected from the image by the processing of step S103.

[0024] Next, in step S104, the identification unit 122 of the control unit 120 calculates the angle of the user's 200 face relative to the front direction relative to the camera 110 based on the relative positions of the feature points of each organ of the user's 200 face, such as the eyes, nose, mouth, and ears, detected in step S103, thereby identifying the facial orientation of the user 200. Examples of feature points include, but are not limited to, the corners of the eyes, the tip of the chin, the tip of the nose, and the corners of the mouth. Calculation of the angle indicating the facial orientation can be realized based on the feature points using well-known technology, and therefore a detailed description thereof will be omitted here.

[0025] Next, in step S105, the identification unit 122 determines whether the angle indicating the facial orientation of the user 200 identified in step S104 is within a predetermined angle range. Here, the predetermined angle range is a range of facial orientation angles within which the control unit 120 does not move the pointer displayed on the display unit 160 even if the user 200 changes the facial orientation from a frontal orientation relative to the camera 110. Therefore, the larger this predetermined angle range, the greater the change in the facial orientation of the user 200 from a frontal orientation relative to the camera 110, which causes the control unit 120 to release the stop of the pointer on the display unit 160 and start moving it. Furthermore, the smaller this predetermined angle range, the less the change in the facial orientation of the user 200 from a frontal orientation relative to the camera 110, which causes the control unit 120 to release the stop of the pointer on the display unit 160 and start moving it. Therefore, the predetermined angle range may be changed as appropriate in accordance with the range of movement of the facial orientation of the user 200. In the following description, this predetermined angle range is referred to as a "dead zone" in which the pointer does not move even if the user 200 changes the direction of the face.

[0026] If the angle indicating the orientation of the face of the user 200 is within the dead zone (S105: YES), the identification unit 122 proceeds to step S106. On the other hand, if the angle indicating the orientation of the face of the user 200 is not within the dead zone (S105: NO), the identification unit 122 proceeds to step S108.

[0027] In step S106, the display control unit 123 of the control unit 120 performs control to suppress movement of the pointer displayed on the display unit 160. Specifically, the display control unit 123 maintains the pointer displayed on the display unit 160 in a stopped state at the currently displayed position. Then, the control unit 120 proceeds to step S107.

[0028] In step S107, the display control unit 123 displays a message notifying the user that the orientation of the user's face is within the dead zone on the display unit 160. An example of the display on the display unit 160 by the display control unit 123 will be described later.

[0029] Furthermore, in step S108, the display control unit 123 performs control to move the pointer displayed on the display unit 160. Specifically, the display control unit 123 moves the pointer displayed on the display unit 160 from the currently displayed position thereof, using a movement direction and movement speed determined based on the face orientation of the user 200 identified in step S104.

[0030] Here, as an example, regarding the movement speed of the pointer, let p be the number of pixels of movement of the pointer per frame based on the frame rate of the display unit 160, α be the angle of the direction of the face of the user 200, and β be the threshold angle within the dead zone (the maximum angle of the angle range considered to be the dead zone). Then, the value of p calculated by the following formula (1) can be used as the moving speed of the pointer.

number

[0031] When the process of step S107 or S108 is completed, the control unit 120 ends the process of the flowchart in FIG. 3, and repeats the process of the flowchart in FIG.

[0032] Next, the processing executed by the control unit 120 will be described with reference to the flowchart of FIG. 4. In this embodiment, it is assumed that the processing executed by the control unit 120 corresponds to the hand gestures identified by the processing described below. It is also assumed that information indicating the correspondence between the hand gestures and the processing to be executed is stored in advance in the storage unit 130. As an example, in the flowchart of FIG. 4, the control unit 120 executes the processing of the flowchart of FIG. 4 on a plurality of images sequentially output from the camera 110, and the processing of the PC 100 is executed based on the hand gestures identified from this series of images. It is to be noted that the execution timing of the processing of the flowcharts of FIG. 3 and FIG. 4 and the images to be executed can be set as appropriate.

[0033] First, in step S201, control unit 120 acquires from camera 110 an image captured by camera 110. Here, similar to when the processing of the flowchart in FIG. 3 is executed, camera 110 captures an image of user 200's face and outputs the captured image to control unit 120.

[0034] In step S202, the image processing unit 121 executes hand detection processing to detect a human hand in each image acquired in step S201. Next, in step S203, the image processing unit 121 determines whether a human hand has been detected by the processing of step S202. If a human hand has been detected in the image (S203: YES), the image processing unit 121 proceeds to the processing of step S204. For example, if the user 200's hand is not included in the image because the user 200 has lowered their hand, the image processing unit 121 cannot detect the hand from the image by the hand detection processing. Therefore, if a hand is not detected in the image (S203: NO), the image processing unit 121 ends the processing of this flowchart.

[0035] Next, in step S204, the identification unit 122 identifies the type of gesture of the hand movement in each image detected in step S202. Examples of hand gestures that can be identified here include well-known gestures such as an open palm gesture, a clenched hand gesture, a waving gesture, and a finger gesture. After identifying the gesture, the identification unit 122 proceeds to step S205.

[0036] In step S205, the identification unit 122 determines whether the hand movement identified in step S204 is a predetermined gesture. Here, the predetermined gesture is a gesture associated with a process to be executed by the control unit 120. As described above, information indicating the correspondence between hand gestures and processes to be executed is stored in the storage unit 130. Therefore, the identification unit 122 refers to the information stored in the storage unit 130 and determines whether the hand movement identified in step S204 is a predetermined gesture indicated by the referenced information. If the identified hand movement is a predetermined gesture (S205: YES), the identification unit 122 The process proceeds to step S206. On the other hand, if the identified hand movement is not a predetermined gesture (S205: NO), the identification unit 122 ends the process of this flowchart. Then, in step S206, the control unit 120 refers to information indicating the correspondence between hand gestures and processes to be executed, which is stored in the storage unit 130, and executes the process corresponding to the gesture identified in step S204. When the process of step S206 is completed, the control unit 120 ends the process of the flowchart of FIG. 4. Then, the control unit 120 repeatedly executes the process of the flowchart of FIG. 4 for another series of images output from the camera 110.

[0037] Next, with reference to FIGS. 5A, 5B, and 6A to 6D, examples of a screen displayed on display unit 160 and an image of user 200 output from camera 110 when the above processing is executed will be described. As shown in FIG. 5A, an input screen 501 displayed on display unit 160 displays a keyboard 502, an input area 503, and a pointer 505. Input area 503 is an area where characters and the like are input in response to key input on keyboard 502. A cursor 504, which is the character input position, is also displayed in input area 503. Here, the position of pointer 505 in FIG. 5A is assumed to be the initial position of pointer 505. Note that user 200 can also operate input unit 150 to move pointer 505 to a desired position and change the moved position to the initial position.

[0038] FIG. 5B shows an image 502 of user 200 output from camera 110 when the display on input screen 501 is in the state shown in FIG. 5A. As shown in FIG. 5B, in image 502, user 200 is slightly shifting the orientation of face 602 from the front with respect to the camera, and the angle of the orientation of face 602 at this time is within the dead zone. In the case shown in FIGS. 5A and 5B, in the processing of the flowchart in FIG. 3 described above, it is determined in step S105 that the angle indicating the orientation of user 200's face is within the dead zone (S105: YES). As a result, even if user 200 shifts the orientation of face 602 from the front with respect to camera 110, display control unit 123 keeps pointer 505 stationary on input screen 501 (step S106), and icon 506, which is an indicator indicating that pointer 505 is inhibited (here, stopped), is displayed on input screen 501 (step S107). Therefore, by checking the input screen 501, the user 200 can recognize that the pointer 505 will not move with the current orientation of the face 602.

[0039] Next, after the state of FIG. 5B, as shown in FIG. 6B, in the image 502 output by the camera 110, the user 200 further shifts the orientation of the face 602 from the front, and the angle of the orientation of the face 602 is not within the dead zone. In this case, in the processing of the flowchart of FIG. 3 described above, it is determined in step S105 that the angle indicating the orientation of the face of the user 200 is not within the dead zone (S105: NO). As a result, as shown in FIG. 6A, on the input screen 501, the pointer 505 moves in a direction and at a speed determined based on the orientation of the face 602 of the user 200 (step S108). Note that in the cases shown in FIGS. 6A and 6B, the user 200 points the face 602 in a direction to the right and toward the camera 110, and therefore the pointer 505 moves in a direction to the right and toward the left on the input screen 501 from the position shown in FIG. 5A. Furthermore, when the angle indicating the orientation of the face of the user 200 is not within the dead zone, the display control unit 123 hides the icon 506 that was displayed on the pointer 505 in FIG. 5A.

[0040] Furthermore, when the user 200 moves the pointer 505 on the input screen 501 to a desired position on the keyboard 502 as shown in FIG. 6A, the user 200 turns the face 602 back to face the camera 110 as shown in FIG. 6D, moves the hand 603 into the imaging range of the camera 110, and makes a gesture (here, a gesture of fisting hands). Also, here, it is assumed that the gesture of fisting hands is associated with the process of pressing a key on the keyboard 502 on the input screen 501. In this case, in the process of the flowchart of FIG. 3, step S In step S105, it is determined that the angle indicating the orientation of the face of the user 200 is within the dead zone (S105: YES). As a result, the display control unit 123 keeps the pointer 505 stationary on the input screen 501 (step S106), and an icon 506, which is an indicator indicating that the pointer 505 is not moving, is displayed on the input screen 501 (step S107).

[0041] Furthermore, in the state shown in FIG. 6D , the process of the flowchart in FIG. 4 determines that the gesture of clenching hands is a predetermined gesture (step S205), and a process corresponding to the determined gesture (here, a process of pressing a key on keyboard 502) is executed (S206). As a result, as shown in FIG. 6C , on input screen 501, a key on keyboard 502 where pointer 505 is located is pressed, and a character is input into input area 503. Therefore, since pointer 505 does not move with the current orientation of face 602 ( FIG. 6D ), a phenomenon in which pointer 505 deviates from the desired position on input screen 501 due to unintentional movement of face 602 from the front is suppressed. As a result, user 200 can accurately perform an operation of moving pointer 505 while changing the orientation of face 602 to input characters on keyboard 502.

[0042] <Other> The above-described embodiment merely exemplifies a configuration example of the present invention. The present invention is not limited to the above-described specific embodiment, and various modifications are possible within the scope of the technical concept thereof. For example, in the PC 100 according to the above-described embodiment, when the orientation of the user 200's face is within the dead zone, the display unit 160 displays the icon 506 as an indicator notifying the user 200 of this. However, instead of or in addition to this, the PC 100 may be configured to notify the user 200 that the orientation of the user 200's face is within the dead zone by changing the display of a part of the input screen 501 or by notifying the user 200 by sound from a speaker (not shown) connected to the PC 100.

[0043] Furthermore, in the above embodiment, the angle range of the face direction that defines the dead zone may be changed according to the distance between camera 110 and user 200. For example, in step S102, control unit 120 calculates the distance between camera 110 and user 200 based on the size of user 200's face in the image acquired from camera 110, and changes the angle of the face direction that serves as the threshold for determining that the position is within the dead zone in step S105 according to the calculated distance. This is expected to optimize the sensitivity of the start of movement of pointer 505 when user 200 changes the direction of his or her face by increasing the threshold angle for determining that the position is within the dead zone when user 200 is closer to camera 110 and decreasing the threshold angle for determining that the position is within the dead zone when user 200 is farther from camera 110.

[0044] Furthermore, in the above embodiment, the movement speed of pointer 505 may be determined according to the screen size of display unit 160. For example, before starting the processing of the flowcharts in Figures 3 and 4, control unit 110 may acquire information about the screen size of display unit 160 and change the value of c in equation (1) above according to the screen size. This allows the movement speed of pointer 505 to be set to increase as the screen size increases, thereby preventing the phenomenon in which it takes a long time to move pointer 505 on the screen even when the screen size increases, and is expected to improve work efficiency when moving pointer 505 according to the direction of the face.

[0045] <Appendix 1> A display unit (160) that displays a movable indicator; a detection unit (110) that detects the orientation of the user's face; a movement control unit (123) that suppresses movement of the indicator on the display unit when the orientation of the user's face detected by the detection unit is within a predetermined angle range from a frontal orientation; An information processing device (100) comprising:

[0046] <Appendix 2> A display step (S101) of displaying a movable indicator on a display unit; a detection step (S104) of detecting the orientation of the user's face; and a movement control step (S106) of suppressing movement of the indicator on the display unit when the detected orientation of the user's face is within a predetermined angle range from a forward orientation. [Explanation of symbols]

[0047] 100 PC, 110 camera, 120 control unit, 121 image processing unit, 122 identification unit, 123 display control unit, 160 display unit

Claims

1. a display unit that displays a movable indicator; a detection unit that detects the orientation of the user's face; a movement control unit that suppresses movement of the indicator on the display unit when the orientation of the user's face detected by the detection unit is within a predetermined angle range from a frontal orientation; Equipped with The information processing device, wherein the movement control unit changes the predetermined angle range depending on the distance between the detection unit and the user.

2. 2. The information processing device according to claim 1, further comprising a notification unit that notifies the user that movement of the indicator will be suppressed when the orientation of the user's face detected by the detection unit is within the predetermined angle range from a frontal orientation.

3. a display step of displaying a movable indicator on a display unit; a detection step of detecting the orientation of the user's face by a detection unit; a movement control step of suppressing movement of the indicator on the display unit when the detected orientation of the user's face is within a predetermined angle range from a front orientation; Including, An information processing method, wherein the movement control step changes the predetermined angle range depending on the distance between the detection unit and the user.

4. A program for causing a computer to execute each step of the information processing method according to claim 3.

Images