Information processing method, information processing device, and program

Abstract

This information processing device: detects the line-of-sight direction of a user; calculates the detection accuracy of the line-of-sight direction for each of a plurality of field-of-view regions included in the field of view of the user; and executes display control processing for controlling display content on a display screen, on the basis of control information pertaining to the detection accuracy for each of the field-of-view regions.

Description

Information processing method, information processing device, and program

[0001] The present disclosure relates to an information processing method, an information processing device, and a program.

[0002] Patent Literature 1 discloses an information processing system according to the background art. The information processing system detects a user's gaze direction based on images captured by multiple cameras and executes processing according to the user's gaze position determined from the gaze direction. The information processing system also calculates the detection accuracy of the user's gaze direction and switches the display format of the operation screen according to the detection accuracy of the gaze direction. For example, the closer the distance between the camera and the user, the higher the detection accuracy is determined, and the more icons are displayed on the operation screen.

[0003] The information processing system disclosed in Patent Document 1 does not consider controlling the display content according to the detection accuracy of the gaze direction for each of multiple visual field areas included in the user's visual field.

[0004] JP 2016-151798 A

[0005] The present disclosure aims to provide an information processing method, an information processing device, and a program that are capable of controlling the content displayed on a display screen according to the detection accuracy of the gaze direction for each field of view area.

[0006] In an information processing method according to one aspect of the present disclosure, an information processing device detects a user's gaze direction, calculates the detection accuracy of the gaze direction for each of a plurality of field of view areas included in the user's field of view, and performs a display control process that controls the display content on a display screen based on control information regarding the detection accuracy for each of the field of view areas.

[0007] FIG. 1 is a diagram showing a simplified configuration of AR glasses according to a first embodiment of the present disclosure. FIG. 1 is a schematic diagram of a left eye unit of the AR glasses viewed from the side of the user's face. FIG. 2 is a flowchart showing processing performed by a processing unit in calibration. FIG. 3 is a diagram showing an image for calibration. FIG. 4 is a diagram showing an example of accuracy information. FIG. 5 is a flowchart showing processing performed by a processing unit in gaze operation. FIG. 6 is a diagram showing an example of a display screen. FIG. 7 is a diagram showing an example of a display screen. FIG. 8 is a flowchart showing processing performed by a processing unit in gaze operation. FIG. 9 is a diagram showing an example of a display screen. FIG. 10 is a diagram showing an example of a display screen. FIG. 11 is a diagram showing a simplified configuration of AR glasses according to a fourth embodiment of the present disclosure. FIG. 12 is a flowchart showing processing performed by a processing unit in relation to guidance processing. FIG. 13 is a diagram showing an example of a display screen. FIG. 14 is a diagram showing a simplified configuration of AR glasses according to a second modified example of the fourth embodiment of the present disclosure. FIG. 15 is a flowchart showing processing performed by a processing unit.

[0008] (Foundation of the Present Disclosure) Gaze control technology for manipulating objects such as icons by the user's gaze is becoming practical.

[0009] For example, a gaze control system detects the gaze direction of a user seated facing a display by capturing an image of the user's eyes with a camera installed on the display. The gaze control system calculates the user's gaze position on the display based on the gaze direction, and performs operations such as selecting an icon displayed on the display based on the gaze position and gaze duration.

[0010] As another example, the gaze control system is mounted on glasses or goggles that use cross-reality technology such as AR (Augmented Reality), VR (Virtual Reality), or MR (Mixed Reality). The gaze control system detects the user's gaze direction by, for example, capturing an image of the eyes of a user wearing the AR glasses with an internal camera that uses a visible light camera or an infrared camera. Alternatively, the gaze control system detects the user's gaze direction by measuring the user's electro-oculography (EOG) using electrodes placed on the AR glasses and estimating the user's eye direction based on the EOG. The gaze control system calculates the user's gaze position on the display based on the gaze direction, and performs operations such as selecting an icon displayed on the display based on the gaze position and gaze duration.

[0011] To detect the user's gaze direction, an estimation model is used that estimates the gaze direction based on the eyeball direction. Because the relationship between eyeball direction and gaze direction varies from person to person, calibration is performed, for example, upon first use, to learn an estimation model tailored to each user.

[0012] However, even with calibration, the estimation accuracy varies among the multiple visual fields included in the user's field of view, resulting in variations in the accuracy of gaze direction detection among visual fields. One possible cause of this is that in the lower visual field, users tend to look down, which can cause part of the eyeball to be hidden by the eyelids or eyelashes, preventing eye direction detection. Other possible causes include pupil size, injury, amblyopia, strabismus, nystagmus, target tracking errors during calibration, or a lack of training data.

[0013] Therefore, in gaze operations, if an object such as an icon is displayed in a field of view where the detection accuracy of gaze direction is low, it is not possible to accurately detect the user's gaze direction or gaze, etc. As a result, situations arise where the operation intended by the user is not executed, or an operation unintended by the user is executed, reducing the operability of gaze operations.

[0014] In order to solve this problem, the inventor discovered that the operability of gaze operations can be improved by calculating the gaze direction detection accuracy for each field of view during calibration, and when performing gaze operations, displaying an object in a display area on the display screen corresponding to a field of view with high gaze direction detection accuracy based on the detection accuracy for each field of view, and this led to the present disclosure.

[0015] Next, each aspect of the present disclosure will be described.

[0016] In an information processing method according to a first aspect of the present disclosure, an information processing device detects a user's gaze direction, calculates the detection accuracy of the gaze direction for each of a plurality of field of view areas included in the user's field of view, and performs a display control process that controls the display content on a display screen based on control information regarding the detection accuracy for each of the field of view areas.

[0017] According to the first aspect, it is possible to control the content displayed on the display screen in accordance with the detection accuracy of the gaze direction for each field of view included in the user's field of view.

[0018] In the information processing method according to the second aspect of the present disclosure, in the first aspect, a gaze operation is further performed, which is an operation based on the detected gaze direction, and the control target in the display control process is an object for gaze operation displayed on the display screen.

[0019] According to the second aspect, the display of the object for line-of-sight operation can be controlled in accordance with the detection accuracy for each field of view area.

[0020] In the information processing method according to the third aspect of the present disclosure, in the second aspect, the display control process may control a display position of the object on the display screen.

[0021] According to the third aspect, the display position of the object for line-of-sight operation can be controlled in accordance with the detection accuracy for each field of view area.

[0022] In an information processing method according to a fourth aspect of the present disclosure, in the third aspect, the control information is accuracy information indicating the detection accuracy for each field of view area, and in the display control process, based on the accuracy information, the object is displayed in a high-precision area, which is a display area on the display screen corresponding to a field of view area in which the detection accuracy is equal to or greater than a predetermined value.

[0023] According to the fourth aspect, by displaying an object in a high-precision area corresponding to a field-of-view area where the detection accuracy is equal to or greater than a predetermined value, it is possible to improve the operability of eye-gaze operations.

[0024] An information processing method according to a fifth aspect of the present disclosure is the fourth aspect, wherein the predetermined value may differ depending on an attribute of the object.

[0025] According to the fifth aspect, the display position of an object can be controlled according to the attributes of the object, such as importance or size.

[0026] In the information processing method according to the sixth aspect of the present disclosure, in the fourth or fifth aspect, an image of the user's field of view is further acquired, and if an object to be operated by gaze contained in the image is located outside the high-precision area on the display screen, a simulated image of the object to be operated by gaze is generated, and in the display control processing, the object may include the simulated image.

[0027] According to the sixth aspect, even for a gaze operation target object whose position cannot or is difficult to move within the display screen, a simulated image can be generated, and the display control process can be executed on the generated simulated image.

[0028] In the information processing method according to the seventh aspect of the present disclosure, in the third aspect, a designated display area on the display screen in which the object is to be displayed is set based on the detection accuracy for each field of view area, the control information is area information indicating the designated display area, and in the display control processing, the object is displayed in the designated display area based on the area information.

[0029] According to the seventh aspect, the object is displayed in the designated display area that is set based on the detection accuracy, thereby improving the operability of the eye-gaze operation.

[0030] An information processing method according to an eighth aspect of the present disclosure is, in the seventh aspect, such that, in setting the display designation area, a display area on the display screen corresponding to a field of view area where the detection accuracy is equal to or greater than a first threshold value is set as a first display designation area, and a display area on the display screen corresponding to a field of view area where the detection accuracy is less than the first threshold value and equal to or greater than a second threshold value is set as a second display designation area, and in the display control process, either the first display designation area or the second display designation area is selected as the display position of the object depending on the attributes of the object.

[0031] According to the eighth aspect, the display position of an object can be controlled according to the attribute of the object, such as importance or size.

[0032] An information processing method according to a ninth aspect of the present disclosure is, in the seventh or eighth aspect, such that, in setting the display designation area, a display area of ​​the display screen corresponding to a field of view area where the detection accuracy is equal to or greater than a first threshold value is set as a first display designation area, and a display area of ​​the display screen corresponding to a field of view area where the detection accuracy is less than the first threshold value and equal to or greater than a second threshold value is set as a second display designation area, and in the display control process, the display manner of the object is made different when the object is displayed in the first display designation area and when the object is displayed in the second display designation area.

[0033] According to the ninth aspect, the display mode of the object displayed in the second display designation area can be made to be a display mode that makes it easier to detect the user's line of sight than the display mode of the object displayed in the first display designation area, thereby avoiding a decrease in operability regarding the second display designation area.

[0034] An information processing method according to a tenth aspect of the present disclosure, in any one of the seventh to ninth aspects, may be such that, in setting the display designation area, a display area on the display screen corresponding to a field of view area where the detection accuracy is equal to or greater than a first threshold value is set as a first display designation area, and a display area on the display screen corresponding to a field of view area where the detection accuracy is less than the first threshold value and equal to or greater than a second threshold value is set as a second display designation area, and in performing the gaze operation, different judgment criteria or judgment methods are used for the object displayed in the first display designation area and the object displayed in the second display designation area.

[0035] According to the tenth aspect, the criteria or method for determining gaze operations for objects displayed in the second display designation area can be made less likely to result in erroneous operations than the criteria or method for determining gaze operations for objects displayed in the first display designation area, thereby suppressing the occurrence of erroneous gaze operations in the second display designation area.

[0036] The information processing method according to the eleventh aspect of the present disclosure, in any one of the seventh to tenth aspects, further comprises acquiring an image of the user's field of view, identifying an inappropriate area that is not suitable as a display position for the object based on the image, and in setting the display designation area, not setting the display designation area in a display area of ​​the display screen that corresponds to the inappropriate area.

[0037] According to the eleventh aspect, it is possible to avoid the occurrence of an erroneous gaze operation caused by an object being displayed in an incompatible area.

[0038] The information processing method according to the twelfth aspect of the present disclosure, in any one of the seventh to eleventh aspects, further comprises acquiring an image of the user's field of view, and generating a simulated image of an object to be operated by gaze control contained in the image when the object is located outside the designated display area on the display screen, and in the display control processing, the object includes the simulated image.

[0039] According to the twelfth aspect, even for a gaze operation target object whose position cannot or is difficult to move within the display screen, a simulated image can be generated, and the display control process can be executed on the generated simulated image.

[0040] An information processing method according to a thirteenth aspect of the present disclosure, in any one of the first to twelfth aspects, may further comprise performing a guidance process for guiding the direction of the user's head based on the detected gaze direction and the control information.

[0041] According to the thirteenth aspect, even for an object to be operated with gaze that cannot or is difficult to move within the display screen, the operability of gaze operation can be improved by guiding the user's head in a direction that increases the detection accuracy of the gaze direction.

[0042] In an information processing method according to a fourteenth aspect of the present disclosure, in the thirteenth aspect, it is preferable to further perform a gaze operation that is an operation based on the detected gaze direction, and to perform the guidance processing before the gaze operation.

[0043] According to the fourteenth aspect, the operability of the gaze operation can be improved by performing the gaze operation after creating a situation in which the detection accuracy of the gaze direction is improved by the guidance process.

[0044] In the information processing method according to the fifteenth aspect of the present disclosure, in the thirteenth or fourteenth aspect, in the guidance processing, the direction of the user's head may be guided by at least one of force feedback, tactile stimulation, display of guidance instructions, and audio output of guidance instructions.

[0045] According to the fifteenth aspect, the user can be directly prompted to change the direction of his or her head to a desired direction by providing a force sense, stimulating a tactile sense, displaying a guidance instruction, or outputting a voice guidance instruction.

[0046] In an information processing method according to a sixteenth aspect of the present disclosure, in any one of the thirteenth to fifteenth aspects, in the guidance process, it is preferable to reduce the visibility of objects included in a display area of ​​the display screen corresponding to a field of view area where the detection accuracy is less than a predetermined value, compared to objects included in a display area of ​​the display screen corresponding to a field of view area where the detection accuracy is equal to or greater than a predetermined value.

[0047] According to the sixteenth aspect, by reducing the visibility of the object, it is possible to indirectly prompt the user to change the head direction to a desired direction.

[0048] In the information processing method according to the seventeenth aspect of the present disclosure, in any one of the thirteenth to sixteenth aspects, it is preferable to further select either the display control process or the guidance process based on at least one of attribute information of the object, information on the user's surrounding environment, information on the user's head posture, information on the user's behavioral status, and information on the user's preferences.

[0049] According to the seventeenth aspect, the display control process or the guidance process can be appropriately performed according to the attribute of the object, the user's surrounding environment, the user's head posture, the user's behavioral status, or the user's preferences.

[0050] An information processing method according to an 18th aspect of the present disclosure, in any one of the 13th to 17th aspects, further comprises performing a gaze operation that is an operation based on the detected gaze direction, determining whether or not the user intends to perform the gaze operation, and if it is determined that the user does not intend to perform the gaze operation, not performing the display control process and the guidance process.

[0051] According to the eighteenth aspect, when the user has no intention of performing a gaze operation, the display control process and the guidance process are not executed, thereby making it possible to avoid the execution of processes that are unnecessary for the user.

[0052] In the information processing method according to a nineteenth aspect of the present disclosure, in any one of the first to eighteenth aspects, the information processing device may be mounted on a glasses-type or goggle-type device worn by the user.

[0053] According to the nineteenth aspect, it is possible to improve the operability of eye-gaze operations in a glasses-type or goggle-type device worn by a user, such as AR glasses or VR goggles.

[0054] In an information processing method according to a twentieth aspect of the present disclosure, an information processing device detects a user's gaze direction, calculates the detection accuracy of the gaze direction for each of a plurality of field of view areas included in the user's field of view, and performs a guidance process to guide the user's head direction based on the detected gaze direction and control information related to the detection accuracy for each of the field of view areas.

[0055] According to the twentieth aspect, the direction of the user's head can be guided in accordance with the detection accuracy of the gaze direction for each field of view included in the user's field of view. As a result, even for a gaze operation target object whose position on the display screen is impossible or difficult to move, the operability of gaze operation can be improved by guiding the user's head in a direction where the detection accuracy of the gaze direction is higher.

[0056] An information processing device according to a 21st aspect of the present disclosure detects a user's gaze direction, calculates the detection accuracy of the gaze direction for each of a plurality of field of view areas included in the user's field of view, and performs a display control process that controls the display content on a display screen based on control information regarding the detection accuracy for each of the field of view areas.

[0057] According to the twenty-first aspect, it is possible to control the content displayed on the display screen in accordance with the detection accuracy of the gaze direction for each field of view included in the user's field of view.

[0058] An information processing device according to a 22nd aspect of the present disclosure detects a user's gaze direction, calculates the detection accuracy of the gaze direction for each of a plurality of field of view areas included in the user's field of view, and performs a guidance process to guide the user's head direction based on the detected gaze direction and control information related to the detection accuracy for each of the field of view areas.

[0059] According to the 22nd aspect, the direction of the user's head can be guided in accordance with the detection accuracy of the gaze direction for each field of view included in the user's field of view. As a result, even for a gaze operation target object whose position on the display screen is impossible or difficult to move, the operability of gaze operation can be improved by guiding the user's head in a direction where the detection accuracy of the gaze direction is higher.

[0060] A program relating to a 23rd aspect of the present disclosure is a program for causing an information processing device to execute a process, which detects a user's gaze direction, calculates the detection accuracy of the gaze direction for each of a plurality of field of view included in the user's field of view, and executes a display control process that controls the display content on a display screen based on control information regarding the detection accuracy for each of the field of view areas.

[0061] According to the twenty-third aspect, the content displayed on the display screen can be controlled in accordance with the detection accuracy of the line-of-sight direction for each field of view included in the user's field of view.

[0062] A program according to a 24th aspect of the present disclosure is a program for causing an information processing device to execute a process, which detects a user's gaze direction, calculates the detection accuracy of the gaze direction for each of a plurality of field of view areas included in the user's field of view, and executes a guidance process for guiding the user's head direction based on the detected gaze direction and control information related to the detection accuracy for each of the field of view areas.

[0063] According to the 24th aspect, the direction of the user's head can be guided in accordance with the detection accuracy of the gaze direction for each field of view included in the user's field of view. As a result, even for a gaze operation target object whose position on the display screen is impossible or difficult to move, the operability of gaze operation can be improved by guiding the user's head in a direction where the detection accuracy of the gaze direction is higher.

[0064] The present disclosure can also be realized as a program that causes a computer to execute each characteristic configuration included in such a method or apparatus, or as a system operated by this program. Needless to say, such a computer program can be distributed on a computer-readable non-transitory recording medium such as a CD-ROM or via a communication network such as the Internet.

[0065] (Embodiments of the Present Disclosure) Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Elements with the same reference numerals in different drawings indicate the same or corresponding elements. Furthermore, the components, the arrangement positions of the components, the connection forms, the order of operations, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. The present disclosure is limited only by the claims. Therefore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concept of the present disclosure are not necessarily required to achieve the objectives of the present disclosure, but are described as constituting more preferred forms.

[0066] In this embodiment, an example will be described in which AR glasses are used as an application target of the present disclosure, but the application target of the present disclosure is not limited to this. The present disclosure is widely applicable to glasses-type or goggle-type devices worn by a user that use cross-reality technologies such as AR, VR, or MR. In addition to devices worn by a user, the present disclosure is also applicable to a stationary gaze control system that performs gaze control by capturing an image of the user's eyes facing a display with a camera near the display.

[0067] (First embodiment) Fig. 1 is a diagram showing a simplified configuration of AR glasses 1 according to a first embodiment of the present disclosure. The AR glasses 1 include a processing unit 10, a storage unit 11, a display unit 12, an internal camera 14, and an external camera 15. The processing unit 10 is configured using a processor (information processing device) such as a CPU. The storage unit 11 is configured using a semiconductor memory or the like. The display unit 12 is configured using a liquid crystal display, an organic EL display or the like. The internal camera 14 and the external camera 15 are configured using an optical system, a CMOS image sensor or the like.

[0068] 2 is a schematic diagram of the left eye unit of the AR glasses 1 viewed from the user's face side. Of the front and back surfaces of the left eye unit, the side facing the user's face (back side) corresponds to the internal side, and the side opposite the user's face (front side) corresponds to the external side. Note that the configuration of the right eye unit is the same as the configuration of the left eye unit, so the right eye unit is not shown in the figure.

[0069] The left eye unit has a frame 40. An internal camera 14 is arranged inside the frame 40. An external camera 15 is arranged outside the frame 40. The area inside the frame 40 corresponds to the field of view V of the user wearing the AR glasses 1. The field of view V corresponds to the actual view in front that the user views through the AR glasses 1. A display screen 13 of the display unit 12 is arranged inside the frame 40.

[0070] The internal camera 14 captures an image of the eyes of the user wearing the AR glasses 1 and outputs the image data. The external camera 15 captures an image of the field of view V of the user wearing the AR glasses 1 and outputs the image data.

[0071] 1 , the storage unit 11 stores a program 31, image data 32, control information 33, and an estimation model 34. Note that the image data 32 and the control information 33 may be stored in a storage device external to the AR glasses 1 instead of in the storage unit 11 inside the AR glasses 1. In this case, the image data 32 and the control information 33 are transmitted and received between the AR glasses 1 and the storage device via wireless communication. The storage device may be a cloud server or the like. The storage unit 11 includes a computer-readable non-volatile storage medium. The program 31 is stored in the storage medium.

[0072] As functions realized by the processor executing the program 31 read from the storage unit 11, the processing unit 10 has a gaze direction detection unit 21, a control information generation unit 22, a gaze operation execution unit 23, a display control unit 24, and a storage control unit 25. Details of the processing content of each of these units will be described later.

[0073] The image data 32 includes image data of objects for eye-gaze operation, such as icons 50A and 50B, displayed on the display screen 13.

[0074] The control information 33 includes a plurality of visual field areas V included in the visual field V of the user. １１ ~V １３ , V ２１ ~V ２３ , V ３１ ~V ３３ The accuracy information 45A indicates the accuracy of the line of sight detection for each field of view V.１１ ~V １３ , V ２１ ~V ２３ , V ３１ ~V ３３ are partial regions obtained by dividing the field of view V, and the plurality of field of view V １１ ~V １３ , V ２１ ~V ２３ , V ３１ ~V ３３ The field of view V is formed by

[0075] The estimation model 34 includes an estimation model that estimates the gaze direction based on the eyeball direction of the user. Since the relationship between the eyeball direction and the gaze direction varies between individuals, the estimation model 34 is learned according to each user by performing calibration when the AR glasses 1 are used for the first time, for example.

[0076] FIG. 3 is a flowchart showing the process executed by the processing unit 10 during calibration.

[0077] First, in step S11, the display control unit 24 displays an image for calibration and an instruction message for the user on the display screen 13.

[0078] 4 is a diagram showing an image for calibration. The display screen 13 has a field of view V １１ ~V １３ , V ２１ ~V ２３ , V ３１ ~V ３３ A plurality of display areas 13 corresponding to １１ ~13 １３ , 13 ２１ ~13 ２３ , 13 ３１ ~13 ３３ For example, the upper left display area 13 １１ is the upper left field of view V １１ , and the central display area 13 ２２ is the central field of view V ２２ , and the lower right display area 13 ３３ is the field of view V at the bottom right ３３In this embodiment, an example will be described in which the display screen 13 is divided into three parts in the left-right direction and the top-bottom direction, but the number of divisions is not limited to this example and may be any plural number. Furthermore, the division shape is not limited to a rectangle, and may be any shape such as concentric circles or radial circles centered at the center position of the display screen 13.

[0079] The display control unit 24 controls each display area 13 １１ ~13 １３ , 13 ２１ ~13 ２３ , 13 ３１ ~13 ３３ The display control unit 24 also displays an instruction message such as "Please focus on the point displayed on the screen" on the display screen 13.

[0080] Next, in step SP12, the gaze direction detection unit 21 detects whether the image P1 is in each display area 13 １１ ~13 １３ , 13 ２１ ~13 ２３ , 13 ３１ ~13 ３３ By acquiring images taken by the internal camera 14 at each time point displayed on the screen, eye information including the direction of the user's eyes at each time point is acquired.

[0081] Next, in step S13, the control information generating unit 22 learns an estimation model 34 that estimates the gaze direction based on the eyeball direction by machine learning using the eyeball information and the display position information of the image P1 as learning data.

[0082] 4, point P2 is the gaze position on display screen 13 obtained from the gaze direction estimated by estimation model 34, and corresponds to the detected position of the user's gaze. The closer the positions of points P1 and P2 are, the higher the detection accuracy of the gaze direction (i.e., the estimation accuracy of estimation model 34).

[0083] The control information generating unit 22 repeats steps S12 and S13 multiple times to generate the control information for each display area 13. １１ ~13 １３ , 13 ２１ ~13 ２３ , 13３１ ~13 ３３ A plurality of points P2 are obtained with respect to

[0084] Next, in step S14, the control information generating unit 22 １１ ~13 １３ , 13 ２１ ~13 ２３ , 13 ３１ ~13 ３３ The parameters used to calculate the detection accuracy include at least one of the following: a statistical value (total, average, maximum, or the like) of the error between the position of point P1 and the position of each point P2; the variance or standard deviation of the multiple points P2; and the number of points P2 that have been acquired.

[0085] Next, in step S15A, the control information generator 22 generates precision information 45A.

[0086] 5 is a diagram showing an example of the accuracy information 45A. The accuracy information 45A is １１ ~13 １３ , 13 ２１ ~13 ２３ , 13 ３１ ~13 ３３ The minimum value of the detection accuracy is 0 and the maximum value is 1, and the larger the value, the higher the detection accuracy. For example, １１ The detection accuracy is "0.6", and the display area 13 ２２ The detection accuracy is "0.9", and the display area 13 ３３ The detection accuracy for is "0.4".

[0087] Next, in step S16A, the storage control unit 25 stores the accuracy information 45A in the storage unit 11 together with the control information 33.

[0088] FIG. 6 is a flowchart showing the processing executed by the processing unit 10 in the eye-gaze operation.

[0089] First, in step S21A, the gaze operation execution unit 23 acquires the accuracy information 45A by reading it from the storage unit 11. Based on the accuracy information 45A, the gaze operation execution unit 23 specifies a high-accuracy area RH1 where the gaze direction detection accuracy is high. The high-accuracy area RH1 corresponds to a display area on the display screen 13 that corresponds to a field of view area where the detection accuracy is equal to or higher than a predetermined value. The predetermined value is set to, for example, "0.7" (first predetermined value). In this case, the gaze operation execution unit 23 specifies a high-accuracy area RH1 where the gaze direction detection accuracy is higher than a predetermined value on the display screen 13. １２ , 13 １３ , 13 ２２ , 13 ２３ is identified as the high-precision region RH1.

[0090] Next, in step S22A, the storage control unit 25 acquires the image data 32 of the icon 50A to be displayed by reading it from the storage unit 11. The display control unit 24 displays the icon 50A within the high-precision region RH1.

[0091] 7 is a diagram showing an example of the display screen 13. Five icons 50A for eye-gaze operation are displayed in the high-precision region RH1 of the display screen 13.

[0092] In a gaze operation, the gaze direction detection unit 21 acquires the trained estimation model 34 by reading it from the storage unit 11. The gaze direction detection unit 21 also acquires an image of the user's eyes captured by the internal camera 14. The gaze direction detection unit 21 detects the gaze direction of the user by estimating the gaze direction based on the eyeball direction using the estimation model 34.

[0093] Based on the detected direction of the user's gaze, the gaze operation execution unit 23 calculates the user's gaze position on the display screen 13. If the user's gaze position does not change for a predetermined gaze determination time or longer, or if the amount of change in the gaze position is less than a predetermined value, the gaze operation execution unit 23 determines that the user is gazing at that position and executes processing according to the gaze position. For example, if an icon 50A with a musical note mark is displayed at the user's gaze position, the gaze operation execution unit 23 executes processing to launch a music playback app.

[0094] The predetermined value for identifying the high-precision area may be set to a different value depending on the attribute of the object to be displayed. The attribute may include importance, size, content, etc. For example, when displaying an icon 50B that is larger than the icon 50A, the predetermined value is set to, for example, "0.6" (second predetermined value). In this case, the line-of-sight operation execution unit 23 determines the accuracy of the display area 13. １１ ~13 １３ , 13 ２１ ~13 ２３ is identified as the high-precision region RH2.

[0095] 8 is a diagram showing an example of the display screen 13. Icons 50A and 50B for eye-gaze operation are displayed in the high-precision region RH2. The display control unit 24 displays five small icons 50A in the display region 13 where the detection accuracy is "0.7" or higher. １２ , 13 １３ In addition, the display control unit 24 displays two icons 50B that are larger than the icon 50A in the display area 13 where the detection accuracy is equal to or greater than "0.6" and less than "0.7". １１ , 13 ２１ As another example, the display control unit 24 may display an important object associated with an application startup process, a menu selection process, or the like, in the display area 13. １２ , 13 １３ , 13 ２２ , 13 ２３ and unimportant objects such as simple information displays are displayed within the display area 13 １１ , 13 ２１ It may be displayed inside.

[0096] According to this embodiment, the visual field area V included in the user's visual field V １１ ~V １３ , V ２１ ~V ２３ , V ３１ ~V ３３ The display content on the display screen 13 can be controlled according to the detection accuracy of each line of sight direction.

[0097] Furthermore, according to this embodiment, the display of the icons 50A and 50B, which are objects for eye-gaze operation, can be controlled according to the detection accuracy for each field of view area.

[0098] Furthermore, according to this embodiment, the display position of the object can be controlled according to the detection accuracy for each field of view area.

[0099] Furthermore, according to this embodiment, by displaying objects in the high-precision areas RH1 and RH2 corresponding to the field-of-view areas where the detection accuracy is equal to or greater than a predetermined value, it is possible to improve the operability of the eye-gaze operation.

[0100] Furthermore, according to this embodiment, the display position of an object can be controlled according to the attributes of the object, such as importance or size.

[0101] Second Embodiment In a second embodiment of the present disclosure, the control information 33 includes area information 45B that indicates a designated display area on the display screen 13 where an object is to be displayed.

[0102] 9 is a flowchart showing the calibration process executed by the processing unit 10. The processes in steps S11 to S14 are the same as those in the first embodiment.

[0103] In step S15B, the control information generator 22 generates the area information 45B. １１ ~13 １３ , 13 ２１ ~13 ２３ , 13 ３１ ~13 ３３ In the example of this embodiment, the control information generation unit 22 sets two display designation areas (a first display designation area RH and a second display designation area RM) according to the detection accuracy. The control information generation unit 22 sets a field of view area V on the display screen 13 where the detection accuracy is equal to or greater than a first threshold value (for example, "0.7"). １２ , V １３ , V ２２ , V ２３ Display area 13 corresponding to １２ , 13 １３ , 13 ２２ , 13 ２３ The control information generator 22 sets the first display designated region RH as the first display designated region RH. The control information generator 22 also sets the first display designated region RH as the first display designated region RH.１１ , V ２１ Display area 13 corresponding to １１ , 13 ２１ is set as the second display designation region RM. The control information generator 22 generates region information 45B as coordinate information indicating the positions of the first display designation region RH and the second display designation region RM within the display screen 13.

[0104] 10 is a diagram showing an example of the region information 45B. The region information 45B is a database that includes the position coordinates of each of the first display designation region RH and the second display designation region RM. The position coordinates include the coordinates of the upper left corner and the coordinates of the lower right corner of each region. The X coordinate values ​​X1 to X4 are coordinate values ​​that indicate the position in the left-right direction on the display screen 13, and the Y coordinate values ​​Y1 to Y4 are coordinate values ​​that indicate the position in the up-down direction on the display screen 13. The origin of the coordinate system is set, for example, to the lower left corner of the display screen 13.

[0105] Next, in step S16B, the storage control unit 25 stores the region information 45B in the storage unit 11 together with the control information 33.

[0106] FIG. 11 is a flowchart showing the processing executed by the processing unit 10 in the eye-gaze operation.

[0107] First, in step S21B, the eye-gaze operation execution unit 23 acquires the region information 45B by reading it from the storage unit 11. The eye-gaze operation execution unit 23 identifies the first display designation region RH and the second display designation region RM based on the region information 45B.

[0108] Next, in step S22B, the storage control unit 25 acquires image data 32 of the icons 50A and 50B to be displayed by reading it from the storage unit 11. The display control unit 24 displays the small icon 50A in the first display designation region RH and the large icon 50B in the second display designation region RM according to the sizes of the icons 50A and 50B. Alternatively, the display control unit 24 may display the icon 50A of high importance in the first display designation region RH and the icon 50B of low importance in the second display designation region RM according to the importance of the icons 50A and 50B. Alternatively, the display control unit 24 may display the icon 50A of high importance in the first display designation region RH and the icon 50B of low importance in the second display designation region RM according to the frequency of use of the icons 50A and 50B.

[0109] When displaying an object in the second display designation region RM according to importance or frequency of use, the display control unit 24 may display the object in a larger size than when the same object is displayed in the first display designation region RH. This allows the display mode of the object displayed in the second display designation region RM to be a display mode that makes it easier to detect the user's line of sight than the display mode of the object displayed in the first display designation region RH. As a result, it is possible to avoid a decrease in the operability of line-of-sight operations related to the second display designation region RM.

[0110] Note that if there is sufficient space in the first display designation region RH to display the icon 50A, the display control unit 24 may display the icon 50B in the first display designation region RH.

[0111] 12 is a diagram showing an example of the display screen 13. An icon 50A and an icon 50B for eye-gaze operation are displayed in the first display designation region RH and the second display designation region RM of the display screen 13, respectively.

[0112] According to this embodiment, the operability of eye-gaze operation can be improved by displaying an object in a designated display area that is set based on the detection accuracy.

[0113] Furthermore, according to this embodiment, the display position of an object can be controlled according to the attributes of the object, such as size, importance, or frequency of use.

[0114] Various modifications of the second embodiment will be described below. The modifications described below can be applied in any combination.

[0115] (First Modification of Second Embodiment) The control information generator 22 may calculate the detection accuracy for each pixel in the display screen 13 by pixel interpolation, and may set the display designation region in pixel units instead of display area units. Furthermore, the control information generator 22 may set the first display designation region RH in a positional relationship that includes the second display designation region RM, or may set the first display designation region RH in a positional relationship that separates it from the second display designation region RM. Furthermore, the control information generator 22 may set the entire region outside the first display designation region RH as the second display designation region RM.

[0116] 13 is a diagram showing an example of the display screen 13. The first display designation region RH and the second display designation region RM are set in pixel units rather than in display area units. The first display designation region RH is set in a positional relationship such that it is included in the second display designation region RM.

[0117] According to this modification, the display setting area is set in pixel units, allowing for precise control of the display position of the object.

[0118] (Second variant of the second embodiment) The gaze operation execution unit 23 may use different criteria or methods for determining gaze operations for the icon 50A displayed in the first display designation area RH and the icon 50B displayed in the second display designation area RM.

[0119] As an example of differentiating the determination criteria, the eye-gaze operation execution unit 23 may set the gaze determination time for the second display designation region RM to be longer than the gaze determination time for the first display designation region RH.

[0120] FIG. 14 is a flowchart showing the processing executed by the processing unit 10 in the eye-gaze operation.

[0121] First, in step S31, the gaze operation execution unit 23 calculates the gaze position of the user on the display screen 13 based on the gaze direction of the user detected by the gaze direction detection unit 21. The gaze operation execution unit 23 determines whether the gaze position of the user is included in the first display designation region RH.

[0122] If the user's gaze position is included within the first display designation area RH (step S31: YES), then in step S32, the gaze operation execution unit 23 sets a first time (e.g., 0.5 seconds) as the threshold value for the gaze determination time.

[0123] If the user's gaze position is included within the second display designation area RM (step S31: NO), then in step S33, the gaze operation execution unit 23 sets a second time (e.g., 1.0 seconds) longer than the first time as the threshold value for the gaze determination time.

[0124] Next, in step S34, the eye-gaze operation execution unit 23 determines whether the user's gaze time is equal to or longer than the threshold set in step S32 or step S33.

[0125] If the gaze time is equal to or longer than the threshold value (step S34: YES), then in step S35, the gaze operation execution unit 23 executes a gaze operation corresponding to the object the user is gazing at.

[0126] If the gaze time is less than the threshold value (step S34: NO), the gaze operation execution unit 23 ends the process without executing the gaze operation.

[0127] As an example of different determination methods, the gaze operation execution unit 23 may operate an object using only the user's gaze information for the first display designation region RH, and may operate an object using the user's gaze information and gesture information for the second display designation region RM. The gesture information includes, for example, information indicating the movement of the user's finger. The gaze operation execution unit 23 acquires the gesture information by analyzing an image captured by the external camera 15.

[0128] According to this modification, the criteria or method for determining a gaze operation for an object displayed in the second display designation area RM can be set to be less prone to erroneous operations than the criteria or method for determining a gaze operation for an object displayed in the first display designation area RH, thereby making it possible to reduce the occurrence of erroneous gaze operations in the second display designation area RM.

[0129] (Third variant of the second embodiment) The control information generation unit 22 may identify an incompatible area RZ that is not suitable as a display position for an object based on an image acquired from the external camera 15, and when setting the display designation area, may exclude the display area of ​​the display screen 13 that corresponds to the incompatible area RZ from the targets for setting the display designation area.

[0130] 15 is a diagram showing an example of the display screen 13. The control information generator 22 identifies an unsuitable region RZ included in the image by analyzing the image acquired from the external camera 15. The control information generator 22 identifies, for example, a region where the luminance value is equal to or greater than a predetermined value (a region that is too bright), or a region where the image complexity indicated by the amount of change in luminance value or color difference value per unit area is equal to or greater than a predetermined value (a cluttered region) as the unsuitable region RZ.

[0131] In the example shown in FIG. 15, the control information generator 22 １３ , 13 １３ The control information generating unit 22 identifies the unsuitable area RZ that straddles the display area 13 where the detection accuracy is equal to or greater than the first threshold. １２ , 13 １３ , 13 ２２ , 13 ２３ , the display area 13 including the non-conforming area RZ. １３ , 13 １３ By excluding ２２ , 13 ２３ is set as the first designated display region RH.

[0132] According to this modification, it is possible to avoid the occurrence of erroneous gaze operations caused by an object being displayed in the incompatible region RZ.

[0133] (Fourth Modification of Second Embodiment) The processing unit 10 may express the position of the field of view using coordinates based on the user's head position or head direction. For example, when detecting the user's line of sight direction by capturing images of the eyes of a user seated facing a display using multiple cameras installed on the display, the processing unit 10 performs the calibration shown in FIG. 9 . In this case, the processing unit 10 acquires head front coordinates as coordinates indicating the user's head position and head direction at the time of calibration. Thereafter, when the processing unit 10 detects a change in the user's head position or head direction, it calculates the amount of change (ΔX, ΔY) from the user's head front coordinates.

[0134] 11 , in step S21B, the processing unit 10 reads out the region information 45B from the storage unit 11. At that time, the processing unit 10 corrects the position coordinates included in the region information 45B based on the amount of change (ΔX, ΔY). The processing unit 10 identifies the first display designation region RH and the second display designation region RM based on the corrected position coordinates.

[0135] According to this modification, even if the user's head position or head direction changes, the first display designation region RH and the second display designation region RM can be set appropriately.

[0136] Third Embodiment Fig. 16 is a diagram showing an example of the display screen 13. As in the first embodiment, the eye-gaze operation execution unit 23 executes the eye-gaze operation in the display area 13. １２ , 13 １３ , 13 ２２ , 13 ２３ is identified as the high-precision region RH1.

[0137] Display area 13 outside high-precision area RH1 ３１ The field of view V corresponding to ３１includes a book 55, which is an example of an object that exists in real space and that is subject to gaze operation. An object that is subject to gaze operation is an example of an object. The actual book 55 cannot be moved by gaze operation. Therefore, the display control unit 24 generates a simulated image 56 that mimics the appearance of the book 55 based on an image acquired from the external camera 15, and displays the simulated image 56 as a pop-up within the high-precision region RH1. During gaze operation, when the gaze operation execution unit 23 detects that the user is gazing at the simulated image 56, the display control unit 24 displays an explanation image 57 that includes an introduction to the book 55 near the simulated image 56 within the high-precision region RH1.

[0138] 17 is a diagram showing an example of the display screen 13. As in the second embodiment, the eye-gaze operation execution unit 23 executes the eye-gaze operation in the display area 13. １２ , 13 １３ , 13 ２２ , 13 ２３ is specified as the first display designated region RH, and the display region 13 １１ , 13 ２１ is specified as the second designated display region RM.

[0139] Display area 13 outside the first display designation area RH and outside the second display designation area RM ３１ The field of view V corresponding to ３１ includes a book 55, which is an example of a gaze operation target object that exists in real space. The actual book 55 cannot be moved by gaze operation. Therefore, the display control unit 24 generates a simulated image 56 that simulates the appearance of the book 55 based on an image acquired from the external camera 15, and pops up the simulated image 56 within the first display designation region RH. During gaze operation, when the gaze operation execution unit 23 detects that the user is gazing at the simulated image 56, the display control unit 24 displays an explanatory image 57 containing an introduction to the book 55 near the simulated image 56 within the first display designation region RH. Note that the display control unit 24 may also display the simulated image 56 and the explanatory image 57 within the second display designation region RM.

[0140] In addition, the same display control as above may be performed not only on objects that are the subject of gaze operation that exist in the real world, but also on objects that are difficult to move by gaze operation, such as objects that are linked to actual objects that exist in the real world.

[0141] According to this embodiment, even for an object to be operated on by gaze, whose position cannot be moved or is difficult to move within the display screen 13 by gaze operation, a simulated image 56 can be generated, and a display control process can be performed on the generated simulated image 56 to control the display position of the object.

[0142] (Fourth embodiment) In a fourth embodiment of the present disclosure, instead of or in addition to the display control according to the third embodiment, a guidance process is executed for a gaze operation target object whose position cannot or is difficult to move within the display screen 13 by gaze operation, to guide the head direction of a user wearing the AR glasses 1. Below, an example in which the fourth embodiment is applied to the first embodiment will be described, but the fourth embodiment can also be applied to the second or third embodiment.

[0143] Fig. 18 is a diagram showing a simplified configuration of AR glasses 1 according to a fourth embodiment of the present disclosure. The AR glasses 1 further include a guidance device 16, a posture sensor 17, and an electroencephalogram sensor 18 in addition to the configuration shown in Fig. 1. The processing unit 10 also includes a guidance processing unit 26 in addition to the configuration shown in Fig. 1. The guidance processing unit 26 executes guidance processing to guide the head direction of the user wearing the AR glasses 1 based on the gaze direction of the user detected by the gaze direction detection unit 21 and control information 33.

[0144] The guidance device 16 may include any device mounted on the temples of the AR glasses 1 that can guide the head direction of a user wearing the AR glasses 1 by providing force or tactile stimulation to the user. Force or tactile stimulation may include Galvanic Vestibular Stimulation (GVS), vibration, gyroscopic effect, torque recoil, or the like. The guidance device 16 may also be a device that guides the head direction of the user by displaying guidance instructions on the display screen 13 or by outputting guidance instructions as audio from a speaker mounted on the AR glasses 1. The guidance instruction may be, for example, a message such as "Please turn your head a little lower." The guidance device 16 may also be a device that guides the head direction of the user by controlling the angle of a movable chair on which the user is seated or its headrest.

[0145] The attitude sensor 17 is configured to include an acceleration sensor or a gyro sensor mounted on the AR glasses 1. The attitude sensor 17 detects the head angle of the user wearing the AR glasses 1, and outputs attitude information including the detected value.

[0146] The EEG sensor 18 is configured with multiple electrodes mounted on the temples of the AR glasses 1. The EEG sensor 18 detects the brain waves of the user wearing the AR glasses 1 and outputs EEG data indicating the detected brain waves. If the AR glasses 1 include a headband mounted on the back or top of the head, the multiple electrodes may be mounted on the headband. The user's EEG includes a readiness potential corresponding to the content of the movement. The readiness potential is a minute potential change that appears just before the start of the movement. By analyzing the EEG data, it is possible to determine whether the user intends to perform a gaze control based on whether a predetermined readiness potential is included in the EEG data. Alternatively, instead of analyzing the EEG data, it is also possible to determine whether the user intends to perform a gaze control by measuring the action potential of the muscles around the user's eyes using electrodes mounted on the AR glasses 1 and analyzing the electromyogram obtained from the measured value.

[0147] FIG. 19 is a flowchart showing the process executed by the processing unit 10 in relation to the guidance process.

[0148] First, in step S41, the guidance processing unit 26 determines, based on the image acquired from the external camera 15 and the accuracy information 45A acquired from the memory unit 11, whether the object to be operated by gaze contained in the image acquired from the external camera 15 is included outside the high-precision region RH1 on the display screen 13.

[0149] If the object to be operated on by gaze is included outside the high-precision region RH1 (step S41: YES), then in step S42, the induction processing unit 26 determines whether the user intends to perform gaze operation based on the brain wave data acquired from the brain wave sensor 18.

[0150] If the user intends to perform a gaze operation (step S42: YES), then in step S43, the guidance processing unit 26 calculates the guidance direction of the user's head. Based on the positional relationship between the gaze operation target and the high-precision region RH1 on the display screen 13, the guidance processing unit 26 calculates the guidance direction of the user's head so that the gaze operation target is included in the high-precision region RH1. For example, if the gaze operation target is located below the high-precision region RH1, the guidance processing unit 26 calculates "downward" as the guidance direction of the user's head. Note that if there are multiple candidate guidance directions, the guidance processing unit 26 may simply adopt the direction that minimizes the amount of change in the user's head direction as the guidance direction. This reduces the burden on the user.

[0151] Next, in step S44, the guidance processing unit 26 performs guidance control by operating the guidance device 16 based on the information on the guidance direction and the attitude information acquired from the attitude sensor 17. The guidance device 16 guides the head direction of the user, for example, by providing the user with a GVS.

[0152] Next, in step S45, the guidance processing unit 26 determines whether the eye-gaze operation target is included in the high-precision region RH1.

[0153] If the eye-gaze operation target object is not included in the high-precision region RH1 (step S45: NO), the guidance processing unit 26 repeatedly executes the processes of steps S42 to S45.

[0154] If the gaze operation target object is included in the high-precision region RH1 (step S45: YES), then in step S46, the gaze operation execution unit 23 executes the same gaze operation as above.

[0155] In the determination process of step S41, if the gaze operation target object is included in the high-precision region RH1 (step S41: NO), the processing unit 10 executes step S46 without executing the guidance process of steps S42 to S45.

[0156] In the determination process of step S42, if the user does not intend to perform a gaze operation (step S42: NO), the processing unit 10 ends the process without performing the guidance process of steps S42 to S45 and the gaze operation of step S46.

[0157] According to this embodiment, the guidance processing unit 26 and the guidance device 16 guide the user's head in a direction that increases the detection accuracy of the gaze direction, thereby improving the operability of gaze operations even for gaze operation objects whose position cannot or is difficult to move within the display screen 13.

[0158] Furthermore, according to this embodiment, the operability of the gaze operation can be improved by performing the gaze operation after creating a situation in which the detection accuracy of the gaze direction is improved by the guidance process.

[0159] Furthermore, according to this embodiment, the user can be directly prompted to change the direction of his or her head to a desired direction by providing a force sense, stimulating a tactile sense, displaying a guidance instruction, or outputting a voice guidance instruction.

[0160] Various modifications of the fourth embodiment will be described below. The modifications described below can be applied in any combination.

[0161] (First variant of the fourth embodiment) The guidance processing unit 26 may perform guidance processing by reducing the visibility of objects outside the high-precision region RH1 on the display screen 13 compared to objects contained within the high-precision region RH1.

[0162] 20 is a diagram showing an example of the display screen 13. The display area 13 outside the high-precision area RH1 ３１ The field of view V corresponding to ３１ The display area 13 in the high-precision area RH1 includes a book 55A, which is an example of an object to be operated by line of sight in the real space. １３ The field of view V corresponding to １３ includes a book 55B, which is an example of an object to be operated by line of sight that exists in real space.

[0163] The guidance processing unit 26 reduces the visibility of the object 55A by reducing the luminance values ​​of pixels in the surrounding area including the object 55A below the actual luminance values ​​of the object 55A and the luminance values ​​of the object 55B. Instead of reducing the luminance values, the guidance processing unit 26 may reduce the visibility of the object by reducing the definition or clarity, or by adding noise. Alternatively, the guidance processing unit 26 may change the display form, such as the color, shape, or size, of the gaze cursor, which indicates the gaze position on the display screen 13, depending on whether the gaze cursor is within the high-precision region RH1 or outside the high-precision region RH1.

[0164] According to this modified example, by reducing the visibility of objects contained outside the high-precision region RH1 or by changing the display form of the gaze cursor, the user can be indirectly encouraged to change the direction of their head to the desired direction that will bring the object into the high-precision region RH1.

[0165] (Second variant of the fourth embodiment) The processing unit 10 may select and execute either the display control process or the guidance process based on at least one of the attribute information of the object, the user's surrounding environment information, the user's head posture information, the user's behavioral status information, and the user's preference information.

[0166] 21 is a diagram showing a simplified configuration of the AR glasses 1 according to a second modified example of the fourth embodiment of the present disclosure. The processing unit 10 further includes a processing selection unit 27 in addition to the configuration shown in FIG.

[0167] FIG. 22 is a flowchart showing the processing executed by the processing unit 10.

[0168] The processes in steps S41 and S42 are the same as those in the fourth embodiment.

[0169] If the user intends to perform a gaze operation (step S42: YES), then in step S51, the processing selection unit 27 determines whether to perform the guidance processing by the guidance processing unit 26 (fourth embodiment) or the display control processing by the display control unit 24 (first to third embodiments).

[0170] As a first example, the process selection unit 27 selects a process to be executed based on attribute information of an object.

[0171] When the object is an object of gaze operation that cannot or is difficult to move within the display screen 13, the process selection unit 27 selects the guidance process if the size of the object is equal to or greater than a predetermined value, and selects the display control process if the size of the object is less than the predetermined value. This is because if a simulated image of a large object is displayed as a pop-up within the high-precision region RH1, the high-precision region RH1 will be occupied by the simulated image, and other important icons, etc. cannot be displayed within the high-precision region RH1.

[0172] The process selection unit 27 selects a guidance process if the duration is less than a predetermined value, and selects a display control process if the duration is equal to or greater than the predetermined value, depending on the duration that the user is expected to gaze at the object. This is because performing a guidance process on an object (such as text or video) that is expected to last a long time will cause the user to maintain an unnatural head posture for a long period of time, which will cause the user to feel tired.

[0173] As a second example, the process selection unit 27 selects a process to be executed based on information about the user's surrounding environment.

[0174] The processing selection unit 27 determines whether the user's current location is indoors or outdoors based on an image acquired from the external camera 15, or, if the AR glasses 1 are equipped with a GPS receiver, based on location information acquired from the GPS receiver. The processing selection unit 27 selects guidance processing if the user's current location is indoors, and selects display control processing if the user's current location is outdoors. This is because guiding the head direction of a user who is outdoors is undesirable from a safety standpoint.

[0175] As a third example, the process selection unit 27 selects a process to be executed based on the head posture information of the user.

[0176] The process selection unit 27 acquires the user's current head angle based on the posture information acquired from the posture sensor 17. The process selection unit 27 selects guidance processing if the user's current head angle, based on the head angle in a basic posture facing forward, is less than a predetermined value, and selects display control processing if the current head angle is equal to or greater than the predetermined value. This is because it would be painful for the user to have to turn their neck further in the same direction when the current head angle is equal to or greater than the predetermined value. Furthermore, if the current head angle is equal to or greater than the predetermined value, it is considered that the user is intentionally turning their face in that direction, and it is therefore not appropriate to change the head direction by guidance processing.

[0177] As a fourth example, the process selection unit 27 selects a process to be executed based on the user's behavioral status information.

[0178] The process selection unit 27 determines whether the user is performing some kind of work based on an image acquired from the external camera 15 or on information about an app currently being executed by the user. The process selection unit 27 selects a guidance process if the user is not performing some kind of work, and selects a display control process if the user is performing some kind of work. This is because guiding the head direction of a user performing some kind of work would interfere with the user's work.

[0179] The processing selection unit 27 determines whether the user is walking, running, or driving based on an image acquired from the external camera 15, or based on position information acquired from a GPS receiver if the AR glasses 1 are equipped with a GPS receiver, or based on movement information acquired from an acceleration sensor or the like if the AR glasses 1 are equipped with an acceleration sensor or the like. The processing selection unit 27 selects guidance processing if the user is not walking or the like, and selects display control processing if the user is walking or the like. This is because guiding the head direction of a user who is walking or the like is undesirable from a safety standpoint.

[0180] As a fifth example, the process selection unit 27 selects a process to be executed based on user preference information.

[0181] The user can arbitrarily set whether to prioritize the guidance process or the display control process, and preference information indicating the setting is stored in the storage unit 11. The process selection unit 27 selects the guidance process or the display control process based on the preference information acquired from the storage unit 11.

[0182] The process selection unit 27 may select a process to be executed by combining the first to fifth examples.

[0183] If the guidance process is selected by the process selection unit 27 (step S51: YES), the processes of steps S43 to S46 similar to those in FIG. 19 are executed.

[0184] If the display control process is selected by the process selection unit 27 (step S51: NO), then in step S52, the display control unit 24 executes the same display control process as in the first to third embodiments. Thereafter, the processes of steps S45 and S46 similar to those in FIG. 19 are executed.

[0185] If the user has no intention of performing a gaze operation (step S42: NO), the processing unit 10 ends the process without executing the display control process and the guidance process.

[0186] According to this modification, the display control process or the guidance process can be appropriately performed according to the attribute of the object, the user's surrounding environment, the user's head posture, the user's behavioral status, or the user's preferences.

[0187] Furthermore, according to this modification, if the user has no intention of performing a gaze operation, the display control process and the guidance process are not executed, thereby making it possible to avoid the execution of processes that are unnecessary for the user.

[0188] (Third variant of the fourth embodiment) In the fourth embodiment, the induction processing unit 26 determines whether or not the user intends to perform a gaze operation based on the electroencephalogram data acquired from the electroencephalogram sensor 18, but the present invention is not limited to such a method.

[0189] Two external cameras 15 may be installed at positions corresponding to both eyes, and may be capable of measuring the depth direction distance of an object present in the external environment by utilizing the parallax between images captured by the two external cameras 15. Alternatively, the external cameras 15 may be capable of measuring the depth direction distance of an object present in the external environment by utilizing time-of-flight (TOF) technology. According to this method, the processing unit 10 can obtain the distance between the user and a book 55, which is an example of an object to be operated by gaze present in real space.

[0190] The convergence angle formed by the user's left and right eyeballs changes depending on the distance between the book 55 and the user. When the user gazes at the book 55, the user's line of sight is focused on the book 55. In addition to determining the user's line of sight direction, the processing unit 10 may also determine whether the user intends to perform a gaze operation based on whether the user's convergence angle matches the distance between the book 55 and the user. In other words, the processing unit 10 may determine that the user intends to perform a gaze operation when the user's line of sight matches the direction of the location of the gaze operation target and the user's convergence angle matches the distance between the gaze operation target and the user.

[0191] The processing unit 10 may also combine determining the intention to perform a gaze control based on the user's electroencephalogram data and determining the intention to perform a gaze control based on the user's convergence angle. The processing unit 10 first detects a match between the convergence angle and the distance to establish a state (provisional determination state) in which the intention to perform a gaze control is provisionally estimated. After establishing the provisional determination state, the processing unit 10 acquires electroencephalogram data from the electroencephalogram sensor 18 to definitively determine the intention to perform a gaze control based on the electroencephalogram data. This method improves the accuracy of determination compared to determining the intention to perform a gaze control based only on the convergence angle or the electroencephalogram data. Note that the measurement of the user's electroencephalogram by the electroencephalogram sensor 18 may be performed continuously, not just after establishing the provisional determination state. Alternatively, the electroencephalogram sensor 18 may be replaced with another biosignal sensor, such as an electromyogram sensor or a heart rate sensor.

[0192] In the flowchart shown in FIG. 22 , the processing unit 10 selects the guidance process or the display control process after determining that there is an intention to perform a gaze operation, but this is not limited to this method. When the processing unit 10 determines that there is an intention to perform a gaze operation, it may execute the gaze operation without performing the guidance process or the display control process. Furthermore, the gaze operation may include selecting a gaze operation target, editing the gaze operation target, viewing information, presenting an internal state, or presenting a list (menu) of executable operations. Furthermore, the operation content of the gaze operation may correspond to the operation content of a right-click operation or a left-click operation using a mouse.

[0193] Furthermore, the gaze operation target is not limited to a target existing in real space, such as the book 55. The gaze operation target may also be a target on a user interface, such as a menu, icon, or button, drawn using computer graphics (CG). The processing unit 10 may display the target on the user interface closer to the user (in front of) the target existing in real space. In this case, the user's convergence angle matches the target on the user interface that is positioned in front of the target in real space, so the intention to perform a gaze operation on the target on the user interface can be determined with higher accuracy.

[0194] Furthermore, the processing unit 10 may change the distance between the user and the gaze operation object after the user begins gazing at the gaze operation object. For example, when the processing unit 10 detects that the user has started gazing at the gaze operation object, it may move the gaze operation object in a direction increasing the distance from the user, as if the gaze had pushed the gaze operation object. When the user continues to gaze at a gaze operation object that is moving away from the user (moving farther away), a change occurs in the user's convergence angle. The processing unit 10 may determine that the user intends to perform a gaze operation on the gaze operation object when it detects that the change in the distance between the user and the gaze operation object and the change in the user's convergence angle are synchronized. This method allows for more accurate determination of the intention to perform a gaze operation on the gaze operation object. The gaze operation object whose distance changes depending on the user's gaze may be an object on a user interface rendered by CG or a simulated image of an object existing in real space.

[0195] Contrary to the above description, the processing unit 10 may move the gaze operation target in a direction that reduces the distance from the user when detecting that the user has started to gaze at the gaze operation target. The direction of movement of the gaze operation target may be arbitrarily set according to the user's preferences. Furthermore, the processing unit 10 may combine and apply the determination of the intention to perform a gaze operation using this method with the determination of the intention to perform a gaze operation based on biological information such as electroencephalograms. This allows for more accurate determination of the intention to perform a gaze operation on the gaze operation target.

[0196] After determining that the user intends to perform a gaze control using this method, the processing unit 10 may specify detailed operations by acquiring a spoken command from the user. For example, the processing unit 10 first determines that the user intends to perform a gaze control on the gaze control object by detecting that a change in distance between the user and the gaze control object is synchronized with a change in the user's convergence angle. Next, the processing unit 10 acquires the user's speech recorded by a microphone and extracts an operation command for the gaze control object by performing audio analysis on the speech content. Next, the processing unit 10 executes an operation corresponding to the extracted operation command on the data indicated by the gaze control object. Although there are existing technologies for operating a computer using spoken commands, these existing technologies require the user to utter a specified prefix before the spoken command in order to enable the computer to distinguish between normal speech and spoken commands. According to this method, the processing unit 10 can determine the user's intention to perform a gaze control by detecting that a change in distance is synchronized with a change in the convergence angle, thereby accurately estimating that the user's speech in that state is a spoken command. As a result, this method eliminates the need to pronounce a prescribed prefix, as in existing techniques.

[0197] Various common modifications common to the first to fourth embodiments will be described below. The modifications described below can be applied in any combination.

[0198] (First Common Modification) The display control unit 24 sets the default display position of the object to the display area 13 at the center of the display screen 13. ２２ If the detection accuracy is not equal to or higher than the allowable lower limit even after calibration, the display control unit 24 may set the display area 13 ２２ Alternatively, if the detection accuracy is equal to or higher than the lower limit of tolerance through calibration, but the detection accuracy falls below the lower limit of tolerance due to a misalignment of the AR glasses 1 or the like, the display control unit 24 displays the object in the display area 13 ２２ Display the object in the

[0199] According to this modification, it is possible to prevent the occurrence of a situation where an object cannot be displayed anywhere on the display screen 13.

[0200] (Second Common Modification) During calibration, the control information generator 22 may search for a depth distance at which the gaze direction detection accuracy is equal to or greater than a predetermined value not only in the in-plane direction of the display screen 13 but also in the depth direction of the display screen 13. During calibration, the display controller 24 virtually displays point P1 at multiple distances in the depth direction. The control information generator 22 calculates the detection accuracy for each region in the in-plane direction and each distance in the depth direction, and generates three-dimensional accuracy information 45A including information on the detection accuracy for the in-plane direction and the depth direction. Based on the accuracy information 45A, the gaze operation execution unit 23 identifies, as a high-precision region RH1, a display region corresponding to a field of view region where the detection accuracy is equal to or greater than a predetermined value in both the in-plane direction and the depth direction.

[0201] This modification can accommodate individual differences in the depth direction distance at which focusing is easy.

[0202] (Third Common Modification) The control information generation unit 22 may calculate the detection accuracy as background processing during actual operation after calibration. The control information generation unit 22 calculates the detection accuracy in a situation where the user's gaze can be determined during actual operation, such as when the user is inputting characters using eye gaze control or when the user is reading text included in an object. When the control information generation unit 22 detects that the detection accuracy has fallen below an allowable lower limit during actual operation, the display control unit 24 displays a message on the display screen 13 indicating that recalibration is required. Alternatively, when the control information generation unit 22 detects that the detection accuracy has fallen below an allowable lower limit during actual operation, the control information generation unit 22 may actively perform recalibration to automatically update the control information 33.

[0203] According to this modification, detection accuracy that has decreased for some reason can be restored by recalibration.

[0204] Furthermore, the control information generator 22 may additionally train the estimation model 34 by using, as training data, the user's gaze direction detected in a situation where the user's gaze can be determined during actual operation and the user's eyeball information acquired at that time. This can improve the estimation accuracy of the estimation model 34.

[0205] The present disclosure is widely applicable to AR glasses, VR goggles, smart glasses, head-mounted displays, computer user interfaces, digital assistants, information terminals, wearable devices, or stationary gaze-controlled systems, etc.

Claims

1. An information processing method in which an information processing device detects a user's gaze direction, calculates the detection accuracy of the gaze direction for each of a plurality of field of view areas included in the user's field of view, and executes a display control process that controls the display content on a display screen based on control information related to the detection accuracy for each of the field of view areas.

2. The information processing method according to claim 1, further comprising: executing a gaze operation which is an operation based on the detected gaze direction; and a control target in the display control process is an object for gaze operation displayed on the display screen.

3. The information processing method according to claim 2, wherein the display control process controls a display position of the object on the display screen.

4. The information processing method of claim 3, wherein the control information is accuracy information indicating the detection accuracy for each field of view area, and in the display control process, the object is displayed in a high accuracy area, which is a display area on the display screen corresponding to a field of view area in which the detection accuracy is equal to or greater than a predetermined value, based on the accuracy information.

5. The information processing method according to claim 4, wherein the predetermined value differs depending on an attribute of the object.

6. The information processing method of claim 4, further comprising: acquiring an image capturing the user's field of vision; and generating a simulated image of an object to be operated by gaze control contained in the image when the object is located outside the high-precision area on the display screen; and in the display control process, the object includes the simulated image.

7. The information processing method of claim 3, further comprising: setting a designated display area on the display screen in which the object is to be displayed based on the detection accuracy for each field of view area; the control information being area information indicating the designated display area; and displaying the object in the designated display area based on the area information in the display control process.

8. The information processing method of claim 7, wherein in setting the display designated area, a display area of ​​the display screen corresponding to a field of view area where the detection accuracy is equal to or greater than a first threshold is set as a first display designated area, and a display area of ​​the display screen corresponding to a field of view area where the detection accuracy is less than the first threshold and equal to or greater than a second threshold is set as a second display designated area, and in the display control process, either the first display designated area or the second display designated area is selected as the display position of the object depending on the attributes of the object.

9. The information processing method of claim 7, wherein in setting the display designated area, a display area of ​​the display screen corresponding to a field of view area where the detection accuracy is equal to or greater than a first threshold is set as a first display designated area, and a display area of ​​the display screen corresponding to a field of view area where the detection accuracy is less than the first threshold and equal to or greater than a second threshold is set as a second display designated area, and in the display control process, the display manner of the object is made different between when the object is displayed in the first display designated area and when the object is displayed in the second display designated area.

10. An information processing method as described in claim 7, wherein, in setting the display designation area, a display area of ​​the display screen corresponding to a field of view area where the detection accuracy is equal to or greater than a first threshold is set as a first display designation area, and a display area of ​​the display screen corresponding to a field of view area where the detection accuracy is less than the first threshold and equal to or greater than a second threshold is set as a second display designation area, and, in performing the gaze operation, a judgment criterion or judgment method for the gaze operation is made different for an object displayed in the first display designation area and an object displayed in the second display designation area.

11. The information processing method of claim 7, further comprising: acquiring an image capturing the user's field of view; identifying an inappropriate area that is not suitable as a display position for the object based on the image; and, in setting the display designation area, not setting the display designation area in a display area of ​​the display screen that corresponds to the inappropriate area.

12. The information processing method of claim 7, further comprising: acquiring an image capturing the user's field of vision; and generating a simulated image of an object to be operated by gaze control contained in the image when the object is located outside the designated display area on the display screen; and in the display control process, the object includes the simulated image.

13. The information processing method according to claim 1, further comprising: executing a guidance process for guiding the head direction of the user based on the detected gaze direction and the control information.

14. The information processing method according to claim 13, further comprising: executing a gaze operation that is an operation based on the detected gaze direction; and executing the guidance process before the gaze operation.

15. The information processing method according to claim 13, wherein in the guiding process, the direction of the user's head is guided by at least one of force feedback, tactile stimulation, display of guiding instructions, and audio output of guiding instructions.

16. The information processing method of claim 13, wherein in the guidance process, visibility of objects included in a display area of ​​the display screen corresponding to a field of view area in which the detection accuracy is less than a predetermined value is reduced compared to objects included in a display area of ​​the display screen corresponding to a field of view area in which the detection accuracy is equal to or greater than a predetermined value.

17. The information processing method of claim 13, further comprising selecting either the display control process or the guidance process based on at least one of attribute information of the object, information on the user's surrounding environment, information on the user's head posture, information on the user's behavioral status, and information on the user's preferences.

18. The information processing method of claim 13, further comprising: executing a gaze operation which is an operation based on the detected gaze direction; determining whether or not the user intends to perform the gaze operation; and if it is determined that the user does not intend to perform the gaze operation, not executing the display control process and the guidance process.

19. The information processing method according to claim 1, wherein the information processing device is mounted on a glasses-type or goggles-type device worn by the user.

20. An information processing method, in which an information processing device detects a user's gaze direction, calculates the detection accuracy of the gaze direction for each of a plurality of field of view areas included in the user's field of view, and executes a guidance process to guide the user's head direction based on the detected gaze direction and control information related to the detection accuracy for each field of view area.

21. An information processing device that detects a user's gaze direction, calculates the detection accuracy of the gaze direction for each of a plurality of field of view areas included in the user's field of view, and executes a display control process that controls the display content on a display screen based on control information related to the detection accuracy for each of the field of view areas.

22. An information processing device that detects a user's gaze direction, calculates the detection accuracy of the gaze direction for each of a plurality of field of view areas included in the user's field of view, and executes a guidance process to guide the user's head direction based on the detected gaze direction and control information related to the detection accuracy for each field of view area.

23. A program for causing an information processing device to execute a process, the process comprising: detecting a user's gaze direction; calculating a detection accuracy of the gaze direction for each of a plurality of field of view areas included in the user's field of view; and executing a display control process that controls the display content on a display screen based on control information regarding the detection accuracy for each of the field of view areas.

24. A program for causing an information processing device to execute a process, the process comprising: detecting a user's gaze direction; calculating a detection accuracy of the gaze direction for each of a plurality of field of view areas included in the user's field of view; and executing a guidance process for guiding the user's head direction based on the detected gaze direction and control information related to the detection accuracy for each field of view area.

Images