Information processing method, information processing device, and program

Abstract

An information processing device according to the present invention acquires line-of-sight information about the line of sight of a user, identifies a target object that is an object on which the line of sight of the user stays from at least one object included in the field of view of the user on the basis of the line-of-sight information, infers a situation of intent for the user with respect to selection of an object, sets a determination threshold value for a line-of-sight stay period on the basis of the inferred situation of intent, and determines whether the target object has been selected on the basis of a stay period that the line of sight of the user stays on the target object and the determination threshold value.

Description

Information processing method, information processing device, and program

[0001] The present disclosure relates to an information processing method, an information processing device, and a program.

[0002] Patent Literature 1 discloses an electronic device according to the background art. The electronic device sets the detection accuracy of gaze input to a low-accuracy mode (power-saving mode) when the user's gaze point is included in a specific area, and sets the detection accuracy of gaze input to a high-accuracy mode (normal mode) when the user's gaze point is not included in the specific area. For example, in the low-accuracy mode, the frame rate of an infrared camera for gaze detection is set lower than in the high-accuracy mode.

[0003] According to the background art, no consideration has been given to dynamically changing the determination threshold value related to the gaze dwell time.

[0004] International Publication No. 2014 / 084224

[0005] The present disclosure aims to provide an information processing method, an information processing device, and a program that are capable of dynamically changing a determination threshold value related to gaze dwell time based on the manner of a user's intention regarding object selection.

[0006] An information processing method according to one aspect of the present disclosure includes an information processing device that acquires gaze information regarding a user's gaze, identifies a target object on which the user's gaze is fixed from at least one object included in the user's field of view based on the gaze information, estimates the user's intention regarding object selection, sets a judgment threshold regarding gaze dwell time based on the estimated intention, and determines whether the target object has been selected based on the dwell time for which the user's gaze is fixed on the target object and the judgment threshold.

[0007] FIG. 1 is a simplified diagram showing the overall configuration of AR glasses according to a first embodiment. FIG. 2 is a schematic diagram of a left eye unit of the AR glasses viewed from the side of a user's face. FIG. 3 is a simplified diagram showing the functional configuration of a processing unit according to the first embodiment. FIG. 4 is a flowchart showing processing executed by a processing unit according to the first embodiment. FIG. 5 is a simplified diagram showing the functional configuration of a processing unit according to a second embodiment. FIG. 6 is a simplified diagram showing the overall configuration of AR glasses according to a third embodiment. FIG. 7 is a simplified diagram showing the functional configuration of a processing unit according to the third embodiment. FIG. 8 is a simplified diagram showing a first example of the functional configuration of a processing unit according to a fourth embodiment. FIG. 9 is a simplified diagram showing a second example of the functional configuration of a processing unit according to the fourth embodiment. FIG. 10 is a simplified diagram showing the overall configuration of AR glasses according to a fifth embodiment. FIG. 11 is a schematic diagram of a left eye unit of the AR glasses viewed from the side of a user's face. FIG. 12 is a simplified diagram showing a first example of the functional configuration of a processing unit according to the fifth embodiment. FIG. 13 is a simplified diagram showing a second example of the functional configuration of a processing unit according to the fifth embodiment. FIG. 14 is a simplified diagram showing the overall configuration of AR glasses according to a sixth embodiment. FIG. 15 is a simplified diagram showing a first example of the functional configuration of a processing unit according to the sixth embodiment. 13 is a diagram showing an example of setting a judgment threshold value. FIG. 14 is a diagram showing, in a simplified form, a second example of the functional configuration of a processing unit according to the sixth embodiment. FIG. 15 is a diagram showing, in a simplified form, the overall configuration of AR glasses according to a modified example of the sixth embodiment. FIG. 16 is a diagram showing, in a simplified form, an example of stay history information. FIG. 17 is a diagram showing, in a simplified form, the functional configuration of a processing unit according to a modified example of the sixth embodiment. FIG. 18 is a diagram showing, in a simplified form, an example of stay history information after correction. FIG. 19 is a diagram showing, in a simplified form, the functional configuration of a processing unit according to a seventh embodiment. FIG. 19 is a diagram showing, in a simplified form, an example of notification information. FIG. 19 is a diagram showing, in a simplified form, an example of notification information. FIG. 19 is a diagram showing, in a simplified form, an example of notification information.

[0008] (Findings forming the basis of the present disclosure) Research is underway into user interfaces for operating objects displayed on a screen in AR (Augmented Reality) glasses, VR (Virtual Reality) glasses, etc. In some products, a user moves their line of sight onto a desired object and then performs a predetermined hand gesture to operate the object.

[0009] However, when AR glasses are used to assist work in a factory, for example, it is difficult for a user to perform hand gestures while working with both hands. Therefore, it is desirable to realize a hands-free user interface that does not rely on hand gestures.

[0010] For example, a so-called gaze selection technique has been proposed in which an object is determined to have been selected by a user when the user gazes at the object for a period of time that the user's gaze remains on the object longer than a predetermined judgment threshold.

[0011] However, users' expectations regarding the gaze dwell time required to select an object vary depending on the usage situation. For example, in usage situations where multiple character objects need to be selected one after another, such as character input, users expect a short gaze dwell time. On the other hand, in usage situations where important objects such as a confirmation button or an agreement button need to be carefully selected, users expect a long gaze dwell time.

[0012] In the background art, the determination threshold for gaze dwell time is set to a fixed value, which results in longer input times and reduced operability in usage situations where a short gaze dwell time is expected, while in usage situations where a long gaze dwell time is expected, there is a possibility of unintended erroneous selections occurring.

[0013] In order to solve such problems, the inventor discovered that the problems of the background art can be solved by estimating the nature of a user's intention regarding object selection and dynamically changing a judgment threshold regarding gaze dwell time based on the nature of the estimated intention, and came up with the present disclosure.

[0014] Next, each aspect of the present disclosure will be described.

[0015] An information processing method according to a first aspect of the present disclosure includes an information processing device that acquires gaze information regarding a user's gaze, identifies a target object on which the user's gaze is fixed from at least one object included in the user's field of view based on the gaze information, estimates the user's intention regarding object selection, sets a judgment threshold regarding gaze dwell time based on the estimated intention, and determines whether the target object has been selected based on the dwell time for which the user's gaze is fixed on the target object and the judgment threshold.

[0016] According to the first aspect, the determination threshold for the gaze dwell time can be dynamically changed based on the user's intention regarding the selection of an object, thereby improving operability or object selection accuracy depending on the usage situation.

[0017] In the information processing method according to the second aspect of the present disclosure, in the first aspect, the aspect of intention may include an expected time of the user regarding a gaze dwell time required to select an object.

[0018] According to the second aspect, the determination threshold value for the gaze dwell time can be appropriately set based on the user's expectation of the gaze dwell time required to select an object.

[0019] In the information processing method according to a third aspect of the present disclosure, in the first or second aspect, the intention state may be estimated based on the line of sight information.

[0020] According to the third aspect, it is possible to appropriately estimate the user's intention regarding the selection of an object based on line-of-sight information regarding the user's line of sight.

[0021] An information processing method according to a fourth aspect of the present disclosure is the third aspect, wherein the line-of-sight information includes a line-of-sight movement speed.

[0022] According to the fourth aspect, the user's intended state is estimated based on the speed of movement of the line of sight, thereby improving the estimation accuracy.

[0023] In the information processing method according to the fifth aspect of the present disclosure, in the first or second aspect, in estimating the state of intention, the state of intention may be estimated based on information regarding the movement or state of the user's eyes.

[0024] According to the fifth aspect, it is possible to appropriately estimate the user's intention regarding the selection of an object based on information relating to the movement or state of the user's eyes.

[0025] In an information processing method according to a sixth aspect of the present disclosure, in the fifth aspect, the information regarding the user's eye movement or state may include blink frequency, eye opening degree, microsaccades, fixational eye movement, eye gestures, or pupil diameter.

[0026] According to the sixth aspect, the estimation accuracy can be improved by estimating the user's intention based on the blink frequency, the degree of eye opening, microsaccades, small eye movements, eye gestures, or pupil diameter.

[0027] In the information processing method according to a seventh aspect of the present disclosure, in the first or second aspect, the estimating of the state of intention may be performed based on biometric information of the user.

[0028] According to the seventh aspect, it is possible to appropriately estimate the user's intention regarding the selection of an object based on the user's biometric information.

[0029] In the information processing method according to the eighth aspect of the present disclosure, in the seventh aspect, the biometric information may include electroencephalograms, electrooculograms, electromyograms, pulse, electrocardiograms, skin resistance, body movement, blood flow, blood pressure, respiration, or voice.

[0030] According to the eighth aspect, the estimation accuracy can be improved by estimating the user's intention based on electroencephalograms, electrooculograms, electromyograms, pulse, electrocardiogram, skin resistance, body movement, blood flow, blood pressure, breathing, or voice.

[0031] In the information processing method according to a ninth aspect of the present disclosure, in the first or second aspect, the determination threshold may be set based on attribute information of the target object.

[0032] According to the ninth aspect, the determination threshold can be appropriately set based on the user's intention regarding the selection of an object and the attribute information of the target object.

[0033] In the information processing method according to a tenth aspect of the present disclosure, in the first or second aspect, the intention mode may be estimated based on attribute information of the target object.

[0034] According to the tenth aspect, it is possible to appropriately estimate the user's intention regarding the selection of an object based on the attribute information of the target object.

[0035] In the information processing method according to the eleventh aspect of the present disclosure, in the first or second aspect, when setting the judgment threshold, it is preferable to further set the judgment threshold based on the relevance between the target object and an object previously selected by the user among the at least one object.

[0036] According to the eleventh aspect, the determination threshold can be appropriately set based on the user's intention regarding the selection of an object and the relevance between the target object and the object previously selected by the user.

[0037] In the information processing method according to the twelfth aspect of the present disclosure, in the first or second aspect, in estimating the aspect of intention, it is preferable that the aspect of intention is estimated based on the relevance between the target object and an object previously selected by the user among the at least one object.

[0038] According to the twelfth aspect, it is possible to appropriately estimate the user's intention regarding the selection of an object, based on the relevance between the target object and an object previously selected by the user.

[0039] In the information processing method according to the thirteenth aspect of the present disclosure, in the first or second aspect, a predictive object is identified from among the at least one object, which is an object that is predicted to be selected next by the user, and in setting the judgment threshold, the judgment threshold is further set based on whether the target object is the predictive object or not.

[0040] According to the thirteenth aspect, the determination threshold can be appropriately set based on the mode of the user's intention regarding the selection of an object and whether or not the target object is a predictive object.

[0041] In the information processing method according to the fourteenth aspect of the present disclosure, in the first or second aspect, a predictive object is identified from among the at least one object, which is an object that is predicted to be the next selection by the user, and in estimating the mode of intention, the mode of intention is estimated based on whether the target object is the predictive object or not.

[0042] According to the fourteenth aspect, it is possible to appropriately estimate the mode of the user's intention regarding the selection of an object based on whether or not the target object is a predictive object.

[0043] An information processing method according to a fifteenth aspect of the present disclosure is the thirteenth or fourteenth aspect, wherein in setting the determination threshold, the higher the prediction probability, the smaller the determination threshold is set.

[0044] According to the fifteenth aspect, the higher the prediction probability, the smaller the determination threshold is set, thereby making it possible to set the determination threshold appropriately.

[0045] In the information processing method according to the sixteenth aspect of the present disclosure, in the thirteenth or fourteenth aspect, further includes storing dwell history information indicating a dwell history of the user's gaze on an object, the dwell history information including a correspondence between an object, a dwell time that the user's gaze dwelled on the object, and the judgment threshold set for the object, and identifying a missing object that is an object that is presumed to have been missing from selection based on two or more objects selected by the user, and correcting the judgment threshold corresponding to the missing object in the dwell history information to be equal to or less than the dwell time corresponding to the missing object.

[0046] According to the sixteenth aspect, the missing object can be processed as a selected object by correcting the determination threshold corresponding to the missing object to a value equal to or less than the dwell time corresponding to the missing object, thereby automatically correcting erroneous inputs due to eye-gaze operations.

[0047] In an information processing method according to a seventeenth aspect of the present disclosure, in any one of the first to sixteenth aspects, notification information indicating the set judgment threshold may be created, and the notification information may be associated with the target object and notified.

[0048] According to the seventeenth aspect, the gaze dwell time corresponding to the determination threshold can be easily notified to the user.

[0049] An information processing device according to an eighteenth aspect of the present disclosure is an information processing device including a processing unit, which acquires gaze information regarding a user's gaze, identifies a target object on which the user's gaze is fixed from at least one object included in the user's field of view based on the gaze information, estimates the user's intention regarding object selection, sets a judgment threshold regarding gaze dwell time based on the estimated intention, and determines whether the target object has been selected based on the dwell time for which the user's gaze is fixed on the target object and the judgment threshold.

[0050] According to the eighteenth aspect, the determination threshold for the gaze dwell time can be dynamically changed based on the user's intention regarding the selection of an object, thereby improving operability or object selection accuracy depending on the usage situation.

[0051] A program according to a 19th aspect of the present disclosure is a program for causing an information processing device to execute a process, the process acquiring gaze information regarding a user's gaze, identifying a target object on which the user's gaze is fixed from at least one object included in the user's field of view based on the gaze information, estimating the user's intention regarding object selection, setting a judgment threshold regarding gaze dwell time based on the estimated intention, and determining whether the target object has been selected based on the dwell time for which the user's gaze is fixed on the target object and the judgment threshold.

[0052] According to the nineteenth aspect, the determination threshold for the gaze dwell time can be dynamically changed based on the user's intention regarding the selection of an object, thereby improving operability or object selection accuracy depending on the usage situation.

[0053] The present disclosure can also be realized as a program that causes a computer to execute each characteristic configuration included in such a method or apparatus, or as a system operated by this program. It goes without saying that such a computer program can be distributed on a computer-readable non-transitory recording medium such as a CD-ROM or via a communication network such as the Internet. (Embodiments of the present disclosure)

[0054] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Elements with the same reference numerals in different drawings indicate the same or corresponding elements. Each embodiment can be applied in any combination. Furthermore, the components, the arrangement positions of the components, the connection forms, and the order of operations shown in the following embodiments are merely examples and are not intended to limit the present disclosure. The present disclosure is limited only by the claims. Therefore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concept of the present disclosure are not necessarily required to achieve the objectives of the present disclosure, but are described as constituting more preferred forms.

[0055] 1 is a simplified diagram showing the overall configuration of AR glasses 1 according to a first embodiment of the present disclosure. The AR glasses 1 include a display unit 11, a processing unit 12, an internal camera 13, and a storage unit 14.

[0056] The display unit 11 is configured using a liquid crystal display, an organic EL display, or the like. The processing unit 12 is configured using a processor (information processing device) such as a CPU. The internal camera 13 is configured using an optical system and a CMOS image sensor, or the like. The internal camera 13 captures moving images of both eyes of a user wearing the AR glasses 1. The storage unit 14 is configured using a semiconductor memory, or the like. The storage unit 14 stores a program 21 and graphic data 22. The storage unit 14 includes a computer-readable non-volatile storage medium. The program 21 is stored in the storage medium. The graphic data 22 includes image data of objects to be displayed on the display unit 11, etc.

[0057] 2 is a schematic diagram of the left eye unit of the AR glasses 1 viewed from the user's face side. Of the front and back surfaces of the left eye unit, the side facing the user's face (back side) corresponds to the internal side, and the side opposite the user's face (front side) corresponds to the external side. Note that the configuration of the right eye unit is the same as the configuration of the left eye unit, so the right eye unit is not shown in the figure.

[0058] The left-eye unit has a frame 30. An internal camera 13 is disposed inside the frame 30. A display unit 11 is disposed inside the frame 30. In the example of usage shown in FIG. 2, the display unit 11 includes display areas 41 and 42. In the display area 41, a character icon object 51 is virtually displayed, the characters of the alphabet "A" to "Z". In the display area 42, a command icon object 52 is virtually displayed, the characters of "convert", "confirm", or "return". Note that the objects 51 and 52 displayed on the display unit 11 are not limited to the example shown in FIG. 2.

[0059] Fig. 3 is a simplified diagram showing the functional configuration of the processing unit 12 according to the first embodiment. The functional configuration shown in Fig. 3 is realized by the processing unit 12 executing the program 21 read from the storage unit 14. The processing unit 12 includes a display control unit 31, a line-of-sight information acquisition unit 32, a target object identification unit 33, a measurement unit 34, an intention estimation unit 35, a setting unit 36, and a line-of-sight operation execution unit 37. Details of the processing content of each of these units will be described later.

[0060] FIG. 4 is a flowchart showing the processing executed by the processing unit 12 according to the first embodiment.

[0061] First, in step S01, the gaze information acquisition unit 32 acquires gaze information including the gaze direction of the user's gaze by calculating the eyeball direction based on image data D1 of the user's both eyes input from the internal cameras 13 of both eyes of the AR glasses 1. The gaze information acquisition unit 32 inputs data D2 indicating the gaze information to the target object identification unit 33, the measurement unit 34, and the intention estimation unit 35.

[0062] Next, in step S02, the target object identification unit 33 identifies a target object from among the multiple objects 51 and 52 included in the user's field of view. The target object is an object that the user is attempting to select through a gaze operation. Data D3 indicating the display positions of the objects 51 and 52 on the display unit 11 is input to the target object identification unit 33 from the display control unit 31. The display positions include the position coordinates of the objects 51 and 52 in the in-plane direction of the display screen of the display unit 11. Based on the data D2 and D3, the target object identification unit 33 identifies the gazed object as the target object when the user's gaze remains continuously at the display position of a specific object for a predetermined period of time or more, that is, when there is a gazed object that the user is gazing at. Specifically, when there is a gazed object whose position coordinates included in the data D3 match the gaze direction indicated by the data D2, the target object identification unit 33 identifies the gazed object as the target object. The target object identification unit 33 inputs data D4 indicating the identified target object to the gaze operation execution unit 37.

[0063] If the target object specifying unit 33 does not specify a target object (step S02: NO), the process ends.

[0064] If the target object identification unit 33 has identified the target object (step S02: YES), then in step S03, the intention estimation unit 35 estimates the user's intention regarding the selection of the object. The intention includes the user's expected gaze dwell time required to select an object. For example, in a usage situation where a user wants to select multiple objects 51 one after another when inputting characters, the user expects a short gaze dwell time, and the gaze movement speed within the display area 41 will be relatively fast. On the other hand, in a usage situation where a user wants to carefully select an object 52, such as when converting or confirming input characters, the user expects a long gaze dwell time, and the gaze movement speed within the display area 42 will be relatively slow. Data D2 indicating gaze information includes the user's gaze movement speed. The intention estimation unit 35 estimates the user's intention based on the gaze movement speed included in the data D2. For example, the intention estimation unit 35 estimates an intention state expecting a short gaze dwell time when the gaze movement speed is equal to or greater than a predetermined value, and estimates an intention state expecting a long gaze dwell time when the gaze movement speed is less than the predetermined value. The intention estimation unit 35 inputs data D6 indicating the estimated intention state of the user to the setting unit 36.

[0065] Next, in step S04, the setting unit 36 ​​sets a judgment threshold for gaze dwell time based on data D6. If the intention indicated by data D6 is one in which a short gaze dwell time is expected, the setting unit 36 ​​sets a first judgment threshold. On the other hand, if the intention indicated by data D6 is one in which a long gaze dwell time is expected, the setting unit 36 ​​sets a second judgment threshold that is longer than the first judgment threshold. The setting unit 36 ​​inputs data D7 indicative of the set judgment threshold to the gaze operation execution unit 37.

[0066] Next, in step S05, the measurement unit 34 measures the elapsed time since the gaze started to dwell on the target object (i.e., the gaze dwell time) based on the data D2. The measurement unit 34 inputs data D5 indicating the gaze dwell time to the gaze operation execution unit 37. The gaze operation execution unit 37 determines whether the gaze dwell time indicated by data D5 on the target object indicated by data D4 is equal to or greater than the determination threshold indicated by data D7.

[0067] If the gaze retention time is less than the determination threshold (step S05: NO), the process ends.

[0068] If the gaze retention time is equal to or longer than the determination threshold (step S05: YES), then in step S06, the gaze operation execution unit 37 determines that a target object has been selected by the user, and executes a gaze operation corresponding to the selected target object. If the selected target object is object 51, the gaze operation execution unit 37 executes a corresponding character input process, and if the selected target object is object 52, it executes a corresponding command process.

[0069] According to this embodiment, the judgment threshold for gaze dwell time can be dynamically changed based on the user's intention regarding object selection. As a result, in a usage situation where a short gaze dwell time is expected, the first judgment threshold can be set to be shorter than the second judgment threshold, thereby shortening the input time and improving operability. Furthermore, in a usage situation where a long gaze dwell time is expected, the second judgment threshold can be set to be longer than the first judgment threshold, thereby improving the accuracy of object selection by avoiding or suppressing erroneous selection.

[0070] Furthermore, according to this embodiment, it is possible to appropriately set a determination threshold value for the gaze dwell time based on the user's expected gaze dwell time required to select an object.

[0071] Furthermore, according to this embodiment, it is possible to appropriately estimate the user's intention regarding the selection of an object based on line-of-sight information regarding the user's line of sight.

[0072] Furthermore, according to this embodiment, the accuracy of estimation can be improved by estimating the mode of the user's intention based on the speed of movement of the line of sight.

[0073] Second Embodiment In the first embodiment, the intention estimation unit 35 estimated the aspect of the user's intention based on the speed of gaze movement. However, the intention estimation unit 35 may also estimate the aspect of the user's intention based on information regarding the movement or state of the user's eyes.

[0074] FIG. 5 is a simplified diagram illustrating the functional configuration of the processing unit 12 according to the second embodiment of the present disclosure. Image data D1 of the user's eyes is input to the intention estimation unit 35 from the internal camera 13. The intention estimation unit 35 calculates the blinking frequency or eye opening degree based on the image data D1. The intention estimation unit 35 estimates the user's intention based on the calculated blinking frequency or eye opening degree. For example, if the blinking frequency is less than a threshold value or the eye opening degree is equal to or greater than a threshold value, the intention estimation unit 35 estimates that the user is mentally concentrating and has or is likely to have the intention to select an object. On the other hand, if the blinking frequency is equal to or greater than the threshold value or the eye opening degree is less than the threshold value, the intention estimation unit 35 estimates that the user is not mentally concentrating and has or is unlikely to have the intention to select an object. The intention estimation unit 35 inputs data D6 indicating the estimated user's intention to the setting unit 36. The intention estimation unit 35 may estimate the state of the user's intention based on microsaccades, small fixational eye movements, eye gestures, or pupil diameter instead of the blink frequency or eye opening degree.

[0075] The setting unit 36 ​​sets a judgment threshold for gaze dwell time based on data D6. When the intention indicated by data D6 indicates an intention to select an object or a possibility of such intention (i.e., a situation in which the user expects a short gaze dwell time), the setting unit 36 ​​sets a first judgment threshold. On the other hand, when the intention indicated by data D6 indicates an intention not to select an object or a possibility of such intention (i.e., a situation in which the user expects a long gaze dwell time), the setting unit 36 ​​sets a second judgment threshold that is longer than the first judgment threshold. The setting unit 36 ​​inputs data D7 indicating the set judgment threshold to the gaze operation execution unit 37. The following processing is the same as in the first embodiment.

[0076] According to this embodiment, it is possible to appropriately estimate the mode of the user's intention regarding the selection of an object based on information regarding the movement or state of the user's eyes.

[0077] Furthermore, according to this embodiment, the estimation accuracy can be improved by estimating the user's intention based on the blink frequency, eye opening degree, microsaccades, fixational eye movement, eye gestures, or pupil diameter.

[0078] Third Embodiment In the first embodiment, the intention estimation unit 35 estimated the state of the user's intention based on the speed of gaze movement. However, the intention estimation unit 35 may also estimate the state of the user's intention based on biometric information of the user.

[0079] 6 is a simplified diagram showing the overall configuration of AR glasses 1 according to a third embodiment of the present disclosure. The AR glasses 1 further include a sensor 15 in addition to the configuration shown in FIG. 1. The sensor 15 is disposed on the AR glasses 1 and detects biometric information of a user wearing the AR glasses 1. The biometric information includes electroencephalograms, electrooculograms, electromyograms, pulse, electrocardiograms, or skin resistance.

[0080] FIG. 7 is a simplified diagram illustrating the functional configuration of the processing unit 12 according to the third embodiment. The processing unit 12 further includes a biometric information acquisition unit 61 in addition to the configuration illustrated in FIG. 3 . The biometric information acquisition unit 61 acquires the user's biometric information based on output data D11 from the sensor 15 and inputs data D12 representing the acquired biometric information to the intention estimation unit 35. The intention estimation unit 35 estimates the user's intention based on the user's biometric information represented by the data D12. For example, if a specific readiness potential is present in the electroencephalogram, or if the electrooculogram, electromyogram, pulse rate, electrocardiogram, or skin resistance is equal to or greater than a threshold, the intention estimation unit 35 estimates that the user has an intention to select an object or that the possibility of such intention is high. On the other hand, if a specific readiness potential is not present in the electroencephalogram, or if the electrooculogram, electromyogram, pulse rate, electrocardiogram, or skin resistance is less than a threshold, the intention estimation unit 35 estimates that the user has no intention to select an object or that the possibility of such intention is low. The intention estimation unit 35 inputs data D6 indicating the estimated mode of the user's intention to the setting unit 36. Note that the biological information is not limited to the above-mentioned electroencephalogram, electrooculography, electromyography, pulse, electrocardiogram, or skin resistance, but may also include body movement (gestures, head movement, etc.), blood flow, blood pressure, respiration, or voice.

[0081] The setting unit 36 ​​sets a judgment threshold for gaze dwell time based on data D6. When the intention indicated by data D6 indicates an intention to select an object or a possibility of such intention (i.e., a situation in which the user expects a short gaze dwell time), the setting unit 36 ​​sets a first judgment threshold. On the other hand, when the intention indicated by data D6 indicates an intention not to select an object or a possibility of such intention (i.e., a situation in which the user expects a long gaze dwell time), the setting unit 36 ​​sets a second judgment threshold that is longer than the first judgment threshold. The setting unit 36 ​​inputs data D7 indicating the set judgment threshold to the gaze operation execution unit 37. The following processing is the same as in the first embodiment.

[0082] According to this embodiment, it is possible to appropriately estimate the user's intention regarding the selection of an object based on the user's biometric information.

[0083] Furthermore, according to this embodiment, the estimation accuracy can be improved by estimating the user's intentions based on electroencephalograms, electrooculograms, electromyograms, pulse, electrocardiograms, skin resistance, body movement, blood flow, blood pressure, breathing, or voice, etc.

[0084] (Fourth embodiment) In the first embodiment, the setting unit 36 ​​sets the judgment threshold for the gaze dwell time based on the manner of the user's intention regarding the selection of an object. However, the setting unit 36 ​​may also set the judgment threshold based on attribute information of the target object.

[0085] FIG. 8 is a simplified diagram illustrating a first example of the functional configuration of the processing unit 12 according to the fourth embodiment of the present disclosure. The processing unit 12 further includes an attribute information acquisition unit 62 in addition to the configuration illustrated in FIG. 3 . The attribute information acquisition unit 62 acquires attribute information indicating attributes of the target object based on data D4. The attributes include an index of visibility or importance of the object. The index of visibility includes the size, display position, brightness, clarity, or saturation of the object. The importance varies depending on the type of object; for example, the importance of a command icon is higher than the importance of a text icon. The importance also includes the complexity of the recovery process in the event of an erroneous selection; for example, the complexity of a page transition icon is higher than the complexity of an action playback icon. The attribute information acquisition unit 62 inputs data D13 indicating the attribute information of the target object to the setting unit 36.

[0086] The setting unit 36 ​​sets a judgment threshold for the gaze dwell time based on the data D6 and D13. For example, the setting unit 36 ​​sets a shorter judgment threshold as the likelihood of the intent to select an object increases, the value of the visibility index decreases, or the importance decreases. On the other hand, the setting unit 36 ​​sets a longer judgment threshold as the likelihood of the intent to select an object decreases, the value of the visibility index increases, or the importance increases. Conversely, the setting unit 36 ​​may set a longer judgment threshold as the value of the visibility index decreases or the importance decreases, or may set a shorter judgment threshold as the value of the visibility index increases or the importance increases. The setting unit 36 ​​inputs data D7 indicating the set judgment threshold to the gaze operation execution unit 37. The following processing is the same as in the first embodiment.

[0087] In addition, in the first embodiment, the intention estimation unit 35 estimated the state of the user's intention based on the speed of gaze movement, but the intention estimation unit 35 may also estimate the state of the user's intention based on attribute information of the target object.

[0088] FIG. 9 is a simplified diagram illustrating a second example of the functional configuration of the processing unit 12 according to the fourth embodiment. The processing unit 12 further includes an attribute information acquisition unit 62 in addition to the configuration illustrated in FIG. 3 . The attribute information acquisition unit 62 inputs data D13 indicating attribute information of the target object to the intention estimation unit 35. The intention estimation unit 35 estimates the user's intention based on the attribute information indicated by the data D13. For example, the intention estimation unit 35 estimates that the smaller the value of the visibility index or the lower the importance, the greater the intention or likelihood of selecting the object. On the other hand, the intention estimation unit 35 estimates that the larger the value of the visibility index or the higher the importance, the less the intention or likelihood of selecting the object. Note that, conversely to the above, the intention estimation unit 35 may estimate that the smaller the value of the visibility index or the lower the importance, the less the intention or likelihood of selecting the object. Alternatively, the intention estimation unit 35 may estimate that the larger the value of the visibility index or the lower the importance, the less the intention or likelihood of selecting the object. The intention estimation unit 35 inputs data D6 indicating the estimated mode of the user's intention to the setting unit 36. The subsequent processing is the same as in the first embodiment.

[0089] According to this embodiment, the determination threshold can be appropriately set based on the user's intention regarding the selection of an object and the attribute information of the target object.

[0090] Furthermore, according to this embodiment, it is possible to appropriately estimate the user's intention regarding the selection of an object based on the attribute information of the target object.

[0091] (Fifth embodiment) In the first embodiment, the setting unit 36 ​​sets the judgment threshold for the gaze dwell time based on the user's intention regarding the selection of an object. However, the setting unit 36 ​​may also set the judgment threshold based on the relevance between the object previously selected by the user and the target object.

[0092] FIG. 10 is a simplified diagram showing the overall configuration of AR glasses 1 according to a fifth embodiment of the present disclosure. The storage unit 14 further stores selection history information 23 and relevance information 24. The selection history information 23 is history information indicating objects previously selected by the user, and includes information on the object previously selected by the user. The relevance information 24 is information indicating the degree of relevance between multiple objects. If a specific object is likely to be selected after a certain object is selected, the relevance between these two objects is high. The relevance information 24 may be information indicating a predetermined rule or may be an estimation model trained by machine learning. The estimation model estimates the probability that the target object will be selected based on information indicating the previously selected object and information indicating the target object.

[0093] 11 is a schematic diagram of the left eye unit of the AR glasses 1 as viewed from the user's face. In the example of usage shown in FIG. 11, the display unit 11 includes a display area 43. An object displaying the text of a predetermined set of terms of use is virtually displayed in the display area 43. In addition, an object 53 displaying the words "I agree" is virtually displayed below the display area 43. Because the user is likely to read the terms of use before selecting the object 53, these two objects are highly related.

[0094] 12 is a simplified diagram showing a first example of the functional configuration of the processing unit 12 according to the fifth embodiment. The processing unit 12 further includes a relevance determination unit 63 in addition to the configuration shown in FIG. 3. The relevance determination unit 63 calculates the degree of relevance between the previously selected object and the target object based on data D4 indicating the target object, data D14 indicating the selection history information 23, and data D15 indicating the relevance information 24. The relevance determination unit 63 inputs data D16 indicating the calculated relevance to the setting unit 36.

[0095] The setting unit 36 ​​sets a judgment threshold for the gaze dwell time based on the data D6 and D16. For example, the setting unit 36 ​​sets a shorter judgment threshold the more likely it is that the intent to select an object is present, or the higher the relationship between the previously selected object and the target object is. On the other hand, the setting unit 36 ​​sets a longer judgment threshold the less likely it is that the intent to select an object is present, or the lower the relationship between the previously selected object and the target object is. The setting unit 36 ​​inputs data D7 indicating the set judgment threshold to the gaze operation execution unit 37. The following processing is the same as in the first embodiment.

[0096] In addition, in the first embodiment, the intention estimation unit 35 estimated the aspect of the user's intention based on the speed of gaze movement, but the intention estimation unit 35 may also estimate the aspect of the user's intention based on the relevance between the object previously selected by the user and the target object.

[0097] FIG. 13 is a simplified diagram illustrating a second example of the functional configuration of the processing unit 12 according to the fifth embodiment. The processing unit 12 further includes a relevance determination unit 63 in addition to the configuration illustrated in FIG. 3 . The relevance determination unit 63 inputs data D16 indicating the calculated relevance to the intention estimation unit 35. The intention estimation unit 35 estimates the user's intention based on the relevance indicated by the data D16. For example, the intention estimation unit 35 estimates that the higher the relevance between the previously selected object and the target object, the higher the intention or possibility of selecting the object. On the other hand, the lower the relevance between the previously selected object and the target object, the lower the intention or possibility of not selecting the object. The intention estimation unit 35 inputs data D6 indicating the estimated user's intention to the setting unit 36. The following processing is similar to that of the first embodiment.

[0098] According to this embodiment, the determination threshold can be appropriately set based on the user's intention regarding the selection of an object and the relevance between the target object and the object previously selected by the user.

[0099] Furthermore, according to this embodiment, it is possible to appropriately estimate the user's intention regarding the selection of an object based on the relevance between the target object and the object previously selected by the user.

[0100] (Sixth embodiment) In the first embodiment, the setting unit 36 ​​sets the judgment threshold for the gaze dwell time based on the manner of the user's intention regarding the selection of an object, but the setting unit 36 ​​may further set the judgment threshold based on whether the target object is a predicted object or not.

[0101] 14 is a simplified diagram showing the overall configuration of AR glasses 1 according to a sixth embodiment of the present disclosure. The storage unit 14 further stores selection history information 23 and dictionary information 25. Various words are registered in advance in the dictionary information 25.

[0102] 15 is a simplified diagram illustrating a first example of the functional configuration of the processing unit 12 according to the sixth embodiment. The processing unit 12 further includes a prediction unit 64 in addition to the configuration illustrated in FIG. 3 . The prediction unit 64 identifies a predictive object, which is an object predicted to be selected next by the user, from among the multiple objects 51, based on data D14 indicating the selection history information 23 and data D17 indicating the dictionary information 25, and calculates the predicted probability. Furthermore, the prediction unit 64 determines whether the target object is a predictive object based on data D4 indicating the target object, and inputs data D18 indicating the determination result and the predicted probability to the setting unit 36.

[0103] The setting unit 36 ​​sets a judgment threshold for the gaze dwell time based on the data D6 and D18. For example, the setting unit 36 ​​sets a shorter judgment threshold as the possibility of the intention to select an object increases, or as the target object matches the predicted object and the predicted probability increases. On the other hand, the setting unit 36 ​​sets a longer judgment threshold as the possibility of the intention to select an object decreases, or as the target object does not match the predicted object and the predicted probability decreases. The setting unit 36 ​​inputs data D7 indicating the set judgment threshold to the gaze operation execution unit 37. The following processing is the same as in the first embodiment.

[0104] 16 is a diagram showing an example of setting the judgment threshold. When the character string "PANASONIC" is input in this input order, the prediction probability increases toward the end of the character string, so the setting unit 36 ​​sets a shorter judgment threshold toward the end of the character string. In other words, the setting unit 36 ​​sets a smaller judgment threshold as the prediction probability increases.

[0105] In addition, in the first embodiment, the intention estimation unit 35 estimated the aspect of the user's intention based on the speed of gaze movement, but the intention estimation unit 35 may also estimate the aspect of the user's intention based on whether the target object is a predicted object or not.

[0106] FIG. 17 is a simplified diagram illustrating a second example of the functional configuration of the processing unit 12 according to the sixth embodiment. The processing unit 12 further includes a prediction unit 64 in addition to the configuration illustrated in FIG. 3 . The prediction unit 64 inputs data D18 indicating a determination result as to whether the target object is a predicted object and the prediction probability to the intention estimation unit 35. The intention estimation unit 35 estimates the user's intention based on the determination result and prediction probability indicated by the data D18. For example, the intention estimation unit 35 estimates that the higher the prediction probability of the target object matching the predicted object, the higher the intention or possibility of selecting the object. On the other hand, the lower the prediction probability of the target object not matching the predicted object, the lower the intention or possibility of selecting the object. The intention estimation unit 35 inputs data D6 indicating the estimated user's intention to the setting unit 36. The following processing is similar to that of the first embodiment.

[0107] According to this embodiment, the determination threshold can be appropriately set based on the mode of the user's intention regarding the selection of an object and whether or not the target object is a predictive object.

[0108] Furthermore, according to this embodiment, it is possible to appropriately estimate the mode of the user's intention regarding the selection of an object based on whether or not the target object is a predictive object.

[0109] 18 is a simplified diagram showing the overall configuration of the AR glasses 1 according to a modification of the sixth embodiment. The storage unit 14 further stores retention history information 26 that indicates a history of the user's gaze retention on an object.

[0110] FIG. 19 is a diagram showing an example of the dwell history information 26. The dwell history information 26 includes a correspondence relationship between an input order, a target object, a judgment threshold set for the target object, and the dwell time that the user's gaze dwells on the target object. In this example, the dwell history information 26 includes the dwell time of the user's gaze corresponding to each character of the string "PANASONIC." For example, the dwell time corresponding to the character "P" is 0.6 seconds, and the dwell time corresponding to the character "S" is 0.4 seconds. Here, the judgment threshold corresponding to the character "S" is 0.5 seconds, so the character "S" is not selected. In other words, the object 51 corresponding to the character "S" is a missing object that was not selected.

[0111] Fig. 20 is a simplified diagram showing the functional configuration of the processing unit 12 according to a modification of the sixth embodiment, based on the first example shown in Fig. 15. However, this modification can also be applied based on the second example shown in Fig. 17.

[0112] The processing unit 12 further includes a correction unit 65 in addition to the configuration shown in Fig. 15. The correction unit 65 identifies a missing object, which is an object that is estimated to have been lost from selection, based on the data D17 and D19. In the example shown in Fig. 19, the correction unit 65 identifies the object 51 corresponding to the letter "S" as the missing object.

[0113] The correction unit 65 corrects the determination threshold value corresponding to the missing object in the stay history information 26 to be equal to or less than the stay time corresponding to the missing object, and re-executes the selection determination of the target object based on the stay history information 26 after the correction.

[0114] 21 is a diagram showing an example of the corrected stay history information 26. The determination threshold corresponding to the missing object of the letter "S" has been corrected from 0.5 seconds before the correction shown in FIG. 19 to 0.4 seconds, which is less than the stay time corresponding to the missing object. In the target object selection determination that is executed again, the stay time for the letter "S" is equal to or greater than the determination threshold (step S05: YES), so the object 51 corresponding to the letter "S" is correctly selected.

[0115] According to this modification, the determination threshold value corresponding to the missing object is corrected to be equal to or less than the dwell time corresponding to the missing object, so that the missing object can be treated as a selected object. As a result, erroneous input due to eye gaze operation can be automatically corrected.

[0116] When an unnecessary character is input by mistake, the correction unit 65 may correct the determination threshold value corresponding to the unnecessary object, which is an object of the unnecessary character, in the stay history information 26 to exceed the stay time corresponding to the unnecessary object, and may re-execute the selection determination of the target object based on the corrected stay history information 26. This allows the unnecessary object to be processed as an object that was not selected, and the unnecessary object to be deleted.

[0117] The correction unit 65 may also determine the retroactive range for automatically correcting an input error based on the user's character input speed (e.g., the number of characters input per unit time). For example, the correction unit 65 sets a first retroactive range for a user who inputs characters faster than a reference value, and sets a second retroactive range that is shorter than the first retroactive range for a user who inputs characters slower than the reference value.

[0118] In addition, instead of automatic correction, the correction unit 65 may pop up a confirmation message near the character string asking whether or not to make correction, and if the user inputs an answer indicating that correction will be made, perform the correction process for the incorrect input.

[0119] Seventh Embodiment The display control unit 31 may create notification information indicating the determination threshold value set by the setting unit 36, and may notify the notification information by associating it with the target object.

[0120] 22 is a diagram illustrating a simplified functional configuration of the processing unit 12 according to the seventh embodiment of the present disclosure. The setting unit 36 ​​inputs data D7 indicating the set determination threshold to the gaze operation execution unit 37 and the display control unit 31.

[0121] The display control unit 31 creates notification information indicating the determination threshold value indicated by the data D7, and displays the notification information on the display unit 11 in association with the target object.

[0122] 23A to 23D are diagrams showing examples of notification information. In this example, the setting unit 36 ​​sets the determination threshold to four levels: 0.3 seconds, 0.5 seconds, 0.7 seconds, and 1.0 seconds.

[0123] When the setting unit 36 ​​sets the judgment threshold to 0.3 seconds, the display control unit 31 displays an arrow graphic 71A having a length equivalent to 0.3 seconds as notification information near the target object, object 51, as shown in Figure 23A.

[0124] Similarly, when the setting unit 36 ​​sets the determination threshold to 0.5 seconds, the display control unit 31 displays an arrow graphic 71B having a length corresponding to 0.5 seconds near the object 51, as shown in FIG. 23B.

[0125] Similarly, when the setting unit 36 ​​sets the determination threshold to 0.7 seconds, the display control unit 31 displays an arrow graphic 71C having a length corresponding to 0.7 seconds near the object 51, as shown in FIG. 23C.

[0126] Similarly, when the setting unit 36 ​​sets the determination threshold to 1.0 second, the display control unit 31 displays an arrow graphic 71D having a length corresponding to 1.0 second near the object 51, as shown in FIG. 23D.

[0127] According to this embodiment, the gaze dwell time corresponding to the determination threshold can be easily notified to the user.

[0128] The display control unit 31 may notify the user of the determination threshold value by the moving speed of the arrow instead of the length of the arrow.

[0129] The notification method is not limited to a display, and the judgment threshold may be notified by the pitch of a sound or the magnitude of vibration, etc.

[0130] The present disclosure is widely applicable to AR glasses, VR glasses, smart glasses, head-mounted displays, computer user interfaces, digital assistants, information terminals, wearable devices, and the like.

Claims

1. An information processing method in which an information processing device acquires gaze information regarding a user's gaze, identifies a target object on which the user's gaze is fixed from at least one object included in the user's field of view based on the gaze information, estimates the user's intention regarding object selection, sets a judgment threshold regarding gaze dwell time based on the estimated intention, and determines whether the target object has been selected based on the dwell time for which the user's gaze is fixed on the target object and the judgment threshold.

2. The information processing method according to claim 1, wherein the aspect of the intention includes the user's expected gaze dwell time required to select an object.

3. The information processing method according to claim 1, wherein the inference of the mode of intention is based on the gaze information.

4. The information processing method according to claim 3, wherein the line-of-sight information includes a line-of-sight movement speed.

5. The information processing method according to claim 1, wherein the inference of the mode of intention is based on information relating to the movement or state of the user's eyes.

6. The information processing method according to claim 5, wherein the information relating to the user's eye movement or state includes blink frequency, eye opening degree, microsaccades, fixational eye movement, eye gestures, or pupil diameter.

7. The information processing method according to claim 1, wherein the inference of the mode of intention is based on biometric information of the user.

8. The information processing method according to claim 7, wherein the biological information includes electroencephalograms, electrooculograms, electromyograms, pulse, electrocardiograms, skin resistance, body movement, blood flow, blood pressure, respiration, or voice.

9. The information processing method according to claim 1, wherein the determination threshold value is set based on attribute information of the target object.

10. The information processing method according to claim 1, wherein the inference of the mode of intention is based on attribute information of the target object.

11. The information processing method according to claim 1, wherein in setting the judgment threshold, the judgment threshold is set based on the relevance between the target object and an object previously selected by the user among the at least one object.

12. The information processing method according to claim 1, wherein the inference of the mode of intention is based on the relevance between the target object and an object previously selected by the user from among the at least one object.

13. The information processing method of claim 1, further comprising identifying a predictive object, which is an object predicted to be selected next by the user, from among the at least one object; and in setting the judgment threshold, further setting the judgment threshold based on whether the target object is the predictive object.

14. The information processing method of claim 1, further comprising identifying a predictive object, which is an object predicted to be selected next by the user, from among the at least one object; and estimating the mode of intention based on whether the target object is the predictive object.

15. The information processing method according to claim 13 or 14, wherein the determination threshold is set to a smaller value as the predicted probability increases.

16. An information processing method according to claim 13 or 14, further comprising: storing dwell history information indicating a dwell history of the user's gaze on an object; the dwell history information including a correspondence between an object, a dwell time that the user's gaze dwelled on the object, and the judgment threshold set for the object; identifying a missing object that is an object presumed to have been omitted from selection based on two or more objects selected by the user; and correcting the judgment threshold corresponding to the missing object in the dwell history information to be equal to or less than the dwell time corresponding to the missing object.

17. The information processing method according to claim 1, further comprising creating notification information indicating the set judgment threshold value, and notifying the notification information in association with the target object.

18. An information processing device having a processing unit, wherein the processing unit: acquires gaze information regarding a user's gaze; identifies a target object on which the user's gaze is fixed from at least one object included in the user's field of view based on the gaze information; estimates the user's intention regarding object selection; sets a judgment threshold regarding gaze dwell time based on the estimated intention; and determines whether the target object has been selected based on the dwell time for which the user's gaze is fixed on the target object and the judgment threshold.

19. A program for causing an information processing device to execute a process, the process comprising: acquiring gaze information regarding a user's gaze; identifying a target object on which the user's gaze is fixed from at least one object included in the user's field of view based on the gaze information; estimating the user's intention regarding object selection; setting a judgment threshold regarding gaze dwell time based on the estimated intention; and determining whether the target object has been selected based on the dwell time for which the user's gaze is fixed on the target object and the judgment threshold.

Images