Information processing system, information processing method, and program

Abstract

To provide an information processing system equipped with a relative change type UI which provides no discrepancy between a user movement and a pointer movement and is excellent in operability.SOLUTION: The present invention is directed to an information processing apparatus 1 having a UI for recognizing an input of direction information based on a movement direction of augen. The apparatus has virtual fixation point calculating means 160 for calculating coordinates of a virtual fixation point based on the augen movement direction, grid generating means 170 for dividing a screen into a plurality of grids G and allocating an index to each grid to thereby acquire center coordinates of each grid, pointer projecting means 180 for setting positional coordinates of a pointer P based on the calculated coordinates of the virtual fixation point, and reference updating means 150 for, if the grid closest to the coordinates of the virtual fixation point is used as the current index and the current index is updated due to movement of the fixation point, resetting the center coordinates of the grid corresponding to the updated current index as the position coordinates of the virtual fixation point to, at this timing, update the augen reference position.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing method, and a program, and more particularly to a technique for inputting information by utilizing the movement of a user's eyes and / or face. [Background technology]

[0002] Conventionally, there has been a technology for operating information processing equipment such as a computer using a tracking device for the user's eyes and face, which calculates the coordinates of the point of gaze from the angle of the eyes or face and the screen position of the information processing device, and displays a pointer at that point (hereinafter referred to as the "pointer method"; see, for example, Patent Document 1).

[0003] FIG. 6(a) shows an example of displaying the gaze point by a pointer P. Displaying the gaze point using a pointer P is widely used because the position of the gaze point can be seen at a glance as shown in the figure, but there is a problem that a device with low precision has a large error, resulting in a discrepancy between the actual gaze point and the position of the pointer. In addition, since the pointer P always moves freely following the movement of the user's eyeball, concentration is required to stop the pointer at the desired position, and it takes some getting used to operating it. Furthermore, calibration is often required to obtain the reference position for gaze input every time the device is started, which is troublesome.

[0004] To solve this problem, the applicant has devised a method (hereinafter referred to as the "relative change method") of moving user interface (hereinafter referred to as the "UI") elements on the screen in accordance with the orientation only when the angle of the eyeballs or face changes by a certain amount or more. With this method, the reference position for gaze input is updated every time the cursor moves, so there is no need to perform calibration. In addition, because this is a one-off behavior, it has the advantage of working practically even if there is a large error. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2020-119093 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, the relative change method has the following problems. In the conventional relative change method, the gaze point is not displayed on the screen, so even if there is a discrepancy between the state of a UI element displayed by the device and the user's gaze point, the user cannot grasp the cause. In other words, in the conventional relative change method, the device does not recognize the user's gaze point and cannot feed back the gaze point to the user, so for example, if an unintended operation is performed due to a discrepancy in the gaze point, the user is confused because he or she cannot grasp the reason for the erroneous operation.

[0007] In addition, when applying the relative change method that does not display the gaze point to an information processing device with a small screen such as a smartphone, it is not possible to arrange many UI elements on the screen from the viewpoint of preventing erroneous operation. As a result, for example, a special UI (UI with 2 × 2 grids) as shown in Figure 6 (b) is created, which makes it difficult to grasp the overall picture and reduces operability.

[0008] The present invention has been made in consideration of such problems, and aims to provide an information processing system, an information processing method, and a program that are equipped with a relative change type UI that is easy to use and eliminates any discrepancy between the user's actions and the movement of the pointer. [Means for solving the problem]

[0009] In order to achieve the above object, an information processing system according to the present invention comprises: An information processing system including a user interface that recognizes an input of directional information based on a moving direction of the eyeballs and / or face from a reference position to an input position of the user's eyeballs and / or face that is acquired after the reference position is acquired, the user interface defining a reference position as a reference position and a position of the user's eyeballs and / or face that is acquired after the reference position is acquired, the user interface recognizing an input of directional information based on a moving direction of the eyeballs and / or face from the reference position to the input position, a virtual gaze point calculation means for calculating coordinates of a virtual gaze point based on a moving direction of the eyeball and / or face of the user; a grid creating means for dividing a screen into a plurality of grids, assigning an index to each grid, and acquiring a center coordinate of each grid in association with the index; a pointer projection means for setting position coordinates of a pointer to be displayed on the screen based on the calculated coordinates of the virtual point of gaze; The present invention is characterized in that it has a reference update means which sets the grid closest to the coordinates of the calculated virtual point of gaze as a current index, and when the current index is updated due to movement of the user's virtual point of gaze, resets the center coordinates of the grid corresponding to the updated current index as the position coordinates of the virtual point of gaze, and updates the reference position at the same timing.

[0010] In this invention, in an information processing system including a user interface that recognizes input of directional information based on the moving direction of the eyeballs and / or face, the coordinates of the virtual gaze point of the user are calculated, and a pointer is displayed on the screen based on the result, so that the gaze point recognized by the information processing system is fed back to the user. Therefore, according to the present invention, no discrepancy in the gaze point occurs between the information processing system and the user, and the user can perform operations using the relative change method without confusion.

[0011] Furthermore, in this invention, the information processing system grasps and utilizes the user's virtual point of gaze as coordinates on the screen, making it possible to accept more precise operations compared to the conventional relative change method that only observed single eye movements (changes in eyeball position).

[0012] An information processing system according to the present invention has the following configuration as a preferred embodiment.

[0013] (1) The virtual gaze point calculation means is characterized in that it sets the center of the screen as the origin and calculates the coordinates and displacement of the user's virtual gaze point from changes in the position of the user's eyeballs and / or face relative to this origin. According to this embodiment, since the coordinates of the virtual gaze point are calculated based on the change in the position of the user's eyeballs and / or face, it is possible to calculate the coordinates of the gaze point in a simpler and less expensive way than a general calculation method of the gaze point on the screen (for example, a method of calculating coordinates from the intersection of the line of sight vector and the screen plane, etc.). Therefore, it is possible to provide an information processing system suitable for application to small devices such as smartphones and tablet terminals.

[0014] (2) The virtual gaze point calculation means is characterized in that the closer the coordinates of the user's virtual gaze point are to the center of the screen, the lower the detection sensitivity of changes in the angle of the user's eyeballs and / or face. According to this embodiment, the closer the virtual gaze point coordinates are to the center of the screen, the lower the detection sensitivity of the change in the angle of the eyeballs and / or face, so that, for example, when the user's gaze is near the center of the screen, the movement of the virtual gaze point acquired by calculation becomes gentler. Therefore, in this embodiment, it becomes easier to focus the gaze on a UI element located near the center of the screen.

[0015] (3) The virtual gaze point calculation means is characterized in that it clips the calculated coordinates of the user's virtual gaze point so that they fall within the screen. According to this embodiment, it is possible to prevent the pointer displayed on the screen from moving off the screen, so that the pointer can be corrected to coincide with the actual point of gaze at the edge of the screen.

[0016] Further, the information processing method according to the present invention comprises: An information processing system including a user interface that recognizes an input of directional information based on a moving direction of the eyeballs and / or face from a reference position to an input position of a user's eyeballs and / or face that has been previously acquired, the input position being a reference position and the position of the user's eyeballs and / or face that has been acquired after the acquisition of the reference position, the input position being an input position, the user interface comprising: a virtual gaze point calculation step of calculating coordinates of a virtual gaze point based on a moving direction of the eyeball and / or face of the user; a grid creation step of dividing the screen into a plurality of grids, assigning an index to each grid, and acquiring a center coordinate of each grid in association with the index; a pointer projection step of setting position coordinates of a pointer to be displayed on the screen based on the calculated coordinates of the virtual gaze point; a reference update step of setting a grid closest to the coordinates of the calculated virtual point of gaze as a current index, and when the current index is updated by a movement of the virtual point of gaze of the user, resetting the center coordinates of the grid corresponding to the updated current index as the position coordinates of the virtual point of gaze, and updating the reference position at the same timing. It is characterized by:

[0017] In addition, the program according to the present invention is a central processing unit of an information processing system including a user interface that recognizes an input of direction information based on a moving direction of the eyeballs and / or face from a reference position to an input position of a user's eyeballs and / or face that has been previously acquired, the input position being a reference position and the position of the user's eyeballs and / or face that is acquired after the acquisition of the reference position being an input position; a virtual gaze point calculation step of calculating coordinates of a virtual gaze point based on a moving direction of the eyeball and / or face of the user; a grid creation step of dividing the screen into a plurality of grids, assigning an index to each grid, and acquiring a center coordinate of each grid in association with the index; a pointer projection step of setting position coordinates of a pointer to be displayed on the screen based on the calculated coordinates of the virtual gaze point; a reference update step of setting the grid closest to the coordinates of the calculated virtual gaze point as a current index, and when the current index is updated by the movement of the virtual gaze point of the user, resetting the center coordinates of the grid corresponding to the updated current index as the position coordinates of the virtual gaze point, and updating the reference position at the same timing. It is characterized by: Effect of the Invention

[0018] According to the present invention, since the user's virtual gaze point is displayed on the screen, the user can operate the UI while checking the position of the displayed gaze point. Therefore, the present invention can provide an information processing system equipped with a relative change type UI that is free of gaze point discrepancies and has excellent operability. In addition, since the UI operation can be performed by the relative change type while displaying the gaze point, for example, the UI can be accurately operated even when many UI elements are displayed on a small screen.

[0019] In addition, in the present invention, since the reference position is updated at the timing when the current index is updated due to the movement of the user's gaze point, for example, by appropriately changing the grid size, it is possible to adjust the update timing of the reference position so that the update frequency of the reference position is not excessive, in other words, so that the data processing load associated with the update of the reference position is not excessive. Therefore, it is possible to provide an information processing system equipped with a relative change type UI that can be smoothly operated even by an information processing device with a low data processing capacity such as a smartphone. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is an explanatory diagram outlining an information processing device to which the present invention is applied, in which FIG. 1(a) shows an example of a situation in which a user uses the information processing device, and FIG. 1(b) shows a schematic diagram of the movement of the user's eyeballs when inputting directional information into the information processing device. [Diagram 2] 2 is a schematic diagram showing a visualized example of a grid that divides the display unit of the information processing apparatus; FIG. [Diagram 3] 13 is a schematic diagram showing a visualized example of another example of the grid that divides the display unit of the information processing device. FIG. [Figure 4] FIG. 2 is a block diagram showing an example of a hardware configuration of the information processing device. [Diagram 5] 2 is a block diagram showing an example of a functional configuration of the information processing device. FIG. [Figure 6] 6 shows UIs of conventional information processing devices, with FIG. 6(a) showing a pointer-based UI and FIG. 6(b) showing a relative change-based UI. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and duplicated explanations will be omitted.

[0022] In addition, all of the matters described in this embodiment are merely examples of the embodiment of the present invention, and the technical scope of the present invention is not limited thereto. The term "system" in this embodiment is not limited to one realized using one computer, but also includes one realized using multiple computers that perform information communication with each other. The information processing system, information processing method, information processing program, and information processing device described in this embodiment can be appropriately changed in terms of their distributed and centralized aspects, and are not limited to those described in this embodiment.

[0023] In this embodiment, an example will be described in which the information processing system is realized by one information processing device 1. However, the information processing system may be realized by a so-called client-server system in which a plurality of information processing devices communicate with each other to perform specific information processing in cooperation with each other. In the following description, a case will be described in which the information processing device 1 is configured to recognize input of directional information based on the moving direction of the position of the user's eyeball.

[0024] (Basic configuration of relative change method) Fig. 1(a) illustrates an example of a situation where a user U uses an information processing device 1. Fig. 1(b) also illustrates a schematic example of the movement of the user's eyeball when inputting directional information. The information processing device 1 recognizes the input of directional information based on the eyeball movement of the user U as illustrated in Fig. 1(b), and performs processing according to a predetermined UI element in accordance with the input directional information.

[0025] Here, the UI element is a display element displayed on the display unit 30, and is related to selection processing, execution processing, and other information processing. The UI element may simply be in a selected state in response to the input of directional information, may perform a predetermined information processing in response to the input of directional information, or may be in a selected state and perform a predetermined information processing in response to the input of directional information. In other words, various display elements can be adopted as the UI element as long as they can execute a predetermined processing such as selection processing or execution processing with the input of directional information as a trigger.

[0026] Examples of UI elements that are simply set to a selected state in response to input of directional information include, for example, a list of selectable icons, cards, character elements, illustrations, etc., radio buttons, check boxes, drop-down lists, keyboards, etc. Examples of elements that perform predetermined information processing in response to input of directional information include, for example, displaying the next photo in a slide show of photos, displaying the next page in an e-book, etc. Examples of elements that are set to a selected state in response to input of directional information and perform predetermined information processing in response to input of directional information include, for example, selection of content in a playlist of music or videos and playback processing, etc.

[0027] The eyeball position is the center of the pupil of the eyeball of the user U. The normal direction between the center of the eyeball and the eyeball position is the direction of the line of sight of the user U. The eyeball position α can be expressed, for example, by an eyeball coordinate system (r, θ, φ) represented by a radial coordinate r, which is the distance from the center of the eyeball, an angular coordinate θ formed by the vertical z-axis and the radial, and an angular coordinate φ formed by the x-axis, which is another axis in a plane perpendicular to the z-axis, and the projection of the radial onto the plane. Here, the radial coordinate r may be omitted as α=(θ, φ) since the distance from the center of the eyeball to the central part of the pupil is always constant.

[0028] The eyeball position may be represented by coordinates projected onto the XY plane of an image acquired by an imaging means such as a camera. In this case, the eyeball position may be represented by an image coordinate system (x, y) with an arbitrary point on the eye E as the origin, or may be represented as an image coordinate system (r, δ) with an arbitrary point on the eye E as the origin, a radial coordinate r that is the distance from the origin to the eyeball position α, and an angular coordinate δ formed by a predetermined axis x passing through the origin and the radial coordinate. In other words, the eyeball position may be represented by an eyeball coordinate system defined on the physical eyeball itself, or may be represented by an image coordinate system defined on an image including the eyeball, as long as it specifies the position of the center of the pupil of the user U, which indicates the direction of the user U's line of sight.

[0029] 1(a), a user U who uses the information processing device 1 sets the information processing device 1 in a position where the eyeball position of the user U can be recognized. As a position where the eyeball position of the user U can be recognized, it is preferable to set the information processing device 1 in a position where the face of the user U is directly facing the information processing device 1. However, the position of the face of the user U and the information processing device 1 may be in an oblique positional relationship as long as the eyeball position of the user U can be acquired by the information processing device 1.

[0030] 1(a), a smartphone is illustrated as an example of the information processing device 1, but the information processing device that realizes the information processing system is not limited to this. For example, the information processing system can be realized using various computers, such as tablet computers, laptop computers, desktop computers, monitors, printers, game machines, ticket vending machines, ATMs, wearable devices such as glasses with information processing functions, or operation panels of various devices.

[0031] FIG. 1(b) shows an example of input of directional information by eye movement. The leftmost row of vertically arranged eyeballs represents input of left directional information, the middle row of eyeballs represents input of up directional information, and the rightmost row of eyeballs represents input of right directional information. In addition, the first row of horizontally arranged eyeballs represents the reference eyeball position (reference position), the second row represents the input eyeball position (input position), and the third row represents the directional information input by directional input. Here, the reference eyeball position represents the eyeball position of user U acquired in the past, and the input eyeball position represents the eyeball position of user U acquired after the acquisition of the reference eyeball position.

[0032] As shown in the leftmost row of eyeballs in FIG. 1(b), the input of directional information by eyeball movement is recognized as an input of left directional information when the eyeball moves left from the reference eyeball position (i.e., right from the information processing device 1, based on the information processing device 1 facing the face of the user U). Similarly, as shown in the middle row of eyeballs in FIG. 1(b), the input of directional information by eyeball movement is recognized as an input of up directional information when the eyeball moves up from the reference eyeball position (i.e., up from the information processing device 1, based on the information processing device 1 facing the face of the user U). Similarly, as shown in the rightmost row of eyeballs in FIG. 1(b), the input of directional information by eyeball movement is recognized as an input of right directional information when the eyeball moves right from the reference eyeball position (i.e., left from the information processing device 1, based on the information processing device 1 facing the face of the user U).

[0033] That is, the input of the directional information in this embodiment is caused by the moving direction of the eyeball. Specifically, the eyeball position of the user U acquired in the past is set as a reference eyeball position, and the eyeball position of the user U acquired after the acquisition of the reference eyeball position is set as an input eyeball position, and the input of the directional information is recognized based on the moving direction of the eyeball from the reference eyeball position to the input eyeball position.

[0034] FIG. 1(b) shows the eyeball position changing in sequence from the top left edge, bottom left edge, top center, bottom center, top right edge, and bottom right edge.

[0035] In this embodiment, the reference eyeball position is updated based on the eyeball position of the user U at the update timing of the current index, which will be described later. For example, the reference eyeball position in the middle of the upper row in FIG. 1B is a reference eyeball position updated based on the input eyeball position in the left end of the lower row, which is the eyeball position of the user U acquired in the past. Also, for example, the reference eyeball position in the right end of the upper row in FIG. 1B is a reference eyeball position updated based on the input eyeball position in the middle of the lower row, which is the eyeball position of the user U acquired in the past. Here, the "previously acquired" eyeball position of the user means a "past" point in time as viewed from the "input eyeball position" acquired after the acquisition of the reference eyeball position. In this embodiment, as will be described later, the eyeball position of the user U acquired at the timing when the current index is updated is the "previously acquired" eyeball position.

[0036] Furthermore, the information processing device 1 performs processing according to a second UI element that is adjacent to the first UI element currently in a selected state and exists in the direction in which the input is received, based on the input of the recognized direction information. As such UI elements and UI processing, various aspects described later can be adopted.

[0037] The distinction between the first UI element and the second UI element is for convenience of description: the UI element that is initially selected is the first UI element, and the UI element adjacent to it in the direction in which the input is received is the second UI element.

[0038] The first UI element in the initially selected state serves as a starting point for identifying a second UI element to be processed by the UI processing means 190. In other words, the starting point for identifying the second UI element is the first UI element in the initially selected state. As the first UI element serving as such a starting point, a UI element that is currently in a selected state when the direction information is input may be adopted, or a UI element that was in a selected state at a predetermined point in the past may be adopted.

[0039] That is, when a UI element that is currently selected is adopted as the first UI element, the UI element that is currently selected may differ depending on the timing of performing UI processing, and it becomes possible to appropriately change the starting point for identifying the second UI element from among the multiple UI elements displayed on the display unit 30. On the other hand, when a UI element that was selected at a predetermined point in the past is adopted as the first UI element, it becomes possible to perform processing according to the second UI element from a consistent starting point without changing the starting point for identifying the second UI element depending on the timing of performing GUI processing.

[0040] In this way, an information processing device that employs the relative change method has the advantage that UI elements can be operated without worrying about deviations in the gaze point, but on the other hand, there are problems as described above. Therefore, the information processing device 1 according to the present invention displays a pointer P indicating the gaze point on the display unit 30 and sets the update timing of the reference eyeball position as follows.

[0041] (Display of pointer P and update of reference eye position) 4 and 5 show an example of a pointer P displayed on the display unit (screen) 30. In these figures, what is shown in a lattice shape is a grid G ​​that divides the display unit 30. The grid G ​​is a theoretical element used for controlling the display of the pointer P, and is not actually displayed on the screen. For convenience of explanation, the grid G ​​is visualized and illustrated in FIGS. 4 and 5.

[0042] In the information processing device 1, when displaying the pointer P on the display unit 30, first, the coordinates of a virtual gaze point (corresponding to) the gaze point of the user U on the screen are calculated. This calculation is performed, for example, based on a predetermined algorithm using the center of the display unit 30 as the origin and the eyeball position of the user U relative to this origin. Specifically, the deviation angle between a reference line connecting the origin on the screen and the center of the eyeball and the line of sight of the user U calculated from the eyeball position is calculated, and the deviation angle is used to calculate the coordinates of the virtual gaze point on the screen.

[0043] Meanwhile, the display unit 30 on which the pointer P is displayed is divided in advance into a plurality of grids G. The grids G are obtained by dividing the screen into a plurality of areas aligned vertically and horizontally, and in the example shown in Fig. 4, the screen is configured with a total of 18 grids G, 6 vertical squares x 3 horizontal squares. Also, in the example shown in Fig. 5, the screen is configured with a total of 128 grids G, 16 vertical squares x 8 horizontal squares. Such grids G are appropriately set according to the size of the display unit 30 and the size of the UI elements displayed on the display unit 30, but it is also possible to configure the display unit 30 so that the user can increase or decrease the number of grids, for example.

[0044] A unique index is assigned to each grid G. The information processing device 1 acquires the center coordinates of each grid in association with each index. Here, the index is information for identifying each grid G, and is composed of character information such as a symbol. The information processing device 1 specifies the position of the corresponding grid G ​​on the screen from this index, and acquires the center coordinates of the grid G.

[0045] The information processing device 1 sets the position coordinates of the pointer P to be displayed on the screen of the display unit 30 based on the calculated coordinates of the virtual point of gaze. Then, the information processing device 1 sets the nearest grid G ​​closest to the calculated coordinates of the virtual point of gaze as a current index, and when the current index is updated due to the movement of the point of gaze of the user U, the information processing device 1 resets the center coordinates of the grid G ​​corresponding to the updated current index as the position coordinates of the virtual point of gaze. As a result, when the point of gaze of the user U moves across the grid G, the pointer P on the screen is displayed at the position of the virtual point of gaze, that is, at the center of the grid G ​​corresponding to the current index (see FIG. 4).

[0046] In this embodiment, the reference eyeball position is updated in accordance with the timing at which the position coordinates of the virtual gaze point are reset. In other words, when the gaze point moves across the grid G, the reference eyeball position is updated at that timing, and the input of directional information is recognized based on the movement direction of the eyeball from that point.

[0047] In this way, in the information processing device 1 of this embodiment, the pointer P is displayed in the center of the grid G ​​closest to the user U's virtual point of gaze, following the movement of the virtual point of gaze. Therefore, the user U can confirm the virtual point of gaze recognized by the information processing device 1 by checking the position of the pointer P, and can perform UI operations without any discrepancies in the point of gaze.

[0048] In addition, in the information processing device 1 of the present embodiment, for example, in calculating the coordinates of the virtual gaze point of the user U, the detection sensitivity of the change in the angle (change in the declination angle) of the eyeball of the user U can be configured to be lowered closer to the center of the screen. By doing so, the movement of the virtual gaze point recognized by the information processing device 1 becomes gentler near the center of the screen, making it easier for the user U to focus his or her gaze on the target UI element.

[0049] Furthermore, in the information processing device 1 of this embodiment, for example, when calculating the coordinates of the virtual gaze point of the user U, if the calculated coordinates of the gaze point of the user U go outside the screen of the display unit 30, it can be configured to clip the coordinates of the gaze point so that they fit within the screen of the display unit 30. By doing so, it is possible to prevent the pointer P displayed on the screen from moving outside the screen and becoming invisible, and since the pointer P stops moving at the edge of the screen, it is possible to perform a correction at that position to match the position of the pointer P with the gaze point of the user U.

[0050] (Hardware configuration of information processing device 1 in the embodiment) Next, an example of the information processing device 1 that realizes the information processing system according to this embodiment will be specifically described.

[0051] FIG. 4 is a hardware configuration diagram of the information processing device 1. As shown in FIG. 4, the information processing device 1 includes a control unit 10, a main memory unit 21 and an auxiliary memory unit 22 that configure a memory unit 20, a display unit 30, and an eyeball position acquisition unit 40. The information processing device 1 in this embodiment has the control unit 10, the memory unit 20, the display unit 30, and the eyeball position acquisition unit 40.

[0052] The control unit 10 can be configured with, for example, one or more arithmetic processing devices such as a central processing unit (CPU), a microprocessor unit (MPU), a graphics processing unit (GPU), or a combination of these. The control unit 10 reads out a program stored in the auxiliary storage unit 22 and cooperates with the storage unit 20 to perform arithmetic processing such as various information processing or control processing.

[0053] The main memory unit 21 can be configured by a temporary storage area such as a static random access memory (SRAM), a dynamic random access memory (DRAM), a cache memory, a flash memory, or a combination of these. The main memory unit 21 temporarily stores data required for the control unit 10 to perform various arithmetic processing.

[0054] The auxiliary storage unit 22 can be configured by, for example, a non-volatile storage area such as a hard disk drive (HDD) or a solid state drive (SSD), or a combination of these. The auxiliary storage unit 22 stores programs and data necessary for the control unit 10 to execute processing, as well as other programs and data.

[0055] The display unit 30 can be configured by, for example, a display means 130 such as a liquid crystal display, an organic electroluminescence display, a touch panel, a projector, or a combination of these. Predetermined information is displayed on the display unit 30 by processing of the control unit 10. In particular, UI elements visually recognized by the user U are displayed on the display unit 30.

[0056] The eyeball position acquisition unit 40 can be configured by, for example, an imaging means such as a camera, an infrared camera, an ultraviolet camera, or other sensor attached to the information processing device 1, a sensor of a wearable device such as glasses with an information processing function attached near the eyeball of the user U to acquire the eyeball position, or a sensor attached to the body of the user U to measure the myoelectric potential near the eyeball of the user U, or a combination of these. The eyeball position acquisition unit 40 acquires the eyeball position of the user U, etc., which is necessary for the control unit 10 to perform processing. Note that the eyeball position acquisition unit 40 may be a sensor or the like that acquires the eyeball position by a single hardware element, or may be a combination of a camera or the like that captures an image including the eyeball and a program that calculates the eyeball position executed by the control unit 10.

[0057] In this embodiment, the eyeball position acquisition unit 40 is configured by combining a camera and a program that calculates the eyeball position, but the eyeball position acquisition unit 40 may also be a sensor that is physically independent of the information processing device 1 and is attached to the body of the user U.

[0058] (Functions of the information processing device 1 in the embodiment and their correlations) 5 is a functional block diagram of the information processing device 1 shown in this embodiment. Each functional element shown here, which is expressed as a means, is realized by the hardware elements of the information processing device 1 working together.

[0059] Specifically, the information processing system has, by using various hardware elements of the information processing device 1, a control means 110, a display means 130, a memory means 120, an input recognition means 140, a reference update means 150, a virtual gaze point calculation means 160, a grid creation means 170, a pointer projection means 180, and a UI processing means 190.

[0060] The control means 110 is configured by, for example, the control unit 10, and a control program and other programs read from the auxiliary storage unit 22. The control means 110 performs arithmetic processing such as various information processing or control processing in the information processing device 1.

[0061] The display means 130 is composed of, for example, the display unit 30, the control unit 10, and a program and display data read from the auxiliary storage unit 22. In this embodiment, the display means 130 displays a plurality of UI elements. It is preferable to provide a predetermined discrimination means for the UI element currently in a selected state so that the user U can understand the UI element being selected. As such a discrimination means, for example, means such as making the UI element have a different brightness or color from other UI elements, displaying a predetermined mark indicating that it is being selected, or surrounding the selected UI element with a frame can be adopted.

[0062] The storage means 120 is composed of, for example, a main storage unit 21, an auxiliary storage unit 22, and a control unit 10 that controls them. In this embodiment, the storage means 120 stores eyeball positions including a reference eyeball position and an input eyeball position, a threshold value, a configuration of grid G, an index of grid G ​​and the center coordinates of each grid, the origin coordinates of the center of the display unit 30, a selected UI element, display data, and the like.

[0063] Here, each piece of information stored in the storage means 120 will be explained. The eyeball position is information on the eyeball position acquired by the eyeball position acquisition means 141. As described above, the eyeball position is stored as coordinate information based on the camera coordinate system. The reference eyeball position (reference position) is the eyeball position at the time when the current index is updated, and is the eyeball position of the user U acquired in the past as viewed from the time when the input of directional information is recognized. The reference eyeball position is repeatedly updated at the timing when the current index is updated to that of a grid with a different index. The input eyeball position (input position) is the eyeball position of the user U acquired after the reference eyeball position is acquired.

[0064] The threshold is a value used to determine whether or not to input directional information, and is compared with the amount of movement of the eyeball position. The threshold may be updated as appropriate by the processing of the information processing device 1, or may be a preset value. The grid configuration is information regarding the arrangement of the grid G ​​on the screen. The index of a grid G ​​is information for identifying each grid G, and the center coordinates of each grid are coordinate information of the center in each grid. The selected UI element is information indicating a UI element that is currently selected among the UI elements displayed on the display unit 30. When the selected UI element is updated, the information indicated by the selected UI element is also updated. The display data is content data for the screen to be displayed on the display unit 30.

[0065] The input recognition means 140 preferably includes, for example, an eyeball position acquisition means 141, a movement direction detection means 142, and a movement amount calculation means 143. The input recognition means 140 is configured, for example, with an eyeball position acquisition unit 40, a control unit 10, and a program read from the auxiliary storage unit 22. The input recognition means 140 recognizes an input of directional information by the user U by acquiring the eyeball position of the user U and detecting the eyeball movement direction by the eyeball position acquisition means 141 and the movement direction detection means 142, and transmits a processing command for performing UI processing to the UI processing means 190. Note that, in the present embodiment, by including the movement amount calculation means 143 as an example, it is possible to recognize the input of suitable directional information using the movement amount in the movement of the eyeball of the user U.

[0066] The eyeball position acquisition means 141 is configured by, for example, the eyeball position acquisition unit 40. The eyeball position acquisition means 141 acquires the eyeball position of the user U viewing the display unit 30. Then, the eyeball position acquisition means 141 acquires the eyeball position of the user U, and stores the acquired eyeball position in the storage means 120. Moreover, it is preferable that the eyeball position acquisition means 141 repeatedly acquires the eyeball position of the user U. In particular, it is preferable that the eyeball position acquisition means 141 repeatedly acquires the eyeball position for each predetermined frame.

[0067] In this embodiment, the eyeball position acquired by the eyeball position acquisition means 141 is represented by an image coordinate system defined in an XY plane image onto which the eyeball is projected. However, for example, the eyeball position may be represented by an eyeball coordinate system defined on the physical eyeball itself of the user U, acquired by a sensor or the like attached near the eyeball.

[0068] The eyeball position acquired by the eyeball position acquisition means 141 is stored in the main memory unit 21 or the auxiliary memory unit 22 by the storage means 120. Then, the eyeball position at the timing when the current index is updated is set as the reference eyeball position by a program executed by the control unit 10, etc. After that, the eyeball position of the user U acquired by the eyeball position acquisition means 141 after the acquisition of the reference eyeball position is set as the input eyeball position by a program executed by the control unit 10, etc.

[0069] The movement direction detection means 142 is composed of, for example, the control unit 10 and the main memory unit 21. The movement direction detection means 142 reads out the reference eyeball position and the input eyeball position stored in the memory means 120, and detects the movement direction of the eyeball from the reference eyeball position to the input eyeball position. In detecting the movement direction, for example, the movement direction can be detected by comparing the coordinates at the reference eyeball position with the coordinates at the input eyeball position. As an example, in this embodiment, the movement directions of the eyeball in four directions, up, down, left, and right, are detected.

[0070] In addition, upon acquiring a new eyeball position, the movement direction detection means 142 may detect the movement direction of the eyeball by treating the new eyeball position as an input eyeball position without storing the new eyeball position in the memory unit 20 and comparing it with a reference eyeball position acquired in the past.

[0071] The movement direction is not limited to the four directions of up, down, left, and right, and may be detected in eight directions including the four directions of up, down, left, and right, plus the upper right, lower right, upper left, and lower left, or may be detected in more directions. Similarly, fewer movement directions, such as two directions, up, down, or left and right, may be detected. Such movement directions are preferably set in relation to the arrangement of UI elements displayed on the display unit 30.

[0072] The movement amount calculation means 143 is composed of, for example, the control unit 10 and the main storage unit 21. The movement amount calculation means 143 calculates the amount of movement of the eyeball from the reference eyeball position to the input eyeball position, and determines whether the amount of movement is equal to or greater than a predetermined threshold value stored in the main storage unit 21. Then, when the amount of movement of the eyeball from the reference eyeball position to the input eyeball position is equal to or greater than the predetermined threshold value, the movement amount calculation means 143 recognizes the input of directional information and transmits a processing command to the UI processing means 190 to perform UI processing.

[0073] Incidentally, even in the case where the information processing system is configured not to include the threshold and movement amount calculation means 143, the movement direction detection means 142 can detect the movement direction of the eyeball from the reference eyeball position to the input eyeball position, recognize this as input of directional information due to the eyeball movement of the user U, and transmit a processing command to the UI processing means 190 to perform UI processing, thereby enabling the information processing system of this embodiment to function.

[0074] The virtual gaze point calculation means 160 is composed of, for example, the control unit 10 and the main storage unit 21. The virtual gaze point calculation means 160 calculates coordinates (coordinates of a virtual gaze point) corresponding to the gaze point of the user U from the change from the eyeball position stored in the storage means 120 to the current eyeball position and the origin at the center of the display unit 30. This calculation is preferably performed repeatedly, and the coordinates corresponding to the gaze point are calculated as the user U's line of sight moves.

[0075] The grid creation means 170 is composed of, for example, the control unit 10 and the main storage unit 21. The grid creation means 170 reads out the configuration of the grid G ​​from the storage means 120, and divides the screen into a plurality of grids G in accordance with this configuration. Then, it assigns a unique index to each divided grid G, and causes the storage means 120 to store the center coordinates of each grid in association with the assigned index.

[0076] The pointer projection means 180 is composed of, for example, the control unit 10 and the main storage unit 21. The pointer projection means 180 sets the position coordinates of the pointer P to be displayed on the screen of the display unit 30 based on the coordinates of the point of interest calculated by the point of interest calculation means 160.

[0077] The reference update means 150 is composed of, for example, the control unit 10 and the storage unit 20. The reference update means 150 sets the nearest grid G ​​closest to the coordinates of the calculated virtual gaze point as the current index, and when the current index is updated due to the movement of the gaze point of the user U, the reference update means 150 reads out the center coordinates of the grid G ​​corresponding to the updated current index from the storage means 120 and resets the read center coordinates as the position coordinates of the virtual gaze point. In addition, the reference update means 150 updates the reference eyeball position at the timing when the position coordinates of the virtual gaze point are reset. In this embodiment, since the eyeball position acquisition means 141 repeatedly acquires the eyeball position of the user U, the eyeball position acquired at the timing when the current index is updated is updated as the reference eyeball position.

[0078] The UI processing means 190 is configured by, for example, the control unit 10 and the storage unit 20. By receiving a processing command for performing UI processing including the recognized directional information from the input recognition means 140, the UI processing means 190 performs processing according to a second UI element that is adjacent to the first UI element currently in a selected state and exists in the direction in which the input is received, based on the input of the recognized directional information.

[0079] As such UI processing, various information processing such as selection processing of a predetermined UI element, execution processing, execution processing performed together with selection, etc. Specific examples include processing for selecting a second UI element adjacent to a first UI element in a selected state and located in the direction in which the input is received based on an input of directional information, execution processing for executing a second UI element adjacent to a first UI element in a selected state and located in the direction in which the input is received based on an input of directional information, and processing for selecting a second UI element adjacent to a first UI element in a selected state and located in the direction in which the input is received based on an input of directional information while performing a predetermined information processing.

[0080] That is, the UI processing includes various types of processing that require directional information, which is generally performed as information processing by inputting directional keys of a computer, flicking a touch panel, and the like. As an example of the UI processing, a selection process based on the input of directional information can be assumed to be a selection of a predetermined UI element such as a list of selectable icons, cards, character elements, illustrations, radio buttons, check boxes, drop-down lists, and keyboards. In addition, a process for executing an operation based on the input of directional information can be assumed to be a display process of the next photo in a slide show of photos, a display process of the next page in a page of an electronic book, and the like. In addition, a process for performing a predetermined information processing together with a selection process based on the input of directional information can be assumed to be a process for selecting content in a playlist of music or videos and a playback process, and the like. Furthermore, in a state in which a second UI element that is adjacent to a first UI element and exists in the direction in which an input is received is not displayed on the display unit 30, the screen may be switched by scrolling or the like to display the second UI element and to select the second UI element.

[0081] In this manner, in the information processing device 1 of the present embodiment, in an information processing system including a user interface that recognizes input of directional information based on the moving direction of the eyeballs and / or face from the reference position to the input position, using the position of the user's eyeballs and / or face acquired in the past as a reference position by cooperation of hardware elements, and using the position of the user's eyeballs and / or face acquired after acquisition of the reference position as an input position, the following steps are executed: a virtual gaze point calculation step of calculating coordinates of a virtual gaze point based on the moving direction of the user's eyeballs and / or face; a grid creation step of dividing a screen into a plurality of grids, assigning an index to each grid, and acquiring the central coordinates of each grid in correspondence with the index; a pointer projection step of setting the position coordinates of a pointer to be displayed on the screen based on the calculated virtual gaze point coordinates; and a reference update step of setting the grid closest to the calculated virtual gaze point coordinates as a current index, and when the current index is updated due to the movement of the user's virtual gaze point, resetting the central coordinates of the grid corresponding to the updated current index as the position coordinates of the virtual gaze point, and updating the reference position at the same timing.

[0082] Therefore, according to the present invention, since the user U can operate the UI while checking the position of the virtual gaze point displayed on the screen, it is possible to provide an information processing system equipped with a relative change type UI with excellent operability without any inconsistency in the gaze point. Also, since the UI operation by the relative change type can be performed while displaying the virtual gaze point, for example, the UI can be accurately operated even when many UI elements are displayed on a small screen.

[0083] Furthermore, according to the present invention, since the reference eyeball position is updated at the timing when the current index is updated due to the movement of the virtual gaze point of the user U, for example, by appropriately changing the grid size, the frequency of updating the reference eyeball position can be adjusted so as not to be excessive, and an information processing system can be provided in which the data processing burden associated with updating the reference eyeball position is small. Therefore, it is possible to provide an information processing system equipped with a relative change type UI that can be smoothly operated even on an information processing device with a low data processing capacity such as a smartphone.

[0084] It should be noted that the above-described embodiments are merely preferred embodiments of the present invention, and the present invention is not limited to these, and various design modifications are possible within the scope of the present invention.

[0085] For example, in the above-described embodiment, the case where the operation of UI elements and the calculation of the coordinates of the virtual gaze point are all performed based on the eyeball position of the user have been shown, but for example, it is also possible to operate UI elements and calculate the coordinates of the virtual gaze point using the face position of the user U instead of or together with the eyeball position. For example, when the face position is used instead of the eyeball position of the user U, the information processing device 1 recognizes the input of directional information based on the moving direction of the face from the reference position to the input position, using the face position (face direction) of the user U acquired in the past as the reference position and the face position (face direction) of the user U acquired after the acquisition of the reference position as the input position. In this case, the virtual gaze point calculation means calculates the coordinates of the virtual gaze point based on the moving direction of the face of the user U (moving direction of the face direction). [Explanation of symbols]

[0086] 1. Information processing device 10 Control section 20 Memory section 30 Display section 40 Eyeball position acquisition unit 110 Control Means 120 Memory means 130 Display means 140 Input Recognition Means 141 Eyeball position acquisition means 142 Movement direction detection means 143 Movement amount calculation means 150 Standard update means 160 Virtual gaze point calculation means 170 Grid Creation Method 180 Pointer projection means 190 UI Processing Methods P Pointer G Grid

Claims

1. An information processing system including a user interface that recognizes an input of directional information based on a moving direction of the eyeballs and / or face from a reference position to an input position, the input position being determined based on a previously acquired position of the eyeballs and / or face of a user, the user interface comprising: a virtual gaze point calculation means for calculating coordinates of a virtual gaze point based on a moving direction of the eyeball and / or face of the user; a grid creating means for dividing a screen into a plurality of grids, assigning an index to each grid, and acquiring a center coordinate of each grid in association with the index; a pointer projection means for setting position coordinates of a pointer to be displayed on the screen based on the calculated coordinates of the virtual point of gaze; a reference update means for setting a grid closest to the coordinates of the calculated virtual gaze point as a current index, and when the current index is updated by the movement of the virtual gaze point of the user, resetting the center coordinates of the grid corresponding to the updated current index as the position coordinates of the virtual gaze point, and updating the reference position at the same timing. An information processing system comprising:

2. The virtual gaze point calculation means calculates the coordinates and displacement of the virtual gaze point of the user based on the center of the screen as an origin and on changes in the position of the eyeball and / or face of the user relative to this origin.

2. The information processing system according to claim 1 .

3. The virtual gaze point calculation means reduces the detection sensitivity of the angle change of the user's eyeballs and / or face as the coordinates of the virtual gaze point of the user are closer to the center of the screen.

3. The information processing system according to claim 2.

4. The virtual gaze point calculation means performs clipping so that the calculated coordinates of the user's virtual gaze point are within the screen.

4. The information processing system according to claim 1, wherein the information processing system is a data processing system.

5. 1. An information processing system including a user interface that recognizes an input of directional information based on a moving direction of the eyeballs and / or face from a reference position to an input position, the user interface defining a previously acquired position of the eyeballs and / or face of a user as a reference position and a position of the eyeballs and / or face of the user acquired after the acquisition of the reference position as an input position, the information processing system comprising: a virtual gaze point calculation step of calculating coordinates of a virtual gaze point based on a moving direction of the eyeball and / or face of the user; a grid creation step of dividing the screen into a plurality of grids, assigning an index to each grid, and acquiring a center coordinate of each grid in association with the index; a pointer projection step of setting position coordinates of a pointer to be displayed on the screen based on the calculated coordinates of the virtual gaze point; a reference update step of setting a grid closest to the coordinates of the calculated virtual gaze point as a current index, and when the current index is updated by a movement of the virtual gaze point of the user, resetting the center coordinates of the grid corresponding to the updated current index as the position coordinates of the virtual gaze point, and updating the reference position at the same timing.

23. An information processing method comprising:

6. a central processing unit of an information processing system including a user interface that recognizes an input of direction information based on a moving direction of the eyeballs and / or face from a reference position to an input position of the user's eyeballs and / or face acquired after the acquisition of the reference position, the input position being a reference position based on a previously acquired position of the user's eyeballs and / or face; a virtual gaze point calculation step of calculating coordinates of a virtual gaze point based on a moving direction of the eyeball and / or face of the user; a grid creation step of dividing the screen into a plurality of grids, assigning an index to each grid, and acquiring a center coordinate of each grid in association with the index; a pointer projection step of setting position coordinates of a pointer to be displayed on the screen based on the calculated coordinates of the virtual gaze point; a reference update step of setting a grid closest to the coordinates of the calculated virtual gaze point as a current index, and when the current index is updated by the movement of the virtual gaze point of the user, resetting the center coordinates of the grid corresponding to the updated current index as the position coordinates of the virtual gaze point, and updating the reference position at the same timing. A program characterized by:

Images