Information processing apparatus, method, and program

Abstract

To provide an information processing apparatus, method and program for allowing a user to perform eye control more easily.SOLUTION: An information processing apparatus 30 includes: an arrangement processing unit 320 which determines an arrangement of an input area on a display surface of a display device 10 based on positions of the eyes of a user; and an input determination unit which determines whether the user has performed predetermined eye control on the input area. The arrangement processing unit may determine the arrangement of the input area vertically above the display surface based on a relationship between the height of the positions of the eyes of the user and the height of a reference position.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device, method, and program. [Background technology]

[0002] In recent years, research has been conducted into systems that enable users to perform operations using their gaze. For example, in the technology described in Non-Patent Document 1, a user gazes at any point on the screen for two seconds or more, and then performs a specific gaze movement to perform input. Patent Document 1 also discloses a technology related to the layout of UIs for operations using gaze. In the technology disclosed in Patent Document 1, when the screen of a display device is small, the UIs are rearranged so that they do not interfere with each other. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-61804 [Patent Document 2] Patent No. 6371475 [Patent Document 3] Japanese Patent Application Laid-Open No. 2011-243108 [Non-patent literature]

[0004] [Non-Patent Document 1] Toshiya Isomoto, Shota Yamanaka, and Buntaro Shizuki, "A gaze-based control method robust against unintentional operations using a series of operations consisting of gaze followed by gestures," Transactions of the Human Interface Society, Vol. 23, No. 1, pp. 5-18, 2021. [Non-patent document 2] Toshiaki Miura, "Visual Attention and Safety - Focusing on Effective Visual Field -", Journal of the Illuminating Engineering Institute of Japan, Vol. 82, No. 3, 1998 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the above-mentioned technology assumes that the user is in a seated position and the distance from the user's eyes to the screen is constant, or that the technology is used with a wearable device such as an HMD (Head Mount Display), and does not take into account the positional relationship between the user and the screen. Therefore, with the above-mentioned technology, depending on the positional relationship between the user and the screen, it is difficult for the user to perform operations using their line of sight.

[0006] Therefore, the present invention has been made in consideration of the above problems, and an object of the present invention is to provide a technique that improves ease of eye-gaze operation by a user. [Means for solving the problem]

[0007] In order to solve the above problem, according to one aspect of the present invention, an information processing device is provided, which includes a placement processing unit that determines the placement of an input area on a display surface based on the position of the user's eyes, and an input determination unit that determines whether the user has performed a predetermined gaze operation on the input area.

[0008] The placement processing unit may determine the placement of the input area in the vertical direction on the display surface based on a relationship between the height of the user's eyes and the height of a reference position.

[0009] The placement processing unit may place the input area at a first height position on the display surface when the user's eye position is higher than the reference position, and may place the input area at a second height position lower than the first height position when the user's eye position is lower than the reference position.

[0010] The reference position may be a central position of the display surface, the first height position may be higher than the reference position, and the second height position may be lower than the reference position.

[0011] The placement processing unit may determine a horizontal size of the input area on the display surface based on a relationship between the position of the user's eyes and a reference position.

[0012] The reference position may be a position on the display surface, and the placement processing unit may determine the horizontal size of the input area based on the distance between the user's eye position and the reference position in the normal direction of the display surface.

[0013] The placement processing unit may calculate an effective field of view, which is the length of the user's horizontal field of view on the display surface, based on the separation distance, and determine the horizontal size of the input area based on the result of comparing the effective field of view with a reference size.

[0014] The placement processing unit may determine the horizontal size of the input area to be the reference size when the effective field of view range is larger than the reference size, and may determine the horizontal size of the input area to be the effective field of view range when the effective field of view range is smaller than the reference size.

[0015] The reference size may be the horizontal size of the display surface.

[0016] The device may further include a data processing unit that measures the position of the user's eyes based on an input from a sensor that captures an image of the user.

[0017] The information processing device may further include a feeling estimation unit that estimates a feeling of the user, and a display control unit that controls display of the display surface including the input area according to a result of estimation by the feeling estimation unit.

[0018] The display control unit may control a following speed of a moving display in the input area that is moved in response to the gaze operation, with respect to the gaze operation, in accordance with a result of estimation by the emotion estimation unit.

[0019] The display control unit may control whether or not to display a display indicating a position viewed by the user, depending on a result of estimation by the feeling estimation unit.

[0020] When the proficiency level of the user is equal to or higher than a standard, the display control unit may execute control according to a result of estimation by the feeling estimation unit.

[0021] The information processing device may further include a display control unit that displays content indicating the content to be confirmed by the user on the display surface, and a time management unit that dynamically sets the waiting time from when the content is displayed until gaze operations on the input area become effective.

[0022] The time management unit may dynamically set the waiting time depending on the user and the content to be displayed.

[0023] The information processing device may further include a machine learning unit that performs machine learning using a set of user identification information, content identification information, and the time that the user gazed at the content, and creates a model that outputs a time corresponding to the user identification information and the content identification information, and the time management unit may set the time output from the model as the waiting time.

[0024] In addition, according to another aspect of the present invention, in order to solve the above problem, a method executed by a computer is provided, which includes determining the placement of an input area on a display surface based on the position of the user's eyes, and determining whether the user has performed a predetermined gaze operation on the input area.

[0025] In addition, according to another aspect of the present invention, in order to solve the above problem, a program is provided to cause a computer to function as an arrangement determination unit that determines the arrangement of an input area on a display surface based on the position of the user's eyes, and an input determination unit that determines whether the user has performed a predetermined gaze operation on the input area. [Effects of the Invention]

[0026] According to the present invention as described above, it becomes easier for the user to perform eye gaze operations. [Brief explanation of the drawings]

[0027] [Figure 1] FIG. 1 is an explanatory diagram illustrating an information processing system according to a first embodiment. [Figure 2] 1 is an explanatory diagram showing the configuration of an information processing device 30 according to an embodiment of the present invention. [Figure 3] FIG. 2 is an explanatory diagram showing a coordinate system. [Figure 4] 10 is a specific example of a display screen that is displayed on the display device 10 when it is determined that the height of the user's eyes is equal to or higher than the reference position. [Figure 5] 10 is a specific example of a display screen that is displayed on the display device 10 when it is determined that the height of the user's eye position is lower than the reference position. [Figure 6] 4 is a flowchart showing the operation of the information processing device 30 according to the first embodiment of the present invention. [Figure 7] FIG. 10 is an explanatory diagram showing the configuration of an information processing device 30-2 according to a second embodiment. [Figure 8] 1 is an explanatory diagram showing a first example of the relationship between a display surface 12 and an effective visual field range G. FIG. [Figure 9] FIG. 10 is an explanatory diagram showing an example of setting an input area. [Figure 10] 10 is an explanatory diagram showing a second example of the relationship between the display surface 12 and the effective visual field range G. FIG. [Figure 11] FIG. 10 is an explanatory diagram showing an example of setting an input area. [Figure 12] 5 is a flowchart showing the operation of the information processing device 30-2 according to the first embodiment of the present invention. [Figure 13] FIG. 10 is an explanatory diagram showing the configuration of an information processing device 30-3 according to a third embodiment of the present invention. [Figure 14] 10 is an explanatory diagram showing a display example of a pointer 60 and the like. FIG. [Figure 15] 10 is an explanatory diagram showing a specific example of a user DB stored in a storage unit 356. FIG. [Figure 16] 10 is a flowchart showing the operation of an information processing device 30-3 according to the third embodiment. [Figure 17] FIG. 10 is an explanatory diagram showing the configuration of an information processing device 30-4 according to a fourth embodiment of the present invention. [Figure 18] FIG. 10 is an explanatory diagram showing a specific example of an instruction DB. [Figure 19] 10 is a flowchart showing the operation of an information processing device 30-4 according to the fourth embodiment. [Figure 20] FIG. 1 is a block diagram showing an example of a hardware configuration 90. DETAILED DESCRIPTION OF THE INVENTION

[0028] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0029] The present invention relates to a technology that enables a user to perform gaze operations. First to fourth embodiments of the technology will be described in detail below. The first embodiment relates to a design for the layout (particularly the position) of an input area that a user gazes at for gaze operations, the second embodiment relates to a design for the layout (particularly the size) of the input area, the third embodiment relates to a design for a display that moves in response to gaze operations, and the fourth embodiment relates to a design for the time until gaze operations are enabled.

[0030] In the following description, the technology according to each embodiment of the present invention is mainly applied to a factory, a display device displays instructions for work in the factory as content on the display surface, and a user worker performs the work while checking the instructions. The instructions may include text, illustrations, or photographs indicating work procedures, or text, illustrations, or photographs indicating important points to note for the work. The worker may, for example, use eye gaze to turn pages of the instructions displayed on the display surface. However, the application of each embodiment of the present invention is not limited to factories. For example, each embodiment of the present invention can also be applied to turning pages of sheet music when a user plays a musical instrument, or turning pages of materials when a user gives a lecture, speech, class, or presentation.

[0031] First Embodiment The height of a user's eyes may vary depending on the user's height, posture, etc. If an input area for eye-gaze operation is arranged at a uniform position on the display surface, depending on the user's eye height, the user may have difficulty performing the eye-gaze operation, which increases the burden on the user. The first embodiment reduces the burden on the user by arranging the input area at the top or bottom of the display surface depending on the user's eye height.

[0032] (Configuration of information processing system) FIG. 1 is an explanatory diagram showing an information processing system according to a first embodiment. As shown in FIG. 1, the information processing system according to the first embodiment includes a display device 10, a gaze sensor 20, and an information processing device 30. The display device 10 and the gaze sensor 20 are installed in a workspace where a user is present. The information processing device 30 may be installed in the same workspace as the display device 10 and the gaze sensor 20, or may be installed on-premise or on a network. The display device 10 and the gaze sensor 20 are connected to the information processing device 30 by wire or wirelessly.

[0033] The display device 10 has a display surface 12 and displays various images on the display surface 12 under control of the information processing device 30. For example, the display device 10 displays work instructions as an example of content on the display surface 12. The display device 10 also displays a UI (User Interface) on the display surface 12 for the user to perform gaze operations. Examples of such UIs include a start UI and an input UI, which will be described later. The display device 10 may be a tablet terminal or a desktop display. Furthermore, the display device 10 may be a projector. In this case, the projection surface of the optical image projected from the projector is treated as the display surface 12.

[0034] The gaze sensor 20 is an example of a sensor that captures an image of a user. The gaze sensor 20 may be, for example, an infrared stereo camera or a visible light monocular camera. As shown in FIG. 1 , the gaze sensor 20 has a bar shape that extends horizontally along the display device 10 and may be installed at the bottom of the display device 10. Data obtained by the gaze sensor 20 is output to the information processing device 30 so that the information processing device 30 can measure the user's gaze information. The user's gaze information includes the position of the user's eyes in three-dimensional space and the position of the gaze point (gaze position), which is the point on the display surface 12 that the user is looking at. Although prior calibration is required to measure such gaze information, the following description will be given assuming that calibration has already been performed.

[0035] The information processing device 30 generates various display screens and outputs the generated display screens to the display device 10, thereby displaying the display screens on the display device 10. The information processing device 30 also measures the user's gaze information based on data input from the gaze sensor 20. When the information processing device 30 determines that the user has performed a predetermined gaze operation, it executes a process linked to the gaze operation. For example, the predetermined gaze operation may be a gaze movement to the left as seen from the user, and the process linked to the gaze operation may be turning the page of an instruction manual. The configuration of such an information processing device 30 will be described in more detail below with reference to FIG. 2.

[0036] (Configuration of information processing device 30) 2 is an explanatory diagram showing the configuration of an information processing device 30 according to one embodiment of the present invention. As shown in FIG. 2, the information processing device 30 according to one embodiment of the present invention has a data processing unit 310, an arrangement processing unit 320, and an operation processing unit 330. The arrangement processing unit 320 has a height determination unit 322 and an arrangement determination unit 324. The operation processing unit 330 has a movement amount calculation unit 332, an input determination unit 334, a memory unit 335, and a display control unit 336.

[0037] The data processing unit 310 is connected to a height determination unit 322 and a movement amount calculation unit 332, the height determination unit 322 is connected to a placement determination unit 324, the placement determination unit 324 is connected to a display control unit 336, the movement amount calculation unit 332 is connected to an input determination unit 334 and a display control unit 336, and the input determination unit 334 is connected to a memory unit 335 and a display control unit 336.

[0038] -Data processing unit 310 The data processing unit 310 measures the user's gaze information based on the data input from the gaze sensor 20. The data processing unit 310 measures and updates the gaze information at each unit time sufficient for processing the gaze information. As described above, the gaze information may include a gaze position and an eye position.

[0039] The gaze position is output as a two-dimensional coordinate position P(x, y) on the display surface 12. The update time unit is Δt, and the gaze position acquired at each time is expressed as Pn(xn, yn) as the coordinate position after Δt×n time has passed. That is, when the data processing unit 310 measures the gaze information over Δt×t time, an array of coordinate positions such as P1(x1, y1), P2(x2, y2), Pt(xt, yt) is obtained as the gaze positions.

[0040] The eye position is output as a three-dimensional coordinate position Q(x, y, z). As shown in FIG. 3, the three-dimensional coordinate system is a coordinate system in which the center position of the display surface 12 of the display device 10 is the origin Q0 (x0, y0, z0). The horizontal direction of the display surface 12 is the x direction, the vertical direction of the display surface 12 is the y direction, and the normal direction of the display surface 12 is the z direction. The position of the user's eyes at time t is expressed as Qt(xt, yt, zt).

[0041] Height determination unit 322 The placement processing unit 320 determines the placement of the input area in the vertical direction on the display surface 12 based on the relationship between the height of the user's eye position measured by the data processing unit 310 and the height of the reference position. To this end, the height determination unit 322 determines whether the height of the user's eye position measured by the data processing unit 310 is equal to or higher than the reference position.

[0042] For example, the height determination unit 322 acquires the user's eye position Qs (xs, ys, zs) at time s from the data processing unit 310 and determines whether ys is equal to or greater than the height of the reference position. If ys is equal to or greater than the height of the reference position, the user is considered to be relatively tall or has relatively good posture. Conversely, if ys is not equal to or greater than the height of the reference position, the user is considered to be relatively short or has a relatively hunched posture. The reference position may be the center position of the display surface 12 or another position.

[0043] -Placement determining section 324 The placement determination unit 324 determines the placement of the input area on the display surface 12 according to the result of the determination by the placement processing unit 320. Specifically, the placement determination unit 324 places the input area at a first height position when the height of the user's eyes is equal to or higher than the reference position, and places the input area at a second height position lower than the first height position when the height of the user's eyes is lower than the reference position. The first height position may be higher than the reference position, and the second height position may be lower than the reference position.

[0044] The layout determination unit 324 outputs the determined layout of the input areas to the display control unit 336, and the display control unit 336 generates a display screen to be displayed on the display device 10 in accordance with the layout of the input areas. Specific examples of layouts of the input areas will be described below with reference to Figs. 4 and 5.

[0045] 4 is a specific example of a display screen displayed on the display device 10 when it is determined that the height of the user's eyes is equal to or higher than the reference position. As shown in the upper part of FIG. 4, when it is determined that the height of the user's eyes is equal to or higher than the reference position, an input area F is arranged above the center position of the display surface 12, and a content display area C is arranged below the input area F. A start UI 40 is displayed in the input area F. When the start UI 40 is gazed at for a predetermined time (for example, 2 seconds), an input UI 50 is displayed in the input area F, as shown in the lower part of FIG. 4. The input UI 50 includes a movement display 52 that moves following the movement of the user's line of sight, and a target display 54.

[0046] FIG. 5 is a specific example of a display screen displayed on the display device 10 when it is determined that the height of the user's eyes is lower than the reference position. As shown in the upper part of FIG. 5, when it is determined that the height of the user's eyes is lower than the reference position, the input area F is arranged below the center position of the display surface 12, and the content display area C is arranged above the input area F. As in the example of FIG. 4, the start UI 40 is displayed in the input area F, and when the start UI 40 is gazed at for a predetermined period of time, the input UI 50 is displayed in the input area F, as shown in the lower part of FIG. 5. Note that, although the input area F is shown in FIGS. 4 and 5 as an area including the start UI 40 or the input UI 50, the start UI 40 and the input UI 50 themselves may be treated as the input area F. That is, the arrangement determination unit 324 may determine the arrangement of the start UI 40 and the input UI 50 according to the result of the determination by the height determination unit 322.

[0047] -Movement amount calculation unit 332 The movement amount calculation unit 332 calculates the movement direction and movement amount of the gaze position P based on the change over time in the gaze position P of the user obtained by the data processing unit 310. Specifically, the movement amount calculation unit 332 calculates the movement direction and movement amount ΔP(Δx, Δy) of the gaze position P in the input area from the difference between the gaze position Pt(xt, yt) at time t and the gaze position Pt+Δt(xt+Δx, yt+Δy) after Δt.

[0048] -input determination unit 334 The input determination unit 334 is a part that determines whether a predetermined gaze operation has been completed based on the gaze movement direction (vector value of ΔP) and movement amount (scalar value of ΔP) of the gaze within the input area calculated by the movement amount calculation unit 332. The input determination unit 334 determines that a gaze operation has started when the start UI within the input area is gazed at. Then, the input determination unit 334 determines that a gaze operation is being performed when the gaze position P is moving in a specific direction associated with the gaze operation stored in the storage unit 335, and determines that the gaze operation has been completed when the movement amount of the gaze position P reaches a predetermined amount.

[0049] For example, in the state of display surface 12 shown in the lower part of Fig. 4, input determination unit 334 determines that a gaze operation is being performed when gaze position P moves leftward from initial position L0 of moving display 52, and display control unit 336 moves moving display 52 leftward to follow gaze position P. Then, when gaze position P (or moving display 52) moves to position L of target display 54 of input UI 50, the absolute value of the amount of movement, "L-L0", reaches predetermined amount Ln, and therefore input determination unit 334 determines that the gaze operation for turning the page has been completed.

[0050] When it is determined that the predetermined eye-gaze operation has been completed, the input determination unit 334 causes the display control unit 336 to execute a process associated with the eye-gaze operation in the storage unit 335.

[0051] -Storage section 335 The storage unit 335 stores the content of a gaze operation in association with the content of a process. The content of the gaze operation may be a gaze movement direction and a gaze movement amount. The content of the process may be page forward, page back, etc. For example, a gaze movement direction "left" and a gaze movement amount "Ln" may be stored as the content of the gaze operation, and "page forward" may be stored in association with the content of the gaze operation.

[0052] -Display control unit 336 The display control unit 336 generates various display screens and outputs the generated display screens to the display device 10. For example, as described with reference to FIGS. 4 and 5, the display control unit 336 generates a display screen including a start UI 40 and a content display area C. The start UI 40 is displayed in the input area F, the layout of which has been determined by the layout determination unit 324. Furthermore, when the start UI 40 is gazed at for a predetermined period of time, the display control unit 336 generates a display screen in which the input UI 50 is displayed in the input area F, and moves the movement display 52 so as to follow the movement of the gaze position P. Furthermore, when the gaze position P (or the movement display 52) moves to the position L of the target display 54 of the input UI 50, the input determination unit 334 determines that the gaze operation has been completed, and the display control unit 336 switches the display in the content display area C to the instruction sheet of the next page.

[0053] (Operation of information processing device 30) The configuration of the information processing device 30 according to the first embodiment of the present invention has been described above. Next, the operation of the information processing device 30 according to the first embodiment of the present invention will be summarized with reference to FIG.

[0054] 6 is a flowchart showing the operation of the information processing device 30 according to the first embodiment of the present invention. First, when the data processing unit 310 measures the position Qs (xs, ys, zs) of the user's eyes (S604), the height determination unit 322 determines whether the height ys of the position Qs of the user's eyes measured by the data processing unit 310 is equal to or greater than a reference position (S608).

[0055] If the height ys of the user's eye position Qs is not equal to or greater than the reference position (S608 / No), the placement determination unit 324 determines the placement of the input area F below the reference position (S612). On the other hand, if the height ys of the user's eye position Qs is equal to or greater than the reference position (S608 / Yes), the placement determination unit 324 determines the placement of the input area F above the reference position (S616).

[0056] Then, the display control unit 336 generates a display screen in which the start UI 40 is displayed in the input area F whose layout has been determined by the layout determination unit 324, and the display device 10 displays the display screen (S620). As a result, the state of the display device 10 becomes an input acceptance state in which it can accept an eye-gaze operation.

[0057] Thereafter, if the start UI 40 is gazed at for a predetermined time (S624 / Yes), the display control unit 336 generates a display screen in which the input UI 50 is displayed in the input area F, and the display device 10 displays the display screen (S628). Then, the input determination unit 334 determines whether the gaze position P is moving in a specific direction associated with the eye-gaze operation stored in the storage unit 335 (S632). If the gaze position P is moving in another direction (S632 / No), the display control unit 336 erases the input UI 50, and the processing from S620 is repeated.

[0058] On the other hand, if the gaze position P is moving in a specific direction (S632 / Yes), the display control unit 336 updates the position of the moving display 52 of the input UI 50 in accordance with the movement of the gaze position P (S636). Updating of the position of the moving display 52 is repeated until the moving display 52 reaches the position of the target display 54 (S640 / No), and when the moving display 52 reaches the position of the target display 54 (S640 / Yes), the display control unit 336 executes a predetermined process (for example, turning the page of the instruction sheet) that is linked in the storage unit 335 to the content of the gaze operation that has been performed (S644).

[0059] (Action and effect) The first embodiment of the present invention described above provides a variety of advantageous effects. For example, according to the first embodiment of the present invention, the placement determination unit 324 determines the placement of the input area in the vertical direction on the display surface 12 based on the relationship between the height of the user's eye position measured by the data processing unit 310 and the height of the reference position. With this configuration, the input area can be placed at a position appropriate to the height of each user's eye position, making it easier for users to perform gaze operations. In particular, it is possible to reduce the impact on the operation burden caused by differences in height and posture between users performing gaze operations in a standing position.

[0060] Furthermore, the placement determination unit 324 places the input area at a first height position when the height of the user's eyes is equal to or higher than the reference position, and places the input area at a second height position lower than the first height position when the height of the user's eyes is lower than the reference position, thereby making it possible to more reliably achieve the above-mentioned effect.

[0061] Furthermore, the center position of the display surface 12 may be used as the reference position, a position higher than the center position of the display surface 12 may be used as the first height position, and a position lower than the center position of the display surface 12 may be used as the second height position. In this case, the user's eyes and the input area are located on the same side (upper or lower side) as viewed from the center position of the display surface 12, making it even easier for the user to perform gaze operations on the input area.

[0062] Furthermore, the data processing unit 310 measures the position of the user's eyes based on the data input from the gaze sensor 20. This configuration is highly convenient because the position of the user's eyes is measured automatically. However, the user's height may be registered in advance, in which case the registered height or a position calculated from the height may be used as the position of the user's eyes.

[0063] <Second embodiment> The first embodiment has been described above with regard to the vertical arrangement of the input area. To further reduce the operational burden, it is desirable to appropriately determine not only the vertical arrangement of the input area but also the horizontal size of the input area. For example, when a one-way gaze movement is used as the gaze operation method, the occurrence of erroneous operations can be reduced as the gaze movement distance required to complete the gaze operation increases. On the other hand, the greater the angle at which the user moves their eyes to move their gaze, the greater the burden on the user. The second embodiment achieves both the reduction of erroneous operations and the reduction of the user's burden by determining the horizontal size of the input area based on the position of the user's eyes.

[0064] (Configuration of information processing device 30-2) 7 is an explanatory diagram showing the configuration of an information processing device 30-2 according to the second embodiment. As shown in FIG. 7, the information processing device 30-2 according to the second embodiment has a data processing unit 310, an operation processing unit 330-2, and an arrangement processing unit 340. The operation processing unit 330-2 has a movement amount calculation unit 332, an input determination unit 334, a storage unit 335, and a display control unit 337. The arrangement processing unit 340 has a screen size acquisition unit 342 and an area size determination unit 344.

[0065] Data processing unit 310 is connected to movement amount calculation unit 332 and area size determination unit 344, screen size acquisition unit 342 is connected to area size determination unit 344, area size determination unit 344 is connected to display control unit 337, movement amount calculation unit 332 is connected to input determination unit 334 and display control unit 337, and input determination unit 334 is connected to storage unit 335 and display control unit 337. The following mainly describes the configuration of information processing device 30-2 that is different from the first embodiment.

[0066] - Screen size acquisition unit 342 The screen size acquisition unit 342 acquires the size of the display surface 12 of the display device 10. More specifically, the screen size acquisition unit 342 acquires the maximum horizontal image size (in centimeters) of the display surface 12 as the horizontal size of the display surface 12. Note that the horizontal size of the display surface 12 is an example of a reference size, and the reference size may be another size, such as a size that is a predetermined percentage of the maximum horizontal image size of the display surface 12.

[0067] -Area size determination unit 344 The area size determination unit 344 determines the horizontal size of the input area on the display surface 12 based on the relationship between the user's eye position and the reference position. For example, the area size determination unit 344 determines the horizontal size of the input area so that the input area falls within the range of a person's effective field of vision. According to Non-Patent Document 2, the effective field of vision of a human is approximately 20 degrees, although this varies somewhat depending on psychological factors. Therefore, the following description will be given assuming that the effective field of vision is 20 degrees and the reference position is a position on the display surface 12.

[0068] The three-dimensional coordinates of the user's eye position are Qs (xs, ys, zs), and the distance between the user and display surface 12 in the normal direction of display surface 12 is zs. At this time, the effective visual field range G, which is the user's visual field range in the horizontal direction on display surface 12, can be calculated using the following formula 1. G=2(zs / tan(20 / 2)) (Formula 1)

[0069] The area size determination unit 344 calculates the effective field of view range G according to equation 1, and determines the horizontal size of the input area based on the comparison result between the effective field of view range G and the horizontal size of the display surface 12 acquired by the screen size acquisition unit 342.

[0070] More specifically, the area size determination unit 344 may determine the horizontal size of the input area to be the horizontal size of the display surface 12 when the effective visual field range G is equal to or larger than the horizontal size of the display surface 12, and may determine the horizontal size of the input area to be the effective visual field range G when the effective visual field range G is smaller than the horizontal size of the display surface 12. Note that the input area may be regarded as a rectangular area circumscribing the input UI 50 when the moving display 52 is at the initial position L0.

[0071] 8 to 11, a specific example of processing by the area size determination unit 344 will be described below. An example will be described below in which the horizontal size of the input area is determined using the effective field of view range G, but the horizontal size of the input area may also be determined using other numerical values.

[0072] Fig. 8 is an explanatory diagram showing a first example of the relationship between the display surface 12 and the effective visual field range G. In the first example shown in Fig. 8, the user's effective visual field range G on the plane on which the display surface 12 is located, which is based on the effective visual field angle θ, is larger than the horizontal size of the display surface 12. In this case, the area size determination unit 344 determines the horizontal size of the input area F to be the horizontal size of the display surface 12, as shown in Fig. 9.

[0073] Fig. 10 is an explanatory diagram showing a second example of the relationship between the display surface 12 and the effective visual field range G. In the second example shown in Fig. 10, the user's effective visual field range G on the plane on which the display surface 12 is located, which is based on the effective visual field angle θ, is smaller than the horizontal size of the display surface 12. In this case, the area size determination unit 344 determines the horizontal size of the input area F as the effective visual field range G, as shown in Fig. 11.

[0074] -Display control unit 337 The display control unit 337 generates various display screens and outputs the generated display screens to the display device 10. For example, the display control unit 337 places the start UI 40 described with reference to FIGS. 4 and 5 within the input area F having the horizontal size determined by the area size determination unit 344. Furthermore, when the start UI 40 is gazed at for a predetermined time, the display control unit 337 generates a display screen in which the input UI 50 is displayed in the input area F and moves the movement indicator 52 so as to follow the movement of the gaze position P. Furthermore, when the gaze position P (or the movement indicator 52) moves to the position L of the target indicator 54 of the input UI 50, the input determination unit 334 determines that the gaze operation is completed, and the display control unit 337 switches the display in the content display area C (see FIGS. 4 and 5) to the instruction sheet of the next page.

[0075] (Operation of information processing device 30-2) The configuration of the information processing device 30-2 according to the second embodiment of the present invention has been described above. Next, the operation of the information processing device 30-2 according to the second embodiment of the present invention will be summarized with reference to FIG.

[0076] 12 is a flowchart showing the operation of the information processing device 30-2 according to the second embodiment of the present invention. First, the data processing unit 310 measures the user's eye position Qs (xs, ys, zs) (S664). Then, the screen size acquisition unit 342 acquires the horizontal size of the display surface 12 (S668).

[0077] Next, the area size determination unit 344 calculates the user's effective visual field range G on the display surface 12 based on the relationship between the positions of the user's eyes and the position of the display surface 12 (S672). The process of S664 and the processes of S668 and S672 may be performed in the reverse order to that shown in Fig. 12, or may be performed simultaneously. Furthermore, the area size determination unit 344 determines whether the effective visual field range G is equal to or greater than the horizontal size of the display surface 12 (S676).

[0078] If the effective visual field range G is equal to or greater than the horizontal size of the display surface 12 (S676 / Yes), the area size determination unit 344 determines the horizontal size of the input area to be the horizontal size of the display surface 12 (S680). On the other hand, if the effective visual field range G is smaller than the horizontal size of the display surface 12 (S676 / No), the area size determination unit 344 determines the horizontal size of the input area to be the effective visual field range G (S684). Thereafter, the processes from S620 to S644 described with reference to FIG. 6 are executed (S688).

[0079] (Action and effect) The second embodiment of the present invention described above provides a variety of advantageous effects. For example, according to the second embodiment of the present invention, the horizontal size of the input area is determined based on the relationship between the position of the user's eyes and a reference position (e.g., the position of the display surface 12). This configuration, when a gaze operation involving horizontal movement is provided, both prevents erroneous operations and reduces the burden on the user. In particular, the area size determination unit 344 determines the horizontal size of the input area to be the horizontal size of the display surface 12 when the effective visual field range G is equal to or larger than the horizontal size of the display surface 12, and determines the horizontal size of the input area to be the effective visual field range G when the effective visual field range G is smaller than the horizontal size of the display surface 12, thereby more reliably achieving the above-described advantageous effects. In other words, it is possible to reduce the impact of differences in the position of the user's eyes and differences in the size of the display surface 12 on the operational burden.

[0080] <Third embodiment> The second embodiment has been described above in terms of the innovations made to the horizontal size of the input area. Next, the background of the third embodiment will be described, followed by a detailed description of the configuration and operation of the third embodiment.

[0081] (background) Patent Document 2 discloses a technique for varying the moving speed of a cursor operated by the line of sight based on the distance from a target to a gaze point or the amount of rotational movement of the eyeball.

[0082] However, in the technology disclosed in Patent Document 2, the display contents and characteristics of the UI for eye gaze operation are set uniformly for all users, which raises concerns that users with a high level of work proficiency may be dissatisfied with the settings.

[0083] The inventors of the present invention have taken the above circumstances into consideration and have come up with a third embodiment of the present invention. According to the third embodiment of the present invention, it is possible to vary the display content and characteristics of the UI for each user.

[0084] (Configuration of information processing device 30-3) 13 is an explanatory diagram showing the configuration of an information processing device 30-3 according to a third embodiment of the present invention. As shown in FIG. 13, the information processing device 30-3 according to the third embodiment of the present invention includes a data processing unit 310, an operation processing unit 330-3, and a display setting unit 350. The operation processing unit 330-3 includes a movement amount calculation unit 332, an input determination unit 334, and a display control unit 338. The display setting unit 350 is a unit that adjusts the display content and characteristics of the UI based on the user's proficiency and emotion, and includes an emotion estimation unit 352, a sensitivity adjustment unit 354, and a storage unit 356.

[0085] The data processing unit 310 is connected to a movement amount calculation unit 332, the movement amount calculation unit 332 is connected to an input determination unit 334 and a display control unit 338, and the input determination unit 334 is connected to the display control unit 338. The emotion estimation unit 352 is connected to a sensitivity adjustment unit 354, the storage unit 356 is connected to the sensitivity adjustment unit 354, and the sensitivity adjustment unit 354 is connected to the display control unit 338. The following mainly describes the configuration of the information processing device 30-3 that is different from the first embodiment.

[0086] -Emotion estimation part 352 The emotion estimation unit 352 is connected to the camera sensor 22 via a wired or wireless connection. The camera sensor 22 is typically implemented as a visible light camera and is installed on a workbench. It captures an image of the face of the user who is the worker and outputs the user's facial image to the emotion estimation unit 352. The emotion estimation unit 352 estimates the user's emotion based on the user's facial image input from the camera sensor 22. For example, the emotion estimation unit 352 estimates a value (e.g., 0 to 100) indicating the strength of each of multiple emotions at every Δt second interval. The multiple emotions include irritation, annoyance, and comfort with gaze control.

[0087] The emotion estimation method is not particularly limited. For example, the emotion estimation unit 352 may estimate the user's emotion using a trained model created in advance by supervised learning, or may estimate the user's emotion by another method.

[0088] -Sensitivity adjustment unit 354 The user identification sensor 24 is connected to the sensitivity adjustment unit 354. The user identification sensor 24 is configured to identify the user of the display device 10. When each user has a uniquely assigned user ID, the user identification sensor 24 may be, for example, a barcode reader that recognizes the user ID held by each user.

[0089] The sensitivity adjustment unit 354 sets the display content and characteristics of the UI according to the result of identification by the user identification sensor 24 and the result of estimation by the emotion estimation unit 352. For example, the sensitivity adjustment unit 354 accesses the storage unit 356 that stores a user DB in which the user ID of a user is associated with information about the user, and sets the display content and characteristics of the UI based on the result of estimation by the emotion estimation unit 352 when the proficiency level of the user identified by the user identification sensor 24 is equal to or higher than a standard.

[0090] Specifically, the sensitivity adjustment unit 354 may set one of a plurality of sensitivity levels as the display contents and characteristics of the UI. For example, the plurality of sensitivity levels may be the following sensitivity levels 1 to 6. Note that the pointer is an indication that indicates the user's gaze position, such as the pointer 60 shown in FIG. 14. The following speed is the speed at which the moving display 52 follows the movement of the user's gaze position.

[0091] Sensitivity level 1 Tracking speed: Low Pointer display: Yes Sensitivity level 2 Tracking speed: Low Pointer display: Off Sensitivity level 3 Tracking speed: Medium Pointer display: Yes Sensitivity level 4 Tracking speed: Medium Pointer display: Off Sensitivity level 5 Tracking speed: High Pointer display: Yes Sensitivity level 6 Tracking speed: High Pointer display: Off

[0092] When "irritation" predominates in the estimation result of the user's emotion, it is considered that the reaction of the moving display 52 is slow at the current sensitivity and the user is feeling dissatisfied. For this reason, the sensitivity adjustment unit 354 increases the sensitivity level when "irritation" predominates in the estimation result of the user's emotion.

[0093] Furthermore, if "comfortable" predominates in the estimation result of the user's emotion, it is considered that the user is not dissatisfied with the current sensitivity. Therefore, the sensitivity adjustment unit 354 maintains the sensitivity level when "comfortable" predominates in the estimation result of the user's emotion.

[0094] Furthermore, if "troubled" is dominant in the estimation result of the user's emotion, it is considered that the current sensitivity makes the moving display 52 too responsive, causing confusion to the user. For this reason, the sensitivity adjustment unit 354 lowers the sensitivity level when "troubled" is dominant in the estimation result of the user's emotion. The sensitivity adjustment unit 354 outputs the set sensitivity level to the display control unit 338.

[0095] It should be noted that the sensitivity adjustment unit 354 sets the sensitivity level to 1 when the user is a beginner who is using the information processing system according to the third embodiment of the present invention for the first time.

[0096] -Storage section 356 The storage unit 356 stores a user DB in which the user ID of a user is associated with information about the user. A specific example of the user DB will be described below with reference to FIG.

[0097] Fig. 15 is an explanatory diagram showing a specific example of the user DB stored in the storage unit 356. As shown in Fig. 15, in the user DB, the user ID, the number of days of use, the emotion at the time of last use, and the sensitivity level at the time of last use are associated with each other.

[0098] The number of days of use is the number of days the user has used the information processing system according to the third embodiment of the present invention. However, the number of days of use is an example of information indicating the user's proficiency, and other information such as the usage time, the number of eye gaze operations, and the number of times one or more tasks have been performed may be stored instead of the number of days of use. The sensitivity adjustment unit 354 can determine whether the user is a beginner and whether the user has a proficiency level equal to or higher than the standard based on the number of days of use.

[0099] The emotion at last use is the emotion of the user estimated when the user last used the information processing system according to this embodiment up to the present time. The sensitivity level at last use is the sensitivity level set when the user finished their last use of the information processing system according to this embodiment up to the present time. When a new user ID is registered in the storage unit 356, the number of days of use and the emotion at last use may be set to "0", and the sensitivity level at last use may be set to "1", as in the entry for user ID "0004" in Figure 15.

[0100] -Display control unit 338 The display control unit 338 generates various display screens and outputs the generated display screens to the display device 10. For example, the display control unit 338 generates a display screen including an input UI 50 and a content display area C, as shown in FIG.

[0101] Furthermore, the display control unit 338 controls the following speed of the moving display 52 in the input UI 50 and whether or not to display the pointer 60, according to the sensitivity level set by the storage unit 356. For example, in sensitivity level 1, the display control unit 338 reduces the following speed of the moving display 52 relative to the movement of the gaze position and displays the pointer 60. On the other hand, in sensitivity level 6, the display control unit 338 increases the following speed of the moving display 52 relative to the movement of the gaze position and does not display the pointer 60.

[0102] (Operation of information processing device 30-3) The configuration of the information processing device 30-3 according to the third embodiment of the present invention has been described above. Next, the operation of the information processing device 30-3 according to the third embodiment will be summarized with reference to FIG.

[0103] Fig. 16 is a flowchart showing the operation of the information processing device 30-3 according to the third embodiment. As shown in Fig. 16, first, when the user identification sensor 24 identifies the user of the display device 10 (S704), the sensitivity adjustment unit 354 refers to the user DB in the storage unit 356 and determines whether the number of days of use of the identified user is greater than 0 (S708).

[0104] If the number of days of use of the identified user is greater than 0 (S708 / Yes), the sensitivity adjustment unit 354 reads out the sensitivity level at the time of last use stored in the user DB for that user, sets the sensitivity level to that sensitivity level at the time of last use, and outputs information indicating that sensitivity level to the display control unit 338 (S712).On the other hand, if the number of days of use of the identified user is 0 (S708 / No), the sensitivity adjustment unit 354 sets the sensitivity level to 1 and outputs information indicating sensitivity level 1 to the display control unit 338 (S716).

[0105] Then, the display control unit 338 causes the display device 10 to display a display screen including the instruction sheet and the start UI 40 (S720). This starts accepting a gaze operation by the user (S724). That is, when the user gazes at the start UI 40, the display control unit 338 causes the display device 10 to display a display screen including the input UI 50. At this time, the display control unit 338 controls the tracking speed of the moving display 52 of the input UI 50 and whether or not to display the pointer 60, in accordance with the sensitivity set by the sensitivity adjustment unit 354. Furthermore, when the moving display 52 reaches the position of the target display 54, a predetermined process (for example, turning the page of the instruction sheet) is executed.

[0106] After the acceptance of the user's gaze operation has begun, for users whose proficiency level is above a standard (for example, the number of days of use is more than a predetermined number of days) (S728 / Yes), the processing for adjusting the sensitivity level is repeated by processing S736 to S756.

[0107] Specifically, the emotion estimation unit 352 estimates the user's emotion based on the user's facial image input from the camera sensor 22 (S736). Then, when "irritation" predominates in the estimation result of the user's emotion (S740 / "irritation" predominates), the sensitivity adjustment unit 354 increases the sensitivity level by one level (S744). On the other hand, when "comfort" predominates in the estimation result of the user's emotion (S740 / "comfort" predominates), the sensitivity adjustment unit 354 maintains the sensitivity level (S748). Furthermore, when "distress" predominates in the estimation result of the user's emotion (S740 / "distress" predominates), the sensitivity adjustment unit 354 decreases the sensitivity level by one level (S752). The display control unit 338 controls the following speed of the moving display 52 and whether or not to display the pointer 60 in accordance with the sensitivity level adjusted as needed by the sensitivity adjustment unit 354.

[0108] The processing of S736 to S756 is then repeated until the work is completed (S756 / Yes), and when the work is completed (S756 / No), the sensitivity adjustment unit 354 updates the user DB in the storage unit 356 (S760). Specifically, the sensitivity adjustment unit 354 increments the number of days of use for the user, stores the user's emotion last estimated in S736 as the emotion at last use, and stores the sensitivity level last set as the sensitivity level at last use.

[0109] On the other hand, for users whose proficiency level is below the standard (S728 / No), no processing is performed to adjust the sensitivity while the work continues (S732 / Yes), and the number of days of use in the user DB is incremented after the work is completed (S760).

[0110] (Action and effect) According to the third embodiment of the present invention described above, various operational effects can be obtained. For example, according to the third embodiment of the present invention, the sensitivity level is adjusted according to the user's emotions, so that the tracking speed of the moving display 52 and whether or not the pointer 60 is displayed are suited to the user. Therefore, it is possible to reduce the stress of the user when performing gaze operations.

[0111] Furthermore, according to the third embodiment of the present invention, the above-mentioned sensitivity level adjustment based on the user's emotion is performed when the user's proficiency level is equal to or higher than the standard. With this configuration, when the user's proficiency level is lower than the standard, the pointer 60 is displayed and the tracking speed of the moving display 52 is slow, so that it is expected that the user will be able to perform gaze operations without any confusion.

[0112] <Fourth embodiment> Having described the third embodiment of the present invention, the background of the fourth embodiment will now be described, followed by a detailed description of the configuration and operation of the fourth embodiment.

[0113] (background) In a system that performs eye-gaze operations, if eye-gaze operations are always accepted, instructions may be skipped or erroneous operations may occur. In this regard, Patent Document 3 discloses a technology that determines whether the user's line of sight is within a cursor display area, so that the cursor is not displayed while the user is reading text on a page, but is displayed after the user has finished reading the text.

[0114] However, in the above technology, the time until the UI for starting an input operation is displayed is set uniformly for all users. Therefore, there is a concern that users who read the instructions slowly relative to the set time may skip over the instructions or make an incorrect operation. Furthermore, users who read the instructions quickly relative to the set time may feel stressed because they have to wait for the UI to be displayed.

[0115] The technology disclosed in Patent Document 3 can be applied to e-books and the like that display only text, but it is not sufficiently effective when applied to content that may include not only text but also illustrations and photographs, such as work instructions in a factory.

[0116] The inventors of the present invention have taken the above circumstances into consideration and have come up with a fourth embodiment of the present invention. According to the fourth embodiment of the present invention, it is possible to set a waiting time until an eye-gaze operation can be accepted for each user.

[0117] (Configuration of information processing device 30-4) 17 is an explanatory diagram showing the configuration of an information processing device 30-4 according to a fourth embodiment of the present invention. As shown in FIG. 17, the information processing device 30-4 according to the fourth embodiment of the present invention includes a data processing unit 310, an operation processing unit 330-4, and a display management unit 360. The operation processing unit 330-4 includes a movement amount calculation unit 332, an input determination unit 334, and a display control unit 339. The display management unit 360 is a part that sets the waiting time until an eye-gaze operation can be accepted to a time suitable for the user by machine learning using the time the user spends looking at (gazing at) the content displayed on the display surface 12, and includes a time management unit 362, a machine learning unit 364, and a memory unit 366.

[0118] The data processing unit 310 is connected to the movement amount calculation unit 332 and the time management unit 362, the movement amount calculation unit 332 is connected to the input determination unit 334 and the display control unit 339, and the input determination unit 334 is connected to the display control unit 339. The time management unit 362 is connected to the machine learning unit 364, the storage unit 366, and the display control unit 339, and the machine learning unit 364 is connected to the storage unit 366. The following mainly describes the configuration of the information processing device 30-4 that is different from the first embodiment.

[0119] -Time management department 362 The time management unit 362 receives two-dimensional coordinates Pt(xt, yt) indicating the gaze position from the data processing unit 310, and measures the time during which the gaze position is within the content display area C shown in FIG. 4 etc., that is, the time during which the user gazes at each instruction sheet after the instruction sheet is displayed. Then, after the measured time has elapsed from the waiting time, the time management unit 362 notifies the display control unit 339 of the elapse of the waiting time. The waiting time is the time until a gaze operation can be accepted, and more specifically, may be a start UI display waiting time, which is the time until the start UI 40 is displayed. Below, an example will be described in which the waiting time is the start UI display waiting time.

[0120] The start UI display wait time is stored for each user and for each page of the instruction book in the instruction book DB stored in the storage unit 366. The time management unit 362 reads out the start UI display wait time corresponding to the user identified by the user identification sensor 24 and the page of the instruction book currently displayed on the display surface 12 from the instruction book DB in the storage unit 366, and dynamically sets the start UI display wait time for each user and for each page of the instruction book. Then, the time management unit 362 determines whether the time the user has been gazing at the instruction book after the display of each page of the instruction book has started has exceeded the start UI display wait time.

[0121] The time management unit 362 also outputs to the machine learning unit 364 the time (gazing time) that the user gazed at the instruction sheet before gazing at the start UI 40, the user ID of the user, and the page number of the instruction sheet.

[0122] -Machine Learning Department 364 The machine learning unit 364 receives a set of a user ID, which is identification information of the user, a page number of the instruction manual, which is identification information of the content, and a gaze time from the time management unit 362, and performs machine learning using the set of the user ID, page number of the instruction manual, and gaze time as training data. When the user ID and page number of the instruction manual are input, the machine learning creates a trained model that outputs the gaze time.

[0123] The machine learning unit 364 uses the trained model to acquire the gaze time for each page of the instruction book for each user, and outputs the acquired gaze time to the memory unit 366 together with the user ID and the page number of the instruction book.

[0124] -Storage unit 366 The storage unit 366 stores an instruction DB that stores a start UI display wait time for each user and for each page of the instruction book. Specifically, the storage unit 366 associates the gaze time received from the machine learning unit 364 with the user ID and page number received from the machine learning unit 364 and stores the gaze time received from the machine learning unit 364 as a start UI display wait time in the instruction DB.

[0125] FIG. 18 is an explanatory diagram showing a specific example of an instruction DB. The left diagram of FIG. 18 shows the UI start display wait time and the number of characters for each page of an instruction for a user with a user ID of "1." The right diagram of FIG. 18 shows the UI start display wait time and the number of characters for each page of an instruction for a user with a user ID of "55." The user with a user ID of "55" has never used the information processing system according to the fourth embodiment of the present invention. Therefore, an arbitrary initial value is set for the UI start display wait time. The initial value may be set by an administrator or may be a value calculated from the number of characters. For example, since the average human reading speed is said to be about 10 characters per second, the result of dividing the number of characters in a page of an instruction by 10 characters per second may be set as the initial value.

[0126] (Operation of information processing device 30-4) The configuration of the information processing device 30-4 according to the fourth embodiment of the present invention has been described above. Next, the operation of the information processing device 30-4 according to the fourth embodiment will be summarized with reference to FIG.

[0127] 19 is a flowchart showing the operation of the information processing device 30-4 according to the fourth embodiment. As shown in Fig. 19, first, when the user identification sensor 24 identifies the user of the display device 10 (S804), the time management unit 362 reads out from the storage unit 366 and sets the start UI display wait time corresponding to the identified user and the page of the instruction sheet displayed on the display surface 12 (S812).

[0128] Then, the display control unit 339 generates a display screen including the instruction sheet and displays the display screen on the display surface 12 of the display device 10 (S816). After that, the time management unit 362 starts measuring the gaze time during which the user gazes at the instruction sheet (S820).

[0129] The time management unit 362 repeats the determination of whether the gaze time has exceeded the start UI display wait time until the gaze time has exceeded the start UI display wait time (S824 / No). If the gaze time has exceeded the start UI display wait time (S824 / Yes), the processing of S620 to S644 shown in FIG. 6, including the start of display of the start UI 40, is executed (S828).

[0130] If the work is to be continued (S832 / No), the process from S812 is repeated. That is, the time management unit 362 sets the start UI display wait time for the next page of instruction sheet (S812), the display control unit 339 displays the display screen including the next page of instruction sheet on the display surface 12 of the display device 10 (S816), and the process from S820 onwards is continued.

[0131] On the other hand, if the work is to be completed (S832 / Yes), the machine learning unit 364 performs machine learning on each page of the instruction sheet based on the user's gaze time on each page of the instruction sheet (S840).Then, the machine learning unit 364 acquires the gaze time on each page of the instruction sheet for each user using the trained model obtained by machine learning, and updates the instruction sheet DB in the storage unit 366 with the acquired gaze time (S844).

[0132] (Action and effect) According to the fourth embodiment of the present invention described above, various operational effects can be obtained. For example, according to the fourth embodiment of the present invention, the waiting time from when the instruction sheet is displayed until the eye gaze operation becomes effective is dynamically set. This configuration can suppress the occurrence of skipping over the text and erroneous operations. In particular, the time management unit 362 can more reliably obtain the above-mentioned effects by setting the start UI display waiting time as the waiting time depending on the user and the instruction sheet to be displayed.

[0133] Furthermore, the machine learning unit 364 performs machine learning using a set of the user ID, the page number of the instruction book, and the gaze duration as training data, and the time management unit 362 sets the time output from the trained model by the machine learning unit 364 as the start UI display wait time. With this configuration, it is possible to set the start UI display wait time to a time suitable for the user and the page of the instruction book.

[0134] <Modification> An embodiment of the present invention has been described above. Below, several modified examples of the above-described embodiment will be described. Note that each modified example described below may be applied alone to the above-described embodiment, or may be applied in combination with the above-described embodiment. Furthermore, each modified example may be applied in place of the configuration of the above-described embodiment, or may be applied in addition to the configuration of the above-described embodiment.

[0135] For example, two or more of the first to fourth embodiments described above, or all of the embodiments, may be combined.

[0136] Furthermore, in the third and fourth embodiments, examples have been described in which the user identification sensor 24 is realized by a barcode reader, but user identification may also be realized using manual input of the user ID or biometric authentication, etc.

[0137] Furthermore, in the third embodiment, an example was described in which multiple values indicating the respective intensities of multiple emotions were output, but a single value indicating the intensity of a single emotion may be output, or a single value indicating the respective intensities of multiple emotions may be output.

[0138] Furthermore, in the third embodiment, an example has been described in which the number of days of use is determined as the level of proficiency, but the administrator may determine the level of proficiency of each user.

[0139] In addition, in the fourth embodiment, an example was described in which the start UI display wait time was set based on the gaze time on the content display area, but the display may also be triggered by the progress of work using user behavior recognition rather than the gaze time.

[0140] <Hardware configuration> The above describes the embodiments of the present invention. Information processing such as the above-described layout determination and display control is realized by a combination of software and hardware. Below, we will describe an example of a hardware configuration that can be applied to the information processing devices 30, 30-2, 30-3, and 30-4 according to the embodiments.

[0141] 20 is a block diagram showing an example of a hardware configuration 90. The hardware configuration 90 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, and a host bus 904. The hardware configuration 90 also includes a bridge 905, an external bus 906, an interface 907, an input device 908, a display device 909, an audio output device 910, a storage device (HDD) 911, a drive 912, and a network interface 915.

[0142] The CPU 901 functions as an arithmetic processing unit and control unit, and controls overall operation in accordance with various programs. The CPU 901 may also be a microprocessor. The ROM 902 stores programs used by the CPU 901, calculation parameters, etc. The RAM 903 temporarily stores programs used in the execution of the CPU 901, parameters that change as appropriate during the execution, etc. These are interconnected by a host bus 904 that includes a CPU bus, etc. Cooperation between the CPU 901, ROM 902, and RAM 903 and software can realize functions such as the data processing unit 310, placement processing unit 320, operation processing units 330, 330-2, 330-3, and 330-4, placement processing unit 340, display setting unit 350, and display management unit 360.

[0143] The host bus 904 is connected to an external bus 906, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 905. It is not necessary to configure the host bus 904, bridge 905, and external bus 906 separately, and these functions may be implemented on a single bus.

[0144] The input device 908 is composed of input means such as a mouse, keyboard, touch panel, buttons, microphone, sensors, switches, and levers that allow the user to input information, and an input control circuit that generates an input signal based on the user's input and outputs it to the CPU 901. By operating the input device 908, the user can input various data and instruct processing operations.

[0145] The display device 909 includes, for example, a display device such as a liquid crystal display (LCD) device, a projector device, an OLED (Organic Light Emitting Diode) device, a lamp, etc. The audio output device 910 includes an audio output device such as a speaker and a headphone.

[0146] The storage device 911 is a data storage device configured as an example of a storage unit according to this embodiment. The storage device 911 may include a storage medium, a recording device that records data on the storage medium, a reading device that reads data from the storage medium, and a deleting device that deletes data recorded on the storage medium. The storage device 911 is configured, for example, by an HDD (Hard Disk Drive) or an SSD (Solid State Drive), or a memory having equivalent functions. This storage device 911 drives storage and stores programs executed by the CPU 901 and various data.

[0147] The drive 912 is a reader / writer for a storage medium, and is built into or externally attached to the hardware configuration 90. The drive 912 reads information recorded on a removable storage medium 84, such as an attached magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs the information to the RAM 903 or the storage device 911. The drive 912 can also write information to the removable storage medium 84.

[0148] The network interface 915 is, for example, a communication interface configured with a communication device for connecting to a network, etc. The network interface 915 may be a wireless LAN (Local Area Network) compatible communication device or a wired communication device that performs wired communication.

[0149] <Supplementary information> Although the preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the present invention is not limited to these examples. It is clear that a person skilled in the art to which the present invention pertains can conceive of various modifications and alterations within the scope of the technical ideas set forth in the claims, and it is understood that these also naturally fall within the technical scope of the present invention.

[0150] For example, the steps in the processing of the information processing devices 30, 30-2, 30-3, and 30-4 in this specification do not necessarily have to be processed in chronological order according to the order described in the flowcharts. For example, the steps in the processing of the information processing devices 30, 30-2, 30-3, and 30-4 may be processed in an order different from the order described in the flowcharts, or may be processed in parallel.

[0151] It is also possible to create a computer program that causes the hardware, such as the CPU, ROM, and RAM, built into the information processing devices 30, 30-2, 30-3, and 30-4 to perform functions equivalent to those of the above-described information processing devices 30, 30-2, 30-3, and 30-4. A non-transitory storage medium storing the computer program is also provided. [Explanation of symbols]

[0152] 10 Display device 12 Display surface 20 Eye Sensor 22 Camera Sensor 24 User identification sensor 30 Information processing equipment 310 Data Processing Unit 320 Placement Processing Unit 322 Height determination unit 324 Placement determination section 330 Operation processing section 332 Travel amount calculation section 334 Input Judgment Unit 335 Storage section 336, 337, 338, 339 Display control unit 340 Placement Processing Unit 342 Screen size acquisition unit 344 Area size determination unit 350 Display setting section 352 Emotion estimation part 354 Sensitivity adjustment unit 356 Storage section 360 Display management department 362 Time Management Department 364 Machine Learning Department 366 Storage section

Claims

1. a placement processing unit that determines the placement of input areas on the display surface based on the positions of the user's eyes; an input determination unit that determines whether a predetermined gaze operation has been performed by the user in the input area; An information processing device comprising:

2. The placement processing unit The information processing apparatus according to claim 1 , wherein the arrangement of the input area in the vertical direction on the display surface is determined based on a relationship between the height of the user's eyes and the height of a reference position.

3. 3. The information processing device according to claim 2, wherein the placement processing unit places the input area at a first height position on the display surface when the position of the user's eyes is higher than the reference position, and places the input area at a second height position lower than the first height position when the position of the user's eyes is lower than the reference position.

4. the reference position is a center position of the display surface, The information processing apparatus according to claim 3 , wherein the first height position is higher than the reference position, and the second height position is lower than the reference position.

5. The placement processing unit The information processing apparatus according to claim 1 , wherein the horizontal size of the input area on the display surface is determined based on a relationship between the position of the user's eyes and a reference position.

6. the reference position is a position on the display surface, The information processing device according to claim 5 , wherein the placement processing unit determines the horizontal size of the input area based on a distance between the position of the user's eyes and the reference position in a normal direction of the display surface.

7. 7. The information processing device according to claim 6, wherein the placement processing unit calculates an effective visual field range, which is a length of the user's horizontal visual field range on the display surface, based on the separation distance, and determines the horizontal size of the input area according to a result of comparing the effective visual field range with a reference size.

8. 8. The information processing device according to claim 7, wherein the placement processing unit determines the horizontal size of the input area to be the reference size when the effective visual field range is larger than the reference size, and determines the horizontal size of the input area to be the effective visual field range when the effective visual field range is smaller than the reference size.

9. The information processing device according to claim 8 , wherein the reference size is a horizontal size of the display surface.

10. The information processing device according to claim 1 , further comprising a data processing unit that measures the position of the user's eyes based on an input from a sensor that captures an image of the user.

11. The information processing device includes: a feeling estimation unit that estimates a feeling of the user; a display control unit that controls display of the display surface including the input area according to a result of estimation by the emotion estimation unit; and The information processing device according to claim 1 , further comprising:

12. The information processing device according to claim 11 , wherein the display control unit controls a tracking speed of the moving display in the input area, which is moved in response to the gaze operation, with respect to the gaze operation, in accordance with a result of estimation by the emotion estimation unit.

13. The information processing device according to claim 11 , wherein the display control unit controls whether or not to display a display indicating a position being viewed by the user, depending on a result of estimation by the feeling estimation unit.

14. The information processing device according to claim 11 , wherein the display control unit executes control according to a result of estimation by the feeling estimation unit when the proficiency level of the user is equal to or higher than a standard.

15. The information processing device includes: a display control unit that displays content indicating the content to be confirmed by the user on the display surface; a time management unit that dynamically sets a waiting time from when the content is displayed until when a gaze operation on the input area becomes valid; The information processing device according to claim 1 , further comprising:

16. The information processing device according to claim 15 , wherein the time management unit dynamically sets the waiting time depending on the user and the content to be displayed.

17. the information processing device further includes a machine learning unit that performs machine learning using a set of user identification information, content identification information, and a time period during which the user gazes at the content, and creates a model that outputs a time period corresponding to the user identification information and the content identification information; The information processing apparatus according to claim 16 , wherein the time management unit sets the time output from the model as the waiting time.

18. determining an arrangement of input areas on a display surface based on the position of the user's eyes; determining whether a predetermined gaze operation has been performed by the user in the input area; A computer-implemented method comprising:

19. Computer, an arrangement determination unit that determines an arrangement of input areas on a display surface based on the position of the user's eyes; an input determination unit that determines whether a predetermined gaze operation has been performed by the user in the input area; A program to function as a

Images