Line of sight guidance method and line of sight guidance program

Abstract

To definitely guide a line of sight.SOLUTION: An information processing apparatus 1 is configured to: display, on a display apparatus 2, an image of a three-dimensional virtual space area seen from a viewpoint of a user from among three-dimensional virtual space areas; detect the position of a user's line of sight in the image and display a predetermined line of sight position image 4 at the position of the line of sight in the image; determine whether the movement of the user's line of sight satisfies a predetermined movement condition in a state where the line of sight position image 4 is displayed; and display, if the movement condition is satisfied, the line of sight position image 4 shifted by a predetermined amount from the position of the line of sight toward a target position 6a based on the relative positional relationship between the direction of the user's line of sight and the target position 6a set in a virtual space.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a gaze guidance method and a gaze guidance program. [Background technology]

[0002] In recent years, virtual reality (VR) technology has been applied to various fields. For example, an image showing a three-dimensional virtual space that recreates a workplace is displayed on a head-mounted display (HMD) worn by the user. By depicting the situation in the virtual space from the user's viewpoint based on the orientation and position of the user's head, the user can experience as if they were actually in the workplace. This technology is being used in training and education to acquire skills and improve them.

[0003] Furthermore, the following proposals have been made regarding technology for displaying images in virtual space. For example, an image processing device has been proposed that displays a guide image on a displayed image to guide the user in the direction in which the user should look. Another proposal has also been made regarding a gaze guidance device that synthesizes and displays a motion stimulus image for guiding the gaze in a peripheral visual field that is preset at the edge of a displayed image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-114822 [Patent Document 2] Japanese Patent Publication No. 2023-160623 Summary of the Invention [Problem to be solved by the invention]

[0005] One method for guiding a user's gaze in a specific direction while a virtual space is displayed is to display an explicit image for guiding the gaze, such as the guide image described above. While this method can guide the gaze relatively reliably, it may reduce the user's sense of immersion, and is therefore not suitable for applications where the gaze needs to be guided naturally. On the other hand, it is difficult to reliably guide the gaze with methods that implicitly guide the gaze.

[0006] In one aspect, the present invention aims to provide a gaze guidance method and a gaze guidance program that can reliably guide the gaze. [Means for solving the problem]

[0007] In one proposal, a gaze guidance method is provided in which a computer displays on a display device an image of an area of ​​a three-dimensional virtual space as seen from the user's viewpoint, detects the user's gaze position in the image, displays a predetermined gaze position image at the gaze position in the image, determines whether the movement of the user's gaze satisfies predetermined movement conditions while the gaze position image is displayed, and if the movement conditions are satisfied, shifts the gaze position image by a predetermined amount from the gaze position toward the target position based on the relative positional relationship between the direction of the user's gaze and a target position set in the virtual space.

[0008] In addition, one proposal provides a gaze guidance program that causes a computer to execute processing similar to the gaze guidance method described above. [Effects of the Invention]

[0009] On the one hand, it can reliably guide the eye. [Brief explanation of the drawings]

[0010] [Figure 1] 1A and 1B are diagrams illustrating an example of a configuration and a processing example of a display control system according to a first embodiment; [Figure 2] FIG. 10 is a diagram illustrating an example of the configuration of a work training system according to a second embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of the configuration of processing functions included in an information processing apparatus. [Figure 4] FIG. 10 is a diagram illustrating an example of gaze guidance processing. [Figure 5] 10 is a flowchart showing the overall processing procedure for gaze guidance. [Figure 6] FIG. 10 is a diagram illustrating a first example of advance preparation processing. [Figure 7] FIG. 10 is a diagram illustrating a second example of the advance preparation process. [Figure 8] 10 is a flowchart illustrating an example of a procedure for advance preparation processing. [Figure 9] FIG. 10 is a diagram illustrating an example of horizontal gaze guidance. [Figure 10] 10A and 10B are diagrams illustrating an example of processing for guiding the user's gaze in accordance with the movement of the user's position and gaze direction. [Figure 11] 10 is a flowchart illustrating an example of a procedure for a gaze guidance process. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [First embodiment] 1 is a diagram showing an example of the configuration and processing of a display control system according to Embodiment 1. The display control system shown in FIG.

[0012] The information processing device 1 is, for example, a computer device such as a personal computer or a server device. The information processing device 1 renders an image showing a three-dimensional virtual space and displays the image on the display device 2 for the user to view. In this image display, the information processing device 1 acquires the direction and position of the user's head in the virtual space, and renders an image of an area of ​​the virtual space seen from the user's viewpoint based on the direction and position. If the direction or position of the user's head changes, the displayed content of the image changes accordingly. The display device 2 may be, for example, an HMD worn on the user's head, or a general display placed on a desk, etc.

[0013] The processing of the information processing device 1 will be described below. As described above, the information processing device 1 displays an image of an area of ​​virtual space seen from the user's viewpoint. At the same time, the information processing device 1 detects the user's gaze position in the displayed image. The user's gaze position is detected, for example, using a dedicated sensor that detects the user's gaze. Alternatively, the forward direction of the user's head may be detected as the gaze position. In the former case, the gaze position may vary in the displayed image, but in the latter case, the gaze position is located at a fixed position (for example, the center of the image) in the displayed image.

[0014] The information processing device 1 displays a predetermined gaze position image at the gaze position in the displayed image. Image 3a shown in FIG. 1 is an example of an image depicting an area of ​​virtual space as seen from the user's viewpoint. As an example, a dot-like gaze position image 4 is displayed at the gaze position on this image 3a. As in this example, it is desirable that the gaze position image be a small image that is reliably visible to the user but does not significantly impede the user's visibility of the image.

[0015] The information processing device 1 determines whether the movement of the user's gaze satisfies a predetermined movement condition while displaying a gaze position image at the detected gaze position. This movement condition is a condition for making the user aware that the displayed gaze position image is the user's gaze position.

[0016] For example, buildings 5a to 5c are displayed in image 3b shown in Fig. 1. When the user's gaze captures all of buildings 5a to 5c (when the gaze position moves and covers all of buildings 5a to 5c), it is determined that the movement condition is met. In this case, for example, each time the user's gaze position is located on one of buildings 5a to 5c, the image of the corresponding building is highlighted, and when the gaze position is located on all of buildings 5a to 5c and the movement condition is satisfied, the user is notified of this.

[0017] When the information processing device 1 determines that the above movement condition is satisfied, it executes a gaze guidance process for guiding the user's gaze to a target position set in the virtual space. In this gaze guidance process, the information processing device 1 displays a gaze position image shifted by a predetermined amount from the detected gaze position toward the target position based on the relative positional relationship between the direction of the user's gaze and the target position in the virtual space.

[0018] In image 3c shown in FIG. 1, target position 6a is set on the side of bridge pier 6. Furthermore, in image 3c, target position 6a is located to the lower right of the detected gaze position. In this case, gaze position image 4 is displayed, for example, at a position shifted a predetermined amount to the lower right from the detected gaze position. Enlarged image 3d shown in FIG. 1 is an enlarged view of a rectangular area centered on the gaze position in image 3c. In this enlarged image 3d, gaze position image 4a when displayed at the gaze position is indicated by a dashed line, and it can be seen that gaze position image 4 is displayed shifted to the lower right from gaze position image 4a at the gaze position.

[0019] In the gaze guidance process shown in image 3c and enlarged image 3d, no explicit image is displayed to guide the gaze. This allows the user's gaze to be naturally guided toward target position 6a. However, unless the user is fully aware that the displayed gaze position image 4 indicates the user's gaze position, it is difficult to reliably guide the gaze.

[0020] Therefore, before starting the gaze guidance process, the information processing device 1 displays the gaze position image 4 at the detected gaze position, as shown in image 3b, and has the user move their gaze until a predetermined movement condition is met. This makes the user fully aware that the gaze position image 4 indicates the user's gaze position. Then, when the gaze guidance process is subsequently executed to shift the gaze position image 4 from the gaze position, the gaze is guided to the position of the gaze position image 4 in an attempt to maintain the recognition once formed by the user's brain that "the gaze position image 4 is the gaze position." As a result, the user's gaze can be reliably guided.

[0021] Second Embodiment Next, a case where the processing of the information processing device 1 is applied to a work training system will be described.

[0022] 2 is a diagram showing an example of the configuration of a work training system according to the second embodiment. The work training system includes an information processing device 100 and an HMD unit 200. Note that the information processing device 100 is an example of the information processing device 1 in FIG. 1, and the HMD unit 200 is an example of the display device 2 in FIG. 1.

[0023] This work training system allows a user wearing an HMD unit 200 to view an image showing a work site and receive work training or education. The information processing device 100 stores three-dimensional data showing a three-dimensional virtual space that recreates the work site, draws an image showing the virtual space based on the three-dimensional data, and displays the image on the HMD unit 200.

[0024] The HMD unit 200 is worn on the head of a user. The HMD unit 200 includes a display 201, a direction / position sensor 202, and a gaze sensor 203. The display 201 is a non-transparent HMD and displays an image based on image data transmitted from the information processing device 100. A display 201 for each of the right and left eyes may be provided. The direction / position sensor 202 is a sensor for detecting the direction (direction pointing forward) and position of the user's head, and includes, for example, an acceleration sensor and an angular velocity sensor (gyro). The gaze sensor 203 is a sensor for detecting the direction of the user's gaze. The gaze sensor 203 is, for example, a near-infrared camera. In this case, the near-infrared camera captures images of the light reflection points on the cornea and the eyeball, and the direction and position of the gaze are estimated from the fluctuations of these images.

[0025] Next, the hardware configuration of the information processing device 100 will be described with reference to Fig. 2. The information processing device 100 is realized as, for example, a computer as shown in Fig. 2. The information processing device 100 includes a processor 101, a random access memory (RAM) 102, a hard disk drive (HDD) 103, a graphics processing unit (GPU) 104, an input interface (I / F) 105, a reader 106, and a communication interface (I / F) 107.

[0026] The processor 101 (processor circuit) performs overall control of the information processing device 100. The processor 101 is, for example, a central processing unit (CPU), a micro processing unit (MPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a programmable logic device (PLD). The processor 101 may also be a combination of two or more elements of a CPU, an MPU, a DSP, an ASIC, or a PLD.

[0027] The RAM 102 is used as a main storage device of the information processing device 100. The RAM 102 temporarily stores at least a part of an OS (Operating System) program and application programs to be executed by the processor 101. The RAM 102 also stores various data necessary for processing by the processor 101.

[0028] The HDD 103 is used as an auxiliary storage device for the information processing device 100. The HDD 103 stores an OS program, application programs, and various data. Note that other types of nonvolatile storage devices, such as a solid state drive (SSD), can also be used as the auxiliary storage device.

[0029] A display 111 is connected to the GPU 104. The GPU 104 displays an image on the display 111 in accordance with an instruction from the processor 101. The display 111 may be a liquid crystal display, an organic EL (ElectroLuminescence) display, or the like.

[0030] An input device 112 is connected to the input interface 105. The input interface 105 transmits a signal output from the input device 112 to the processor 101. The input device 112 may be a keyboard or a pointing device. The pointing device may be a mouse, a touch panel, a tablet, a touch pad, a trackball, or the like.

[0031] A portable recording medium 113 is detachably attached to the reading device 106. The reading device 106 reads data recorded on the portable recording medium 113 and transmits it to the processor 101. The portable recording medium 113 may be an optical disk, a semiconductor memory, or the like.

[0032] The communication interface 107 transmits and receives data to and from other devices. In the present embodiment, for example, the communication interface 107 performs wireless communication and is capable of wirelessly communicating with the HMD unit 200. As another example, the information processing device 100 and the HMD unit 200 may be connected via a communication cable such as a USB (Universal Serial Bus).

[0033] The processing functions of the information processing device 100 can be realized by the above hardware configuration. 3 is a diagram showing an example of the configuration of processing functions of the information processing device 100. The information processing device 100 includes a storage unit 120, a direction / position detection unit 131, a gaze detection unit 132, a virtual space display unit 133, and a gaze guidance processing unit 134.

[0034] The storage unit 120 is a storage area allocated in a storage device included in the information processing device 100, such as the RAM 102 or the HDD 103. The storage unit 120 stores three-dimensional data 121. The three-dimensional data 121 is data representing the three-dimensional shape of a virtual space that is a virtualized work site that is the subject of work training or education. The three-dimensional data 121 includes, for example, point cloud data representing the three-dimensional coordinates of a plurality of feature points set in the virtual space, and data of texture images that are applied to each polygon based on the point cloud data. The virtual space may be a virtualized space of a real work site, or a space representing a fictitious work site created for training or education.

[0035] The processing of the direction / position detection unit 131, the line-of-sight detection unit 132, the virtual space display unit 133, and the line-of-sight guidance processing unit 134 is realized, for example, by the processor 101 executing a predetermined program.

[0036] The direction / position detection unit 131 detects the direction and position of the user's head based on the detection result of the direction / position sensor 202. Note that the direction and position of the user's head may be detected in response to a user operation, instead of being calculated based on actual measurements by the direction / position sensor 202. For example, the user may virtually change the direction and position of the user's own head in the up, down, left, and right directions in the virtual space using a stick-shaped input device or a key input device.

[0037] The gaze detection unit 132 detects the direction of the user's gaze based on the detection result of the gaze sensor 203. The virtual space display unit 133 renders an image showing the virtual space of the work site based on the three-dimensional data 121 and the direction and position of the user's head, and displays the image on the display 201 of the HMD unit 200.

[0038] In this image display, the direction and position of the user's head are associated with the coordinates of the virtual space, and the situation in the virtual space as seen from the user's viewpoint is reproduced on the image based on these directions and positions. For example, three-dimensional objects in the virtual space are placed on the image with a posture and position that corresponds to the direction and position of the user's head. Furthermore, if the direction or position of the user's head changes, the posture and position of the objects on the image also change accordingly. This allows the user to experience as if they were actually present at the work site and visually observing the environment while moving around the work site.

[0039] The virtual space display unit 133 may render the image of the virtual space as a panoramic image, a celestial sphere image, or a celestial sphere image centered on the head, or a part of such an image (for example, an image of a certain field of view centered in front of the head).

[0040] The image display by the virtual space display unit 133 as described above allows the user to grasp the situation at the work site in detail without actually going to the actual work site. This allows the user to receive training or education on work at the work site in a location other than the work site, such as an office. Also, three-dimensional data showing an appropriate work site for training or education can be prepared, and the user can receive training or education on that work site.

[0041] The information processing device 100 may be configured to allow the user to virtually experience work at a work site by detecting, for example, the movement of the user's hands. In this embodiment, an object that the user should pay attention to is set in the virtual space. Then, the information processing device 100 causes the user to perform the task of finding the set object as a training or education task. The gaze guidance processing unit 134 performs gaze guidance processing to guide the user's gaze toward the object based on the relative position of the object with respect to the direction of the user's gaze.

[0042] Hereinafter, the above-mentioned object to which the line of sight is guided may be referred to as a “target object.” Data relating to the target object set in the virtual space is included in the three-dimensional data 121, for example.

[0043] In order to enable users to reliably and efficiently master the above tasks and skills in a virtual space, it is effective to provide training and education while maintaining a sense of self-control. For example, rather than passive learning where users are shown images showing the work process, it is important to provide learning that gives users the feeling that they are performing the work of their own volition.

[0044] For example, a user with low skill in finding the target object described above may not be able to easily find the target object in the virtual space. In such a case, the gaze guidance processing unit 134 can guide the user's gaze in the direction of the target object, giving the user the feeling that they have found the target object themselves. This can enhance the effect of the user's acquisition and improvement of skills.

[0045] One possible method for guiding the gaze when displaying images in a virtual space is to explicitly display an image for guiding the gaze, such as an arrow. However, this method may reduce the sense of immersion in the video content, and as a result, may also reduce the user's sense of self-control. Therefore, while it is desirable to implicitly guide the gaze, this method presents the problem of difficulty in reliably guiding the gaze in the desired direction.

[0046] In this embodiment, the following method is first used to implicitly guide the gaze. The gaze guidance processing unit 134 constantly displays a gaze position image representing the user's gaze position on an image depicting the virtual space. The gaze guidance processing unit 134 then displays the gaze position image shifted a predetermined amount from its original position in the direction in which the target object exists. With this method, the user is unlikely to be aware that their gaze has been guided, making it possible to naturally guide their gaze in the direction of the target object.

[0047] FIG. 4 is a diagram illustrating an example of the gaze guidance process. First, the display of a gaze position image indicating the gaze position will be described. As described above, the virtual space display unit 133 renders an image showing the situation of the virtual space as seen from the user's viewpoint based on the direction and position of the user's head, and displays the image on the display 201 of the HMD unit 200. Image 210 shown in FIG. 4 is an example of an image obtained by rendering a portion of the virtual space in this manner. Image 210 shows the situation of a certain field of view area of ​​the virtual space, centered in front of the user's head. Therefore, the center of image 210 indicates the forward direction of the user's head.

[0048] Furthermore, the gaze guidance processing unit 134 detects the gaze position (position of the point of gaze) in the image 210 based on the detection result of the gaze detection unit 132. The gaze position is the position where the detected gaze direction intersects with the screen. The gaze guidance processing unit 134 displays a gaze position image 211 at the gaze position in the image 210. The gaze position image 211 is displayed as a dot-like image (point image) centered on the gaze position, as shown in FIG. 4, for example. This dot-like image is sufficiently visible to the user, but has a small radius that does not significantly affect the visibility of the background virtual space, and is drawn in a predetermined color.

[0049] Next, gaze guidance will be described. When a target object is set and gaze guidance processing is started, the gaze position image 211 is displayed at a position shifted by a predetermined amount in the direction of the target object from the original gaze position. The gaze guidance processing unit 134 first performs gaze guidance in the horizontal direction, and then performs gaze guidance in the vertical direction as necessary.

[0050] The reason for initially guiding the gaze only horizontally is to prevent the onset of symptoms known as "VR sickness." Because the human sense of balance is primarily controlled by the vestibular system of the inner ear, humans are more sensitive to vertical movement than horizontal movement in the image they are viewing. For this reason, humans are generally more susceptible to sickness when vertical movement occurs in the image they are viewing than when horizontal movement occurs. The gaze guidance processing unit 134 guides the gaze only horizontally as much as possible, minimizing vertical guidance, thereby making it less likely for the user to feel sick.

[0051] In FIG. 4, an example of a user's work is assumed to be inspecting cracks in a bridge set in a virtual space, and the location of a crack in a bridge pier 212 is set as a target object 213. An image of the target object 213 appears in the image 210. In this example, the target object 213 is located to the upper right of the user's line of sight. Therefore, the line of sight guidance processing unit 134 first executes processing to guide the user's line of sight to the right.

[0052] 4 shows an enlarged image 210a of the area of ​​image 210, with the detected gaze position at its center. In this enlarged image 210a, a gaze position image 211 indicating the detected original gaze position is indicated by a dotted line. As shown in this enlarged image 210a, the gaze guidance processing unit 134 first displays the gaze position image 211a shifted a predetermined amount to the right from the gaze position. This guides the user's gaze to the right.

[0053] Furthermore, the gaze guidance processing unit 134 executes processing to guide the user's gaze in the vertical direction, for example, when the horizontal coordinate of the gaze position reaches the vicinity of the position of the target object 213 but the vertical coordinate does not reach the vicinity of the position of the target object 213. The enlarged image 210b shown in FIG. 4 illustrates an example of a state in which the horizontal coordinate of the gaze position reaches the vicinity of the position of the target object 213 but the vertical coordinate does not reach the vicinity of the position of the target object 213. The center of the enlarged image 210b is the detected gaze position, and the enlarged image 210b shows a gaze position image 211b indicating the original gaze position by a dotted line. In this enlarged image 210b, the target object 213 is located to the upper right of the user's gaze position, so the gaze guidance processing unit 134 displays the gaze position image 211 shifted upward by a predetermined amount from the gaze position. This guides the user's gaze upward.

[0054] 4 illustrates a case where the target object 213 is included in the user's field of view (i.e., included in the displayed image). However, even if the target object 213 is outside the user's field of view, a gaze guidance process is performed to shift the gaze position image 211 from the gaze position based on the relative positional relationship between the gaze position and the target object 213.

[0055] According to the above-described gaze guidance process, the user's gaze can be naturally guided in the direction of the target object 213 without displaying an explicit image for guiding the gaze. This prevents a decrease in the sense of immersion in the video content, and as a result, prevents a decrease in the user's sense of self-operation. This makes it possible to efficiently acquire and improve work skills. On the other hand, the above-described gaze guidance process has the problem that it is difficult to reliably guide the gaze unless the user is fully aware that the displayed gaze position image indicates the user's gaze position.

[0056] 5 is a flowchart showing the overall processing procedure for gaze guidance. To solve the above problem, the gaze guidance processing unit 134 executes a preparatory process (step S11) to make the user fully aware that the gaze position image indicates the user's gaze position, before executing a gaze guidance process (step S12) in which the gaze position image is displayed shifted from the gaze position.

[0057] In the advance preparation process (step S11), the gaze guidance processing unit 134, while displaying a gaze position image at the detected gaze position, moves the user's gaze until a predetermined movement condition is met. This makes the user fully aware that the gaze position image indicates the user's gaze position. Then, when the gaze guidance process (step S12) is subsequently executed to shift the gaze position image from the gaze position, the gaze is guided to the position of the gaze position image in an attempt to maintain the recognition once formed by the user's brain that "the gaze position image is the gaze position." As a result, it is possible to reliably guide the user's gaze toward the target object, even though an image for guiding the gaze is not explicitly displayed.

[0058] An example of the advance preparation process will be described below. FIG. 6 is a diagram illustrating a first example of the advance preparation process. One example of a movement condition that can be applied is that the user visually recognizes one or more predetermined objects set in the virtual space. In the image 220 illustrated in FIG. 6, four buildings 221a to 221d are shown as examples of objects. The gaze guidance processing unit 134 determines that the movement condition is satisfied when the user's gaze captures all of buildings 221a to 221d (when the gaze position moves and is located on all of buildings 221a to 221d). Note that the gaze position being located on the object may be determined, for example, when the gaze position overlaps the area of ​​the object, or when the gaze image 222 overlaps part of the area of ​​the object.

[0059] In the case of FIG. 6, the gaze guidance processing unit 134 provides visual feedback to the user, for example, each time the user's gaze position is positioned on one of the buildings 221a to 221d. For example, an image of the building where the gaze position is positioned is highlighted. For example, the highlighting may be achieved by changing the color of the building or by thickening the outline of the building. Alternatively, the name of the building where the gaze position is positioned may be displayed. Furthermore, auditory feedback may be provided to the user each time the user's gaze position is positioned on one of the buildings 221a to 221d. For example, a sound indicating the name of the building where the gaze position is positioned is output. Such visual or auditory feedback stimulates the user's brain, enabling the user to form a strong awareness that the gaze position image is the user's own gaze position.

[0060] The gaze guidance processing unit 134 may determine that the movement condition is satisfied not when the gaze captures all of the predetermined objects as described above, but, for example, when the gaze captures a certain number of objects among a set number of objects. Alternatively, the gaze guidance processing unit 134 may determine that the movement condition is satisfied when the gaze captures at least a certain number of objects among a set number of objects and a certain amount of time has elapsed since the start of the advance preparation process. Furthermore, the objects to be viewed may be selected and set from objects arranged in the workspace, or may be objects (e.g., dedicated markers) specifically provided to be viewed during the advance preparation process.

[0061] Fig. 7 is a diagram showing a second example of the advance preparation process. As another example of the movement condition, a condition that the execution of a predetermined task is completed by the movement of the user's line of sight can be applied. Fig. 7 shows an example in which the task of drawing a predetermined figure is executed by the movement of the user's line of sight.

[0062] 7, the user's line of sight image 231 moves from the upper left to the upper right, lower right, and lower left, and then returns to its original position, thereby drawing a rectangle. The line of sight guidance processing unit 134, for example, displays movement trajectories 232a to 232d in accordance with the movement of the line of sight image 231, allowing the user to visually recognize the figure drawn by the line of sight movement. The line of sight guidance processing unit 134 determines that the movement condition is satisfied when it determines that a rectangle has been drawn.

[0063] 6, the gaze guidance processing unit 134 may draw a figure by moving the gaze to a plurality of pre-set objects in a specified order. Furthermore, the gaze guidance processing unit 134 can measure the degree to which the user recognizes that the gaze position image is the user's own gaze position by evaluating task performance. Therefore, the gaze guidance processing unit 134 may evaluate the time and accuracy required to draw the figure as task performance, and determine that the movement condition is satisfied when the degree of recognition estimated from the evaluation result is equal to or greater than a predetermined value.

[0064] As described above, by having the user perform a specified task by moving the gaze position image in accordance with the movement of the gaze, the user's brain is stimulated and it becomes possible for the user to form a strong awareness that the gaze position image is the user's own gaze position.

[0065] As other examples of the movement condition, it is possible to apply a condition that the line of sight moves for a certain period of time or more, or that the line of sight moves horizontally by a certain angle or more, for example. 8 is a flowchart illustrating an example of the procedure for advance preparation processing. The processing in FIG. 8 corresponds to step S11 in FIG.

[0066] [Step S21] The virtual space display unit 133 draws an image showing the virtual space as seen from the user's viewpoint based on the three-dimensional data 121 and the direction and position of the user's head, and displays the image on the display 201 of the HMD unit 200.

[0067] [Step S22] The gaze detection unit 132 detects the direction of the user's gaze based on the detection result of the gaze sensor 203. The gaze guidance processing unit 134 detects the gaze position in the displayed image based on the direction and position of the user's head and the detected gaze direction. The gaze guidance processing unit 134 draws a gaze position image at the detected gaze position.

[0068] [Step S23] The line-of-sight guidance processing unit 134 determines whether the line-of-sight movement condition is satisfied. If the movement condition is not satisfied, the process proceeds to step S21. Note that the processes of steps S21 to S23 are executed, for example, at an image frame cycle. On the other hand, if the movement condition is satisfied, the advance preparation process ends.

[0069] Next, the gaze guidance process will be explained. In the following explanation, the X and Z axes are taken as the horizontal direction of the virtual space, and the Y axis is taken as the vertical direction. The Z axis is taken in the direction forward of the user's head, the X axis is taken to the right of the user, and the Y axis is taken to be above the user.

[0070] Fig. 9 is a diagram showing an example of gaze guidance in the horizontal direction. In Fig. 9, as an example, it is assumed that the virtual space is rendered as a spherical image. In this case, the gaze position and the target position serving as the guidance target are expressed as coordinates on a spherical surface centered on the user's head. In the following description, the coordinates of the gaze position Pe are defined as (xe, ye, ze), and the coordinates of the target position Pt are defined as (xt, yt, zt).

[0071] In the following description, a predetermined point within the image area in which the target object is drawn is set as the target position. For example, the target position is set at the center of a rectangular area inscribed with the target object.

[0072] 9 shows a case where the user is viewed downward from above the head. In the state shown in the upper part of FIG. 9, as an example, the detected gaze position Pe is on the Z axis, and the target position Pt is located to the right and rear of the user's head. In this case, since the target position Pt is located relatively to the right of the gaze position Pe, the gaze position image is displayed shifted a predetermined amount to the right of the gaze position Pe.

[0073] In the lower part of FIG. 9, the center point of the gaze position image during the gaze guidance process is indicated as Pe'. The center point Pe' of the gaze position image is set, for example, at a position shifted by an angle θ1 from the detected gaze position Pe, with the position of the user's head as the center. In the example in the lower part of FIG. 9, the center point Pe' is set at a position shifted by an angle θ1 to the right from the gaze position Pe, and the gaze position image is displayed at this center point Pe'. On the other hand, although not shown, when the target position Pt is located relatively to the left of the gaze position Pe, the center point Pe' of the gaze position image is set at a position shifted by an angle θ1 to the left from the gaze position Pe.

[0074] Furthermore, in vertical gaze guidance, when the target position Pt is located relatively above the gaze position Pe, the center point Pe' of the gaze position image is set to a position shifted upward from the gaze position Pe by an angle θ2. On the other hand, when the target position Pt is located relatively below the gaze position Pe, the center point Pe' of the gaze position image is set to a position shifted downward from the gaze position Pe by an angle θ2. Here, with regard to the angles θ1 and θ2, θ1 may be equal to θ2. However, as described above, since humans are more sensitive to vertical image movement than horizontal image movement, setting θ1 > θ2 can make the user less likely to feel sick.

[0075] In the above explanation, the shift amount of the gaze position image from the detected gaze position Pe is defined by angles θ1 and θ2, but as another example, the shift amount may be defined by the number of pixels in the displayed image.

[0076] Next, a guidance end condition for ending the gaze guidance process and a vertical guidance start condition for starting the gaze guidance in the vertical direction will be described. The gaze guidance processing unit 134 terminates the gaze guidance processing, for example, when the horizontal and vertical coordinates of the gaze position Pe stay near the target object for a certain period of time. Specifically, for example, when both conditions of the following formulas (1-1) and (1-2) are satisfied continuously for a period of time T1, it determines that the guidance termination condition has been met and terminates the gaze guidance processing. xt-Δx <xe<xt+Δx ···(1-1) yt-Δy <ye<yt+Δy ···(1-2) Furthermore, the gaze guidance processing unit 134 starts vertical gaze guidance, for example, when the horizontal coordinate of the gaze position Pe stays near the target object for a certain period of time but the vertical coordinate of the gaze position Pe is not located near the target object. Specifically, for example, when the condition of the following formula (2) is satisfied continuously for a period of time T2 but the condition of formula (1-2) is not satisfied, it determines that the vertical guidance start condition has been met and starts vertical gaze guidance. xt-Δxh <xe<xt+Δxh ···(2) Between the above equations (1-1), (1-2) and (2), for example, Δx, Δy, Δxh, and Δyh may be set so that Δx≧Δxh and Δy≧Δyh.

[0077] 10 is a diagram showing an example of processing for guiding the gaze in response to the movement of the user's position and gaze direction, which illustrates an example of guiding the gaze in the horizontal direction. 10, a bridge 241 and a pier 242 are placed in a virtual space, and a target position Pt is set on the side of the pier 242. In this virtual space, it is assumed that the user's head position moves from Ph1 to Ph4. The arrows indicated at the head positions Ph1 to Ph4 indicate the user's line of sight.

[0078] At head position Ph1, the target position Pt is located relatively to the right of the gaze position Pe, so the center point Pe' of the gaze position image is set to a position shifted to the right from the gaze position Pe by an angle θ1. As a result, the user's viewpoint is guided to the right, and the user's head moves to head position Pt2.

[0079] At head position Ph2, the target position Pt is located relatively to the right of the gaze position Pe, so the center point Pe' of the gaze position image is set to a position shifted to the right from the gaze position Pe by an angle θ1. As a result, the user's viewpoint is guided to the right, and the user's head moves to head position Pt3.

[0080] At head position Ph3, the target position Pt is located relatively to the left of the gaze position Pe, so the center point Pe' of the gaze position image is set to a position shifted to the left from the gaze position Pe by an angle θ1. As a result, the user's viewpoint is guided to the left, and the user's head moves to head position Pt4.

[0081] At head position Ph4, it is assumed that the relationship between gaze position Pe and target position Pt satisfies the guidance termination condition using the above-mentioned formulas (1-1) and (1-2). In this case, the gaze guidance process ends, and the center point Pe' of the gaze position image is located at the gaze position Pe. Note that when the guidance termination condition is satisfied, not only the gaze guidance process ends, but also the display of the gaze position image may be terminated.

[0082] Although the above formulas (1-1), (1-2), and (2) show the conditions when the target position is set to one point, the target position may be a three-dimensional area having lengths in both the horizontal and vertical directions. In this case, for example, the minimum value xt1 and maximum value xt2 of the X coordinate and the minimum value yt1 and maximum value yt2 of the Y coordinate in this three-dimensional area may be used to set the guidance end condition and vertical guidance start condition as follows:

[0083] The guidance termination condition is when both of the following formulas (1-1a) and (1-2a) are satisfied continuously for a time T1. xt1-Δx <xe<xt2+Δx ···(1-1a) yt1-Δy <ye<yt2+Δy ···(1-2a) The vertical guidance start condition is when the condition of the following formula (2a) is satisfied continuously for a time T2, but the condition of formula (1-2a) is not satisfied. xt1-Δxh <xe<xt2+Δxh ···(2a) 11 is a flowchart showing an example of the procedure for the gaze guidance process. The process in FIG. 11 corresponds to the process in step S12 in FIG.

[0084] [Step S31] The virtual space display unit 133 draws an image showing the virtual space as seen from the user's viewpoint based on the three-dimensional data 121 and the direction and position of the user's head, and displays the image on the display 201 of the HMD unit 200.

[0085] [Step S32] The gaze detection unit 132 detects the direction of the user's gaze based on the detection result of the gaze sensor 203. The gaze guidance processing unit 134 detects the gaze position in the displayed image based on the direction and position of the user's head and the detected gaze direction. The gaze guidance processing unit 134 also detects the target position for gaze guidance.

[0086] The gaze guidance processing unit 134 detects the horizontal and vertical relative positions of the target position with respect to the gaze position. In this detection, if the gaze position and the target position do not match, it detects whether the target position is located to the right or left of the gaze position and the difference value of the coordinates of each position in the horizontal direction. It also detects whether the target position is located above or below the gaze position and the difference value of the coordinates of each position in the vertical direction.

[0087] [Step S33] The gaze guidance processing unit 134 calculates the drawing position of the gaze position image based on the detection result of the relative position with respect to the horizontal direction in step S32, and draws the gaze position image at that drawing position. If the target position is located to the right of the gaze position, the gaze position image is drawn at a position to the right of the gaze position by an angle θ1 centered on the user's head. On the other hand, if the target position is located to the left of the gaze position, the gaze position image is drawn at a position to the left of the gaze position by an angle θ1 centered on the user's head. This guides the user's gaze horizontally.

[0088] [Step S34] The gaze guidance processing unit 134 determines whether the gaze movement status satisfies the guidance end condition based on the detection results of the relative positions in the horizontal and vertical directions in step S33 at the most recent time T1. If the guidance end condition is not satisfied, the process proceeds to step S35. If the guidance end condition is satisfied, the gaze guidance process ends. Note that even in the latter case, the virtual space display unit 133 may continue drawing the image in the virtual space.

[0089] [Step S35] The gaze guidance processing unit 134 determines whether the vertical guidance start condition is satisfied based on the result of detecting the relative position with respect to the horizontal direction in step S33 at the most recent time T2 and the result of detecting the relative position with respect to the vertical direction in the most recent step S33. If the vertical guidance start condition is not satisfied, the process proceeds to step S31. In this case, the loop process of steps S31 to S35 is executed, for example, at a frame cycle. On the other hand, if the vertical guidance start condition is satisfied, the process proceeds to step S36.

[0090] [Step S36] The virtual space display unit 133 draws an image showing the virtual space as seen from the user's viewpoint based on the three-dimensional data 121 and the direction and position of the user's head, and displays the image on the display 201 of the HMD unit 200.

[0091] [Step S37] The gaze detection unit 132 detects the direction of the user's gaze based on the detection result of the gaze sensor 203. The gaze guidance processing unit 134 detects the gaze position in the displayed image based on the direction and position of the user's head and the detected gaze direction. The gaze guidance processing unit 134 also detects a target position for gaze guidance. The gaze guidance processing unit 134 detects the horizontal and vertical positions of the target position relative to the gaze position using the same procedure as in step S32.

[0092] [Step S38] The gaze guidance processing unit 134 calculates the drawing position of the gaze position image based on the detection result of the relative position with respect to the vertical direction in step S37, and draws the gaze position image at that drawing position. If the target position is located above the gaze position, the gaze position image is drawn at a position above the gaze position by an angle θ2 centered on the user's head. On the other hand, if the target position is located below the gaze position, the gaze position image is drawn at a position below the gaze position by an angle θ2 centered on the user's head. This guides the user's gaze in the vertical direction.

[0093] Thereafter, the process proceeds to step S34, where it is determined whether the gaze movement state satisfies the guidance end condition based on the detection results of the relative positions in the horizontal and vertical directions in step S37 at the most recent time T2. Note that the loop process of steps S34 to S38 is executed, for example, at a frame cycle.

[0094] In step S38, the gaze position image may be shifted from the gaze position not only in the vertical direction but also in the horizontal direction. At least when Δxh>Δx is set, it is desirable that gaze guidance in the horizontal direction is also performed in this way in step S38.

[0095] According to the second embodiment described above, it is possible to reliably perform natural gaze guidance while maintaining a sense of immersion and self-operation, thereby enabling users to efficiently acquire and improve the skills of the task being trained or learned.

[0096] In the second embodiment described above, the direction of the user's gaze is calculated based on the detection result of gaze sensor 203. However, the direction of the user's gaze may be set to the forward direction of the user's head. In this case, the gaze position in the displayed image is the center of the image.

[0097] Furthermore, at least a part of the above-described processing executed by the information processing device 100 may be executed by a processor (not shown) included in the HMD unit 200. For example, if the HMD unit 200 includes all of the processing functions shown in Fig. 3, the above-described processing is realized by the HMD unit 200 alone.

[0098] Furthermore, although the second embodiment above has described gaze guidance in VR, the processing of the second embodiment can also be applied to gaze guidance in mixed reality or augmented reality.

[0099] The processing functions of the devices (e.g., information processing devices 1, 100) shown in each of the above embodiments can be realized by a computer. In this case, a program describing the processing contents of the functions that each device should have is provided, and the processing functions are realized on the computer by executing the program on the computer. The program describing the processing contents can be recorded on a computer-readable recording medium. Examples of computer-readable recording media include magnetic storage devices, optical discs, and semiconductor memories. Examples of magnetic storage devices include hard disk drives (HDDs) and magnetic tapes. Examples of optical discs include CDs (Compact Discs), DVDs (Digital Versatile Discs), and Blu-ray Discs (BD, registered trademark).

[0100] When distributing a program, for example, the program is recorded on a portable recording medium such as a DVD or CD and sold. Alternatively, the program can be stored in a storage device of a server computer and transferred from the server computer to other computers via a network.

[0101] A computer that executes a program stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. The computer then reads the program from its own storage device and executes processing in accordance with the program. Note that the computer can also read the program directly from a portable recording medium and execute processing in accordance with that program. The computer can also execute processing in accordance with the program received each time a program is transferred from a server computer connected via a network. [Explanation of symbols]

[0102] 1. Information processing equipment 2 Display device 3a~3c images 3D enlarged image 4,4a Gaze position image 5a~5c Building 6. Pier 6a Target position

Claims

1. The computer displaying on a display device an image of an area of ​​the three-dimensional virtual space seen from the user's viewpoint; Detecting a gaze position of the user in the image and displaying a predetermined gaze position image at the gaze position in the image; determining whether a movement of the user's gaze satisfies a predetermined movement condition while the gaze position image is displayed; When the movement condition is satisfied, the gaze position image is displayed by shifting it by a predetermined amount from the gaze position toward the target position based on a relative positional relationship between the gaze direction of the user and the target position set in the virtual space. Eye guidance method.

2. the movement condition indicates that the user's line of sight has moved to each of one or more predetermined objects arranged in the virtual space; The method for guiding the gaze according to claim 1.

3. the movement condition indicates that a predetermined task has been performed by a movement of the gaze position image accompanying a gaze movement of the user. The method for guiding the gaze according to claim 1.

4. When the movement condition is satisfied, the gaze position image is displayed shifted horizontally from the gaze position; when a predetermined horizontal position condition indicating that the horizontal coordinate of the gaze position has reached the vicinity of the target position in the horizontal direction is satisfied but a predetermined vertical position condition indicating that the vertical coordinate of the gaze position has reached the vicinity of the target position in the vertical direction is not satisfied, the gaze position image is displayed shifted vertically from the gaze position; The method for guiding the gaze according to claim 1.

5. On the computer, displaying on a display device an image of an area of ​​the three-dimensional virtual space seen from the user's viewpoint; Detecting a gaze position of the user in the image and displaying a predetermined gaze position image at the gaze position in the image; determining whether a movement of the user's gaze satisfies a predetermined movement condition while the gaze position image is displayed; When the movement condition is satisfied, the gaze position image is displayed by shifting it by a predetermined amount from the gaze position toward the target position based on a relative positional relationship between the gaze direction of the user and the target position set in the virtual space. A gaze guidance program that executes the process.

Images