Information processing apparatus, method for controlling information processing apparatus, and storage medium

Abstract

To detect mounting deviation of a head-mounted display device in a more suitable mode.SOLUTION: An image is displayed in a display area which is a display screen of a HMD120 used by being mounted on the head of a user and supported so as to be positioned in front of the eyes of the user by mounting the display apparatus. The gaze point calculation unit 102 acquires the line-of-sight information according to the calculation result of the line of sight of the user. The line-of-sight information holding unit 103 holds in advance first line-of-sight information acquired in a state in which the user is gazing at a predetermined position in the display area. The line-of-sight information comparison unit 104 compares the second line-of-sight information acquired in a state where the user is gazing at a predetermined position in the display area with the first line-of-sight information held in the line-of-sight information holding unit 103. When the difference between the second line-of-sight information and the first line-of-sight information is equal to or larger than a threshold, the mixed-reality image generation unit 109 notifies the user that the wearing state of the display device has changed.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, a control method for an information processing device, and a program. [Background technology]

[0002] In recent years, technologies such as virtual reality (VR) and augmented reality (AR) have been used in games and various simulations. Virtual reality is a technology that allows users to experience a virtual world created in a computer as if it were real. Augmented reality is a technology that adds digital information to a virtual space (virtual object) created using CG (Computer Graphics) and presents the virtual space to users by superimposing it on the real world so that the virtual space is reflected in the real world. A head-mounted display (HMD) may be used by users to experience virtual reality or augmented reality. In recent years, HMDs have become smaller and lighter, and not only large goggle-type displays but also flip-up and handy eyeglass-type displays are now available, making them suitable for use while traveling or working. In such usage scenarios, significant head movement or vibrations caused by the user's movements may cause a misalignment between the HMD and the head. Patent Document 1 discloses a technology for determining misalignment of an HMD by detecting a misalignment between the HMD's display screen and the user's eyes. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 5414946 Summary of the Invention [Problem to be solved by the invention]

[0004] On the other hand, the technology disclosed in Patent Document 1 determines whether the HMD is being worn by detecting the inner or outer corner of the user's eye from a captured image of the user's eyes, which tends to place a greater processing load on processors such as the CPU and GPU. In particular, in situations where an HMD is used while traveling in a vehicle or while performing work, a standalone HMD is used, and it is possible that the processing power of the processor may be limited.

[0005] In view of the above problems, an object of the present invention is to make it possible to detect misalignment of a head-mounted display device in a more suitable manner. [Means for solving the problem]

[0006] The information processing device of the present invention is characterized by having a display control means for displaying an image in a display area of ​​a display device that is worn on a user's head and is supported so that it is positioned in front of the user's eyes when the display device is worn, an acquisition means for acquiring gaze information according to a calculation result of the user's gaze, a storage means for storing first gaze information acquired by the acquisition means in advance when the user is gazing at a predetermined position in the display area, a comparison means for comparing second gaze information acquired by the acquisition means when the user is gazing at a predetermined position in the display area with the first gaze information stored in the storage means, and a notification means for notifying the user that the wearing state of the display device has changed when the difference between the second gaze information and the first gaze information is equal to or greater than a threshold. [Effects of the Invention]

[0007] According to the present invention, it is possible to detect misalignment of a head-mounted display device in a more suitable manner. [Brief explanation of the drawings]

[0008] [Figure 1]FIG. 2 is a diagram illustrating an example of a functional configuration of the system. [Figure 2] FIG. 1 illustrates an example of a hardware configuration of an information processing device. [Figure 3] 10A and 10B are diagrams illustrating an example of processing related to determination of misalignment of the HMD. [Figure 4] 10 is a flowchart illustrating an example of processing by an information processing device. [Figure 5] 10 is a flowchart illustrating an example of processing by an information processing device. [Figure 6] 10A and 10B are diagrams illustrating an example of processing related to determination of misalignment of the HMD. [Figure 7] 10 is a flowchart illustrating an example of processing by an information processing device. [Figure 8] FIG. 2 is a diagram illustrating an example of a functional configuration of the system. [Figure 9] 10A and 10B are diagrams illustrating an example of a process related to calculation of a line of sight angle. [Figure 10] 10 is a flowchart illustrating an example of processing by an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0010] First Embodiment An information processing apparatus according to a first embodiment of the present disclosure will be described below with reference to FIGS. First, an example of the functional configuration of a system according to this embodiment will be described with reference to Fig. 1. The system according to this embodiment includes an information processing device 100 and an HMD (Head Mounted Display) 120. The information processing device 100 includes a gaze detection unit 101, a gaze point calculation unit 102, a gaze information storage unit 103, a gaze information comparison unit 104, a captured image acquisition unit 105, a position and orientation measurement unit 106, a virtual data storage unit 107, a virtual space generation unit 108, and a mixed reality image generation unit 109. The above-mentioned series of components of the information processing device 100 are connected via a bus 110 so as to be able to transmit and receive data to and from each other. The HMD 120 also includes a pupil image capture unit 121, an image capture unit 122, and an image display unit 123. The information processing device 100 and the HMD 120 are connected via a predetermined network or transmission path so as to be able to transmit and receive data to and from each other. Note that the configuration for connecting the information processing device 100 and the HMD 120 so as to be able to transmit and receive data to and from each other is not particularly limited. For example, the information processing device 100 and the HMD 120 may be connected via a wired network or transmission path, or via a wireless network.

[0011] First, the configuration of the information processing device 100 will be described. The gaze detection unit 101 acquires an image (hereinafter also referred to as a pupil image) corresponding to the image capture result of the user's pupil by the pupil imaging unit 121, and calculates the user's gaze vector by performing gaze detection from the image. The gaze detection unit 101 inputs the calculated gaze vector to the gaze point calculation unit 102. In this embodiment, gaze detection will be described in various ways assuming that it is achieved by the corneal reflex method, which is a known technology, but the method is not particularly limited as long as gaze detection is possible, and other technologies may be applied. The corneal reflex method is a method in which near-infrared light is irradiated toward the eyeball, the reflected light from the corneal surface and the position of the pupil center are found by image processing, and the gaze vector is calculated by using the relationship between their relative positions.

[0012] The gaze point calculation unit 102 calculates the angle of the gaze with respect to the display screen of the HMD (hereinafter also referred to as the "gaze angle") and the coordinates of the position where the gaze is directed (hereinafter also referred to as the "gaze point") from the gaze vector acquired by the gaze detection unit 101. The gaze point calculation unit 102 inputs gaze information including the calculated gaze angle and gaze point to the gaze information storage unit 103 and the gaze information comparison unit 104.

[0013] The gaze information storage unit 103 stores the gaze information acquired by the gaze point calculation unit 102, and inputs the gaze information to the gaze information comparison unit 104. The gaze information stored in the gaze information storage unit 103 corresponds to information acquired when the HMD is properly worn.

[0014] The gaze information comparison unit 104 compares the gaze information stored in the gaze information storage unit 103, i.e., the gaze information acquired and stored when the HMD is properly worn, with the gaze information acquired by the gaze point calculation unit 102 (e.g., gaze information while the user is using the HMD). Based on the comparison result, the gaze information comparison unit 104 determines whether the coordinates of the gaze points included in the respective gaze information match. If they match, the gaze information comparison unit 104 compares the gaze angles at the target gaze points to determine whether the difference between the gaze angles to be compared is within a threshold. If the difference in the gaze angles exceeds a predetermined range, the gaze information comparison unit 104 determines that the HMD has shifted, and instructs the mixed reality image generation unit 109 to present information on the display screen to encourage the user to wear the HMD properly. Note that, although the present embodiment determines whether the gaze points are at the same coordinates, it may also determine whether the gaze points are in a predetermined region centered on the same coordinates. Among the gaze information to be compared by the gaze information comparison unit 104, the gaze information stored in the gaze information storage unit 103, i.e., the gaze information acquired and stored while the HMD is properly worn, corresponds to an example of first gaze information. Also, among the gaze information to be compared by the gaze information comparison unit 104, the gaze information acquired by the gaze point calculation unit 102 (for example, gaze information while the user is viewing content using the HMD) corresponds to an example of second gaze information.

[0015] Here, an example of a process for determining whether the HMD is misaligned will be described with reference to FIG. 3A is a schematic diagram showing a state in which a user is wearing an HMD properly. Reference numeral 300 indicates a user wearing the HMD. Reference numeral 301 indicates the left eye of the user 300. Reference numeral 302 indicates the HMD worn by the user 300. FIG. 3B shows an example of a gaze angle relative to the display screen of the HMD when the HMD is worn as shown in FIG. 3A. Reference numeral 303 denotes the display screen on the left eye side of the HMD 302. Reference numeral 304 denotes a specific position located in the upper right region of the display screen 303 when the user 300 views the display screen 303. The specific position is a fixed coordinate position on the display screen, and multiple specific positions are provided. In the example shown in FIG. 3, the specific values ​​are illustrated as points, but they do not need to be displayed so that they are visible to the user. In addition, in this embodiment, various descriptions will be given assuming that nine specific positions are set, but the locations and number of specific positions set are not particularly limited. Reference numeral 305 schematically denotes a gaze vector when the left eye 301 gazes at the specific position 304. Reference numeral 306 denotes the gaze angle of the gaze vector 305 relative to the display screen 303.

[0016] Fig. 3(C) shows a schematic diagram of a state in which the HMD is misaligned. In the example shown in Fig. 3(C), the HMD 302 is worn by the user 300 in a state inclined in the roll direction, compared to the example shown in Fig. 3(A). Fig. 3(D) shows an example of the gaze angle with respect to the display screen of the HMD when the HMD is worn as shown in Fig. 3(C). Reference numeral 307 schematically shows the gaze vector when the left eye 301 gazes at a specific position 304. Reference numeral 308 is the gaze angle of the gaze vector 307 with respect to the display screen 303.

[0017] As described above, the gaze angle 306 when the specific position 304 is gazed at while the HMD is properly worn is different from the gaze angle 308 when the specific position 304 is gazed at while the HMD is misfitted. Therefore, it is possible to determine whether or not the HMD is misfitted based on the difference between the gaze angles 306 and 308.

[0018] The captured image acquisition unit 105 acquires an image of real space (real space image) captured by the image capturing unit 122, and inputs the image to the position and orientation measurement unit 106 and the mixed reality image generation unit 109, respectively. The position and orientation measurement unit 106 measures the position and orientation of the HMD 120 and creates an environmental map by extracting feature points in the image using the real space image acquired by the captured image acquisition unit 105. The position and orientation measurement unit 106 inputs the position and orientation information of the HMD 120 and the environmental map to the mixed reality image generation unit 109. The position and orientation measurement of the HMD 120 and the creation of an environmental map can be achieved, for example, by a known image processing measurement method called VSLAM (Visual Simultaneous Localization and Mapping). VSLAM is a technology that can simultaneously estimate the self-position of an imaging device in an unknown environment and create environmental map coordinates. Note that the method used for this purpose is not particularly limited as long as it is possible to achieve position and orientation measurement and the creation of an environmental map. Specific examples that may be used include a measurement method using infrared light, a measurement method using ultrasound, and a measurement method using an inertial sensor.

[0019] The virtual data storage unit 107 stores data about the virtual space, such as data about virtual objects that make up the virtual space (e.g., shape information, position and orientation information, etc.) and data about the light source that illuminates the virtual space, and inputs the data to the virtual space generation unit 108. The virtual space generation unit 108 generates a virtual space based on the virtual data held in the virtual data holding unit 107 , and inputs information about the virtual space to the mixed reality image generation unit 109 . The mixed reality image generation unit 109 generates a virtual space image based on the virtual space information acquired from the virtual space generation unit 108 and the measurement results acquired from the position and orientation measurement unit 106 (for example, position and orientation information of the HMD 120 and an environmental map, etc.). The mixed reality image generation unit 109 also superimposes the virtual space image on the real space image acquired from the captured image acquisition unit 105 to generate an image of the mixed reality space (mixed reality image). Note that a known technology can be applied as a technology for generating a virtual space visible from a predetermined position, and therefore a detailed description of this technology will be omitted. Thereafter, the mixed reality image generation unit 109 outputs the generated mixed reality image to the image display unit 123 of the HMD 120.

[0020] Next, an example of the configuration of the HMD 120 will be described. The HMD 120 is provided with display screens, such as liquid crystal screens, for the right and left eyes. These display screens (display areas on which images, etc. are displayed) are attached to the HMD 120 so as to be positioned in front of the user's right and left eyes when the HMD 120 is worn on the user's head. In other words, when the HMD 120 is worn on the user's head, the display screens (for example, the display screens for the right and left eyes) are held (supported) by the housing of the HMD 120 or the like so as to be positioned in front of the user's eyes. Furthermore, images that reflect the parallax between the left and right screens are displayed in stereo. The pupil image capturing unit 121 captures an image of the pupil of the user viewing the display screen of the HMD, and inputs the pupil image to the information processing device 100. When capturing an image of the pupil, near-infrared light is irradiated onto the eyeball, and an image of the pupil and the light reflected on the surface of the cornea is captured. The image capturing unit 122 captures an image of the real space around the HMD 120, and inputs the captured image to the information processing device 100 as a real space image. The image display unit 123 displays the mixed reality image generated by the information processing device 100.

[0021] As described above, in this embodiment, various explanations will be given assuming that the HMD 120 is realized as a video see-through HMD that displays, on a display device, a mixed reality image generated based on an image captured by the image capturing unit 122. Alternatively, the HMD 120 may be realized as an optical see-through HMD that displays a virtual space image superimposed on a display medium that allows observation through real space. It goes without saying that, when an optical see-through HMD is used instead of a video see-through HMD, the control related to the display of the virtual space image is appropriately changed. Furthermore, since known techniques can be applied to realize the control related to the display of the virtual space image when an optical see-through HMD is used, a detailed description of this control will be omitted.

[0022] An example of the hardware configuration of the information processing device 100 according to this embodiment will be described with reference to FIG. The CPU 2010 comprehensively controls the operation of each component connected via the bus 2000. The ROM (Read Only Memory) 2020 stores the operating system, various processing programs according to this embodiment, device drivers, etc. The RAM (Random Access Memory) 2030 is used as a temporary storage area for the CPU 2010 to load various data and programs. The input I / F 2040 accepts input of an input signal from an external device (such as an imaging device or an operation device) in a format that can be processed by the information processing device 100. The output I / F 2050 outputs an output signal to an external device (such as a display device) in a format that can be processed by the device. Each of the functional units described above can be realized by the CPU 2010 loading a program stored in the ROM 2020 into the RAM 2030 and executing processing in accordance with each flowchart described below. For example, when configuring hardware as an alternative to software processing using the CPU 2010, it is sufficient to configure a calculation unit or circuit corresponding to the processing of each functional unit described here.

[0023] Next, an example of the processing of the information processing device 100 according to this embodiment will be described with reference to Fig. 4, focusing particularly on the processing related to determining whether the HMD is misaligned. The series of processing shown in Fig. 4 is started when a predetermined start condition is satisfied. As a specific example, the series of processing shown in Fig. 4 may be started when a user puts on an HMD and starts viewing content.

[0024] In S401, the information processing device 100 executes a calibration process to calculate a gaze angle at a specific position when the HMD is properly worn and to store the calculated gaze angle in advance. Details of the calibration process will be described separately later. In S407, the information processing device 100 determines whether a predetermined end condition is satisfied. As a specific example, the information processing device 100 may determine that the end condition is satisfied when an instruction to end viewing of the content is received from the user. If the information processing device 100 determines in S407 that the predetermined termination condition is not satisfied, the process proceeds to S402. On the other hand, if it is determined in S407 that the predetermined termination condition is satisfied, the information processing apparatus 100 terminates the series of processes shown in FIG.

[0025] In S402, the gaze detection unit 101 performs gaze detection based on the pupil image of the user acquired by the pupil image capturing unit 121, and calculates a gaze vector based on the result of the gaze detection. In S403, the gaze point calculation unit 102 calculates the gaze angle with respect to the display screen of the HMD and the gaze point based on the gaze vector acquired in S402.

[0026] In S404, the line-of-sight information comparison unit 104 determines whether the coordinates of the point of gaze acquired in S403 are a specific position. If the line-of-sight information comparison unit 104 determines in S404 that the coordinates of the point of gaze are at a specific position, the process proceeds to S405. On the other hand, if the line-of-sight information comparison unit 104 determines in S404 that the coordinates of the gaze point are not at a specific position, the process proceeds to S407. In this case, the processes from S407 onwards are executed again.

[0027] In S405, the gaze information comparison unit 104 compares the gaze angle acquired in S403 with the gaze angle at a specific position previously stored in the gaze information storage unit 103 in the processing of S401, and determines whether the difference between these gaze angles to be compared is within a threshold value. If the line-of-sight information comparison unit 104 determines in S405 that the difference in line-of-sight angle is within the threshold, the process proceeds to S407. In this case, the processes from S407 onwards are executed again. On the other hand, if the line-of-sight information comparison unit 104 determines in S405 that the difference in line-of-sight angle is not within the threshold (the difference exceeds the threshold), the process proceeds to S406.

[0028] In S406, the mixed reality image generation unit 109 generates a message indicating that misalignment of the HMD has been detected, and notifies (announces) the user of the misalignment of the HMD (in other words, a change in the wearing state) by superimposing the message on the display image. The mixed reality image generation unit 109 then proceeds to S407. In this case, the processes from S407 onward are executed again.

[0029] As an example of processing by the information processing device 100 according to this embodiment, the calibration processing shown as the processing of S401 in Fig. 4 will be described in detail with reference to Fig. 5. The calibration processing is performed when the user is properly wearing the HMD, and is a processing in which specific positions are displayed sequentially on the display screen and the user is prompted to gaze at the specific positions, thereby calculating and storing the gaze angle at each specific position.

[0030] In S501, the mixed reality image generation unit 109 sequentially displays the specific positions shown in FIG. 3(B) on the display image. In S502, the gaze detection unit 101 performs gaze detection based on the pupil image of the user acquired by the pupil image capturing unit 121, and calculates a gaze vector based on the result of the gaze detection. In S503, the gaze point calculation unit 102 calculates the gaze angle with respect to the display screen of the HMD and the gaze point based on the gaze vector acquired in S502.

[0031] In S504, the line-of-sight information comparison unit 104 determines whether the coordinates of the point of gaze acquired in S503 are a specific position. If the line-of-sight information comparison unit 104 determines in S504 that the coordinates of the point of gaze are at a specific position, the process proceeds to S505. On the other hand, if the line-of-sight information comparison unit 104 determines in S504 that the coordinates of the gaze point are not at a specific position, the process proceeds to S501. In this case, the processes from S501 onwards are executed again. In S505, the line-of-sight information storage unit 103 stores the line-of-sight angle calculated in S503 in association with a specific position displayed on the display image.

[0032] In S506, the information processing device 100 determines whether the gaze angles of all the specific positions have been stored. If it is determined in S506 that the gaze angles of all the specific positions have not been stored, the information processing device 100 proceeds to S501. In this case, the processes from S501 onward are executed again for the specific positions for which the gaze angles have not been stored. If it is determined in S506 that the gaze angles of all the specific positions have been stored, the information processing device 100 ends the series of processes shown in Fig. 5. In this case, the process of S402 shown in Fig. 4 is executed continuously.

[0033] As described above, the information processing device 100 according to this embodiment can determine whether the HMD is being worn out of place by performing gaze detection processing with a relatively low processing load, and can prompt the user to wear the HMD properly based on the results of the determination.

[0034] <Second embodiment> As an information processing device according to a second embodiment of the present disclosure, an example of a case in which, when a user gives an instruction by directing their gaze toward a virtual object, the gaze is utilized to determine whether the HMD is being worn out will be described below with reference to Figures 6 and 7. Note that the functional configuration of the system (information processing device and HMD) according to this embodiment is substantially the same as the configuration described with reference to Figure 1, and therefore detailed description thereof will be omitted.

[0035] First, with reference to FIG. 6, an example of a process for determining misalignment of the HMD by utilizing the line of sight when the user points to a virtual object will be described. 6(A) shows an example of the gaze angle with respect to the display screen of the HMD when the user is wearing the HMD properly. Reference numeral 601 indicates the user's left eye. Reference numeral 602 indicates the display screen on the left eye side of the HMD. Reference numeral 603 indicates a specific position located in the lower center area of ​​the display screen 602 when the user views the display screen 602. Reference numeral 604 schematically shows the gaze vector when the left eye 601 is gazing at the specific position 603. Reference numeral 605 is the gaze angle of the gaze vector 604 with respect to the display screen 602. Fig. 6(B) shows a state in which a virtual object is displayed on a display screen 602 such that an instruction portion of the virtual object is located at a specific position 603 shown in Fig. 6(A). Reference numeral 606 denotes a virtual object for prompting a user to give instructions. Reference numeral 607 denotes an instruction portion within the virtual object 606 for receiving instructions relating to the execution of processing from the user. In the example shown in Fig. 6(B), the display of the virtual object 606 is controlled so that the instruction portion 607 is located at the same position on the display screen 602 as the specific position 603 shown in Fig. 6(A). By applying the above-described control, when the user specifies the instruction unit 607, the user will gaze at the display position of the instruction unit 607. Therefore, by comparing the gaze angle at the time when the instruction unit 607 is specified, it becomes possible to determine whether the HMD is being worn out of place.

[0036] Next, an example of the processing of the information processing device according to this embodiment will be described with reference to Fig. 7, focusing particularly on the processing related to determining whether the HMD is misaligned. The series of processing shown in Fig. 7 is started when a predetermined start condition is satisfied. As a specific example, the series of processing shown in Fig. 7 may be started when a user puts on an HMD and starts viewing content.

[0037] The process of S701 is substantially the same as the process described with reference to FIG. 5 in the first embodiment, and therefore a detailed description thereof will be omitted. In S709, the information processing device 100 determines whether a predetermined end condition is satisfied. As a specific example, the information processing device 100 may determine that the end condition is satisfied when an instruction to end viewing of the content is received from the user. If the information processing apparatus 100 determines in S709 that the predetermined termination condition is not satisfied, the process proceeds to S702. On the other hand, if the information processing device 100 determines in S709 that the predetermined termination condition is satisfied, it terminates the series of processes shown in FIG.

[0038] In S702, the virtual space generation unit 108 determines whether or not there is an instruction to display a virtual object from the system (for example, an instruction to display a virtual object accompanying playback of content, etc.). If it is determined that there is an instruction to display a virtual object, the virtual space generation unit 108 places the virtual object for the instruction stored in the virtual data storage unit 107 in the virtual space, and then the process proceeds to S703. On the other hand, if the virtual space generation unit 108 determines that there is no instruction to display a virtual object, the process proceeds to S709. In this case, the processes from S709 onwards are executed again.

[0039] In S703, the mixed reality image generation unit 109 generates an image of the mixed reality space by superimposing the virtual space acquired in S702 on the real space image acquired by the captured image acquisition unit 105. The display position of the instruction portion of the virtual object for instruction displayed at this time is controlled so that it is positioned at the same position as the specific position. In S704, the information processing apparatus 100 determines whether or not an instruction from the user to the instruction portion of the virtual object has been accepted. If the information processing apparatus 100 determines in S704 that an instruction from the user to the instruction portion of the virtual object has not been accepted, the information processing apparatus 100 continues to wait for an instruction from the user. Then, if it is determined in S704 that an instruction from the user to the instruction portion of the virtual object has been accepted, the information processing apparatus 100 advances the process to S705.

[0040] In S705, the gaze detection unit 101 performs gaze detection based on the pupil image of the user acquired by the pupil image capturing unit 121, and calculates a gaze vector based on the result of the gaze detection. In S706, the gaze point calculation unit 102 calculates the gaze angle with respect to the display screen of the HMD based on the gaze vector acquired in S705.

[0041] In S707, the gaze information comparison unit 104 compares the gaze angle acquired in S706 with the gaze angle at a specific position previously stored in the gaze information storage unit 103 in the processing of S701, and determines whether the difference between these gaze angles to be compared is within a threshold value. If the line-of-sight information comparison unit 104 determines in S707 that the difference in line-of-sight angle is within the threshold, the process proceeds to S709. In this case, the processes from S709 onwards are executed again. On the other hand, if the line-of-sight information comparison unit 104 determines in S707 that the difference in line-of-sight angle is not within the threshold, the process proceeds to S708.

[0042] In S708, the mixed reality image generation unit 109 generates a message indicating that misalignment of the HMD has been detected and notifies the user of the misalignment of the HMD by superimposing the message on the display image. The mixed reality image generation unit 109 then proceeds to S709. In this case, the processes from S709 onward are executed again.

[0043] As described above, the information processing device according to this embodiment makes it possible to determine whether the HMD is being worn incorrectly when the user directs their gaze toward a virtual object to give an instruction. Furthermore, the determination of whether the HMD is being worn incorrectly is achieved by gaze detection processing, which has a relatively low processing load, and the user can be prompted to wear the HMD properly based on the results of the determination.

[0044] <Third embodiment> As an information processing device according to a third embodiment of the present disclosure, an example of a configuration having a function of generating a line of sight angle at an arbitrary display position on a display screen will be described with reference to FIGS.

[0045] First, an example of the functional configuration of the system according to this embodiment will be described with reference to Fig. 8. Note that the configuration shown in Fig. 8 differs from the example described in the first embodiment with reference to Fig. 1 in that it includes a line-of-sight angle generation unit 811, but the other components are substantially similar to the example shown in Fig. 1. Therefore, the description will focus on the parts that are different from the example shown in Fig. 1, and detailed description of parts that are substantially similar to the example shown in Fig. 1 will be omitted. The gaze angle generation unit 811 interpolates and calculates the gaze angle at any position on the display screen from the gaze information for each of the multiple specific positions stored in the gaze information storage unit 803, and inputs the gaze angle calculation result to the gaze information comparison unit 804.

[0046] Here, an example of a process for calculating the gaze angle at an arbitrary position on the display screen based on gaze information at a plurality of specific positions will be described with reference to FIG. Reference numeral 901 indicates the left eye of the user. Reference numeral 902 indicates the display screen on the left eye side of the HMD. Reference numeral 903 indicates a specific position located in the upper central area of ​​the display screen 902 when the display screen 902 is viewed from the user. Similarly, reference numeral 904 indicates a specific position located in the upper right area of ​​the display screen 902. Furthermore, reference numeral 905 indicates a specific position located in the central area of ​​the display screen 902. Furthermore, reference numeral 906 indicates a specific position located in the right area of ​​the display screen 902. Reference numeral 907 schematically indicates an arbitrary position (gazing point position) at which the user is gazing on the display screen 902. Reference numeral 908 schematically indicates a gaze vector when the left eye 901 is gazing at the gaze point position 907. Reference numeral 909 is the gaze angle of the gaze vector 908 with respect to the display screen 902. A gaze angle 909 when the user is wearing the HMD properly is calculated by interpolating gaze information at each of specific positions 903, 904, 905, and 906 for a gaze point position 907. Note that, in this embodiment, a case will be described in which the gaze angle at the gaze point position is calculated from the gaze angles of four specific positions around the gaze point position, but it may also be calculated from two or more specific positions.

[0047] 10, an example of the processing of the information processing device according to the embodiment will be described, focusing in particular on the processing when the gaze angle at an arbitrary gaze point is calculated and the misalignment of the HMD is determined based on the gaze angle. The series of processing shown in FIG. 10 is started when a predetermined start condition is satisfied. As a specific example, the series of processing shown in FIG. 10 may be started when a user puts on an HMD and starts viewing content.

[0048] The process of S1001 is substantially the same as the process described with reference to FIG. 5 in the first embodiment, and therefore a detailed description thereof will be omitted. In S1007, the information processing device 100 determines whether a predetermined end condition is satisfied. As a specific example, the information processing device 100 may determine that the end condition is satisfied when an instruction to end viewing of the content is received from the user. If the information processing apparatus 100 determines in S1007 that the predetermined termination condition is not satisfied, the process proceeds to S1002. On the other hand, if it is determined in S1007 that the predetermined termination condition is satisfied, the information processing apparatus 100 terminates the series of processes shown in FIG.

[0049] In S1002, the gaze detection unit 101 performs gaze detection based on the pupil image of the user acquired by the pupil image capturing unit 121, and calculates a gaze vector based on the result of the gaze detection. In S1003, the gaze point calculation unit 102 calculates the gaze angle with respect to the display screen of the HMD and the gaze point based on the gaze vector acquired in S1002. In S1004, the gaze angle generation unit 811 generates a gaze angle by interpolating the gaze angles at each of multiple specific positions located closer to the gaze point acquired in S1003 and stored in the gaze information storage unit 803.

[0050] In S1005, the gaze information comparison unit 104 compares the gaze angle of the gaze point acquired in S1003 with the gaze angles generated from multiple specific positions closer to the gaze point in S1004, and determines whether the difference between these gaze angles to be compared is within a threshold value. If the line-of-sight information comparison unit 104 determines in S1005 that the difference in line-of-sight angle is within the threshold, the process proceeds to S1007. In this case, the processes from S1007 onwards are executed again. On the other hand, if the line-of-sight information comparison unit 104 determines in S1005 that the difference in line-of-sight angle is not within the threshold, the process proceeds to S1006.

[0051] In S1006, the mixed reality image generation unit 109 generates a message indicating that misalignment of the HMD has been detected and notifies the user of the misalignment of the HMD by superimposing the message on the display image. The mixed reality image generation unit 109 then proceeds to S1007. In this case, the processes from S1007 onward are executed again.

[0052] As described above, the information processing device according to this embodiment makes it possible to generate a gaze angle at any position on the display screen from the gaze angles at each of multiple specific positions acquired when the user is wearing the HMD properly. This makes it possible to determine whether the HMD is being worn properly by comparing the gaze angle at any point of interest on the display screen with the gaze angle when the user is wearing the HMD properly.

[0053] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0054] The disclosure of this embodiment also includes the following configurations, methods, and programs. (Configuration 1) An information processing device comprising: a display control means for displaying an image in a display area of ​​a display device that is worn on a user's head and is supported so that the display device is positioned in front of the user's eyes when worn; an acquisition means for acquiring gaze information according to a calculation result of the user's gaze; a storage means for storing first gaze information acquired by the acquisition means in advance when the user is gazing at a predetermined position in the display area; a comparison means for comparing second gaze information acquired by the acquisition means when the user is gazing at a predetermined position in the display area with the first gaze information stored in the storage means; and an alert means for alerting the user that the wearing state of the display device has changed when the difference between the second gaze information and the first gaze information is equal to or greater than a threshold. (Configuration 2) The information processing device according to configuration 1, wherein the line-of-sight information includes coordinates of a position where the line of sight is directed and an angle of the line of sight with respect to the display area. (Configuration 3) The information processing device described in Configuration 2, characterized in that the angle of the line of sight with respect to the display area is calculated based on an image of the user's eyeball captured in a state in which the eyeball is illuminated with near-infrared light. (Configuration 4) The information processing device described in any one of configurations 1 to 3, characterized in that the display control means displays a virtual object for receiving instructions from the user by gaze, and at least one of the first gaze information and the second gaze information is acquired when the instruction is received, with the position where the virtual object is displayed being the predetermined position. (Configuration 5) An information processing device described in any one of configurations 1 to 4, characterized in that it has a generation means that generates new first gaze information for another position different from each of a plurality of positions in the display area based on the first gaze information acquired for each of the plurality of different positions stored by the storage means, wherein the comparison means, when the second gaze information is acquired by the acquisition means while the user is gazing at the other position, compares the second gaze information with the new first gaze information generated by the generation means for the other position, and when the difference between the second gaze information and the new first gaze information is equal to or greater than a threshold, notifies the user that the wearing state has changed. (Configuration 6) An information processing device described in any one of configurations 1 to 5, characterized in that the display control means displays in the display area at least one of an image of a virtual space, an image in which a virtual object is superimposed on an image of a real space, and a virtual object presented to the user so as to be superimposed on the real space. (Method 1) A control method for an information processing device, comprising: a display control step of displaying an image in a display area of ​​a display device worn on a user's head when used, the display area being supported so as to be positioned in front of the user's eyes when the display device is worn; an acquisition step of acquiring gaze information according to a calculation result of the user's gaze; a retention step of pre-storing first gaze information acquired in the acquisition step while the user is gazing at a predetermined position in the display area; a comparison step of comparing second gaze information acquired in the acquisition step while the user is gazing at a predetermined position in the display area with the first gaze information retained in the retention step; and a notification step of notifying the user that the wearing state of the display device has changed when the difference between the second gaze information and the first gaze information is equal to or greater than a threshold. (Program 1) A program for causing a computer to function as an information processing device, comprising: a display control means for displaying an image in a display area of ​​a display device worn on a user's head when used, the display area being supported so as to be positioned in front of the user's eyes when the display device is worn; an acquisition means for acquiring gaze information according to the calculation result of the user's gaze; a storage means for pre-storing first gaze information acquired by the acquisition means when the user is gazing at a predetermined position in the display area; a comparison means for comparing second gaze information acquired by the acquisition means when the user is gazing at a predetermined position in the display area with the first gaze information stored in the storage means; and a notification means for notifying the user that the wearing state of the display device has changed when the difference between the second gaze information and the first gaze information is greater than or equal to a threshold. [Explanation of symbols]

[0055] 100 Information processing device 101 Gaze detection unit 102 gaze point calculation unit 103 Gaze information storage unit 104 Gaze information comparison section 109 Mixed reality image generation unit

Claims

1. a display control means for displaying an image in a display area of ​​a display device that is worn on a user's head and is supported so as to be positioned in front of the user's eyes when the display device is worn; and an acquisition means for acquiring line-of-sight information according to a calculation result of the user's line of sight; a storage means for storing in advance first line-of-sight information acquired by the acquisition means while the user is gazing at a predetermined position in the display area; a comparison means for comparing second line-of-sight information acquired by the acquisition means while the user is gazing at a predetermined position in the display area with the first line-of-sight information stored in the storage means; a notification means for notifying the user that a wearing state of the display device has changed when a difference between the second line of sight information and the first line of sight information is equal to or greater than a threshold; An information processing device comprising:

2. The information processing device according to claim 1 , wherein the line-of-sight information includes coordinates of a position where the line of sight is directed and an angle of the line of sight with respect to the display area.

3. The information processing device according to claim 2 , wherein the angle of the line of sight with respect to the display area is calculated based on an image of the user's eyeball captured in a state where the eyeball is illuminated with near-infrared light.

4. the display control means displays a virtual object for receiving an instruction from the user by a line of sight; At least one of the first line-of-sight information and the second line-of-sight information is acquired when the instruction is accepted, with the position where the virtual object is displayed being the predetermined position.

2. The information processing device according to claim 1, wherein:

5. a generating means for generating new first line-of-sight information for positions different from each other, based on the first line-of-sight information acquired for each of a plurality of positions different from each other in the display area, which is stored by the storing means; the comparing means, when the acquiring means acquires the second line of sight information while the user is gazing at the other position, compares the second line of sight information with the new first line of sight information generated by the generating means for the other position; The notifying means notifies the user that the wearing state has changed when a difference between the second line of sight information and the new first line of sight information is equal to or greater than a threshold.

2. The information processing device according to claim 1, wherein:

6. 2. The information processing device according to claim 1, wherein the display control means causes at least one of an image of a virtual space, an image in which a virtual object is superimposed on an image of real space, and a virtual object presented to the user so as to be superimposed on real space to be displayed in the display area.

7. A control method for an information processing device, comprising: a display control step of displaying an image in a display area of ​​a display device that is worn on a user's head and is supported so as to be positioned in front of the user's eyes when the display device is worn; an acquisition step of acquiring line-of-sight information according to a calculation result of the user's line of sight; a holding step of holding in advance the first line-of-sight information acquired in the acquiring step while the user is gazing at a predetermined position in the display area; a comparing step of comparing the second line of sight information acquired in the acquiring step with the first line of sight information stored in the storing step while the user is gazing at a predetermined position in the display area; a notifying step of notifying the user that a wearing state of the display device has changed when a difference between the second line of sight information and the first line of sight information is equal to or greater than a threshold; 10. A method for controlling an information processing device, comprising:

8. Computer, a display control means for displaying an image in a display area of ​​a display device that is worn on a user's head and is supported so as to be positioned in front of the user's eyes when the display device is worn; and an acquisition means for acquiring line-of-sight information according to a calculation result of the user's line of sight; a storage means for storing in advance first line-of-sight information acquired by the acquisition means while the user is gazing at a predetermined position in the display area; a comparison means for comparing second line-of-sight information acquired by the acquisition means while the user is gazing at a predetermined position in the display area with the first line-of-sight information stored in the storage means; a notification means for notifying the user that a wearing state of the display device has changed when a difference between the second line of sight information and the first line of sight information is equal to or greater than a threshold; A program for causing an information processing device to function, comprising:

Images