Information processing device

Abstract

To provide a technology which makes user operation possible with a simple constitution and high operability.SOLUTION: An information processing device comprises control means which controls a head-mounted type display device according to touch operation with respect to a touch operation surface. In calibration of the touch operation, the control means: controls the head-mounted type display device so as to identify a first display position which is a position in a video displayed by the head-mounted type display device; makes a user perform touch on the touch operation surface while paying attention to the first display position, and acquires information on a first touch position which is the position to which the touch has been performed; controls the head-mounted type display device so as to identify a second display position in a prescribed positional relation with respect to the first display position; and makes the user perform touch on the touch operation surface while paying attention to the second display position, and acquires the information on a second touch position which is the position to which the touch has been performed.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, and more particularly to a method for operating a head-mounted display device.

Background Art

[0002] There are various display devices that display a virtual space, such as a head-mounted display (HMD) and a game device. A user can operate an object in the virtual space displayed on the display device using an operation member in the real world. In order to improve the reality, it is required to be able to perform operations with the same feeling as in real life in the real world.

[0003] Patent Document 1 discloses a technique for enabling an operation on a 3D displayed object without a sense of strangeness. In the technique disclosed in Patent Document 1, an operation member is imaged by a camera, a relative position of the operation member with respect to the camera is calculated based on the captured image of the operation member, and processing is performed based on the display position of the object and the relative position of the operation member.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the technique disclosed in Patent Document 1, it is necessary to use a camera, which causes complication and cost increase of the apparatus (system). Furthermore, it is necessary to keep the operation member within the viewing angle of the camera, which restricts the user's movement.

[0006] In addition, when a user wears a head-mounted display device such as an HMD, the user may not be able to view the scenery of the outside world (real space), and the operability of user operations using operation members may be significantly reduced. For example, when a user is wearing an HMD and experiencing a virtual space unrelated to the real space, the user cannot touch while looking at an operation member (e.g., a touch pad) capable of receiving a touch operation, and cannot easily touch a desired location.

[0007] An object of the present invention is to provide a technology that enables a user operation with high operability with a simple configuration (a user operation that has no restrictions on the user's actions and can easily give a desired instruction).

Means for Solving the Problems

[0008] A first aspect of the present invention includes control means for controlling a head-mounted display device according to a touch operation on a touch operation surface. In the calibration of the touch operation, the control means controls the head-mounted display device so as to be able to identify a first display position that is a position in an image displayed by the head-mounted display device, causes the user to touch the touch operation surface while being conscious of the first display position, and acquires information on a first touch position that is a position where the touch is performed. The head-mounted display device is controlled so as to be able to identify a second display position that has a predetermined positional relationship with respect to the first display position, and the user is caused to touch the touch operation surface while being conscious of the second display position, and information on a second touch position that is a position where the touch is performed is acquired. It is an information processing apparatus characterized by the above.

[0009] A second aspect of the present invention is a control method for an information processing apparatus that controls a head-mounted display device according to a touch operation on a touch operation surface, the method having a step of performing calibration of the touch operation, the step of performing calibration including: controlling the head-mounted display device so as to be able to identify a first display position that is a position in an image displayed by the head-mounted display device; causing a user to perform a touch on the touch operation surface while being aware of the first display position, and acquiring information on a first touch position that is a position where the touch is performed; controlling the head-mounted display device so as to be able to identify a second display position that has a predetermined positional relationship with respect to the first display position; and causing the user to perform a touch on the touch operation surface while being aware of the second display position, and acquiring information on a second touch position that is a position where the touch is performed.

[0010] A third aspect of the present invention is a program for causing a computer to function as each means of the information processing apparatus.

Advantages of the Invention

[0011] According to the present invention, a user operation with high operability can be performed with a simple configuration (a user operation that has no restrictions on the user's actions and can easily give a desired instruction).

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Mode for Carrying Out the Invention

[0013] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0014] <Description of Configuration> Figs. 1(A) and 1(B) are external views showing the configuration of the display system according to the present embodiment. The display system according to the present embodiment includes a head-mounted display (HMD) 100 which is a head-mounted display device, and a controller 200 which is an operating device separate from the HMD 100. Fig. 1(A) is a front perspective view of the display system (HMD 100 and controller 200), and Fig. 1(B) is a rear perspective view of the display system.

[0015] The HMD 100 has imaging lenses 101a and 101b on the front side, and can image the external world (real space, the front side of the HMD 100) through the imaging lenses 101a and 101b. Further, the HMD 100 has display units 102a and 102b on the rear side, and various images can be displayed on the display units 102a and 102b. The display units 102a and 102b may have a display panel such as a liquid crystal panel or an organic EL panel, or may not. The display units 102a and 102b may have a laser light source that directly projects an image onto the user's retina.

[0016] For example, when the HMD 100 is worn on the user's head, the arrangement of the display units 102a and 102b is determined such that the display unit 102a faces the user's left eye and the display unit 102b faces the user's right eye. And the arrangement of the imaging lens 101a is determined such that the visual field of the left eye of the user wearing the HMD 100 (the visual field when it is assumed that the user is not wearing the HMD 100) is imaged through the imaging lens 101a. Similarly, the arrangement of the imaging lens 101b is determined such that the visual field of the right eye of the user wearing the HMD 100 (the visual field when it is assumed that the user is not wearing the HMD 100) is imaged through the imaging lens 101b.

[0017] Note that the display units 102a and 102b may display the video of the real space captured through the imaging lenses 101a and 101b in real time. That is, the HMD 100 may operate as a video see-through type HMD. The display units 102a and 102b may display the video of the virtual space (for example, the video of the virtual space unrelated to the real space). The present invention is applicable to any of VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality).

[0018] The controller 200 has a touch operation member 201 (for example, a touch pad). The touch operation member 201 has a touch operation surface and can receive a touch operation on the touch operation surface. In the present embodiment, the information of the operation on the controller 200 (for example, the touch operation on the touch operation member 201 (touch operation surface)) is transmitted to the HMD 100 and used for the control of the HMD 100. The controller 200 may be a controller dedicated to the HMD 100 or a non-dedicated device such as a smartphone or a tablet terminal.

[0019] Note that in the present embodiment, the operation member for controlling the HMD 100 is provided in a controller 200 separate from the HMD 100, but the operation member may be provided integrally with the HMD 100. Further, in the present embodiment, the HMD 100 controls itself, but an information processing device (for example, a server) separate from the HMD 100 may receive the information of the user operation and control the HMD 100 by transmitting a control signal to the HMD 100.

[0020] FIG. 2 is a block diagram showing the configuration of the display system (HMD 100 and controller 200) according to the present embodiment. The HMD 100 has imaging lenses 101a and 101b, display units 102a and 102b, imaging elements 103a and 103b, a display unit driving circuit 104, a CPU 105, a memory unit 106, and a wireless communication circuit 107. The controller 200 has a touch operation member 201, a display unit 202, a display unit driving circuit 204, a CPU 205, a memory unit 206, and a wireless communication circuit 207.

[0021] In the HMD 100, the CPU 105 controls the imaging elements 103a and 103b, the display unit driving circuit 104, the memory unit 106, and the wireless communication circuit 107. The imaging element 103a images the real space through the imaging lens 101a, and the imaging element 103b images the real space through the imaging lens 101b. The display unit driving circuit 104 drives the display units 102a and 102b so that various images are displayed. The memory unit 106 stores various information. For example, a control program is stored in the memory unit 106 in advance, and the CPU 105 reads out the control program from the memory unit 106 and executes it to control each part of the HMD 100. The wireless communication circuit 107 enables wireless communication between the HMD 100 and an external device (including communication via the Internet).

[0022] In the controller 200, the CPU 205 receives signals from the touch operation member 201 (signals output in response to touch operations) and controls the display unit driving circuit 204, the memory unit 206, and the wireless communication circuit 207. The display unit driving circuit 204 drives the display unit 202 so that various images (various screens) are displayed. The memory unit 206 stores various information. For example, a control program is stored in the memory unit 206 in advance, and the CPU 205 reads out the control program from the memory unit 206 and executes it to control each part of the controller 200. The wireless communication circuit 207 enables wireless communication between the controller 200 and an external device (including communication via the Internet).

[0023] In this embodiment, wireless communication is performed between the wireless communication circuit 107 of the HMD 100 and the wireless communication circuit 207 of the controller 200, but wired communication may be performed between the HMD 100 and the controller 200. In the controller 200, the touch operation member 201 and the display unit 202 may be separate bodies or may be integrally provided like a touch panel. In addition to the touch operation member 201, the controller 200 may have one or more operation members such as a lever, a direction key, or a button.

[0024] <Description of User Operations> In this embodiment, the HMD 100 is controlled according to a touch operation on the touch operation member 201 (touch operation surface) of the controller 200. When the user touches the touch operation member 201, the CPU 205 of the controller 200 recognizes the touch position (the position (coordinates) where the touch is made), and notifies the CPU 105 of the HMD 100 via the wireless communication circuits 207 and 107. Then, the CPU 105 determines that the position (coordinates, display position) corresponding to the notified touch position in the currently displayed video is specified, and performs control according to the determination result.

[0025] <Description of Touch Operation Calibration> Generally, when performing a touch operation on a touch panel, the user touches a desired location on the touch panel (touch operation surface) while looking at the video displayed on the touch panel. Similarly, in the case of the absolute coordinate system in which the coordinates of the video and the coordinates of the touch operation surface are associated one-to-one, the user wants to touch a desired location while looking at the touch operation surface. However, when the user wears the HMD 100 and performs a touch operation on the touch operation member 201, the user cannot see the controller 200 and cannot easily touch a desired location on the touch operation member 201 (touch operation surface). This is the first problem.

[0026] In addition to the absolute coordinate system, there is also a relative coordinate system for touch operation methods. In the case of the relative coordinate system, for example, various controls are performed based on the positional relationship between the previous touch position and the current touch position. However, since the size of the video viewed by the user is different from the size of the touch operation member 201 (touch operation surface), the user cannot easily grasp how far the distance from the previous touch position to the current touch position is treated as in the video. This is the second problem.

[0027] Therefore, in the present embodiment, calibration to be described later is executed so that even when the controller 200 cannot be seen in the touch operation in the absolute coordinate system, it is possible to easily touch a desired position on the touch operation member 201 (touch operation surface).

[0028] FIG. 3(A) is a flowchart showing the operation of the HMD 100. The operation in FIG. 3(A) is started, for example, in response to an instruction to start the HMD 100 from the user or the HMD 100 being worn on the user's head.

[0029] In step S101, the CPU 105 performs a startup process. The startup process includes initial settings such as initialization of various parameters used for controlling the HMD 100.

[0030] In step S102, the CPU 105 determines whether to execute calibration. If it is determined to execute calibration, the process proceeds to step S103; otherwise, the process proceeds to step S104.

[0031] Calibration is performed when the user starts (or resumes) holding the controller 200 It is preferably executed when the user changes or when a new user is detected. For example, when the user wears the HMD 100, the CPU 105 determines to execute (start) the calibration. The wearing of the HMD 100 by the user is detected using, for example, a sensor (not shown) provided in the HMD 100. The CPU 105 may also determine to execute the calibration when the user instructs the execution of the calibration. The user may instruct the execution of the calibration using, for example, the controller 200. The CPU 105 may also determine to execute the calibration when the information on the user's line of sight is obtained and the position on which the user is focusing changes by a change amount greater than a predetermined change amount. The information on the line of sight is obtained using, for example, a sensor (not shown) provided in the HMD 100. The information on the line of sight may be angle information indicating the direction of the line of sight or coordinate information indicating the position that the user is looking at. The position on which the user is focusing is, for example, the position where the user's line of sight stays longer than a predetermined time. The CPU 105 may also determine to execute the calibration when the duration in a state where no user operation is performed is counted and the duration reaches a predetermined time (when a predetermined time has elapsed in a state where no user operation is performed). Here, the user operation may be interpreted as a user operation on the controller 200 or as a touch operation on the touch operation member 201.

[0032] When the CPU 105 does not execute the calibration, it may use the results of the calibration (calibration results) executed in the past. The CPU 105 may obtain the user's information and store the calibration results in association with the user in the memory unit 106. The user's information is obtained using, for example, known personal authentication processing. Then, the CPU 105 may read out and use the calibration results associated with the current user from the memory unit 106. The CPU 105 may also determine not to execute the calibration when the calibration results associated with the current user are stored in the memory unit 106.

[0033] In step S103, the CPU 105 performs calibration. FIG. 3(B) is a flowchart of the calibration.

[0034] In step S201, the CPU 105 causes the user to specify a reference display position. For example, the CPU 105 causes the display units 102a and 102b to display guidance such as "Please specify a point in the video."

[0035] The user specifies the reference display position using, for example, an operating member different from the touch operation member 201 (an operating member capable of receiving a user operation different from a touch operation, such as a lever, a direction key, or a button). The CPU 105 may acquire information on the user's line of sight and determine the position at which the user is looking as the reference display position. The CPU 105 may acquire information on the posture of the controller 200 and determine the position toward which the controller 200 (a predetermined surface of the controller 200) is directed as the reference display position. Information on the posture of the controller 200 is acquired using, for example, a sensor (such as a gyro sensor) (not shown) provided in the controller 200 and is transmitted from the controller 200 to the HMD 100.

[0036] In step S202, the CPU 105 acquires information (coordinate information) on the reference display position specified in step S201 and stores it in the memory unit 108.

[0037] Note that the reference display position may be a predetermined position, and information on the reference display position may be stored in the memory unit 106 in advance. In that case, the CPU 105 may omit steps S201 and S202. The predetermined reference display position may be a position predetermined by the manufacturer or a position specified by the user in the past. The CPU 105 may determine whether to omit steps S201 and S202 according to whether information on the reference display position is stored in the memory unit 106.

[0038] In step S203, the CPU 105 makes the reference display position identifiable based on the information of the reference display position. For example, the CPU 105 causes the display units 102a and 102b to display an item indicating the reference display position.

[0039] In step S204, the CPU 105 causes the user to perform a touch (a touch on the touch operation member 201 (touch operation surface)) while being aware of the reference display position. For example, the CPU 105 causes the display units 102a and 102b to display guidance such as "Please touch the touch operation surface so as to specify the first item."

[0040] FIG. 4 is a schematic diagram showing an image 400 displayed by the HMD 100 (display units 102a and 102b). The image 400 includes video content 401 and a graphic 402 simulating the touch operation member 201 (touch operation surface). In the present embodiment, it is assumed that the touch operation member 201 and the display unit 202 are integrally provided like a touch panel. The graphic 402 shows the image displayed on the display unit 202. In FIG. 4, an item 403 is displayed at the reference display position. In step S204, the user touches the touch operation member 201 while looking at the item 403.

[0041] In step S205, the CPU 105 acquires the information (coordinate information) of the position touched in step S204 from the controller 200 and stores it in the memory unit 108 as the information of the reference touch position.

[0042] In step S206, the CPU 105 acquires, as the information of the relative display position, the information (coordinate information) of a position having a predetermined positional relationship with respect to the reference display position based on the information of the reference display position. Then, the CPU 105 stores the information of the relative display position in the memory unit 108.

[0043] Note that, similar to the reference display position, the relative display position may be a predetermined position, or information on the relative display position may be stored in advance in the memory unit 106. In that case, the CPU 105 may omit step S206. The CPU 105 may determine whether to omit step S206 according to whether information on the relative display position is stored in the memory unit 106.

[0044] In step S207, the CPU 105 makes the relative display position identifiable based on the information on the relative display position. For example, the CPU 105 causes the display units 102a and 102b to display an item indicating the relative display position.

[0045] In step S208, the CPU 105 causes the user to perform a touch (a touch on the touch operation member 201 (touch operation surface)) while being aware of the relative display position. For example, the CPU 105 causes the display units 102a and 102b to display guidance such as "Please touch the touch operation surface so as to specify the second item."

[0046] FIG. 5 shows an image 500 displayed by the HMD 100 (display units 102a and 102b). Similar to FIG. 4, the image 500 includes video content 501 and a graphic 502 (a graphic indicating the image displayed on the display unit 202) simulating the touch operation member 201 (touch operation surface). In FIG. 5, an item 504 is displayed at the relative display position. In step S208, while looking at the item 504, the user touches the touch operation member 201.

[0047] Note that when making the relative display position identifiable, the CPU 105 may stop the process of making the reference display position identifiable, or may continue to make the reference display position identifiable. By continuing to make the reference display position identifiable, the user can easily be aware of the positional relationship between the reference display position and the relative display position and perform a touch. In FIG. 5, an item 503 is displayed at the reference display position. In FIG. 5, the relative display position is set below the reference display position, but the positional relationship between the reference display position and the relative display position is not particularly limited.

[0048] Also, the aspect of item 503 indicating the reference display position (for example, color, luminance, shape, size, or a combination thereof) and the aspect of item 504 indicating the relative display position may be the same or different. By the aspects of item 503 and item 504 being different from each other, it is possible to prevent the user from misunderstanding the position of item 503 as the relative display position or misunderstanding the position of item 504 as the reference display position.

[0049] In step S209, the CPU 105 acquires the information (coordinate information) of the position touched in step S208 from the controller 200 and stores it in the memory unit 108 as the information of the relative touch position.

[0050] In step S210, the CPU 105 calculates the distance from the reference display position to the relative display position as the display distance based on the information of the reference display position and the information of the relative display position. Then, the CPU 105 stores the information (distance information) of the display distance in the memory unit 108.

[0051] In step S211, the CPU 105 calculates the distance from the reference touch position to the relative touch position as the touch distance based on the information of the reference touch position and the information of the relative touch position. Then, the CPU 105 stores the information (distance information) of the touch distance in the memory unit 108.

[0052] In step S212, the CPU 105 calculates the ratio of the display distance calculated in step S210 to the touch distance calculated in step S211 and stores the ratio (ratio information) in the memory unit 108. It is also possible to calculate the ratio of the touch distance calculated in step S211 to the display distance calculated in step S210.

[0053] FIG. 6 is a schematic diagram showing a video 600 displayed by the HMD 100 (display units 102a, 102b) and a controller 200. FIG. 6 shows a reference display position 601, a relative display position 602, a reference touch position 603, and a relative touch position 604. The distance from the reference display position 601 to the relative display position 602 is a distance L1, and the distance from the reference touch position 603 to the relative touch position 604 is a distance L2. In step S212, the CPU 105 calculates the ratio L1 / L2.

[0054] Note that step S212 may not be included in the calibration and may be included in the designated position determination process described later.

[0055] Also, since the ease of moving a finger varies depending on the direction, the user may be made to perform a plurality of touches while being aware of a plurality of relative display positions respectively, and information on a plurality of relative touch positions may be acquired. Then, a plurality of display distances respectively corresponding to the plurality of relative display positions, a plurality of touch distances respectively corresponding to the plurality of relative touch positions, and a plurality of ratios respectively corresponding to a plurality of combinations of the relative display positions and the relative touch positions may be calculated.

[0056] Returning to the description of FIG. 3(A). In step S104, the CPU 105 determines whether the touch operation member 201 (touch operation surface) is being touched. As described above, when the user touches the touch operation member 201, the touch position is notified from the controller 200 to the HM D100. The CPU 105 can determine whether the touch operation member 201 is being touched by determining whether the touch position has been notified. If it is determined that the touch operation member 201 is being touched, the process proceeds to step S105; otherwise, the process proceeds to step S107.

[0057] In step S105, the CPU 105 performs a designated position determination process for determining the display position designated by the user based on the calibration result. FIG. 3(C) is a flowchart of the designated position determination process.

[0058] In step S301, the CPU 105 determines the direction (touch direction) from the touch reference position to the current touch position based on the information on the touch reference position obtained by calibration and the information on the current display position obtained in step S104. For example, the CPU 105 determines a coordinate system with the touch reference position as the origin as the touch coordinate system, and acquires the information expressed in the touch coordinate system as the information on the touch direction.

[0059] In step S302, the CPU 105 determines the distance (touch distance) from the touch reference position to the current touch position based on the information on the touch reference position obtained by calibration and the information on the current display position obtained in step S104.

[0060] In step S303, the CPU 105 adjusts the touch distance obtained in step S302 at a magnification based on the ratio obtained by calibration. Thereby, the touch distance (the distance on the touch operation surface) is converted into the display distance (the distance on the video displayed by the HMD 100). In the present embodiment, one ratio is obtained by calibration, and in step S303, the touch distance obtained in step S302 is adjusted at a magnification equal to the ratio (the ratio obtained by calibration is multiplied by the touch distance obtained in step S302). Note that the ratio and the magnification may be different. For example, when a plurality of ratios are obtained by calibration, the magnification may be determined by synthesizing the plurality of ratios.

[0061] In step S303, the CPU 105 determines the display position specified by the user based on the information of the display reference position obtained by calibration, the touch direction determined in step S301, and the display distance (adjusted distance) obtained in step S303. The CPU 105 determines that a position separated by the display distance obtained in step S303 in the direction (display direction) based on the touch direction determined in step S301 is specified from the reference display position. For example, the CPU 105 determines a coordinate system with the display reference position as the origin as the display coordinate system. The CPU 105 converts the touch direction (direction on the touch operation surface) determined in step S301 into the display direction (direction on the video displayed by the HMD 100) based on the correspondence relationship between the display coordinate system and the touch coordinate system. Then, the CPU 105 determines that a position separated by the display distance obtained in step S303 in the display direction from the reference display position is specified.

[0062] In the present embodiment, an axis parallel to the horizontal direction (lateral direction, left - right direction) of the touch operation surface is defined as the X - axis (horizontal axis) of the touch coordinate system, and an axis parallel to the vertical direction (perpendicular direction, up - down direction) of the touch operation surface is defined as the Y - axis (vertical axis) of the touch coordinate system. Then, an axis parallel to the horizontal direction of the video displayed by the HMD 100 is defined as the X - axis of the display coordinate system, and an axis parallel to the vertical direction of the video is defined as the Y - axis of the display coordinate system.

[0063] Note that the touch coordinate system and the display coordinate system are not limited to the coordinate systems described above. For example, an axis parallel to the straight line passing through the reference touch position and the relative touch position may be defined as the X - axis of the touch coordinate system, and an axis perpendicular to the X - axis may be defined as the Y - axis of the touch coordinate system. Similarly, an axis parallel to the straight line passing through the reference display position and the relative display position may be defined as the X - axis of the display coordinate system, and an axis perpendicular to the X - axis may be defined as the Y - axis of the display coordinate system.

[0064] As described above, there may be a plurality of relative display positions and a plurality of relative touch positions corresponding to the plurality of relative display positions respectively. In that case, an axis parallel to the straight line passing through the reference touch position and the first relative touch position may be defined as the X-axis of the touch coordinate system, and an axis parallel to the straight line passing through the reference touch position and the second relative touch position may be defined as the Y-axis of the touch coordinate system. Similarly, an axis parallel to the straight line passing through the reference display position and the first relative display position may be defined as the X-axis of the display coordinate system, and an axis parallel to the straight line passing through the reference display position and the second relative display position may be defined as the Y-axis of the display coordinate system.

[0065] The direction from the reference touch position to the first relative touch position may be defined as the positive direction of the X-axis of the touch coordinate system, and the direction from the reference touch position to the second relative touch position may be defined as the negative direction of the X-axis of the touch coordinate system. And the direction from the reference touch position to the third relative touch position may be defined as the positive direction of the Y-axis of the touch coordinate system, and the direction from the reference touch position to the fourth relative touch position may be defined as the negative direction of the Y-axis of the touch coordinate system. Similarly, the direction from the reference display position to the first relative display position may be defined as the positive direction of the X-axis of the display coordinate system, and the direction from the reference display position to the second relative display position may be defined as the negative direction of the X-axis of the display coordinate system. And the direction from the reference display position to the third relative display position may be defined as the positive direction of the Y-axis of the display coordinate system, and the direction from the reference display position to the fourth relative display position may be defined as the negative direction of the Y-axis of the display coordinate system.

[0066] Return to the description of FIG. 3(A). In step S106, the CPU 105 performs control according to the determination result (the display position specified by the user) obtained in step S105.

[0067] In step S107, the CPU 105 determines whether to stop the HMD 100 (turn off the power of the HMD 100). For example, when the user instructs to stop the HMD 100 or the HMD 100 is removed from the user's head, the CPU 105 determines to stop the HMD 100. If it is determined to stop the HMD 100, the operation of FIG. 3(A) is terminated and the HMD 100 is stopped. Otherwise, the process proceeds to step S102.

[0068] As described above, according to the present embodiment, by performing calibration that causes the user to perform a touch while being aware of the display position for each of a plurality of display positions with a simple configuration that does not use a camera or the like, a highly operable user operation becomes possible. After calibration, a user operation is possible in which there is no restriction on the user's movement and a desired instruction can be easily given.

[0069] According to the present embodiment, by causing the user to perform a touch while being aware of the display position for each of a plurality of display positions, the display position intended by the user can be associated with the touch position. Therefore, the user can easily grasp and touch a desired location (a location corresponding to the desired display position) on the touch operation surface without looking at the touch operation surface. That is, the above-described first problem can be solved. In addition, the user can easily grasp the correspondence between the distance on the video and the distance on the touch operation surface. That is, the above-described second problem can also be solved.

[0070] Note that the above-described embodiments (including modified examples) are merely examples, and configurations obtained by appropriately modifying or changing the configurations of the above-described embodiments within the scope of the gist of the present invention are also included in the present invention. Configurations obtained by appropriately combining the configurations of the above-described embodiments are also included in the present invention.

[0071] (Other Embodiments) The present invention provides a program for realizing one or more functions of the above-described embodiments via a network or It can also be realized by a process of supplying to a system or device via a storage medium and having one or more processors in a computer of the system or device read and execute a program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0072] The disclosure of this embodiment includes the following configurations, methods, and programs. (Configuration 1) It has control means for controlling a head-mounted display device according to a touch operation on a touch operation surface, In the calibration of the touch operation, the control means controls the head-mounted display device so as to be able to identify a first display position that is a position in an image displayed by the head-mounted display device, causes the user to touch the touch operation surface while being conscious of the first display position, and acquires information on a first touch position that is the position where the touch is made, controls the head-mounted display device so as to be able to identify a second display position that has a predetermined positional relationship with respect to the first display position, causes the user to touch the touch operation surface while being conscious of the second display position, and acquires information on a second touch position that is the position where the touch is made An information processing apparatus characterized by the above. (Configuration 2) In the calibration, the control means further calculates a ratio of the distance from the first display position to the second display position to the distance from the first touch position to the second touch position. The information processing apparatus according to Configuration 1, characterized by the above. (Configuration 3) After the calibration, when a touch on the touch operation surface is performed, the control means determines a direction from the first touch position toward a third touch position that is the position where the touch is made, determines a distance from the first touch position to the third touch position, Adjust the determined distance by a magnification based on the ratio of the distance from the first display position to the second display position to the distance from the first touch position to the second touch position, Determine that a position is specified that is separated from the first display position by the adjusted distance in the direction based on the determined direction from the first display position, The information processing apparatus according to Configuration 1 or 2, characterized in that. (Configuration 4) After the calibration, when a touch is performed on the touch operation surface, the control means Determine the direction from the first touch position to the third touch position, Determine the distance from the first touch position to the third touch position, Adjust the determined distance by a magnification equal to the ratio, Determine that a position is specified that is separated from the first display position by the adjusted distance in the direction based on the determined direction, The information processing apparatus according to Configuration 3, characterized in that. (Configuration 5) In the calibration, the control means further causes the user to specify the first display position, The information processing apparatus according to any one of Configurations 1 to 4, characterized in that. (Configuration 6) The control means acquires information on the user's line of sight and determines the position that the user is looking at as the first display position specified by the user, The information processing apparatus according to Configuration 5, characterized in that. (Configuration 7) The control means acquires information on the posture of the controller having the touch operation surface and determines the position that the controller is directed at as the first display position specified by the user, The information processing apparatus according to Configuration 5, characterized in that. (Configuration 8) The first display position is specified using an operation member that can receive a user operation different from the touch operation.​ The information processing apparatus according to configuration 5, characterized by the above. (Configuration 9) The first display position is a predetermined position. The information processing apparatus according to any one of configurations 1 to 4, characterized by the above. (Configuration 10) The control means executes the calibration in response to the user wearing the head-mounted display. The information processing apparatus according to any one of configurations 1 to 9, characterized by the above. (Configuration 11) The control means executes the calibration in response to the user instructing the execution of the calibration. The information processing apparatus according to any one of configurations 1 to 9, characterized by the above. (Configuration 12) The control means executes the calibration in response to a predetermined time elapsing in a state where no user operation is being performed. The information processing apparatus according to any one of configurations 1 to 9, characterized by the above. (Configuration 13) The control means acquires information on the user's line of sight and executes the calibration in response to the position at which the user is looking changing by a change amount greater than a predetermined change amount. The information processing apparatus according to any one of configurations 1 to 9, characterized by the above. (Configuration 14) The control means controls the head-mounted display so as to continue to make the first display position distinguishable when making the second display position distinguishable. The information processing apparatus according to any one of configurations 1 to 13, characterized by the above. (Configuration 15) The control means makes the first display position distinguishable by displaying a first item at the first display position, and makes the second display position distinguishable by displaying a second item at the second display position. The information processing apparatus according to any one of configurations 1 to 14, characterized by the above. (Method) A control method for an information processing apparatus that controls a head-mounted display device according to a touch operation on a touch operation surface, comprising a step of performing calibration of the touch operation, The step of performing the calibration is a step of controlling the head-mounted display device so as to be able to identify a first display position which is a position in an image displayed by the head-mounted display device; a step of causing the user to touch the touch operation surface while being conscious of the first display position, and acquiring information on a first touch position which is the position where the touch is performed; a step of controlling the head-mounted display device so as to be able to identify a second display position having a predetermined positional relationship with respect to the first display position; and a step of causing the user to touch the touch operation surface while being conscious of the second display position, and acquiring information on a second touch position which is the position where the touch is performed, and including thereby being characterized as a control method. (Program) (Program) A program for causing a computer to function as each means of the information processing apparatus according to any one of Configurations 1 to 15.

Description of Signs

[0073] 100: Head-Mounted Display (HMD) 105: CPU

Claims

1. It has control means for controlling a head-mounted display device in accordance with a touch operation on a touch operation surface, In the calibration of the touch operation, the control means, Controls the head-mounted display device so as to be able to identify a first display position which is a position in an image displayed by the head-mounted display device, Causes the user to perform a touch on the touch operation surface being aware of the first display position, and acquires information on a first touch position which is the position where the touch is performed, Controls the head-mounted display device so as to be able to identify a second display position which has a predetermined positional relationship with respect to the first display position, Causes the user to perform a touch on the touch operation surface being aware of the second display position, and acquires information on a second touch position which is the position where the touch is performed An information processing apparatus characterized by the above.

2. In the calibration, the control means further calculates a ratio of a distance from the first display position to the second display position with respect to a distance from the first touch position to the second touch position The information processing apparatus according to claim 1, characterized by the above.

3. After the calibration, when a touch on the touch operation surface is performed, the control means, Determines a direction from the first touch position toward a third touch position which is the position where the touch is performed, Determines a distance from the first touch position to the third touch position, Adjusts the determined distance by a magnification based on a ratio of a distance from the first display position to the second display position with respect to a distance from the first touch position to the second touch position, Determines that a position separated by the adjusted distance in the direction based on the determined direction from the first display position is designated The information processing apparatus according to claim 1, characterized by the above.

4. After the calibration, when a touch on the touch operation surface is performed, the control means, Determines a direction from the first touch position toward the third touch position, Determines a distance from the first touch position to the third touch position, Adjusts the determined distance by a magnification equal to the ratio, Determines that a position separated by the adjusted distance in the direction based on the determined direction from the first display position is designated The information processing apparatus according to claim 3, characterized by the above.

5. In the calibration, the control means further causes the user to specify the first display position. The information processing apparatus according to claim 1, characterized in that.

6. The control means acquires information on the user's line of sight and determines the position on which the user is focusing as the first display position specified by the user. The information processing apparatus according to claim 5, characterized in that.

7. The control means acquires information on the posture of the controller having the touch operation surface, and determines the position toward which the controller is directed as the first display position specified by the user. The information processing apparatus according to claim 5, characterized in that.

8. The first display position is specified using an operation member capable of receiving a user operation different from the touch operation. The information processing apparatus according to claim 5, characterized in that.

9. The first display position is a predetermined position. The information processing apparatus according to claim 1, characterized in that.

10. The control means executes the calibration in response to the user wearing the head-mounted display device. The information processing apparatus according to claim 1, characterized in that.

11. The control means executes the calibration in response to the user instructing the execution of the calibration. The information processing apparatus according to claim 1, characterized in that.

12. The control means executes the calibration in response to a predetermined time elapsing in a state where no user operation is being performed. The information processing apparatus according to claim 1, characterized in that.

13. The control means acquires information on the user's line of sight and executes the calibration in response to the position on which the user is focusing changing by a change amount greater than a predetermined change amount. The information processing apparatus according to claim 1, characterized in that.

14. The control means controls the head-mounted display device so as to continue to make the first display position identifiable when making the second display position identifiable. The information processing apparatus according to claim 1, characterized in that.

15. The control means makes the first display position identifiable by displaying a first item at the first display position, and makes the second display position identifiable by displaying a second item at the second display position. The information processing apparatus according to claim 1, characterized in that...

16. A control method for an information processing apparatus that controls a head-mounted display device according to a touch operation on a touch operation surface, comprising a step of performing calibration of the touch operation, wherein the step of performing the calibration includes a step of controlling the head-mounted display device so as to make it possible to identify a first display position that is a position in an image displayed by the head-mounted display device, a step of causing the user to touch the touch operation surface while being conscious of the first display position, and acquiring information on a first touch position that is the position where the touch is made, a step of controlling the head-mounted display device so as to make it possible to identify a second display position that has a predetermined positional relationship with respect to the first display position, and a step of causing the user to touch the touch operation surface while being conscious of the second display position, and acquiring information on a second touch position that is the position where the touch is made. The control method is characterized by including the above steps.

17. A program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 15. ​

Images