Program and image display system

The system addresses the challenge of understanding positional relationships in virtual spaces by using a head-mounted display to control a virtual camera, displaying stereoscopic images and adjusting display modes based on head movements for intuitive interaction with virtual objects.

JP7580834B2Active Publication Date: 2024-11-12KONAMI DIGITAL ENTERTAINMENT CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2023190851
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2023-11-08
Publication Date
2024-11-12
Estimated Expiration
2038-01-22

AI Technical Summary

Technical Problem

Users have difficulty grasping the positional relationship between objects in a virtual space and reference lines when using a head-mounted display, particularly in interactions with characters.

Method used

A system that captures images in a virtual space using a virtual camera controlled by a head-mounted display, displaying stereoscopic images with binocular parallax and adjusting the display mode of an instruction image based on the positional relationship between a reference line and objects, allowing intuitive interaction through head movements.

Benefits of technology

Enhances user understanding of positional relationships and facilitates intuitive interaction with virtual space objects by visually indicating contact points and changing display modes based on head movements, providing a more immersive experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007580834000001
    Figure 0007580834000001
  • Figure 0007580834000002
    Figure 0007580834000002
  • Figure 0007580834000003
    Figure 0007580834000003
Patent Text Reader

Abstract

To allow a user to easily grasp a positional relationship between an object and a reference line in a virtual space.SOLUTION: A program disclosed herein causes a computer to function as: a display control unit 42B configured to display, on a head-mounted display, an image of a virtual space captured by a virtual camera whose orientation is controlled in accordance with an orientation of the head-mounted display; and an operation control unit 43B configured to cause an object in the virtual space to make motions according to a position where a reference line whose orientation in the virtual space changes according to the orientation of the head-mounted display intersects with the object.SELECTED DRAWING: Figure 11
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a technique for displaying an image captured in a virtual space. [Background technology]

[0002] A technique for displaying an image captured by a virtual camera (hereinafter referred to as a "virtual camera") installed in a virtual space on a display device has been proposed. For example, Patent Document 1 discloses a configuration for displaying a menu image in a virtual space according to the inclination of a head mounted display. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-115122 A Summary of the Invention [Problem to be solved by the invention]

[0004] Contents such as games that allow communication between objects such as characters in a virtual space have been proposed. When this type of content is displayed using a head-mounted display, there is a problem that it is difficult for a user to grasp the positional relationship between a virtual straight line (hereinafter referred to as a "reference line") such as the optical axis of a virtual camera or the user's line of sight, and an object such as a character in the virtual space. In consideration of the above circumstances, the present invention aims to make it easy for a user to grasp the positional relationship between an object in a virtual space and a reference line. [Means for solving the problem]

[0005] In order to solve the above problems, a program according to a preferred embodiment of the present invention is a program that causes a computer to function as a display control unit that displays, on a display device of the head-mounted display, a stereoscopic image utilizing binocular parallax, which is an image captured of a virtual space by a virtual camera whose direction is controlled according to the direction of a head-mounted display, and the display control unit displays an instruction image in the direction of a reference line whose direction in the virtual space changes according to the direction of the head-mounted display, and changes the display mode of the instruction image according to the positional relationship between the reference line and an object in the virtual space.

[0006] An image display system according to a preferred embodiment of the present invention is an image display system comprising a head-mounted display and an information processing device, wherein the information processing device comprises a display control unit that causes a display device of the head-mounted display to display a stereoscopic image utilizing binocular parallax, the stereoscopic image being an image captured of a virtual space by a virtual camera whose direction is controlled according to the direction of the head-mounted display, and the display control unit causes an indication image to be displayed in the direction of a reference line whose direction in the virtual space changes according to the direction of the head-mounted display, and changes the display mode of the indication image according to the positional relationship between the reference line and an object in the virtual space. [Brief description of the drawings]

[0007] [Figure 1] 1 is a perspective view illustrating an example of the appearance of an image display system according to a first embodiment of the invention. [Diagram 2] FIG. 1 is a block diagram illustrating a configuration of an image display system. [Diagram 3] FIG. 2 is a block diagram illustrating an example of a functional configuration of the image display system. [Figure 4] FIG. 1 is an explanatory diagram of a virtual space. [Diagram 5] FIG. 2 is an explanatory diagram of an image displayed by a display device. [Figure 6] 4 is a flowchart illustrating the contents of a process executed by a control device. [Figure 7]FIG. 13 is an explanatory diagram of the rotation of the HMD in the roll direction. [Figure 8] FIG. 2 is an explanatory diagram of an image displayed by a display device. [Figure 9] FIG. 13 is an explanatory diagram of the movement of the virtual camera in modification example A4. [Figure 10] FIG. 13 is an explanatory diagram of a change in the imaging range in modification A5. [Figure 11] FIG. 11 is a block diagram illustrating an example of a functional configuration of an image display system in a second embodiment. [Figure 12] FIG. 11 is an explanatory diagram of an image displayed when the reference line does not intersect with the character in the second embodiment. [Figure 13] FIG. 11 is an explanatory diagram of an image displayed when a reference line intersects with a character in the second embodiment. [Figure 14] FIG. 11 is an explanatory diagram of the relationship between a character and an instruction image in the second embodiment. [Figure 15] 10 is a flowchart illustrating the contents of a process executed by a control device of a second embodiment. [Figure 16] FIG. 13 is an explanatory diagram of a change over time in an instruction image in modification B2. [Figure 17] FIG. 13 is an explanatory diagram of an instruction image in modification example B3. [Figure 18] FIG. 13 is an explanatory diagram of an instruction image in modification example B4. [Figure 19] FIG. 13 is a perspective view illustrating an example of the appearance of an image display system in Modification C5. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] [A: First embodiment] FIG. 1 is a perspective view illustrating an example of the appearance of an image display system 1A according to a first embodiment of the present invention. The image display system 1A is a video device for displaying images, and includes a terminal device 10 and a wearing tool 20. The terminal device 10 is a portable information terminal such as a smartphone or a tablet terminal. The terminal device 10 is detachably installed in the wearing tool 20. The wearing tool 20 is a tool for wearing the terminal device 10 on the head of a user U. For example, as illustrated in FIG. 1, a goggle-type attachment having a belt that is wrapped around the head of the user U is suitable as the wearing tool 20.

[0009] Fig. 2 is a block diagram illustrating the configuration of the terminal device 10. As illustrated in Fig. 2, the terminal device 10 is a computer system including a control device 11, a storage device 12, a display device 13, and a detection device 14. The control device 11 is a processor such as a CPU (Central Processing Unit) and generally controls each element of the image display system 1A. The storage device 12 stores programs executed by the control device 11 and various data used by the control device 11. For example, the storage device 12 is configured by a known recording medium such as a magnetic recording medium or a semiconductor recording medium, or a combination of multiple types of recording media.

[0010] The display device 13 displays an image under the control of the control device 11. For example, a flat display such as a liquid crystal display panel or an organic EL (Electroluminescence) display panel is used as the display device 13. As can be understood from FIG. 1, when the wearing device 20 is worn on the head of the user U, the display device 13 is disposed in front of both eyes of the user U with the display surface facing the head of the user U. The display device 13 of the first embodiment displays a stereoscopic image that allows the user U to perceive a stereoscopic effect. The stereoscopic image is composed of a right-eye image and a left-eye image to which binocular parallax is imparted. The user U perceives a stereoscopic effect by having the right eye of the user U view the right eye image and the left eye image of the user U view the left eye of the user U.

[0011] 2, the display device 13, the detection device 14, and the wearing equipment 20 constitute a head mounted display (hereinafter referred to as "HMD") 30, and the control device 11 and the storage device 12 constitute an information processing device 40 that displays an image on the HMD 30. That is, the image display system 1A includes the HMD 30 and the information processing device 40. As described above, in the first embodiment, the information processing device 40, the display device 13, and the detection device 14 are realized by a single terminal device 10. Note that the information processing device 40 may be understood as an element of the HMD 30.

[0012] The detection device 14 in FIG. 2 is a sensor that outputs a detection signal according to the attitude of the HMD 30. Specifically, the detection device 14 is composed of one or more types of sensors selected from a plurality of types of sensors, such as a gyro sensor that detects angular velocity, an acceleration sensor that detects acceleration, an inclination sensor that detects inclination angle, and a geomagnetic sensor that detects direction by geomagnetism. By analyzing the detection signal output from the detection device 14, it is possible to identify the attitude or change in attitude of the HMD 30. In other words, the detection device 14 is a sensor that outputs a detection signal according to the attitude of the display device 13 or the terminal device 10. Note that an amplifier that amplifies the detection signal and an A / D converter that converts the detection signal from analog to digital are omitted from the illustration for convenience.

[0013] As illustrated in FIG. 1, the posture of the HMD 30 is determined by three mutually orthogonal axes (X-axis, Y-axis, and Z-axis) at the origin O. The X-axis is an axis perpendicular to the display surface of the display device 13 and corresponds to the front-rear direction of the user U. The Y-axis and Z-axis are axes parallel to the display surface of the display device 13. The Y-axis corresponds to the left-right direction of the user U, and the Z-axis corresponds to the up-down direction of the user U. Focusing on the display surface of the display device 13, the X-axis corresponds to the depth direction of the display surface, the Y-axis corresponds to the horizontal direction of the display surface, and the Z-direction corresponds to the vertical direction of the display surface. The direction of the circumference around the X-axis is the roll direction, the direction of the circumference around the Y-axis is the pitch direction, and the direction of the circumference around the Z-axis is the yaw direction. The direction of the X-axis is determined by the angle in the pitch direction and the angle in the yaw direction. The rotation in the roll direction corresponds to the action of tilting the head left and right while the user U faces forward. That is, for example, when the user U tilts his / her head, the HMD 30 rotates in the roll direction. In the following description, one side in the roll direction is referred to as the "first side" and the other side is referred to as the "second side" as shown in Fig. 1. Either the clockwise or counterclockwise direction around the X-axis is the first side, and the other is the second side.

[0014] The control device 11 in Fig. 2 executes a program stored in the storage device 12 to function as a posture analysis unit 41, a display control unit 42A, and a motion control unit 43A, as illustrated in Fig. 3. Note that some or all of the functions of the control device 11 may be realized by a dedicated electronic circuit.

[0015] The posture analysis unit 41 identifies the posture of the HMD 30 by analyzing the detection signal output by the detection device 14. Specifically, the posture analysis unit 41 sequentially generates posture data related to the posture of the HMD 30 at a predetermined cycle. The posture data is data that represents the posture of the HMD 30 or a change in posture.

[0016] The display control unit 42A causes the display device 13 to display an image captured by a virtual camera E in the virtual space V. FIG. 4 is an explanatory diagram of the virtual space V. As illustrated in FIG. 4, the virtual camera E is installed in the virtual space V and captures an image of a specific range (hereinafter referred to as an "imaging range") R in the virtual space V. The imaging range R is a range that spans a predetermined angle in the vertical and horizontal directions with respect to the optical axis of the virtual camera E. The optical axis of the virtual camera E corresponds to the virtual line of sight of the user U in the virtual space V. The display control unit 42A of the first embodiment causes the display device 13 to display a stereoscopic image (an image for the right eye and an image for the left eye) using binocular parallax. Therefore, the virtual camera E includes a first virtual camera that captures an image for the left eye and a second virtual camera that captures an image for the right eye. Image data representing the stereoscopic image captured by the virtual camera E is sequentially supplied from the display control unit 42A to the display device 13, whereby the stereoscopic image is displayed on the display device 13.

[0017] The display control unit 42A controls the direction of the virtual camera E in the virtual space V according to the direction of the HMD 30 identified by the posture analysis unit 41. Specifically, the direction of the optical axis of the virtual camera E (and further the imaging range R) changes in conjunction with the rotation of the X axis around the origin O. For example, when the HMD 30 rotates in the yaw direction, the optical axis of the virtual camera E rotates left and right around the origin O, and when the HMD 30 rotates in the pitch direction, the optical axis of the virtual camera E rotates up and down around the origin O. In the first embodiment, even when the HMD 30 rotates in the roll direction, the posture of the virtual camera E (angle around the optical axis) does not change. However, the inclination of the virtual camera E around the optical axis may be changed according to the rotation of the HMD 30 in the roll direction.

[0018] 4, a character C is placed in a virtual space V in which a virtual camera E is installed. The character C in the first embodiment is an object representing a virtual creature (e.g., a human, an animal, or a monster) active in the virtual space V, and is configured to include a head and a torso. A game in which the user U interacts with the character C by making contact (skinship) with the character C at any time in the virtual space V is provided to the user U by the image display system 1A.

[0019] A reference line Q is set in the virtual space V. The reference line Q is a straight line whose direction in the virtual space V changes depending on the direction of the HMD 30 (the direction of the X-axis). In the first embodiment, as illustrated in FIG. 4, the optical axis of the virtual camera E is exemplified as the reference line Q. For example, if the user U turns his head to the right, the reference line Q rotates to the right, and if the user U turns his head to the left, the reference line Q rotates to the left. As can be understood from the above explanation, the reference line Q is the virtual line of sight of the user U in the virtual space V.

[0020] 5 is a schematic diagram of an image displayed on the display device 13. As illustrated in FIG. 5, the display control unit 42A causes the display device 13 to display a stereoscopic image captured by the virtual camera E in the virtual space V. When a character C is located within the imaging range R, the character C is displayed on the display device 13. The display control unit 42A also causes the display device 13 to display an image (hereinafter referred to as an "indication image") G indicating the direction of a reference line Q. The indication image G is disposed at a predetermined position on the reference line Q in the virtual space V. Since the reference line Q in the first embodiment is the optical axis of the virtual camera E, the indication image G is displayed at a predetermined position on the display surface of the display device 13.

[0021] The movement control unit 43A in Fig. 3 controls the movement of the character C in the virtual space V. The movement control unit 43A controls the movement of the character C in accordance with the progress of the game, and also makes the character C execute a movement (hereinafter referred to as a "reaction movement") corresponding to a touch by the user U in the virtual space V. Specifically, the movement control unit 43A determines whether the user U has touched the character C in the virtual space V, and makes the character C execute a reaction movement to the touch by the user U. For example, various movements such as a change in posture, a change in facial expression, or the pronunciation of a specific line are typical examples of the reaction movement.

[0022] The motion control unit 43A determines whether the user U has contacted the character C in the virtual space V according to the change in the posture of the HMD 30 identified by the posture analysis unit 41. The motion control unit 43A of the first embodiment determines that the user U has contacted the character C in the virtual space V when the HMD 30 has rotated in the roll direction. That is, the rotation of the HMD 30 in the roll direction corresponds to an instruction (motion instruction) for the user U to contact the character C in the virtual space V. Specifically, when the HMD 30 has rotated in the roll direction, the motion control unit 43A determines that the user U has contacted a point P where the reference line Q and the surface of the character C intersect (hereinafter referred to as the "contact point") when the HMD 30 has rotated in the roll direction. The contact point P is located on the surface of the character C. Note that when the reference line Q intersects with the surface of the character C at multiple points, the point closest to the virtual camera E (i.e., the user U) among the multiple points is selected as the contact point P.

[0023] As described above, in the first embodiment, the movement of the character C in the virtual space V is controlled in response to the rotation of the HMD 30 in the roll direction. Therefore, it is possible to reflect the instruction from the user U in the movement of the character C without the need to prepare an operation device into which the user U inputs an instruction to control the movement of the character C separately from the HMD 30. In particular, in the first embodiment, it is determined that the user U has come into contact with the character C when the reference line Q and the character C are in an intersecting relationship, so that the user U can operate the character C while keeping the character C within his or her field of vision. In other words, there is an advantage that the user U can operate the character C more intuitively.

[0024] Fig. 6 is a flow chart illustrating the specific contents of the process (hereinafter referred to as "first control process") executed by the movement control unit 43A. The movement control unit 43A repeatedly executes the first control process of Fig. 6 at a predetermined cycle. The storage device 12 stores contact determination data indicating whether or not the user U is in contact with the character C in the virtual space V (hereinafter referred to as "contact state"). The contact determination data is, for example, a flag indicating either a contact state or a non-contact state.

[0025] When the first control process is started, the operation control unit 43A determines whether or not the contact determination data indicates a contact state (Sa1). If the contact determination data indicates a non-contact state (Sa1: NO), the operation control unit 43A determines whether or not the HMD 30 has rotated in the roll direction (Sa2). Specifically, the operation control unit 43A refers to the posture data sequentially supplied from the posture analysis unit 41 to determine whether or not the HMD 30 has rotated to the first side (e.g., clockwise) in the roll direction.

[0026] FIG. 7 is an explanatory diagram of a process for determining whether or not the HMD 30 has rotated in the roll direction. As illustrated in FIG. 7, the operation control unit 43A determines that the HMD 30 has rotated in the roll direction when the angle θ at which the HMD 30 has rotated to the first side in the roll direction exceeds a threshold θt. The angle θ is an angle at which the Z axis or the Y axis has rotated around the X axis, and is defined, for example, with the vertical direction as a reference (θ=0). On the other hand, when the rotation angle θ of the HMD 30 is below the threshold θt, the operation control unit 43A does not determine that the HMD 30 has rotated in the roll direction. As described above, in the first embodiment, the HMD 30 is not determined to have rotated in the roll direction when the rotation is below the threshold θt. Therefore, it is possible to reduce the possibility that the HMD 30 is determined to have rotated excessively frequently. In addition, there is also an advantage that it is possible to reduce the possibility that the HMD 30 is determined to have rotated even though the user U does not intend to rotate the HMD 30 (i.e., the possibility of erroneous operation).

[0027] In addition, not only when rotation in the roll direction alone occurs in the HMD 30, but also when rotation in the roll direction occurs together with rotation in the pitch direction or yaw direction, if the angle of rotation in the roll direction exceeds the threshold value θt, it is determined that the HMD 30 has rotated in the roll direction.

[0028] In the first control process of FIG. 6, when it is determined that the HMD 30 has rotated to the first side in the roll direction (Sa2: YES), the operation control unit 43A determines whether or not the reference line Q intersects with the character C in the virtual space V (Sa3). That is, it is determined whether or not the character C exists on the line of the reference line Q. As illustrated in FIG. 8, when the reference line Q intersects with the character C in the virtual space V (Sa3: YES), the operation control unit 43A determines that the user U is in a contact state in which the user U has contacted the contact point P of the character C (Sa4). Specifically, the contact determination data is changed to a numerical value indicating the contact state. As can be understood from the above description, when the HMD 30 has rotated in the roll direction, the operation control unit 43A determines that the user U has contacted the contact point P of the character C in the virtual space V. That is, the user U can instruct contact with the contact point P of the character C by rotating the HMD 30 in the roll direction.

[0029] When the contact state is entered, the action control unit 43A makes the character C execute a reaction action (Sa5). Specifically, the action control unit 43A makes the character C execute a reaction action according to the position of the contact point P where the reference line Q intersects with the character C. That is, the reaction action executed by the character C changes according to the position of the contact point P. For example, a table that specifies the type of reaction action for each of a plurality of regions into which the surface of the character C is divided is stored in the storage device 12. The action control unit 43A makes the character C execute a reaction action specified in the table for the region including the contact point P among the plurality of regions on the surface of the character C. For example, when the contact point P is located on the head of the character C, the action control unit 43A makes the character C execute a reaction action (e.g., a favorable action such as being happy, laughing, or approaching) that accepts contact by the user U. On the other hand, when the contact point P is located on the torso of the character C, the action control unit 43A makes the character C execute a reaction action (e.g., a negative action such as being angry, sad, or moving away) that rejects contact by the user U. According to the above configuration, it is possible to vary the reaction action of the character C in a variety of ways depending on the position of the contact point P.

[0030] On the other hand, if the HMD 30 does not rotate to the first side in the roll direction (Sa2: NO), or if the reference line Q does not intersect with the character C (Sa3: NO), the operation control unit 43A maintains the contact determination data at a value indicating a non-contact state. That is, even if the HMD 30 rotates to the first side in the roll direction, when the reference line Q does not intersect with the character C as in the example of FIG. 5, it is determined that the user U is not in contact with the character C.

[0031] When the user U contacts the character C in the virtual space V, in step Sa1 of the first control process thereafter, the contact determination data is determined to indicate a contact state. When the contact determination data indicates a contact state (Sa1: YES), the motion control unit 43A determines whether the HMD 30 has rotated to the second side in the roll direction (for example, counterclockwise) by referring to the posture data sequentially supplied from the posture analysis unit 41. Specifically, the motion control unit 43A determines that the HMD 30 has rotated to the second side in the roll direction when the rotation angle θ with respect to the second side in the roll direction exceeds a predetermined threshold value θt. When the HMD 30 has rotated to the second side in the roll direction (Sa6: YES), the motion control unit 43A determines that the contact with the character C has been released (Sa7). Specifically, the motion control unit 43A changes the contact determination data to a numerical value indicating a non-contact state. On the other hand, when the HMD 30 does not rotate to the second side in the roll direction (Sa6: NO), the motion control unit 43A maintains the contact determination data at a numerical value indicating a contact state.

[0032] In addition, when contact with character C is released, it is preferable that the action control unit 43A causes character C to perform an action that responds to the release of contact, but a configuration in which character C is not caused to perform an action that responds to the release of contact is also envisioned.

[0033] As can be understood from the above description, when the HMD30 rotates to the first side in the roll direction (Sa2: YES), the operation control unit 43A determines that the user U has contacted the character C (Sa4). On the other hand, when the HMD30 rotates to the second side in the roll direction (Sa6: YES) in a state in which the user U is in contact with the character C (Sa1: YES), the operation control unit 43A determines that the contact with the character C has been released. With the above configuration, the user U can instruct the generation and release of contact with the character C in the virtual space V according to the direction of rotation of the HMD30 in the roll direction. Since rotations in opposite directions in the roll direction correspond to opposite actions of the generation and release of contact with the character C, there is also an advantage that the user U can easily intuitively grasp the generation and release of contact with the character C.

[0034] In the first embodiment, for example, the following configuration may be adopted.

[0035] [Variation A1] The motion control unit 43A in the modified example A1 causes the character C to execute a reaction action according to the time length during which contact with the character C is maintained in the virtual space V. Specifically, the motion control unit 43A changes the reaction action over time during the period during which the contact state continues (hereinafter referred to as the "contact period"). The contact period is the period from when the HMD 30 rotates to the first side in the roll direction to when it rotates to the second side. The contact period may be the period from when the contact determination data is set to a value indicating a contact state to when it is changed to a value indicating a non-contact state. The motion control unit 43A changes the reaction action of the character C over time from the start point to the end point of the contact period. Therefore, the reaction action of the character C changes according to the time length of the contact period. According to the above configuration, the reaction action of the character C can be varied in a variety of ways according to the time length during which the user U contacts the character C.

[0036] [Variation A2] In the storage device 12 in the modification A2, parameters related to the character C are stored. Specifically, the favorability rating of the character C for the user U is stored as a parameter in the storage device 12. The favorability rating is a parameter indicating the degree of favorable feelings from the character C to the user U. The operation control unit 43A changes the favorability rating according to the progress of the game, and also changes the favorability rating from the character C to the user U according to contact with the character C. Specifically, the greater the degree of contact (e.g., number of times or time) of the user U with the character C, the greater the favorability rating from the character C to the user U is set to. Note that, in the above example, the favorability rating of the character C for the user U has been described, but the favorability rating stored in the storage device 12 may be the favorability rating from the character C to the character (player character) used by the user U. The user U may have multiple player characters in the virtual space V.

[0037] Furthermore, the action control unit 43A changes the reaction action according to the likeability stored for the character C. That is, the type of reaction action that the character C executes (Sa4) when the user U touches the character C (Sa2: YES, Sa3: YES) changes according to the likeability of the character C. For example, when the number of contacts or the time with the character C is insufficient and the likeability is low, the character C executes a reaction action of rejecting the contact when the user U touches the head. On the other hand, when the number of contacts or the time with the character C is sufficiently secured and the likeability is high, the character C executes a reaction action of accepting the contact when the user U touches the head. With the above configuration, the reaction action of the character C can be varied in a variety of ways according to the parameters related to the character C.

[0038] In addition, the parameters that affect the reaction behavior of the character C are not limited to the favorability level exemplified above. For example, various parameters such as the growth level (level) of the character C or the intimacy between the user U and the character C change according to the contact of the character C by the user U, and the reaction behavior of the character C is controlled according to the parameters.

[0039] [Variation A3] The motion control unit 43A in the modification A3 causes the character C to perform a reaction action according to the angle θ at which the HMD 30 rotates in the roll direction. For example, the rotation angle θ of the HMD 30 corresponds to the virtual pressure of the user U contacting the character C in the virtual space V. For example, when the rotation angle θ is small, it means that the user U is lightly contacting the character C, and when the rotation angle θ is large, it means that the user U is strongly pressing the character C. For example, when the rotation angle θ is small, the motion control unit 43A causes the character C to perform a reaction action to accept the contact by the user U, and when the rotation angle θ is large, it causes the character C to perform a reaction action to reject the contact by the user U. According to the above configuration, the reaction action of the character C can be varied in various ways according to the rotation angle θ of the HMD 30 (i.e., the virtual pressure of the user U contacting the character C).

[0040] [Variation A4] The display control unit 42A in modification A4 moves the virtual camera E (i.e., the virtual viewpoint in the virtual space V) in the virtual space V in response to the rotation of the HMD 30 in the roll direction. Specifically, the display control unit 42A changes the positional relationship between the virtual camera E and the character C in the virtual space V in conjunction with the rotation of the HMD 30. For example, the distance between the virtual camera E and the character C is controlled in response to the rotation angle θ of the HMD 30. That is, the virtual camera E moves toward the object in conjunction with the rotation of the HMD 30 in the roll direction.

[0041] For example, when the HMD 30 rotates to the first side in the roll direction, the display control unit 42A moves the virtual camera E closer to the character C in the virtual space V by an amount of movement corresponding to the rotation angle θ (θ>0) of the HMD 30, as shown by an arrow a1 in Fig. 9. On the other hand, when the HMD 30 rotates to the second side in the roll direction, the display control unit 42A moves the virtual camera E away from the character C in the virtual space V by an amount of movement corresponding to the rotation angle θ (θ<0) of the HMD 30, as shown by an arrow a2 in Fig. 9. With the above configuration, the user U can move the virtual camera E in the virtual space V by a simple operation of rotating the HMD 30 in the roll direction.

[0042] [Variation A5] The display control unit 42A in the modified example A5 changes the imaging range R of the virtual camera E (i.e., the angle of view of the virtual camera E) according to the rotation of the HMD 30 in the roll direction. That is, the range displayed on the display device 13 in the virtual space V changes according to the rotation of the HMD 30 in the roll direction. For example, when the HMD 30 rotates to the first side in the roll direction, the display control unit 42A reduces the imaging range R at a ratio according to the rotation angle θ of the HMD 30 as shown by the arrow b1 in FIG. 10. Therefore, for example, the character C in the virtual space V is zoomed in (enlarged). On the other hand, when the HMD 30 rotates to the second side in the roll direction, the display control unit 42A expands the imaging range R at a ratio according to the rotation angle θ of the HMD 30 as shown by the arrow b2 in FIG. Therefore, for example, the character C in the virtual space V is zoomed out (reduced). According to the above configuration, the user U can change the imaging range R of the virtual camera E in the virtual space V by a simple operation of rotating the HMD 30 in the roll direction.

[0043] [Variation A6] In the first embodiment, the contact with the character C is determined to be released when the HMD 30 rotates to the second side in the roll direction (Sa6: YES), but the condition for determining that the contact with the character C is released is not limited to the above example. For example, the operation control unit 43A may determine that the contact with the character C is released when the HMD 30 returns to the original state from the state rotated in the roll direction (Sa2: YES). Specifically, in step Sa6 of FIG. 6, the operation control unit 43A determines that the contact with the character C is released when the rotation angle θ of the HMD 30 changes to an initial angle (e.g., θ=0) at the time when the rotation for contacting the character C is started (Sa7). On the other hand, in step Sa2, the operation control unit 43A determines that the user U has contacted the character C when the rotation angle θ changes, regardless of the direction of rotation of the HMD 30 (first side / second side). As can be understood from the above description, by monitoring the rotation angle θ of the HMD 30, it is possible to determine the occurrence and release of contact with the character C without determining the direction of rotation (first side / second side) of the HMD 30. In other words, in determining contact with the character C, the action control unit 43A does not necessarily need to determine the direction of rotation of the HMD 30.

[0044] [Variation A7] In the first embodiment, when the reference line Q intersects with the character C in the virtual space V (Sa3: YES), the character C is made to perform a reaction action to the contact by the user U, but the condition regarding the positional relationship between the reference line Q and the character C is not limited to the above example. For example, the character C may be made to perform a reaction action on the condition that the distance between the reference line Q and the character C is below a predetermined threshold. When the distance between the reference line Q and the character C is below a predetermined threshold, this includes not only the case where the reference line Q intersects with the character C, but also the case where the reference line Q is separated from the character C within a range below the threshold. Also, the character C may be made to perform a reaction action on the condition that the reference line Q intersects with a specific part of the character C. When the reference line Q intersects with a part other than the specific part of the character C, the movement control unit 43A does not make the character C perform a reaction action. As can be understood from the above description, the movement control unit 43A of the first embodiment is comprehensively expressed as an element that controls the movement of the character C according to the positional relationship between the reference line Q and the character C (for example, the relationship where the reference line Q intersects with the character C).

[0045] [B: Second embodiment] A second embodiment of the present invention will be described. In the following examples, elements having the same functions as those in the first embodiment will be designated by the same reference numerals as those in the first embodiment, and detailed descriptions thereof will be omitted as appropriate.

[0046] The image display system in the second embodiment has the same configuration as that in the first embodiment illustrated in Figures 1 and 2. That is, the HMD 30 in the second embodiment includes the display device 13 and the detection device 14 of the terminal device 10 and the wearing tool 20, and the information processing device 40 includes the control device 11 and the storage device 12 of the terminal device 10.

[0047] Fig. 11 is a block diagram illustrating a functional configuration of the terminal device 10 of the second embodiment. As illustrated in Fig. 11, the control device 11 of the second embodiment functions as a posture analysis unit 41, a display control unit 42B, and an operation control unit 43B by executing a program stored in the storage device 12. Note that a part or the whole of the control device 11 may be realized by a dedicated electronic circuit. The posture analysis unit 41 sequentially generates posture data related to the posture of the HMD 30 by analyzing a detection signal output by the detection device 14, as in the first embodiment.

[0048] Similar to the display control unit 42A in the first embodiment, the display control unit 42B causes the display device 13 to display an image within the imaging range R captured by the virtual camera E in the virtual space V, and an instruction image G indicating the direction of the reference line Q. As described above in the first embodiment, the reference line Q is a virtual straight line whose direction in the virtual space V changes depending on the orientation of the HMD 30.

[0049] The display control unit 42B of the second embodiment changes the display mode of the instruction image G according to the positional relationship between the reference line Q and a character C (an example of an object) in the virtual space V. Specifically, the display control unit 42B makes the display mode of the instruction image G different between a case where the reference line Q intersects with the character C and a case where the reference line Q does not intersect with the character C. The display mode refers to the properties of an image that can be visually distinguished by the user U. For example, the three attributes of color, that is, hue (tone), saturation, and brightness (gradation), as well as size and image content (e.g., pattern or shape) are also included in the concept of the display mode.

[0050] When the reference line Q does not intersect with the character C, the display control unit 42B causes the display device 13 to display the image G1 as the instruction image G, as illustrated in FIG. 12. The image G1 is a circular or dot-like object arranged in the virtual space V, and is arranged on the reference line Q in the virtual space V. On the other hand, when the reference line Q intersects with the character C, the display control unit 42B causes the display device 13 to display an image G2 different from the image G1 as the instruction image G, as illustrated in FIG. 13. The image G2 is a planar object that typically represents the hand of the user U. The display control unit 42B places the image G2 at a point (i.e., a contact point P) on the surface of the character C in the virtual space V that intersects with the reference line Q, as illustrated in FIG. 14. That is, the image G2 contacts the contact point P on the surface of the character C in the virtual space V. The contact point P can also be said to be a point where the image G2 contacts the surface of the character C.

[0051] In the above description, switching between images G1 and G2 is exemplified, but the change in the display mode of the instruction image G also includes switching between display and non-display. That is, the display control unit 42B may display the image G2 as the instruction image G when the reference line Q intersects with the character C, and may not display the instruction image G when the reference line Q does not intersect with the character C.

[0052] As can be understood from the above description, in the second embodiment, the display mode of the instruction image G indicating the direction of the reference line Q in the virtual space V changes according to the positional relationship between the reference line Q and the character C. Therefore, the user U can easily grasp the positional relationship between the reference line Q and the character C in the virtual space V. In addition, since the instruction image G (image G2) is arranged on the surface of the character C in the virtual space V, there is an advantage that the user U can easily perceive the state of contacting the surface of the character C in the virtual space V. Since the instruction image G is arranged on the surface of the character C, there is also an advantage that the user U can easily and accurately visually grasp the contact point P.

[0053] 11 controls the movement of the character C in the virtual space V, similar to the movement control unit 43A in the first embodiment. Specifically, the movement control unit 43B causes the character C to perform a movement (i.e., a reaction movement) that reacts to a touch by the user U in the virtual space V. Typical examples of the reaction movement are a change in posture, a change in facial expression, or the pronunciation of a specific line.

[0054] Specifically, the action control unit 43B causes the character C to perform an action according to the position of the contact point P where the reference line Q intersects with the character C. For example, when the contact point P is located on the head of the character C, the action control unit 43B causes the character C to perform a reaction action to accept contact by the user U, and when the contact point P is located on the torso of the character C, the action control unit 43B causes the character C to perform an action to reject contact by the user U. With the above configuration, it is possible to vary the action of the character C in various ways according to the position of the contact point P.

[0055] Fig. 15 is a flowchart illustrating a specific example of a process (hereinafter referred to as a "second control process") executed by the display control unit 42B and the operation control unit 43B in the second embodiment. The second control process in Fig. 15 is repeatedly executed at a predetermined cycle.

[0056] When the second control process is started, the display control unit 42B determines whether or not the reference line Q intersects with the character C in the virtual space V (Sb1). If the reference line Q intersects with the character C (Sb1: YES), the display control unit 42B causes the display device 13 to display an image G2 arranged at a contact point P where the reference line Q and the character C intersect as an instruction image G (Sb2). On the other hand, if the reference line Q does not intersect with the character C (Sb1: NO), the display control unit 42B causes the display device 13 to display an image G1 arranged on the reference line Q as an instruction image G (Sb3).

[0057] In a state in which image G2 overlapping character C is displayed, the action control unit 43B determines whether or not an action instruction (hereinafter referred to as an "action instruction") for character C has been received from the user U (Sb4). The action instruction is an instruction to cause an action to occur for character C in the virtual space V. The action instruction in the second embodiment means an instruction for user U to touch character C in the virtual space V.

[0058] The motion control unit 43B of the second embodiment accepts a specific change in the posture of the HMD30 as a motion instruction from the user U. Specifically, the motion control unit 43B determines that the motion instruction has been received when the change in the posture of the HMD30 satisfies a predetermined condition (hereinafter referred to as an "instruction determination condition"). For example, similar to the first embodiment, the motion control unit 43B determines that the motion instruction has been given from the user U when the HMD30 has rotated in the roll direction. That is, the rotation of the HMD30 in the roll direction is the instruction determination condition. On the other hand, when the change in the posture of the HMD30 does not satisfy the instruction determination condition, the motion control unit 43B determines that the motion instruction has not been received. Note that, as described above with reference to FIG. 7, it is preferable to configure the HMD30 to be rotated when the rotation angle θ of the HMD30 exceeds the threshold θt.

[0059] When an action instruction is received from the user U (Sb4: YES), the action control unit 43B determines that the user U has contacted the character C in the virtual space V, and causes the character C to perform an action reacting to the contact by the user U (Sb5). Specifically, as described above, the action control unit 43B causes the character C to perform an action according to the position of the contact point P where the reference line Q intersects with the character C. As can be understood from the above explanation, in the second embodiment, the user U can contact a desired position of the character C in the virtual space V by giving an action instruction in a state where the reference line Q has been moved so that it intersects with the desired position of the character C.

[0060] In addition, in a configuration in which a change in the posture of the HMD 30 is recognized as an action instruction, there is a problem that since a direct operation such as a touch operation on the display surface is not required, the user U has difficulty in feeling the sensation of contact. In the second embodiment, since the display mode of the instruction image G changes according to the positional relationship between the reference line Q and the character C (specifically, whether or not the reference line Q intersects with the character C), there is an advantage that the user U can easily feel the sensation of contacting the character C in the virtual space V, compared to a configuration in which the instruction image G is displayed in a fixed manner.

[0061] In the second embodiment, for example, the following configuration may be adopted.

[0062] [Variation B1] In the second embodiment, the character C is caused to perform a reaction action according to the position of the contact point P, but similarly to the first embodiment, the character C may be caused to perform a reaction action according to the amount of change in the attitude of the HMD30 (for example, the rotation angle θ in the roll direction). For example, the amount of change in the attitude of the HMD30 is assumed to be a virtual pressure of the user U contacting the character C in the virtual space V. The action control unit 43B of the modification B1 causes the character C to perform a reaction action to accept contact by the user U when the amount of change in the attitude of the HMD30 is small, and causes the character C to perform a reaction action to reject contact by the user U when the amount of change in the attitude is large. According to the above aspect, it is possible to cause the character C to perform a variety of reaction actions according to the attitude of the HMD30.

[0063] [Variation B2] The movement control unit 43B of the modified example B2 accepts movement instructions from the user U within a specific period on the time axis (hereinafter referred to as the "instruction receiving period"). That is, the movement instructions given by the user U during the instruction receiving period are effectively reflected in the movement of the character C, but the movement instructions given by the user U after the instruction receiving period has elapsed are ignored. The instruction receiving period is, for example, a period of a predetermined time length (for example, several seconds) starting from the point at which the reference line Q begins to intersect with the character C. Note that the start point or end point of the instruction receiving period is not limited to the above examples. For example, the start point of the instruction receiving period may be the point at which a specific event starts in the game. By limiting the acceptance of movement instructions to the instruction receiving period as described above, an appropriate sense of tension is given to the user U, and as a result, it is possible to suppress boredom of the game.

[0064] The display control unit 42B changes the display mode of the instruction image G (image G2) over time during the instruction acceptance period. For example, as illustrated in FIG. 16, assume that the image G2 displayed as the instruction image G when the reference line Q intersects with the character C includes an image G21 and an image G22. The image G21 is an image representing the hand of the user U, similar to the image G2 in the first embodiment, and the image G22 is a circular image arranged behind the image G21. The display control unit 42B reduces the size of the image G22 over time from the start point to the end point of the instruction acceptance period. The size of the image G21 does not change. The above configuration has the advantage that the user U can visually grasp the instruction acceptance period (e.g., the remaining time) during which acceptance of an operation instruction is permitted from the display mode of the instruction image G.

[0065] [Variation B3] The display control unit 42B of the modification B3 changes the display mode of the instruction image G according to the position of the contact point P of the character C in the virtual space V where the reference line Q intersects. Specifically, as illustrated in Fig. 17, the display mode of the instruction image G differs between a case where the contact point P is at a first position on the surface of the character C in the virtual space V and a case where the contact point P is at a second position on the surface of the character C. The first position is, for example, a position where the character C accepts contact by the user U (for example, the torso of the character C), and the second position is, for example, a position where the character C rejects contact by the user U (for example, the head of the character C).

[0066] 17 illustrates an example in which the contact point P is at the first position or the second position, but the display control unit 42B may change the display mode of the instruction image G over time in parallel with the change in the position of the contact point P on the character C. That is, when the user U moves the contact point P on the surface of the character C in accordance with the posture of the HMD 30, the display control unit 42B changes the display mode of the instruction image G over time in conjunction with the movement of the contact point P. For example, within a period in which the contact point P moves from the first position to the second position in FIG. 17, the display control unit 42B changes the display mode of the instruction image G continuously or stepwise from the display mode at the first position to the display mode at the second position.

[0067] In the modified example B3, the display mode of the instruction image G changes depending on the position of the contact point P. Therefore, by visually checking the display mode of the instruction image G, the user U can guess the action of the character C at the time of contact. In other words, the user U can be provided with the fun of gradually moving the contact point P while guessing the action of the character C at the time of contact by checking the display mode of the instruction image G, and giving an action instruction while maintaining the contact point P at the position where the desired action is expected.

[0068] [Variation B4] The display control unit 42B of the modification B4 controls the character C so that the line of sight of the character C tracks the reference line Q. For example, the character C is controlled so that the line of sight is directed toward the contact point P where the reference line Q intersects with the character C. Specifically, as illustrated in FIG. 18, the eyeballs (e.g., pupils) and head of the character C rotate so as to track the reference line Q. Note that only one of the eyeballs or the head of the character C may track the reference line Q. The modification B4 has the advantage that the user U can easily grasp the sense of the influence the user U is having on the character C.

[0069] [Variation B5] In the second embodiment, the display mode of the instruction image G is changed depending on whether the reference line Q intersects with the character C, but the conditions for changing the display mode of the instruction image G are not limited to the above examples. For example, the display control unit 42B may change the display mode of the instruction image G continuously or stepwise depending on the distance between the reference line Q and the character C. The intersection of the reference line Q and the character C is not an essential condition for changing the display mode of the instruction image G. In addition, the display mode of the instruction image G may be changed depending on whether the reference line Q intersects with a specific part of the character C. In a state in which the reference line Q intersects with a part other than the specific part of the character C, the instruction image G is displayed in the same display mode as when the reference line Q does not intersect with the character C. As can be understood from the above description, the operation control unit 43B of the second embodiment is comprehensively expressed as an element that changes the display mode of the instruction image G depending on the positional relationship between the reference line Q and the character C.

[0070] [Variation B6] The configuration of the second embodiment in which the display mode of the instruction image G is changed according to the positional relationship between the reference line Q and the character C is also applied to the case where the user U uses the terminal device 10 (e.g., a smartphone) while holding it in his / her hand. As in the second embodiment, the imaging range R of the virtual space V displayed on the display device 13 changes according to the attitude of the terminal device 10.

[0071] The terminal device 10 of the modification B6 includes a touch panel that detects the contact of the user U with the display surface of the display device 13. In the second embodiment, the position of the contact point P in the virtual space V is changed according to the attitude of the HMD 30, but in the modification B6, the point where the user U contacts the display surface of the display device 13 is set as the contact point P, and an instruction image G is displayed. For example, a reference line Q passing through the contact point P where the user U contacts is set in the virtual space V. That is, a virtual straight line in the direction indicated by the user U by contacting (i.e., touch operation) the display surface of the display device 13 is set as the reference line Q in the virtual space V. When the reference line Q intersects with the character C, the display control unit 42B causes the display device 13 to display the image G2 as the instruction image G, and when the reference line Q does not intersect with the character C, causes the display device 13 to display the image G1 as the instruction image G. Even with the above configuration, there is an advantage that the user U can easily grasp the positional relationship between the reference line Q and the character C in the virtual space V (for example, a state where the user U is in contact with the character C in the virtual space V).

[0072] As can be understood from the example of Modification Example B6, in the second embodiment, the configuration in which the display device 13 is attached to the head of the user U may be omitted. In a configuration that does not assume that the display device 13 is attached to the head of the user U, the display control unit 42B is expressed as an element that causes the display device 13 to display an image of the virtual space V captured by a virtual camera E whose attitude is controlled according to the attitude of the display device 13. In addition, in a display device 13 that is used while being held in the hand of the user U, a virtual straight line in a direction indicated by an operation (for example, a touch operation) on the display device 13 is the "reference line."

[0073] [C: Other variations] Specific examples of modifications to the first or second embodiment are given below. Two or more examples selected from the following examples may be combined as appropriate to the extent that they are not mutually contradictory.

[0074] [Variation C1] The reference line Q is not limited to the optical axis of the virtual camera E exemplified in each of the above-mentioned embodiments. For example, a straight line perpendicular to the display surface of the display device 13 is used as the reference line Q. Also, a straight line forming a predetermined angle with respect to the optical axis of the virtual camera E or a straight line perpendicular to the display surface may be used as the reference line Q. Note that, in an HMD 30 equipped with an eye tracking (gaze measurement) function for estimating the direction of the gaze of the user U, the gaze estimated by the function may be used as the reference line Q. As can be understood from the above examples, the reference line Q is comprehensively expressed as a virtual straight line set in the virtual space V.

[0075] [Variation C2] In each of the above-mentioned embodiments, a character C is exemplified as an object placed in the virtual space V, but the object that the user U comes into contact with in the virtual space V is not limited to the character C. For example, inanimate elements such as buildings and natural objects in the virtual space V are also included in the concept of an object. For a living object (character C), an "action" is, for example, the behavior, action, appearance, or attitude of the object. Also, for an inanimate object, an "action" is, for example, a change in the form of the element. For example, the action of opening a door of a building, or the action of a natural object such as a rock deforming or moving are exemplified as the action of an object.

[0076] [Variation C3] In each of the above-mentioned embodiments, when the HMD30 rotates in the roll direction, it is determined that the user U has contacted the character C in the virtual space V, and the character C is made to perform a reaction action, but the trigger for the reaction action (i.e., the action instruction) is not limited to the above examples. For example, the rotation of the HMD30 in the pitch direction or yaw direction may be regarded as a contact by the user U, and the character C may be made to perform a reaction action. In addition, the character C may be made to perform a reaction action on the condition that, for example, the reference line Q intersects within a specific range of the character C (i.e., the user U stares within the range for a long time) continues for a predetermined time. The character C may be made to perform a reaction action on the condition that the user U operates an operating device (not shown) connected to the HMD30. As can be understood from the above examples, the configuration in which the change in the attitude of the HMD30 (especially the rotation in the roll direction) is used as a trigger for the reaction action of the character C may be omitted.

[0077] [Variation C4] In each of the above-mentioned embodiments, the rotation of the HMD 30 in the roll direction is accepted as a motion instruction (an instruction to contact the character C), but the change in the attitude of the HMD 30 that is determined as a motion instruction is not limited to the rotation in the roll direction. For example, the rotation of the HMD 30 in the pitch direction or the yaw direction may be accepted as a motion instruction.

[0078] [Variation C5] The modified examples (Modification A1 to Modification A7) exemplified for the first embodiment are similarly applied to the second embodiment. Moreover, the modified examples (Modification B1 to Modification B6) exemplified for the second embodiment are similarly applied to the first embodiment.

[0079] [Variation C6] In each of the above-described embodiments, the display device 13 and the detection device 14 of the HMD 30 and the information processing device 40 that controls the HMD 30 are implemented as a single terminal device 10, but the HMD 30 and the information processing device 40 may be implemented as separate devices as in the image display system 1B of FIG. 19. The HMD 30 and the information processing device 40 can communicate with each other by wire or wirelessly. Any method of communication between the HMD 30 and the information processing device 40 may be used, but short-range wireless communication such as Bluetooth (registered trademark) is preferable.

[0080] The HMD 30 includes a display device 13, a detection device 14, and a wearing tool 20, and is worn on the head of the user U, as in the first embodiment. The information processing device 40 is a control device that communicates with the HMD 30 to display various images on the HMD 30, and includes a control device 11 and a storage device 12. For example, various information terminals such as a mobile phone, a smartphone, a tablet terminal, a personal computer, or a home game device are used as the information processing device 40. It does not matter whether the information processing device 40 is a portable type or a stationary type. The control device 11 executes a program stored in the storage device 12, and functions as a posture analysis unit 41, a display control unit 42 (42A, 42B), and a motion control unit 43 (43A, 43B), as in the first or second embodiment. Therefore, the configuration of FIG. 19 also achieves the same effects as the first or second embodiment.

[0081] [Variation C7] A preferred embodiment of the present invention is realized by cooperation between a computer (specifically, the control device 11) and a program, as exemplified in each of the above-mentioned embodiments. The program according to each of the above-mentioned embodiments is provided in a form stored in a computer-readable recording medium and installed in the computer. The recording medium is, for example, a non-transitory recording medium, and is exemplified by an optical recording medium (optical disk) such as a CD-ROM, but includes any known type of recording medium such as a semiconductor recording medium or a magnetic recording medium. Note that a non-transitory recording medium includes any recording medium except for a transitory, propagating signal, and does not exclude a volatile recording medium. It is also possible to provide the program to the computer in the form of distribution via a communication network.

[0082] [D:Additional Notes] From the above description, for example, preferred embodiments of the present invention can be understood as follows. In order to facilitate understanding of each embodiment, reference numerals in the drawings are written in parentheses for convenience, but the present invention is not limited to the illustrated embodiments.

[0083] [Aspect 1] A program relating to a preferred embodiment (embodiment 1) of the present invention is a program that causes a computer (11) to function as a display control unit (42B) that causes a display device (13) of the head mounted display (30) to display a stereoscopic image utilizing binocular parallax, the stereoscopic image being an image captured of a virtual space (V) by a virtual camera (E) whose direction is controlled according to the direction of a head mounted display (30), and the display control unit (42B) displays an instruction image (G) in the direction of a reference line (Q) whose direction in the virtual space (V) changes according to the direction of the head mounted display (30), and changes the display mode of the instruction image (G) according to the positional relationship between the reference line (Q) and an object (C) in the virtual space (V). In the above configuration, the display mode of the instruction image (G) indicating the direction of the reference line (Q) in the virtual space (V) changes depending on the positional relationship between the reference line (Q) and the object (C), which has the advantage that the user (U) can easily grasp the positional relationship between the reference line (Q) and the object (C) in the virtual space (V).

[0084] The "reference line" is a virtual straight line set in the virtual space (V). For example, the optical axis of the virtual camera (E), the center line of the display screen of the display device (13), or a straight line forming a predetermined angle with respect to these straight lines are suitable examples of the reference line (Q). In a configuration in which an eye tracking function for estimating the line of sight of the user (U) can be used, in addition to the straight lines exemplified above, the line of sight of the user (U) may be used as the reference line (Q).

[0085] An "object" is a virtual object placed in a virtual space (V). A typical example of an object (C) is a character (e.g., a human, an animal, or a monster), but the concept of an object can also include inanimate elements such as buildings or natural objects in the virtual space (V).

[0086] "Display mode" refers to the visually distinguishable characteristics of an image. For example, the three attributes of color, hue (tone), saturation, and brightness (gradation), as well as size and image content (e.g., pattern or shape), are all included in the concept of display mode. Changes in display mode also include switching between display and non-display.

[0087] [Aspect 2] In a preferred example (Aspect 2) of Aspect 1, the display control unit (42B) displays the instruction image (G) at a point (P) on the surface of the object (C) that intersects with the reference line (Q). According to the above aspect, since the instruction image (G) is placed on the surface of the object (C) in the virtual space (V), there is an advantage that the user (U) can easily perceive the state of contact with the surface of the object (C) from the instruction image (G).

[0088] [Aspect 3] A program according to a preferred example (Aspect 3) of Aspect 1 or Aspect 2 causes the computer (11) to function as a motion control unit (43B) that controls the motion of the object (C) in accordance with the positional relationship between the reference line (Q) and the object (C). According to the above aspect, it is possible to cause the object (C) to execute various motions that reflect instructions from the user (U).

[0089] The "movement of an object" refers to, for example, the behavior, action, appearance, or attitude of a character that is an example of an object (C). With respect to an inanimate element, the "movement of an object" refers to, for example, a change in the form of the element (for example, a door or window of a building opening, or a natural object such as a rock deforming or moving, etc.).

[0090] [Aspect 4] In a preferred example (Aspect 4) of Aspect 3, when the motion control unit (43B) receives a motion instruction from the user (U), it controls the motion of the object (C) in accordance with the positional relationship between the reference line (Q) and the object (C). According to the above aspect, the user (U) can cause the object (C) to execute a motion corresponding to the desired position by giving a motion instruction in a state where the reference line (Q) has been moved to a desired position.

[0091] An "action instruction" is an instruction to cause an action (e.g., contact) to occur on an object (C) in the virtual space (V). It can also be said as an instruction to confirm the provisional direction represented by the instruction image (G) as the direction desired by the user (U). For example, it is determined that an action instruction has been accepted when a specific change (e.g., rotation in the roll direction) occurs in the direction of the head-mounted display (30). It may also be determined that an action instruction has been accepted when, for example, the reference line (Q) remains stationary in the virtual space (V) for a predetermined period of time (e.g., 2 seconds).

[0092] [Aspect 5] In a preferred example (aspect 5) of aspect 4, the action control unit (43B) determines that the action instruction has been received when a change in the attitude of the head mounted display (30) satisfies a predetermined condition, and causes the object (C) to perform an action corresponding to the amount of change in the attitude, and the display control unit (42B) changes the display mode of the instruction image (G) according to the change in the attitude of the head mounted display (30) before receiving the action instruction. According to the above aspect, it is possible to cause the object (C) to perform various actions according to the attitude of the head mounted display (30). In addition, since the display mode of the instruction image (G) changes according to the attitude of the head mounted display (30), there is an advantage that the user (U) can easily grasp the sense of the user (U) influencing the object (C) (for example, the sense of contact). Note that the "predetermined condition" is a condition of the attitude change that is determined to be an action instruction by the user (U), and is, for example, a rotation in the roll direction in this embodiment.

[0093] [Aspect 6] In a preferred example (Aspect 6) of Aspect 4 or Aspect 5, the display control unit (42B) changes the display mode of the instruction image (G) over time during an instruction acceptance period during which acceptance of the operation instruction by the user (U) is permitted. According to the above aspect, the user (U) can visually grasp the instruction acceptance period during which acceptance of the operation instruction is permitted (e.g., the remaining time of the instruction acceptance period) from the display mode of the instruction image (G). Note that the "instruction acceptance period" is, for example, the remaining time during which one contact with a character is permitted, or the remaining time of an operation mode during which contact with a character is permitted.

[0094] [Aspect 7] In a preferred example (Aspect 7) of any of Aspects 3 to 6, the display control unit (42B) changes the display mode of the instruction image (G) according to the position of the point (P) of the object (C) where the reference line (Q) intersects. According to the above aspects, since the display mode of the instruction image (G) changes according to the position of the point (P) of the object (C) where the reference line (Q) intersects, the user (U) can infer the movement of the object (C) from the display mode of the instruction image (G).

[0095] [Aspect 8] In a preferred example (aspect 8) of aspect 7, the display control unit (42B) changes the display mode of the indication image (G) over time in parallel with the change in the position of the point (P) of the object (C) where the reference line (Q) intersects. According to the above aspect, the display mode of the indication image (G) changes over time in parallel with the change in the position of the point (P) of the object (C) where the reference line (Q) intersects, so that the user (U) can be provided with the interest of gradually changing the reference line (Q) while checking the display mode of the indication image (G). Note that "change in the display mode over time" does not mean that the display mode changes only in a binary manner at a specific point in time, but that the display mode changes gradually over time. Note that it does not matter whether the display mode changes continuously or in stages.

[0096] [Aspect 9] In a preferred example (Aspect 9) of any of Aspects 1 to 8, the motion control unit (43B) controls the object (C) so that the line of sight of the object (C) tracks the reference line (Q). According to the above aspects, the user (U) can easily grasp the sensation of the user (U) influencing the object (C) (e.g., the sensation of touching). Note that "the line of sight of the object tracks the reference line" includes not only the case where the eyeball (e.g., the pupil) of the object (C) rotates to track the reference line (Q), but also the case where the head of the object (C) rotates according to the reference line (Q).

[0097] [Aspect 10] An image display system (1A, 1B) according to a preferred embodiment (embodiment 10) of the present invention is an image display system (1A, 1B) comprising a head mounted display (30) and an information processing device (40), wherein the information processing device (40) comprises a display control unit (42B) that causes a display device (13) of the head mounted display (30) to display a stereoscopic image utilizing binocular parallax, the stereoscopic image being an image captured of a virtual space (V) by a virtual camera (E) whose direction is controlled according to the direction of the head mounted display (30), and the display control unit (42B) causes an instruction image (G) to be displayed in the direction of a reference line (Q) whose direction in the virtual space (V) changes according to the direction of the head mounted display (30), and changes the display mode of the instruction image (G) according to the positional relationship between the reference line (Q) and an object (C) in the virtual space (V). In the above configuration, the display mode of the instruction image (G) indicating the direction of the reference line (Q) in the virtual space (V) changes depending on the positional relationship between the reference line (Q) and the object (C), which has the advantage that the user (U) can easily grasp the positional relationship between the reference line (Q) and the object (C) in the virtual space (V). Note that the head mounted display (30) and the information processing device (40) may be either integrated or separate. [Explanation of symbols]

[0098] 1A, 1B...image display system, 10...terminal device, 11...control device, 12...storage device, 13...display device, 14...detection device, 20...wearing device, 30...HMD, 40...information processing device, 41...posture analysis unit, 42A, 42B...display control unit, 43A, 43B...motion control unit, V...virtual space, E...virtual camera, C...character, R...imaging range, Q...reference line, G...instruction image, P...contact point.

Claims

1. a display control unit that causes an image of a virtual space captured by a virtual camera whose direction is controlled according to a direction of a head mounted display to be displayed on the head mounted display; and a motion control unit that causes the object to execute a motion corresponding to a position where a reference line, the direction of which in the virtual space changes according to the direction of the head mounted display, intersects with the object in the virtual space; A program for causing a computer to function as The display control unit displays a first pointing image in a direction of the reference line when the reference line does not intersect with the object, and displays a second pointing image having a display mode different from that of the first pointing image in a direction of the reference line when the reference line intersects with the object. program.

2. The second instruction image is in contact with a surface of the object in the virtual space. The program of claim 1.

3. a display control unit that causes an image of a virtual space captured by a virtual camera whose direction is controlled according to a direction of a head mounted display to be displayed on the head mounted display; and a motion control unit that causes the object to execute a motion corresponding to a position where a reference line, the direction of which in the virtual space changes according to the direction of the head mounted display, intersects with the object in the virtual space; A program for causing a computer to function as The action control unit causes the object to execute the action when a change in the direction of the head mounted display satisfies a predetermined condition. program.

4. The action control unit causes the object to perform a first action when the reference line intersects a first portion of the object, and causes the object to perform a second action different from the first action when the reference line intersects a second portion of the object that is different from the first portion. The program according to any one of claims 1 to 3.

5. An image display system including a head mounted display and an information processing device, The information processing device includes: a display control unit that causes the head mounted display to display an image of a virtual space captured by a virtual camera whose direction is controlled according to a direction of the head mounted display; an action control unit that causes the object to perform an action according to a position where a reference line, the direction of which in the virtual space changes according to a direction of the head mounted display, intersects with the object in the virtual space; Equipped with The display control unit displays a first pointing image in a direction of the reference line when the reference line does not intersect with the object, and displays a second pointing image having a display mode different from that of the first pointing image in a direction of the reference line when the reference line intersects with the object. Image display system.

6. An image display system including a head mounted display and an information processing device, The information processing device includes: a display control unit that causes the head mounted display to display an image of a virtual space captured by a virtual camera whose direction is controlled according to a direction of the head mounted display; an action control unit that causes the object to perform an action according to a position where a reference line, the direction of which in the virtual space changes according to a direction of the head mounted display, intersects with the object in the virtual space; Equipped with The action control unit causes the object to execute the action when a change in the direction of the head mounted display satisfies a predetermined condition. Image display system.

Citation Information

Patent Citations

  • Head-mounted display system, method of displaying on head-mounted display, and program

    JP2016115122A

  • Program and image formation device

    JP2017058971A

  • Information processing apparatus, information processing system, control method thereof, and program

    JP2017102685A