IMAGE DISPLAY SYSTEM, IMAGE DISPLAY METHOD, AND GOGGLE-TYPE DEVICE

Abstract

To provide an image display system, an image display program, an image display method, and a display device capable of improving the convenience for presenting a user interface image.SOLUTION: An image display system includes a display device having a display screen, and a goggle device on which the display device can be mounted. In first display mode, the system displays, on the display screen, a first image including a first user interface image and a content image which is a non-stereoscopic image. In second display mode, the system displays, on the display screen, a second image including a content image composed of a left-eye image and a right-eye image with parallax and a second user interface image corresponding to the first user interface image. In the second display mode, the system displays the second user interface image in a position which is different from a position on the display screen where the first user interface image is displayed in the first display mode.SELECTED DRAWING: Figure 16

Description

[Technical field]

[0001] The present invention relates to an image display system capable of displaying stereoscopic images. , painting Image display method, and Goggle type Regarding the device. [Background technology]

[0002] Conventionally, there is a stereoscopic image display device that displays a stereoscopic image by having a user view two images with a parallax with each other with the user's left and right eyes, respectively (see, for example, Patent Document 1). For example, in the above Patent Document 1, a smartphone is stored in a goggle device that can be worn by a user, and the user can view a stereoscopic image by peering into the stereoscopic image displayed on the display screen of the smartphone through the goggle device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-198651 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the stereoscopic image display device disclosed in Patent Document 1 does not take into consideration the presentation of a user interface image that accepts touch operations by the user when displaying a stereoscopic image on the display screen, and therefore there is room for improving convenience in the method of presenting the user interface image.

[0005] Therefore, an object of the present invention is to provide an image display system capable of improving the convenience of presenting a user interface image. , painting Image display method, and Goggle type The present invention provides an apparatus for detecting a [Means for solving the problem]

[0006] In order to achieve the above object, the present invention may adopt, for example, the following configurations: When interpreting the claims, it is understood that the scope should be interpreted only by the claims, and in the event of a contradiction between the claims and this section, the claims shall take precedence.

[0007] One configuration example of the image display system of the present invention includes a display device having a display screen for displaying an image, and a goggle device on which the display device can be worn. The image display system includes a mode setting means, a first display control means, and a second display control means. The mode setting means sets the display device to a first display mode or a second display mode different from the first display mode. The first display control means causes the display screen to display a first image including a content image, which is a non-stereoscopic image, and a first user interface image, in the first display mode. The second display control means causes the display screen to display a second image including a content image consisting of a left-eye image and a right-eye image having parallax therebetween, and a second user interface image corresponding to the first user interface image, in the second display mode. The second display control means causes the second user interface image to be displayed in a position on the display screen different from the position at which the first user interface image is displayed in the first display mode, in the second display mode.

[0008] According to the above, when the display mode is switched, the user interface image is displayed at a different position, so that it is possible to improve convenience regarding the presentation of the user interface image.

[0009] The image display system further includes a detection means for detecting whether the display device is attached to the goggle device or not, or whether the display device is in the middle of being attached to the goggle device or not. In this case, the mode setting means may switch the display device to the second display mode based on a detection result of the detection means when the display device is set to the first display mode.

[0010] According to the above, the display mode is switched based on the mounting state or the intermediate mounting state of the display device to the goggle device, so that seamless switching of the display mode is possible.

[0011] The display device may also include an illuminance sensor. In this case, the goggle device may also include a light blocking section. The light blocking section blocks light from reaching the illuminance sensor of the display device when the display device is in a state where it is attached to the goggle device or in a state where it is being attached to the goggle device. The detection means may detect whether the display device is in a state where it is attached to the goggle device or in a state where it is being attached to the goggle device based on a detection result by the illuminance sensor.

[0012] Based on the above, it is possible to easily detect whether the display device is in an attached state or not, or whether the display device is in a partially attached state or not, based on a light blocking state for the display device.

[0013] The display device may include a touch panel provided on a display screen. In this case, a first user interface image displayed on the display screen may be capable of providing an operational instruction in response to a touch operation on the touch panel. A second user interface image displayed on the display screen may be capable of providing an operational instruction in response to a touch operation on the touch panel in a state in which the display device is attached to a goggle device.

[0014] Based on the above, it is possible to improve convenience regarding the presentation of a user interface image that accepts a touch operation.

[0015] Moreover, the second display control means may cause an image corresponding to the content image displayed as a non-stereoscopic image by the first display control means to be displayed on the display screen as a content image consisting of a left eye image and a right eye image.

[0016] Based on the above, it is possible to seamlessly switch between a stereoscopic image and a non-stereoscopic image in the same content image.

[0017] In addition, in the second display mode, the second display control means may convert the content image that was displayed in the first display mode immediately before the second display mode was set to the second display mode into a stereoscopic image consisting of an image for the left eye and an image for the right eye, and display it on the display screen.

[0018] Based on the above, it is possible to seamlessly switch between a stereoscopic image and a non-stereoscopic image in the same content image.

[0019] In addition, the second display control means may, in the second display mode, display an image for the left eye in a first region of the display screen, display an image for the right eye in a second region of the display screen different from the first region, and display a second user interface image in a third region of the display screen different from the first and second regions.

[0020] According to the above, it is possible to prevent the first and second areas for displaying a stereoscopic image from being soiled by touch operations, and it is also possible to prevent a finger for touching the second user interface image from coming into view when viewing a stereoscopic image.

[0021] The second display control means may display, as the second user interface image, a user interface image that has substantially the same function as the first user interface image but a different shape.

[0022] Based on the above, it is possible to display a user interface image having a shape suitable for operation and stereoscopic image display.

[0023] In addition, in the second display mode, the second display control means may adjust the shape of the second user interface image to fit the shape of a third area of ​​the display screen that is different from the first area of ​​the display screen that displays an image for the left eye and the second area of ​​the display screen that displays an image for the right eye, and display the second user interface image in the third area.

[0024] Based on the above, it is possible to display a user interface image in an appropriate shape.

[0025] The second display control means may set a third area above or below the display screen between the first area and the second area on the display screen.

[0026] Based on the above, it is possible to display a user interface image that does not interfere with stereoscopic image display.

[0027] The second display control means may set a third area below the display screen between the first area and the second area on the display screen.

[0028] Based on the above, it is possible to display a user interface image that does not interfere with stereoscopic image display and that is easy to operate.

[0029] The goggle device may have an opening that exposes at least a third region, which is a part of the display screen, to the outside when the display device is attached to the goggle device.

[0030] Based on the above, a touch operation of touching a user interface image can be performed in a state where the user is wearing the goggle device.

[0031] The opening may be formed at a position corresponding to the nose of a user when the user wears the goggle device.

[0032] According to the above, the opening can be formed without impairing the light blocking property.

[0033] Furthermore, the first display control means may, in the first display mode, display the first user interface image so as to be superimposed on a content image displayed on the display screen.

[0034] Based on the above, a relatively large content image can be displayed without being affected by the display of a user interface image.

[0035] Furthermore, the second display control means may cause the second user interface image to be displayed as a non-stereoscopic image on the display screen in the second display mode.

[0036] Based on the above, the operation of touching the user interface image becomes easy.

[0037] The display device may further include a display device-side connection section electrically connectable to another device. In this case, the goggle device may include a goggle device-side connection section electrically connectable to the display device-side connection section. The detection means may detect whether the display device is attached to the goggle device or is in the process of being attached in response to the display device-side connection section and the goggle device-side connection section being connected.

[0038] Based on the above, it is possible to reliably detect that the display device has been attached to the goggle device, or that the display device is in the process of being attached.

[0039] Furthermore, the present invention may be embodied in the form of an image display program, an image display method, and a display device. Effect of the Invention

[0040] According to the present invention, since a user interface image is displayed at a different position when the display mode is switched, it is possible to improve the convenience regarding the presentation of the user interface image. [Brief description of the drawings]

[0041] [Figure 1] A diagram showing the left controller 3 and the right controller 4 attached to the main unit 2. [Diagram 2]FIG. 1 shows an example of a state in which the left controller 3 and the right controller 4 are removed from the main unit 2. [Diagram 3] Six-sided views showing an example of the main unit 2 [Figure 4] Six-sided diagram showing an example of the left controller 3 [Diagram 5] Six-sided diagram showing an example of the right controller 4 [Figure 6] A block diagram showing an example of the internal configuration of the main unit 2. [Figure 7] A block diagram showing an example of the internal configuration of the main unit 2, the left controller 3, and the right controller 4. [Figure 8] FIG. 1 is a perspective view showing an example of the appearance of a goggle device 150. [Figure 9] FIG. 1 is a front view showing an example of a state in which the main unit 2 is attached to the goggle device 150. [Figure 10] FIG. 1 is a front view showing an example of a state of the main unit 2 attached to the goggle device 150. [Figure 11] FIG. 13 is a diagram showing an example of the shape of a front contact portion 151b that comes into contact with a part of the front surface of the main unit 2. [Figure 12] FIG. 1 is a diagram showing an example of the internal structure of a goggle device 150. [Figure 13] FIG. 1 is a side view showing an example of a state of the main unit 2 attached to the goggle device 150. [Figure 14] FIG. 1 is a diagram showing an example of a user viewing an image displayed on an image display system. [Figure 15] FIG. 1 is a diagram showing an example of a user holding an image display system; [Figure 16] FIG. 13 shows an example of an image displayed on the main unit 2 in a stereoscopic display mode and a non-stereoscopic display mode. [Figure 17] FIG. 1 shows an example of a data area set in the DRAM 85 of the main unit 2. [Figure 18] A flowchart showing an example of a game process executed by the game system 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0042] An image display system according to an example of this embodiment will be described below. The example of the image display system in this embodiment is configured by a game system 1 (as a minimum configuration, a main unit device 2 included in the game system 1) and a goggle device 150. The example of the game system 1 includes a main unit device (information processing device; in this embodiment, it functions as a game device main unit) 2, a left controller 3, and a right controller 4. The left controller 3 and the right controller 4 are detachable from the main unit device 2. That is, the game system 1 can be used as an integrated device by attaching the left controller 3 and the right controller 4 to the main unit device 2. In addition, the game system 1 can also be used as a separate device from the main unit device 2, the left controller 3, and the right controller 4 (see FIG. 2). The hardware configuration of the game system 1 of this embodiment will be described below, and then the control of the game system 1 of this embodiment will be described.

[0043] Fig. 1 is a diagram showing an example of a state in which a left controller 3 and a right controller 4 are attached to a main unit 2. As shown in Fig. 1, the left controller 3 and the right controller 4 are each attached to and integrated with the main unit 2. The main unit 2 is a device that executes various processes (e.g., game processes) in the game system 1. The main unit 2 includes a display 12. The left controller 3 and the right controller 4 are devices that include an operation unit that allows the user to perform input.

[0044] Fig. 2 is a diagram showing an example of a state in which the left controller 3 and the right controller 4 have been removed from the main unit 2. As shown in Figs. 1 and 2, the left controller 3 and the right controller 4 are detachable from the main unit 2. In the following, the left controller 3 and the right controller 4 may be collectively referred to as "controller."

[0045] Fig. 3 is a six-sided view showing an example of the main unit 2. As shown in Fig. 3, the main unit 2 includes a substantially plate-shaped housing 11. In this embodiment, the main surface of the housing 11 (in other words, the front surface, i.e., the surface on which the display 12 is provided) is generally rectangular in shape.

[0046] The shape and size of the housing 11 are arbitrary. As an example, the housing 11 may be of a size that is portable. Furthermore, the main unit 2 alone or an integrated device in which the left controller 3 and the right controller 4 are attached to the main unit 2 may be a portable device. Furthermore, the main unit 2 or the integrated device may be a handheld device. Furthermore, the main unit 2 or the integrated device may be a portable device.

[0047] 3, the main unit 2 includes a display 12 provided on a main surface of a housing 11. The display 12 displays an image generated by the main unit 2. In this embodiment, the display 12 is a liquid crystal display (LCD). However, the display 12 may be any type of display device.

[0048] The main unit 2 also includes a touch panel 13 on the screen of the display 12. In this embodiment, the touch panel 13 is of a type that allows multi-touch input (e.g., a capacitive type). However, the touch panel 13 may be of any type, and may be of a type that allows single-touch input (e.g., a resistive film type), for example.

[0049] The main unit 2 includes a speaker (i.e., speaker 88 shown in FIG. 6) inside the housing 11. As shown in FIG. 3, speaker holes 11a and 11b are formed in the main surface of the housing 11. The output sound of the speaker 88 is output from these speaker holes 11a and 11b, respectively.

[0050] The main unit 2 also has a left side terminal 17 which is a terminal through which the main unit 2 performs wired communication with the left controller 3, and a right side terminal 21 which is a terminal through which the main unit 2 performs wired communication with the right controller 4.

[0051] As shown in FIG. 3, the main unit 2 includes a slot 23. The slot 23 is provided on the upper side of the housing 11. The slot 23 has a shape that allows a predetermined type of storage medium to be attached thereto. The predetermined type of storage medium is, for example, a storage medium (e.g., a dedicated memory card) dedicated to the game system 1 and the same type of information processing device. The predetermined type of storage medium is used, for example, to store data used in the main unit 2 (e.g., application save data, etc.) and / or programs executed in the main unit 2 (e.g., application programs, etc.). The main unit 2 also includes a power button 28.

[0052] The main unit 2 includes a lower terminal 27. The lower terminal 27 is a terminal through which the main unit 2 communicates with the cradle. In this embodiment, the lower terminal 27 is a USB connector (more specifically, a female connector). When the all-in-one device or the main unit 2 alone is placed on the cradle, the game system 1 can display images generated and output by the main unit 2 on a stationary monitor. In this embodiment, the cradle also has a function of charging the all-in-one device or the main unit 2 alone that is placed on it. The cradle also has a function of a hub device (more specifically, a USB hub).

[0053] The main unit 2 includes an illuminance sensor 29. In this embodiment, the illuminance sensor 29 is provided under the main surface of the housing 11, and detects the illuminance (brightness) of light incident from the main surface side of the housing 11. Note that, depending on the illuminance of light detected by the illuminance sensor 29, the display 12 can be adjusted to an appropriate brightness to display an image, and in this embodiment, whether or not the main unit 2 is attached to a goggle device 150 (described later) is determined based on the detected illuminance.

[0054] FIG. 4 is a six-sided view showing an example of the left controller 3. As shown in FIG. 4, the left controller 3 includes a housing 31. In this embodiment, the housing 31 has a vertically long shape, that is, a shape that is long in the up-down direction (i.e., the y-axis direction shown in FIG. 1 and FIG. 4). The left controller 3 can also be held in a vertically long orientation when removed from the main unit 2. The housing 31 has a shape and size that allows it to be held in one hand, particularly the left hand, when held in a vertically long orientation. The left controller 3 can also be held in a horizontally long orientation. When the left controller 3 is held in a horizontally long orientation, it may be held with both hands.

[0055] The left controller 3 includes an analog stick 32. As shown in FIG. 4, the analog stick 32 is provided on the main surface of the housing 31. The analog stick 32 can be used as a direction input unit capable of inputting a direction. By tilting the analog stick 32, the user can input a direction according to the tilt direction (and input a magnitude according to the tilt angle). Note that the left controller 3 may include a cross key or a slide stick capable of slide input, instead of an analog stick, as the direction input unit. Also, in this embodiment, input is possible by pressing the analog stick 32.

[0056] The left controller 3 includes various operation buttons. The left controller 3 includes four operation buttons 33 to 36 (specifically, a right button 33, a down button 34, an up button 35, and a left button 36) on the main surface of the housing 31. Furthermore, the left controller 3 includes a record button 37 and a - (minus) button 47. The left controller 3 includes a first L button 38 and a ZL button 39 on the upper left of the side of the housing 31. The left controller 3 also includes a second L button 43 and a second R button 44 on the side of the housing 31 that is attached when the left controller 3 is attached to the main unit 2. These operation buttons are used to give instructions according to various programs (for example, OS programs and application programs) executed on the main unit 2.

[0057] In addition, the left controller 3 is equipped with a terminal 42 that enables the left controller 3 to communicate with the main unit 2 via wire.

[0058] FIG. 5 is a six-sided view showing an example of the right controller 4. As shown in FIG. 5, the right controller 4 includes a housing 51. In this embodiment, the housing 51 has a vertically long shape, that is, a shape that is long in the up-down direction. The right controller 4 can also be held in a vertically long orientation when removed from the main unit 2. The housing 51 has a shape and size that allows it to be held in one hand, particularly the right hand, when held in a vertically long orientation. The right controller 4 can also be held in a horizontally long orientation. When the right controller 4 is held in a horizontally long orientation, it may be held with both hands.

[0059] The right controller 4, like the left controller 3, includes an analog stick 52 as a direction input unit. In this embodiment, the analog stick 52 has the same configuration as the analog stick 32 of the left controller 3. The right controller 4 may include a cross key or a slide stick capable of slide input, instead of the analog stick. The right controller 4, like the left controller 3, includes four operation buttons 53 to 56 (specifically, an A button 53, a B button 54, an X button 55, and a Y button 56) on the main surface of the housing 51. The right controller 4 further includes a + (plus) button 57 and a home button 58. The right controller 4 also includes a first R button 60 and a ZR button 61 on the upper right of the side surface of the housing 51. The right controller 4 also includes a second L button 65 and a second R button 66, like the left controller 3.

[0060] In addition, the right controller 4 is equipped with a terminal 64 for enabling the right controller 4 to communicate with the main unit 2 via wire.

[0061] Fig. 6 is a block diagram showing an example of the internal configuration of main unit 2. In addition to the configuration shown in Fig. 3, main unit 2 includes components 81-91, 97, and 98 shown in Fig. 6. Some of these components 81-91, 97, and 98 may be mounted on an electronic circuit board as electronic components and housed in housing 11.

[0062] The main unit 2 includes a processor 81. The processor 81 is an information processing unit that executes various types of information processing executed in the main unit 2, and may be composed of only a CPU (Central Processing Unit), or may be composed of a SoC (System-on-a-chip) including multiple functions such as a CPU function and a GPU (Graphics Processing Unit) function. The processor 81 executes various types of information processing by executing an information processing program (e.g., a game program) stored in a storage unit (specifically, an internal storage medium such as a flash memory 84, or an external storage medium inserted in the slot 23, etc.).

[0063] The main unit 2 includes a flash memory 84 and a dynamic random access memory (DRAM) 85 as examples of internal storage media built into the main unit 2. The flash memory 84 and the DRAM 85 are connected to the processor 81. The flash memory 84 is a memory used mainly for storing various data (which may be programs) saved in the main unit 2. The DRAM 85 is a memory used for temporarily storing various data used in information processing.

[0064] The main device 2 includes a slot interface (hereinafter abbreviated as "I / F") 91. The slot I / F 91 is connected to the processor 81. The slot I / F 91 is connected to the slot 23, and reads and writes data from and to a predetermined type of storage medium (e.g., a dedicated memory card) inserted in the slot 23 in response to an instruction from the processor 81.

[0065] The processor 81 appropriately reads and writes data from and to the flash memory 84, DRAM 85, and each of the above storage media to execute the above information processing.

[0066] The main unit 2 includes a network communication unit 82. The network communication unit 82 is connected to the processor 81. The network communication unit 82 communicates with an external device via a network (specifically, wireless communication). In this embodiment, the network communication unit 82 connects to a wireless LAN and communicates with an external device using a method conforming to the Wi-Fi standard as a first communication mode. The network communication unit 82 also performs wireless communication with other main units 2 of the same type using a predetermined communication method (for example, communication using a unique protocol or infrared communication) as a second communication mode. Note that the wireless communication using the second communication mode enables wireless communication with other main units 2 located within a closed local network area, and realizes a function that enables so-called "local communication" in which data is transmitted and received by directly communicating between multiple main units 2.

[0067] The main unit 2 includes a controller communication unit 83. The controller communication unit 83 is connected to the processor 81. The controller communication unit 83 performs wireless communication with the left controller 3 and / or the right controller 4. Any communication method may be used between the main unit 2 and the left controller 3 and right controller 4, but in this embodiment, the controller communication unit 83 performs communication with the left controller 3 and the right controller 4 in accordance with the Bluetooth (registered trademark) standard.

[0068] The processor 81 is connected to the left terminal 17, the right terminal 21, and the lower terminal 27. When the processor 81 performs wired communication with the left controller 3, it transmits data to the left controller 3 via the left terminal 17 and receives operation data from the left controller 3 via the left terminal 17. When the processor 81 performs wired communication with the right controller 4, it transmits data to the right controller 4 via the right terminal 21 and receives operation data from the right controller 4 via the right terminal 21. When the processor 81 performs communication with the cradle, it transmits data to the cradle via the lower terminal 27. Thus, in this embodiment, the main unit 2 can perform both wired communication and wireless communication with the left controller 3 and the right controller 4. When the main unit 2 alone or an integrated device with the left controller 3 and the right controller 4 attached to the main unit 2 is attached to the cradle, the main unit 2 can output data (e.g., image data and audio data) to a stationary monitor or the like via the cradle.

[0069] Here, the main unit 2 can communicate with a plurality of left controllers 3 simultaneously (in other words, in parallel). The main unit 2 can also communicate with a plurality of right controllers 4 simultaneously (in other words, in parallel). Therefore, a plurality of users can simultaneously input to the main unit 2 using each set of left controllers 3 and right controllers 4. As an example, a first user can input to the main unit 2 using a first set of left controllers 3 and right controllers 4, while a second user can input to the main unit 2 using a second set of left controllers 3 and right controllers 4.

[0070] The main device 2 includes a touch panel controller 86, which is a circuit that controls the touch panel 13. The touch panel controller 86 is connected between the touch panel 13 and the processor 81. The touch panel controller 86 generates data indicating, for example, the position where a touch input has been performed based on a signal from the touch panel 13, and outputs the data to the processor 81.

[0071] The display 12 is also connected to the processor 81. The processor 81 displays on the display 12 an image generated (for example, by executing the above-mentioned information processing) and / or an image acquired from the outside.

[0072] The main unit 2 includes a codec circuit 87 and speakers (specifically, a left speaker and a right speaker) 88. The codec circuit 87 is connected to the speaker 88 and the audio input / output terminal 25, and is also connected to the processor 81. The codec circuit 87 is a circuit that controls the input and output of audio data to and from the speaker 88 and the audio input / output terminal 25.

[0073] The main body device 2 also includes an acceleration sensor 89. In this embodiment, the acceleration sensor 89 detects the magnitude of acceleration along three predetermined axial directions (for example, the x, y and z axes shown in FIG. 1). The acceleration sensor 89 may detect acceleration along one axial direction or two axial directions.

[0074] The main body device 2 also includes an angular velocity sensor 90. In this embodiment, the angular velocity sensor 90 detects angular velocities around three predetermined axes (for example, the x, y and z axes shown in FIG. 1). Note that the angular velocity sensor 90 may detect angular velocities around one axis or two axes.

[0075] The acceleration sensor 89 and the angular velocity sensor 90 are connected to the processor 81, and the detection results of the acceleration sensor 89 and the angular velocity sensor 90 are output to the processor 81. The processor 81 can calculate information related to the movement and / or attitude of the main unit 2 based on the detection results of the acceleration sensor 89 and the angular velocity sensor 90.

[0076] The illuminance sensor 29 is connected to the processor 81, and the detection result of the illuminance sensor 29 is output to the processor 81. The processor 81 is capable of calculating information regarding the brightness around the main body device 2 based on the detection result of the illuminance sensor 29.

[0077] The main unit 2 includes a power control unit 97 and a battery 98. The power control unit 97 is connected to the battery 98 and the processor 81. Although not shown, the power control unit 97 is also connected to each unit of the main unit 2 (specifically, each unit that receives power from the battery 98, the left terminal 17, and the right terminal 21). The power control unit 97 controls the supply of power from the battery 98 to each of the above-mentioned units based on instructions from the processor 81.

[0078] Furthermore, battery 98 is connected to lower terminal 27. When an external charging device (e.g., a cradle) is connected to lower terminal 27 and power is supplied to main unit 2 via lower terminal 27, battery 98 is charged with the supplied power.

[0079] Fig. 7 is a block diagram showing an example of the internal configuration of the main unit 2, the left controller 3, and the right controller 4. Note that details of the internal configuration of the main unit 2 are omitted in Fig. 7 because they are shown in Fig. 6.

[0080] The left controller 3 includes a communication control unit 101 that communicates with the main unit 2. As shown in FIG. 7, the communication control unit 101 is connected to each component including the terminal 42. In this embodiment, the communication control unit 101 can communicate with the main unit 2 by both wired communication via the terminal 42 and wireless communication not via the terminal 42. The communication control unit 101 controls the communication method by which the left controller 3 communicates with the main unit 2. That is, when the left controller 3 is attached to the main unit 2, the communication control unit 101 communicates with the main unit 2 via the terminal 42. Also, when the left controller 3 is detached from the main unit 2, the communication control unit 101 performs wireless communication with the main unit 2 (specifically, the controller communication unit 83). The wireless communication between the controller communication unit 83 and the communication control unit 101 is performed according to, for example, the Bluetooth (registered trademark) standard.

[0081] The left controller 3 also includes a memory 102, such as a flash memory. The communication control unit 101 is configured with, for example, a microcomputer (also called a microprocessor), and executes firmware stored in the memory 102 to perform various processes.

[0082] The left controller 3 includes buttons 103 (specifically, buttons 33 to 39, 43, 44, and 47). The left controller 3 also includes an analog stick (referred to as "stick" in FIG. 7) 32. Each button 103 and analog stick 32 repeatedly outputs information relating to operations performed on them to the communication control unit 101 at appropriate timing.

[0083] The communication control unit 101 acquires information related to the input (specifically, information related to the operation, or the detection results by the sensors) from each input unit (specifically, each button 103, analog stick 32, each sensor 104 and 105). The communication control unit 101 transmits operation data including the acquired information (or information obtained by performing a specified process on the acquired information) to the main unit 2. The operation data is repeatedly transmitted once every specified time. The interval at which the information related to the input is transmitted to the main unit 2 may or may not be the same for each input unit.

[0084] By transmitting the above operation data to the main unit 2, the main unit 2 can obtain the input performed on the left controller 3. In other words, the main unit 2 can determine the operations performed on each button 103 and analog stick 32 based on the operation data.

[0085] The left controller 3 is equipped with a power supply unit 108. In this embodiment, the power supply unit 108 has a battery and a power control circuit. Although not shown, the power control circuit is connected to the battery and to each part of the left controller 3 (specifically, each part that receives power from the battery).

[0086] As shown in FIG. 7, the right controller 4 includes a communication control unit 111 that communicates with the main unit 2. The right controller 4 also includes a memory 112 that is connected to the communication control unit 111. The communication control unit 111 is connected to each component including the terminal 64. The communication control unit 111 and the memory 112 have the same functions as the communication control unit 101 and the memory 102 of the left controller 3. Therefore, the communication control unit 111 can communicate with the main unit 2 by both wired communication via the terminal 64 and wireless communication (specifically, communication in accordance with the Bluetooth (registered trademark) standard) that does not go through the terminal 64, and controls the method of communication that the right controller 4 uses with the main unit 2.

[0087] The right controller 4 has input units similar to those of the left controller 3. Specifically, it has buttons 113 and an analog stick 52. These input units have the same functions as those of the left controller 3, and operate in the same manner.

[0088] The right controller 4 is equipped with a power supply unit 118. The power supply unit 118 has the same functions as the power supply unit 108 of the left controller 3, and operates in the same manner.

[0089] Next, with reference to Figs. 8 to 15, a goggle device 150, which is an example of a device constituting an image display system by wearing the game system 1 (specifically, the main body device 2), will be described. Fig. 8 is a perspective view showing an example of the appearance of the goggle device 150. Fig. 9 is a front view showing an example of a state in which the main body device 2 is worn on the goggle device 150. Fig. 10 is a front view showing an example of a state in which the main body device 2 is worn on the goggle device 150. Fig. 11 is a diagram showing an example of a shape of a front contact portion 151b that comes into contact with a part of the front surface of the main body device 2. Fig. 12 is a diagram showing an example of an internal structure of the goggle device 150. Fig. 13 is a side view showing an example of a state in which the main body device 2 is worn on the goggle device 150. Fig. 14 is a diagram showing an example of a user viewing an image displayed on the image display system. Fig. 15 is a diagram showing an example of a user holding the image display system. Note that Figure 11 is a view from the same direction as Figure 10, with parts of the goggle device 150 (part of the main body 151, lens frame member 152, lens 153, and plate-shaped member 154) removed (transparent state) in order to clearly show the front abutment portion 151b.

[0090] 8 to 13, the goggle device 150 has a main body 151, a lens frame member 152, a lens 153, and a plate-shaped member 154. Here, the goggle device, which is an example of a device constituting an image display system, is not limited to the configuration described below as long as it is worn by fitting the user's face so as to cover the user's left and right eyes, and has a function of blocking at least a part of external light and a function of supporting the user's stereoscopic vision by a pair of lenses. For example, the goggle device may be used in various states, such as a type that fits the user's face by being held by the user, a type that fits the user's face by being fixed to the user's head, and a type in which the user looks into the device in a placed state. In addition, the goggle device may function as a so-called head-mounted display by being worn on the user's head with the main body device 2 attached, and may have a helmet-type shape in addition to a goggle-type shape. In the following description of the goggle device 150, a goggle-type goggle device that is held by the user and worn to fit the user's face is used.

[0091] The main body 151 has an attachment section that detachably fixes the main body device 2 by contacting the front, rear, top, and bottom of the main body device 2. The attachment section has a front contact section that contacts a part of the front surface (the surface on which the display 12 is provided) of the main body device 2, a rear contact section that contacts the rear surface of the main body device 2, an upper contact section that contacts the top surface of the main body device 2, and a lower contact section that contacts the bottom surface of the main body device 2, and is formed in a rectangular tube shape with both left and right side surfaces open and with a gap formed by being surrounded by the front contact section, rear contact section, upper contact section, and lower contact section. The attachment section has both side surfaces (the side surface on the x-axis positive direction side and the side surface on the x-axis negative direction side in the figure) open so that it can be attached from the left side surface side or the right side surface side of the main body device 2. 9, when the goggle device 150 is attached from the opening on the right side of the main device 2, the front contact portion contacts the front surface of the main device 2, the rear contact portion contacts the rear surface of the main device 2, the upper contact portion contacts the upper surface of the main device 2, and the lower contact portion contacts the lower surface of the main device 2. Note that, as shown in FIG. 11, the front contact portion 151b of the main body 151 has an opening formed therein so as not to obstruct at least the view of the images (left eye image and right eye image) displayed on the display 12 when the main device 2 is attached.

[0092] 9 and 13, the main unit 2 is attached to the goggle device 150 by sliding the main unit 2 from the left or right side of the main unit 2 into the gap of the mounting portion of the main body 151 along the front abutment portion, rear abutment portion, upper abutment portion, and lower abutment portion of the mounting portion. The main unit 2 can be removed from the goggle device 150 by sliding the main unit 2 left or right along the front abutment portion, rear abutment portion, upper abutment portion, and lower abutment portion of the mounting portion from the state where the main unit 2 is attached to the goggle device 150. In this manner, the main unit 2 can be detachably attached to the goggle device 150.

[0093] The lens frame member 152 is fixed to the opening formed on the front surface of the main body 151. The lens frame member 152 has a pair of lens frames that are opened so as not to obstruct the field of view of the display images (left eye image IML and right eye image IMR) displayed on the display 12 of the main body device 2 attached to the main body 151. Furthermore, the outer edges formed on the top, bottom, left and right sides of the lens frame member 152 are formed with bonding surfaces for bonding to the main body device 2, and a V-shaped recess for contacting the nose of a user wearing the goggle device 150 is formed in the center of the outer edge formed on the lower side.

[0094] The lens 153 is composed of a pair of left-eye lenses 153L and right-eye lenses 153R, and is, for example, a pair of Fresnel lenses. The left-eye lenses 153L and right-eye lenses 153R are fitted into the lens frames of the lens frame member 152. Specifically, the left-eye lens 153L is fitted into one of the lens frames that is opened so as not to obstruct the field of view of the left-eye image IML displayed on the display 12 of the main unit device 2 attached to the main body 151, and when the user looks into the left-eye lens 153L with his left eye, the left-eye image IML can be seen. The right-eye lens 153R is fitted into the other lens frame that is opened so as not to obstruct the field of view of the right-eye image IMR displayed on the display 12 of the main unit device 2 attached to the main body 151, and when the user looks into the right-eye lens 153R with his right eye, the right-eye image IMR can be seen. The left eye lens 153L and the right eye lens 153R may be typically circular or elliptical magnifying lenses, and may be lenses that distort an image and allow a user to view it. For example, the left eye lens 153L may distort a left eye image IML (described later) displayed in a circular or elliptical shape in a direction opposite to the distortion of the image, and the right eye lens 153R may distort a right eye image IMR (described later) displayed in a circular or elliptical shape in a direction opposite to the distortion of the image, thereby allowing a user to view the image stereoscopically. The left eye lens 153L and the right eye lens 153R may also be configured to be integrally formed.

[0095] The main body 151 has an abutment portion provided so as to protrude outward from the front side of the main body 151 so as to surround the outer edge of the lens frame member 152 in a rectangular cylindrical shape. An end face of the abutment portion protruding outward from the front side is disposed on the near side of the lens 153 when the lens 153 is viewed from outside the goggle device 150, and the end face is disposed on the near side (z-axis negative direction side) of the goggle device 150 when the main body device 2 is worn. The abutment portion of the main body 151 has an end face shape that fits the user's face (typically, the area around the user's eyes) when the user looks into the goggle device 150 with the main body device 2 worn, and has a function of fixing the positional relationship between the user's eyes and the lens 153 by the end face abutting against the user's face.

[0096] In addition, the contact portion can block external light to the left eye lens 153L and the right eye lens 153R when viewing a stereoscopic image displayed on the display 12 using an image display system. This can improve the immersive feeling of the user viewing the stereoscopic image displayed on the display 12. The light blocking by the contact portion does not have to completely block external light. For example, as shown in FIG. 15, a recess may be formed in a part of the contact portion configured in a cylindrical shape. The recess of the contact portion illustrated in FIG. 15 is formed at a position that is a lower part of the midpoint between the left eye lens 153L and the right eye lens 153R, which is a position where the nose of a user viewing a stereoscopic image displayed on the display 12 will contact. That is, the recess of the contact portion can prevent the contact portion from contacting the user's nose strongly, and even if the light blocking ability is somewhat deteriorated, the discomfort caused by the contact portion and the nose can be reduced.

[0097] As shown in FIG. 10, the plate-like member 154 is fixed inside the main body 151 between the lens frame member 152 and the display 12 when the main body device 2 is attached to the attachment portion of the main body 151. For example, the plate-like member 154 is arranged like a wall (hereinafter, referred to as a first wall portion) that has a shape such that a part of the plate-like member 154 fits the V-shaped recess in the lens frame member 152 and connects the recess and the display 12 of the attached main body device 2. The space surrounded by the first wall portion is an opening 154h that exposes a part of the display 12 of the main body device 2 attached to the main body 151 to the outside and functions as an operation window that allows a user to touch the part through the space. Note that the first wall portion of the plate-like member 154 may be partially open as shown in FIG. 12.

[0098] 12, the plate-like member 154 is, for example, vertically erected between the left-eye lens 153L and the right-eye lens 153R, and is arranged as a wall (hereinafter, referred to as a second wall portion) connecting the recess and the display 12 of the attached main body device 2. When the main body device 2 is attached to the main body 151, the second wall portion is arranged so as to separate the image between the left-eye image IML and the right-eye image IMR displayed on the display 12, and functions as a partition wall provided between the left-eye image IML and the right-eye image IMR. The plate-like member 154 is provided by extending the first wall portion to the second wall portion, and the first wall portion and the second wall portion are formed by an integrated member.

[0099] 10, 13, 14, and 15, the image display system is configured by mounting the main unit 2 on the goggle device 150. Here, in this embodiment, the main unit 2 is mounted so that the entire main unit 2 is covered by the goggle device 150. When the main unit 2 is mounted on the goggle device 150, it is possible to view only the left-eye image IML displayed in the left region of the display 12 through the left-eye lens 153L, and only the right-eye image IMR displayed in the right region of the display 12 through the right-eye lens 153R. Therefore, a user of the image display system can view the left-eye image IML and the right-eye image IMR by viewing the left-eye lens 153L with the left eye and the right-eye lens 153R with the right eye, and therefore a stereoscopic image that gives a sense of stereoscopic feeling to the user can be displayed by displaying the left-eye image IML and the right-eye image IMR with parallax on the display 12.

[0100] 14 and 15, when a user holds an image display system in which main unit 2 is attached to goggle device 150 and watches a stereoscopic image displayed on display 12, the user can hold the left side of goggle device 150 to which main unit 2 is attached with the left hand and hold the right side of goggle device 150 with the right hand. By the user holding the left and right side parts of goggle device 150 in this manner, main unit 2 can be kept stably attached.

[0101] Furthermore, even when the main body device 2 is attached to the goggle device 150, the image display system can touch and operate a part of the touch panel 13 provided on the screen of the display 12 (a third region of the display 12, which will be described later) through the opening 154h formed by being surrounded by the first wall portion of the plate-like member 154. Furthermore, the image display system can calculate information on the movement and / or attitude of the main body device 2, that is, the movement and / or attitude of the image display system including the goggle device 150, based on the detection result of the acceleration sensor 89 and / or the angular velocity sensor 90 provided in the main body device 2. Therefore, the image display system can calculate the attitude of the user's head looking into the goggle device 150 on which the main body device 2 is attached, based on the gravity direction. Furthermore, when the attitude or direction of the head of the user looking into the goggle device 150 on which the main body device 2 is attached changes, the image display system can calculate the changed direction or angle. Furthermore, when a user looking into the goggle device 150 on which the main body device 2 is attached applies a vibration as if hitting the image display system, the image display system can detect the vibration. Therefore, when the main device 2 is attached to the goggle device 150 and a stereoscopic image displayed on the display 12 is viewed through the left eye lens 153L and the right eye lens 153R, a play style is realized in which touch operations through the opening 154h, operations based on the attitude of the image display system based on the gravity direction, operations to change the attitude of the image display system, and operations to apply vibration to the image display system are possible.

[0102] When using the image display system in this embodiment, at least one of the left controller 3 and the right controller 4 detached from the main unit 2 may be used for operation. For example, when operating the image display system using the left controller 3, the user holds the goggle device 150 to which the main unit 2 is attached with the right hand while viewing the stereoscopic image displayed on the display 12, and holds the detached left controller 3 alone with the left hand to perform operation. In this case, operation information operated on the left controller 3 and / or the right controller 4 detached from the main unit 2 is transmitted to the main unit 2 by wireless communication with the main unit 2. Specifically, operation information operated on the left controller 3 is wirelessly transmitted from the communication control unit 101 of the left controller 3 and received by the controller communication unit 83 of the main unit 2. Operation information operated on the right controller 4 is wirelessly transmitted from the communication control unit 111 of the right controller 4 and received by the controller communication unit 83 of the main unit 2.

[0103] In this way, in this embodiment, by attaching the main unit 2 to the goggle device 150, a portable image display system that the user can hold and view a stereoscopic image can be configured. In addition, in the image display system of this embodiment, the user views a stereoscopic image displayed on the display 12 of the main unit 2 while placing the user's face against the goggle device 150, so that the positional relationship between the stereo speakers (left speaker 88L and right speaker 88R) provided in the main unit 2 and the user's ears is also fixed, and the left and right speakers are disposed near the user's ears. Therefore, the main unit 2 can output sound based on the positional relationship between the sound output device and the viewer's ears without forcing the use of earphones or speakers. For example, the main unit 2 can control a sound source using so-called stereophonic technology based on the positional relationship between the sound output device and the viewer's ears.

[0104] Next, images displayed on the main unit 2 will be described with reference to Fig. 9, Fig. 10, and Fig. 16. Fig. 16 shows examples of images displayed on the main unit 2 in the stereoscopic display mode and the non-stereoscopic display mode.

[0105] The image display system in this embodiment is set to either a stereoscopic display mode used when the main device 2 is attached to the goggle device 150 and an image displayed on the display 12 is stereoscopically viewed, or a non-stereoscopic display mode used when the main device 2 is detached from the goggle device 150 and an image displayed on the display 12 is viewed directly for non-stereoscopic viewing. The image display system then displays an image corresponding to the set mode on the display 12 of the main device 2. Here, the stereoscopic image for stereoscopic viewing may be one that the user views stereoscopically by viewing a right-eye image and a left-eye image having parallax with each other with the right eye and the left eye. In this case, the non-stereoscopic image that is viewed non-stereoscopically is an image other than the above-mentioned two-image display (stereoscopic display), and may typically be one in which the user views a single image with the right eye and the left eye. In this embodiment, the non-stereoscopic display mode is used as an example of the first display mode. In this embodiment, the stereoscopic display mode is used as an example of the second display mode.

[0106] In the stereoscopic display mode, the image display system configures a content image to be displayed (for example, an image for displaying a part of a virtual space or a real space) by a left-eye image IML and a right-eye image IMR having a parallax therebetween, displays the left-eye image IML in a left region of the display 12, and displays the right-eye image IML in a right region of the display 12. Specifically, as shown in FIG. 9, in the stereoscopic display mode, the left-eye image IML is a substantially elliptical region viewable by the left-eye lens 153L when the main device 2 is attached to the goggle device 150, and is displayed in a first region that is a part of the left region of the display 12. Also, in the stereoscopic display mode, the right-eye image IMR is a substantially elliptical region viewable by the right-eye lens 153R when the main device 2 is attached to the goggle device 150, and is displayed in a second region that is a part of the right region of the display 12.

[0107] Here, as described above, when the main body device 2 is attached to the goggle device 150, the second wall portion of the plate-shaped member 154 is disposed between the left eye image IML displayed in the first area and the right eye image IMR displayed in the second area of ​​the display 12. Therefore, the left eye image IML and the right eye image IMR are separated by the second wall portion of the plate-shaped member 154 as a partition, so that it is possible to prevent the right eye image IMR from being viewed through the left eye lens 153L or the left eye image IML from being viewed through the right eye lens 153R.

[0108] As an example, images of the virtual space viewed from a pair of virtual cameras (left virtual camera and right virtual camera) arranged in the virtual space and having a parallax between them are generated as a left-eye image IML and a right-eye image IMR, respectively. The pair of virtual cameras are arranged in the virtual space so as to correspond to the attitude of the main unit 2 based on the direction of gravity in the real space. The pair of virtual cameras change their attitude in the virtual space so as to correspond to the change in attitude of the main unit 2 in the real space, and the line-of-sight direction of the virtual camera is controlled according to the attitude of the main unit 2. As a result, a user wearing the image display system can change the display range of the virtual space viewed stereoscopically by changing the attitude of the main unit 2 (image display system) as if looking around, and can look around the virtual space viewed stereoscopically, so that the user can experience being at the location of the virtual camera. In this embodiment, the main unit 2 matches the direction of gravitational acceleration acting on the main unit 2 with the direction of gravity in the virtual space acting on the virtual camera, and matches the amount of change in the attitude of the main unit 2 with the amount of change in the line-of-sight direction of the virtual camera. This increases the realism of the operation of looking around a stereoscopic virtual space depending on the orientation of the main unit 2.

[0109] The image display system also displays on the display 12 a user interface image IMU for receiving a touch operation on the touch panel 13 of the main unit 2. For example, the user interface image IMUa displayed in the stereoscopic display mode is displayed in a display area of ​​the display 12 that can be touched through the opening 154h of the goggle device 150. For example, as described above, the opening 154h is formed surrounded by the first wall of the plate-like member 154, and enables a touch operation to a part of the display 12 of the main unit 2 attached to the goggle device 150 (specifically, an area near the center of the lower part of the display 12) from a V-shaped recess in the lens frame member 152 that abuts against the user's nose. As an example, as shown in FIG. 9, the opening 154h enables a touch operation to a third area set below the display 12 between the first area and the second area of ​​the display 12, even when the main unit 2 is attached to the goggle device 150.

[0110] For example, in the stereoscopic display mode, two user interface images IMUa1 and IMUa2 are displayed in the third region of the display 12. As an example, the user interface image IMUa1 is an operation icon that, when the display position is touched via the touch panel 13, is an operation icon that is an operation instruction to retry the game from the beginning. Also, the user interface image IMUa2 is an operation icon that, when the display position is touched via the touch panel 13, is an operation icon that is an operation instruction to end the game. The two user interface images IMUa1 and IMUa2 are displayed in the third region of the display 12 side by side in a size that fits the shape of the third region. This allows the user to touch one of the two user interface images IMUa1 and IMUa2 through the opening 154h, thereby enabling multiple operation instructions by touch operation even when the main body device 2 is attached to the goggle device 150. The two user interface images IMUa1 and IMUa2 may be displayed near the third region that allows touch operation by exposing a part of the display 12 to the outside. That is, a portion of the two user interface images IMUa1 and / or IMUa2 may be displayed outside the third area. In this embodiment, the user interface image IMUa is used as an example of the second user interface image.

[0111] The two user interface images IMUa1 and IMUa2 are images displayed in the stereoscopic display mode, but are displayed in a third area outside the first area visible to the user's left eye and the second area visible to the user's right eye on the display 12 of the main unit 2 attached to the goggle device 150. Therefore, the user interface images IMUa1 and IMUa2 displayed in the third area are displayed outside the field of view of the user viewing through the goggle device 150, and are not composed of two images with parallax between them, so that they are displayed as non-stereoscopic images that cannot be viewed stereoscopically. In addition, since the user interface images IMUa1 and IMUa2 that are the subject of touch operations are displayed outside the first and second areas for displaying stereoscopic images, the first and second areas are less likely to be touched. Therefore, it is possible to prevent the first and second areas for displaying stereoscopic images from being soiled by touch operations on the display 12, and it is also possible to prevent fingers for touch operations from entering the field of view when viewing stereoscopic images.

[0112] In another embodiment, the user interface images IMUa1 and IMUa2 may be displayed on the display 12 as stereoscopic images that can be viewed stereoscopically in the stereoscopic display mode. In this case, the user interface images IMUa1 and IMUa2 are displayed on the display 12 as stereoscopic images by being composed of two images each having parallax with respect to each other, and typically, one image for stereoscopic viewing is displayed in a part of the first area, and the other image for stereoscopic viewing is displayed in a part of the second area.

[0113] As shown in Figure 16, in the non-stereoscopic display mode, the image display system constructs the content image to be displayed as a single image IMS, which is a non-stereoscopic image, and displays the single image IMS over the entire display area of ​​the display 12, as an example.

[0114] As an example, an image of a virtual space viewed from a single virtual camera arranged in the virtual space is generated as a single image IMS. The single virtual camera is arranged in the virtual space so as to correspond to the attitude of the main unit 2 based on the gravity direction in the real space. The single virtual camera changes its attitude in the virtual space so as to correspond to the change in attitude of the main unit 2 in the real space, and controls the line of sight of the virtual camera according to the attitude of the main unit 2. This allows a user holding the main unit 2 removed from the goggle device 150 to change the display range of the virtual space displayed on the display 12 by changing the attitude of the main unit 2 as if looking around, and to look around the virtual space, so that the user can have an experience as if they were actually at the location of the virtual camera. Note that in this embodiment, even in the non-stereoscopic display mode, the main unit 2 matches the direction of the gravitational acceleration acting on the main unit 2 with the direction of gravity in the virtual space acting on the virtual camera, and matches the amount of change in the attitude of the main unit 2 with the amount of change in the line of sight of the virtual camera. This increases the reality of the operation of looking around the non-stereoscopic virtual space depending on the attitude of the main unit 2. In this embodiment, a single image IMS is used as an example of a non-stereoscopic image.

[0115] Furthermore, the user interface image IMUb displayed in the non-stereoscopic display mode is displayed on the display 12, for example, superimposed on a content image (single image IMS) displayed on the display 12. For example, as shown in FIG. 16, two user interface images IMUb1 and IMUb2 are displayed on the display 12 even in the non-stereoscopic display mode. As an example, the user interface image IMUb1 is an image corresponding to the user interface image IMUa1, and is an operation icon that becomes an operation instruction to retry the game from the beginning when the display position is touched via the touch panel 13. Furthermore, the user interface image IMUb2 is an image corresponding to the user interface image IMUa2, and is an operation icon that becomes an operation instruction to end the game when the display position is touched via the touch panel 13. Here, the image corresponding to the user interface image IMUa indicates that the image has a different design and / or size but is substantially the same in function as the user interface image IMUa (for example, the operation instruction content by the touch operation is the same). Note that the image corresponding to the user interface image IMUa may not only be substantially identical in function to the user interface image IMUa displayed in the non-stereoscopic display mode, but may also be the same in design and size, i.e., completely identical. Note that in this embodiment, the user interface image IMUb is used as an example of the first user interface image.

[0116] The two user interface images IMUb1 and IMUb2 are displayed in corner regions (for example, the upper left corner region and the upper right corner region) of the display 12 that are different from the third region. Note that the two user interface images IMUb1 and IMUb2 are not limited by the region in which touch operations are possible, and therefore can be enlarged and displayed more than the user interface image IMUa displayed in the stereoscopic display mode, and can be displayed in a size and shape that is easy for the user to touch, and at a position where the visibility of the content image (single image IMS) is not likely to be impaired by touch operations.

[0117] The image display system in this embodiment can automatically switch between the stereoscopic display mode and the non-stereoscopic display mode based on a detection result of whether the main device 2 is in a state of being attached to the goggle device 150 or is in a state of being in the middle of being attached. For example, the main device 2 is provided with an illuminance sensor 29 that detects the illuminance (brightness) of light incident from the main surface side of the housing 11, and based on the illuminance detection result by the illuminance sensor 29, it can detect whether the main device 2 is in a state of being attached to the goggle device 150 or is in a state of being in the middle of being attached. Specifically, when the main device 2 is attached to the goggle device 150 or is in a state of being in the middle of being attached, the illuminance detected by the illuminance sensor 29 becomes dark, so that it is possible to detect whether the main device 2 is in a state of being attached or is in a state of being in the middle of being attached by setting a threshold value capable of detecting the darkened illuminance and detecting whether the illuminance is equal to or greater than the threshold value. Here, the "being in a state of being attached" detected by the main device 2 based on the illuminance detection result by the illuminance sensor 29 means that the main device 2 is in a state of being completely attached to the goggle device 150. In addition, based on the illuminance detection result by the illuminance sensor 29, the “in the middle of being attached state” detected by the main unit 2 means that the main unit 2 is in a state preceding the state where it is completely attached to the goggle device 150.

[0118] There are several possible modes for the illuminance sensor 29 to detect whether the main device 2 is in a worn state or in a half-wearing state. As a first example, as shown in Fig. 11, in a wearing section of the goggle device 150 that detachably fixes the main device 2, if a front abutment portion 151b of the main body 151 that comes into contact with a part of the front surface (the surface on which the display 12 is provided) of the main device 2 in the worn state is formed so as to cover the light receiving surface and light receiving hole of the illuminance sensor 29, whether the main device 2 is in a worn state or in a half-wearing state on the goggle device 150 is detected by the front abutment portion 151b covering the light receiving surface and light receiving hole of the illuminance sensor 29. As an example, if the light receiving surface or light receiving hole of the illuminance sensor 29 is provided at a corner of the main unit 2, when the main unit 2 is inserted into the mounting portion of the goggle device 150 from the corner side, the light receiving surface or light receiving hole is covered by the front abutment portion 151b at the beginning of the mounting operation, so that it is possible to detect that the main unit 2 is "in the middle of mounting" based on the illuminance detection result by the illuminance sensor 29. Also, when the main unit 2 is inserted into the mounting portion of the goggle device 150 from the side opposite to the corner where the light receiving surface or light receiving hole of the illuminance sensor 29 is provided, the light receiving surface or light receiving hole is covered by the front abutment portion 151b at the end of the mounting operation, so that it is possible to detect that the main unit 2 is "in the mounting state" based on the illuminance detection result by the illuminance sensor 29. As another example, in the case where the light receiving surface or light receiving hole of the illuminance sensor 29 is provided on the lower central side of the main unit 2, when the main unit 2 is inserted into the wearing portion of the goggle device 150, the light receiving surface or light receiving hole is covered by the front abutment portion 151b in the initial to middle stages of the wearing operation regardless of the direction from which the main unit 2 is inserted, so that it becomes possible to detect that the main unit 2 is "in the middle of being worn" based on the illuminance detection result by the illuminance sensor 29. In the above first example, the front abutment portion 151b of the main body 151 is used as an example of a light blocking portion.

[0119] As a second example, even if the front contact portion 151b is not formed so as to cover the light receiving surface or light receiving hole of the illuminance sensor 29, it is possible to detect whether the main unit 2 is in a state of being attached to the goggle device 150 or in a state of being in the middle of being attached by utilizing the fact that the inside of the main unit 151 is relatively dark. For example, a state in which external light is incident on the main unit 151 of the goggle device 150 through the left eye lens 153L and the right eye lens 153R is considered, and even in this state, the wearing state of the main unit 2 is detected by detecting that the illuminance is darker than the outside of the main unit 151. As an example, if the light receiving surface or light receiving hole of the illuminance sensor 29 is provided at a corner of the main unit 2, when the main unit 2 is inserted into the mounting portion of the goggle device 150 from the corner side, the light receiving surface or light receiving hole is inserted inside the main unit 151 at the beginning of the mounting operation, so that it is possible to detect that the main unit 2 is in a "state of being in the middle of being attached" based on the illuminance detection result by the illuminance sensor 29. In addition, when the main unit 2 is inserted into the mounting portion of the goggle device 150 from the side opposite to the corner where the light receiving surface and light receiving hole of the illuminance sensor 29 are provided, the light receiving surface and light receiving hole are inserted into the main unit 150 at the end of the mounting operation, so that it is possible to detect that the main unit 2 is "in a mounting state" based on the illuminance detection result by the illuminance sensor 29. As another example, when the light receiving surface and light receiving hole of the illuminance sensor 29 are provided on the lower center side of the main unit 2, when the main unit 2 is inserted into the mounting portion of the goggle device 150, the light receiving surface and light receiving hole are inserted into the main unit 151 at the beginning to middle of the mounting operation regardless of the direction from which the main unit 2 is inserted, so that it is possible to detect that the main unit 2 is "in the middle of mounting" based on the illuminance detection result by the illuminance sensor 29. In the above second example, the main unit 151 is used as another example of a light blocking portion.

[0120] As a third example, even if the front contact portion 151b is not formed to cover the light receiving surface or light receiving hole of the illuminance sensor 29, the fact that the inside of the main body 151 becomes darker when the user wears the goggle device 150 to which the main body 2 is attached is utilized to detect that the main body 2 is attached to the goggle device 150. For example, when the user is not wearing the goggle device 150, it is considered that external light enters the main body 151 of the goggle device 150 through the left eye lens 153L and the right eye lens 153R. In this case, the external light entering through the left eye lens 153L and the right eye lens 153R is blocked by the user wearing the goggle device 150, and the wearing state of the main body 2 can be detected by detecting that the illuminance inside the main body 151 becomes darker. As an example, when light incident on the light receiving surface or light receiving hole of the illuminance sensor 29 arranged in the main body 151 of the goggle device 150 is blocked by the main body 151 and the user's face, it is possible to detect that the main body device 2 is "in a worn state" based on the illuminance detection result by the illuminance sensor 29. Moreover, in this example, it is possible to detect not only that the main body device 2 is in a worn state on the goggle device 150, but also whether or not the user is wearing the goggle device 150 to which the main body device 2 is attached. In the above third example, the main body 151 is used as another example of a light blocking section.

[0121] When the image display system in this embodiment determines that the main unit 2 is not attached to the goggle device 150, it sets the display mode of the main unit 2 to a non-stereoscopic display mode. On the other hand, when it is determined that the main unit 2 has changed from a non-attached state to an attached state in the main unit 2 set to the non-stereoscopic display mode, the main unit 2 changes the displayed content image of the non-stereoscopic image to a stereoscopic image, thereby displaying the same content image on the display 12. As an example, when a single virtual camera is set in the virtual space to display a single image IMS in the non-stereoscopic display mode and a virtual space image is generated, the main unit 2 changes the single virtual camera to a pair of virtual cameras (left virtual camera and right virtual camera) having parallax with each other without changing the position and line of sight direction of the single virtual camera, and sets virtual cameras for displaying the left eye image IML and the right eye image IMR, thereby switching to generation of a virtual space image in the stereoscopic display mode. Furthermore, when it is determined that the main unit 2 has changed from a state where the main unit 2 is attached to the goggle device 150 to a state where the main unit 2 is not attached to the goggle device 150 in the stereoscopic display mode, the main unit 2 changes the content image of the displayed stereoscopic image to a non-stereoscopic image, thereby displaying the same content image corresponding to the content image of the stereoscopic image as a non-stereoscopic image on the display 12. As an example, when a pair of virtual cameras is set in the virtual space to display the left-eye image IML and the right-eye image IMR in the stereoscopic display mode and a virtual space image is generated, the main unit 2 changes the pair of virtual cameras to a single virtual camera without changing the position and line-of-sight direction, and sets a virtual camera for displaying the single image IMS, thereby switching to the generation of a virtual space image in the non-stereoscopic display mode. In this way, the content image (e.g., virtual space image) of a non-stereoscopic image corresponding to the content image (e.g., virtual space image) of a stereoscopic image, or the content image (e.g., virtual space image) of a stereoscopic image corresponding to the content image (e.g., virtual space image) of a non-stereoscopic image, shows that the only difference in either case is whether the image is a stereoscopic image or a non-stereoscopic image.However, the display ranges of a non-stereoscopic image content image corresponding to a stereoscopic image content image, or a stereoscopic image content image corresponding to a non-stereoscopic image content image, may be different, and typically the display range of a stereoscopic image content image may be narrower than that of a non-stereoscopic image content image.

[0122] Furthermore, when the display mode is switched, the image display system in this embodiment changes the size, shape, and position of the user interface image IMU and displays it on the display 12. For example, in the case of the main body device 2 set to the non-stereoscopic display mode, when it is determined that the main body device 2 has changed from a non-mounted state to a mounted state on the goggle device 150, the main body device 2 changes the shape of the user interface images IMUb1 and IMUb2, which are displayed overlapping with the content image in the corner region of the display 12, to user interface images IMUa1 and IMUa2, and moves the display position into the third region of the display 12, thereby displaying the user interface image IMU with the same function. In addition, when it is determined that the main unit 2, which is set to the stereoscopic display mode, has changed from a state in which it is attached to the goggle device 150 to a state in which it is not attached to the goggle device 150, the main unit 2 changes the shape of the user interface images IMUa1 and IMUa2 displayed in the third area of ​​the display 12 to user interface images IMUb1 and IMUb2, and moves the display position so that it overlaps with the content image in the corner area of ​​the display 12, thereby displaying the user interface image IMU with the same function.

[0123] In the above embodiment, an example is used in which it is detected whether the main device 2 is in a state of being attached to the goggle device 150 or in a state of being in the middle of being attached based on the illuminance detection result by the illuminance sensor 29. However, it may be detected whether the main device 2 is in a state of being attached to the goggle device 150 or in a state of being in the middle of being attached based on other detection results. As an example, it may be detected whether the main device 2 is in a state of being attached to the goggle device 150 or in a state of being in the middle of being attached based on a detection result of a connection terminal provided in the main device 2 and a connection terminal provided in the goggle device 150 being electrically connected by being in the above-mentioned attached state or the above-mentioned middle of being attached, or a detection result of a predetermined switch mechanism provided in the main device 2 being turned on or off by being in the above-mentioned attached state or the above-mentioned middle of being attached. As another example, it may be detected whether the main device 2 is in a state of being attached to the goggle device 150 or in a state of being in the middle of being attached by determining whether a predetermined image is captured or not based on an imaging result by an imaging means (imaging element) provided in the main device 2, or by determining whether an imaged luminance is equal to or greater than a threshold value. As another example, when the main unit 2 is attached to the goggle device 150 or in the process of being attached to the goggle device 150, the user may be prompted to perform a specified operation, and based on the specified operation being performed, it may be detected whether the main unit 2 is attached to the goggle device 150 or in the process of being attached to the goggle device 150.

[0124] In the above embodiment, the third area is set below the center of the display 12 sandwiched between the first area and the second area on the display 12, and a touch operation on the third area is enabled. However, the third area may be set in another area on the display 12. As a first example, the third area may be set above the display 12 above the center of the display 12 sandwiched between the first area and the second area on the display 12. As a second example, the third area may be set above (i.e., the upper left corner area of ​​the display 12) or below (i.e., the lower left corner area of ​​the display 12) between the first area and the left edge of the display 12. As a third example, the third area may be set above (i.e., the upper right corner area of ​​the display 12) or below (i.e., the lower right corner area of ​​the display 12) between the second area and the right edge of the display 12. Regardless of the region in which the third region is set, a user interface image IMUa that fits the shape of the third region is displayed in the third region, and an opening 154h that allows touch operation within the third region is formed in the goggle device 150, thereby enabling the same operation as described above. Note that when the third region is set between the first region and the second region sandwiched by the first region and the second region, the third region may be shifted to the left or right from the midpoint between the first region and the second region.

[0125] In the above-described embodiment, the function of the user interface image is, as an example, what is displayed on the display 12 to receive touch operations on the touch panel 13, but the image may not receive touch operations. For example, the user interface image may be text information or icons that present some information to the user but cannot be touched. For example, the user interface image may be an image showing instructions corresponding to button operations, stick operations, operations to move the controller body, etc. on the left controller 3 or the right controller 4.

[0126] The left eye image IML and the right eye image IMR may also be displayed outside the display area of ​​the display 12 visible through the left eye lens 153L and the right eye lens 153R (typically outside the first area and / or outside the second area), and may be partially displayed within the third area where touch operation is possible. The left eye image IML and the right eye image IMR may also be displayed in an area smaller than the display area of ​​the display 12 visible through the left eye lens 153L and the right eye lens 153R (typically the first area and / or the second area).

[0127] Next, an example of a specific process executed by the game system 1 in this embodiment will be described with reference to Fig. 17 to Fig. 18. Fig. 17 is a diagram showing an example of a data area set in the DRAM 85 of the main unit 2 in this embodiment. In addition to the data shown in Fig. 17, the DRAM 85 also stores data used in other processes, but detailed description thereof will be omitted.

[0128] The program storage area of ​​the DRAM 85 stores various programs Pa executed by the game system 1. In this embodiment, the various programs Pa store a communication program for wireless communication between the left controller 3 and the right controller 4 described above, and an application program for performing information processing (e.g., game processing) based on data acquired from the operation unit (left controller 3, right controller 4, touch panel 13, acceleration sensor 89, angular velocity sensor 90) and the illuminance sensor 29, etc. Note that the various programs Pa may be stored in advance in the flash memory 84, or may be acquired from a storage medium (e.g., a predetermined type of storage medium attached to the slot 23) that is detachable from the game system 1 and stored in the DRAM 85, or may be acquired from another device via a network such as the Internet and stored in the DRAM 85. The processor 81 executes the various programs Pa stored in the DRAM 85.

[0129] Furthermore, the data storage area of ​​the DRAM 85 stores various data used in communication processing, information processing, and other processing executed in the game system 1. In this embodiment, the DRAM 85 stores operation data Da, angular velocity data Db, acceleration data Dc, illuminance data Dd, posture data De, operation object data Df, virtual camera data Dg, left-eye virtual space image data Dh, right-eye virtual space image data Di, stereoscopic UI (user interface) image data Dj, non-stereoscopic virtual space image data Dk, non-stereoscopic UI (user interface) image data Dm, and image data Dn, etc.

[0130] The operation data Da is operation data appropriately acquired from the left controller 3 and / or the right controller 4 and the touch panel 13. As described above, the operation data transmitted from the left controller 3 and / or the right controller 4 includes information on input from each input unit (specifically, each button, analog stick, each sensor) (specifically, information on operation and detection results by each sensor). In this embodiment, the operation data is transmitted from the left controller 3 and / or the right controller 4 at a predetermined period by wireless communication, and the operation data Da is appropriately updated using the received operation data. The update period of the operation data Da may be updated every frame, which is the period of processing executed by the game system 1 described later, or may be updated every period in which the operation data is transmitted by the wireless communication. Furthermore, the operation data indicating that the touch panel 13 has been operated is acquired every period of the processing, and is stored in the operation data Da and updated in response to the acquisition.

[0131] The angular velocity data Db is data indicating an angular velocity occurring in the main unit 2 detected by the angular velocity sensor 90. For example, the angular velocity data Db includes data indicating an angular velocity occurring in the main unit 2 about the x, y and z axes.

[0132] The acceleration data Dc is data indicating the acceleration occurring in the main device 2 detected by the acceleration sensor 89. For example, the acceleration data Dc includes data indicating the acceleration occurring in the main device 2 in the x, y and z axis directions.

[0133] The illuminance data Dd is data indicating the illuminance around the main device 2 detected by the illuminance sensor 29.

[0134] The attitude data De is data that indicates the attitude in real space of the main unit 2. As an example, the attitude data De includes data that indicates an attitude that is used as a reference for a gravity vector that indicates gravitational acceleration occurring in the main unit 2, and data that indicates a change in attitude of the main unit 2.

[0135] The operation object data Df is data that indicates the position, direction, posture, motion, etc., in the virtual space of the object operated by the user.

[0136] The virtual camera data Dg is data that indicates the position, direction, viewing angle, magnification, etc. of a virtual camera (a pair of left and right virtual cameras in stereoscopic display mode, or a single virtual camera in non-stereoscopic display mode) set in the virtual space.

[0137] The left-eye virtual space image data Dh is data for generating a left-eye image IML in the stereoscopic display mode. The right-eye virtual space image data Di is data for generating a right-eye image IMR in the stereoscopic display mode. The stereoscopic UI image data Dj is data indicating the position, shape, size, etc. of the user interface image IMUa in the stereoscopic display mode.

[0138] The non-stereoscopic virtual space image data Dk is data for generating a single image IMS in the non-stereoscopic display mode. The non-stereoscopic UI image data Dm is data that indicates the position, shape, size, etc. of the user interface image IMUb in the non-stereoscopic display mode.

[0139] The image data Dn is data for displaying images (for example, images of virtual objects, user interface images, information images, field images, background images, etc.) on the display screen during a game.

[0140] Next, a detailed example of information processing (game processing) in this embodiment will be described with reference to Fig. 18. Fig. 18 is a flow chart showing an example of game processing executed by the game system 1. In this embodiment, the series of processes shown in Fig. 18 are performed by the processor 81 executing a communication program and a predetermined application program (game program) included in the various programs Pa. In addition, the game processing shown in Fig. 18 can be started at any timing.

[0141] Note that the processing of each step in the flowchart shown in FIG. 18 is merely an example, and the order of processing each step may be changed, or another processing may be performed in addition to (or instead of) the processing of each step, as long as the same result is obtained. In addition, in this embodiment, the processing of each step in the above flowchart is described as being executed by the processor 81, but the processing of some steps in the above flowchart may be executed by a processor other than the processor 81 or a dedicated circuit. In addition, some of the processing executed in the main unit 2 may be executed by another information processing device that can communicate with the main unit 2 (for example, a server that can communicate with the main unit 2 via a network). In other words, each processing shown in FIG. 18 may be executed by a plurality of information processing devices including the main unit 2 working together.

[0142] In FIG. 18, the processor 81 performs initial settings in the game processing (step S200) and proceeds to the next step. For example, in the initial settings, the processor 81 initializes parameters for performing the processing described below. As an example, the processor 81 uses the acceleration data stored in the acceleration data Dc to calculate the direction of the gravity vector of the gravitational acceleration acting on the main unit 2, sets an initial attitude of the main unit 2 based on the direction of the gravity vector, and updates the attitude data De. The processor 81 also sets an initial attitude of the virtual camera in the virtual space and updates the virtual camera data Dg so that the direction is the same as the relationship between the direction of the gravity vector of the gravitational acceleration acting on the main unit 2 and the x, y, and z-axis directions of the main unit 2. Here, being in the same direction as the x, y, and z-axes of the main unit 2 means that the z-axis positive direction (screen depth direction) based on the gravitational acceleration direction in the real space is the same as the line of sight direction of the virtual camera based on the gravitational acceleration direction in the virtual space, and the x-axis positive direction (screen left direction) based on the gravitational acceleration direction in the real space is the same as the left direction of the virtual camera based on the gravitational acceleration direction in the virtual space.

[0143] Next, the processor 81 acquires various data to update the operation data Da, angular velocity data Db, acceleration data Dc, and illuminance data Dd (step S201), and proceeds to the next step. For example, the processor 81 acquires operation data from the left controller 3 and / or the right controller 4 to update the operation data Da. The processor 81 also acquires touch operation data from the touch panel 13 to update the operation data Da. The processor 81 also acquires inertial data (acceleration data and angular velocity data) from the inertial sensors (acceleration sensor 89 and angular velocity sensor 90) provided in the main unit 2 to update the acceleration data Dc and angular velocity data Db. The processor 81 also acquires illuminance data from the illuminance sensor 29 to update the illuminance data Dd.

[0144] Next, the processor 81 calculates the attitude of the main unit 2 (step S202) and proceeds to the next step. For example, the processor 81 uses the acceleration data and angular velocity data stored in the angular velocity data Db and the acceleration data Dc to calculate the direction of the gravitational vector of the gravitational acceleration acting on the main unit 2, and updates the attitude data De. The processor 81 also calculates the rotation direction and rotation amount from the initial attitude of the main unit 2, and updates the attitude data De. For example, the processor 81 calculates the rotation direction and rotation amount around the x, y and z axis directions of the main unit 2 in the initial attitude, and updates the attitude data De. Note that the rotation direction can be represented by the positive and negative values ​​of the rotation amount, so that only data indicating the rotation amount may be stored in the attitude data De. For example, the processor 81 may add the rotation amount based on the angular velocity data acquired in the current step S202 to the rotation amount calculated in the previous process, and calculate it as a new rotation amount.

[0145] Next, the processor 81 performs a process of determining whether or not the main device 2 is attached to the goggle device 150 (step S203), and proceeds to the next step. For example, when the illuminance indicated by the illuminance data Dd indicates a dark illuminance less than a threshold for detecting the illuminance when the main device 2 is attached to the goggle device 150, the processor 81 determines in the above step S203 that the main device 2 is attached to the goggle device 150. The detection means detects whether or not the display device is attached to the goggle device, or detects whether or not it is in the middle of being attached, and corresponds to the processor 81 that performs the process of step S203, for example.

[0146] Next, the processor 81 determines whether or not the display mode is stereoscopic (step S204). For example, when the processor 81 determines that the main device 2 is attached to the goggle device 150 in the wearing determination process in the above step S203, the processor 81 makes a positive determination in the above step S204 and performs processing in the stereoscopic display mode. On the other hand, when the processor 81 determines that the main device 2 is not attached to the goggle device 150 in the wearing determination process in the above step S203, the processor 81 makes a negative determination in the above step S204 and performs processing in the non-stereoscopic display mode. Then, when the processor 81 performs processing in the stereoscopic display mode, the processor 81 advances the processing to step S205. On the other hand, when the processor 81 performs processing in the non-stereoscopic display mode, the processor 81 advances the processing to step S213. The mode setting means executes processing to set the display device to the first display mode or a second display mode different from the first display mode, and corresponds to the processor 81 performing the processing in step S204, for example.

[0147] In step S205, the processor 81 performs object movement processing and proceeds to the next step. For example, when an operation is performed to move a control object in a virtual space with reference to the operation data Da, the processor 81 sets the movement of the control object according to the operation. Then, based on the set movement of the control object, the processor 81 sets the position, direction, attitude, movement, etc. of the control object in the virtual space, and updates the control object data Df.

[0148] In the object operation process in step S205, the following object operation controls are possible. As a first example, a predetermined operation object is set as an operation target and the operation object is operated. In this case, the predetermined operation object is moved, operated, or deformed based on an input to each input unit in the left controller 3 and / or the right controller 4. As a second example, an operation object is selected as an operation target based on an operation, and the operation object is operated. In this case, an operation object arranged at a predetermined display position (for example, an operation object displayed overlapping with a sign displayed in the center of the display screen) is set as an operation target, and the operation object selected as the operation target is moved, operated, or deformed based on an input to each input unit in the left controller 3 and / or the right controller 4. As a third example, the operation object set as the operation target is operated based on vibration to the goggle device 150 to which the main unit 2 is attached. For example, the gravitational acceleration components are removed from the acceleration in the x, y and z-axis directions in the main unit 2 indicated by the acceleration data Dc, and when the acceleration after the removal indicates that a vibration of a predetermined magnitude or more is being applied to the main unit 2, the operation object that is the subject of the operation is moved in response to the vibration. Note that any method may be used to extract the gravitational acceleration, and for example, the acceleration components occurring on average in the main unit 2 may be calculated and extracted as the gravitational acceleration.

[0149] Next, the processor 81 performs a process of operating the pair of left and right virtual cameras (step S206), and proceeds to the next step. For example, the processor 81 sets the orientation of the pair of left and right virtual cameras in the virtual space by rotating the pair of left and right virtual cameras by the rotation amount calculated in step S202 from the initial orientation, and updates the virtual camera data Dg. For example, while fixing the positional relationship of the pair of left and right virtual cameras from the initial orientation, the processor 81 rotates the virtual camera around the left and right direction of the virtual camera as an axis by the same amount of rotation around the left and right axis direction (x-axis direction) of the main unit 2 calculated in step S202, rotates the virtual camera around the up and down direction of the virtual camera as an axis by the same amount of rotation around the up and down axis direction (y-axis direction) of the main unit 2 calculated in step S202, and rotates the virtual camera around the line of sight direction of the virtual camera as the same amount of rotation around the screen depth axis direction (z-axis direction) of the main unit 2 calculated in step S202, thereby setting the orientation of the pair of left and right virtual cameras in the virtual space.

[0150] Next, the processor 81 performs a process of generating a virtual space image for the left eye (step S207), and proceeds to the next step. For example, the processor 81 places an operation object in a virtual space based on the operation object data Df. Then, the processor 81 generates a virtual space image seen from the left virtual camera of the pair of left and right virtual cameras set by the virtual camera data Dg as a virtual space image for the left eye, and updates the virtual space image data Dh for the left eye.

[0151] Next, the processor 81 performs a process of generating a virtual space image for the right eye (step S208), and proceeds to the next step. For example, the processor 81 generates a virtual space image seen from the right virtual camera of the pair of left and right virtual cameras set by the virtual camera data Dg as a virtual space image for the right eye, and updates the virtual space image data Di for the right eye.

[0152] Next, the processor 81 performs a process of generating a stereoscopic user interface image (step S209), and proceeds to the next step. For example, the processor 81 generates a stereoscopic user interface image that matches the shape of the third area (see FIG. 9) of the display 12, and updates the stereoscopic UI image data Dj.

[0153] Next, the processor 81 performs a display control process to display the left-eye virtual space image in the first region of the display 12 (step S210), and proceeds to the next step. For example, the processor 81 displays the left-eye virtual space image set in the left-eye virtual space image data Dh as the left-eye image IML in the entire first region of the display 12 (see FIG. 9).

[0154] Next, the processor 81 performs a display control process to display the right-eye virtual space image in the second area of ​​the display 12 (step S211), and proceeds to the next step. For example, the processor 81 displays the right-eye virtual space image set in the right-eye virtual space image data Di as the right-eye image IMR in the entire second area (see FIG. 9) of the display 12.

[0155] Next, the processor 81 performs a display control process to display the stereoscopic UI image in the third region of the display 12 (step S212), and proceeds to step S219. For example, the processor 81 displays the user interface image set in the stereoscopic UI image data Dj as the user interface image IMUa (for example, two user interface images IMUa1 and IMUa2) at a predetermined position in the third region (see FIG. 9) of the display 12. Note that the second display control means executes a process to display, on the display screen in the second display mode, a second image including a content image consisting of a left eye image and a right eye image having parallax therebetween and a second user interface image corresponding to the first user interface image, and corresponds to the processor 81 performing the processes of steps S205 to S212, for example.

[0156] On the other hand, if it is determined in step S204 that the display mode is non-stereoscopic, in step S213, the processor 81 performs object movement processing and proceeds to the next step. For example, when an operation is performed to move a control object in a virtual space with reference to the operation data Da, the processor 81 sets the movement of the control object according to the operation. Then, based on the set movement of the control object, the processor 81 sets the position, direction, attitude, movement, etc. of the control object in the virtual space, and updates the control object data Df. Note that the object movement processing in step S213 is similar to the object movement processing in step S205 described above, and therefore a detailed description thereof will be omitted.

[0157] Next, the processor 81 performs a process of operating a single virtual camera (step S214), and proceeds to the next step. For example, the processor 81 sets the orientation of the single virtual camera in the virtual space by rotating it from the initial orientation by the rotation amount calculated in step S202, and updates the virtual camera data Dg. For example, the processor 81 sets the orientation of the single virtual camera in the virtual space by rotating the single virtual camera from the initial orientation by the left-right direction of the virtual camera as an axis by the same amount of rotation about the left-right axis direction (x-axis direction) of the main unit 2 calculated in step S202, rotating the single virtual camera by the same amount of rotation about the up-down axis direction (y-axis direction) of the main unit 2 calculated in step S202, and rotating the single virtual camera by the same amount of rotation about the line of sight direction of the virtual camera as the same amount of rotation about the screen depth axis direction (z-axis direction) of the main unit 2 calculated in step S202.

[0158] Next, the processor 81 performs a process of generating a virtual space image (step S215), and proceeds to the next step. For example, the processor 81 generates a virtual space image seen from a virtual camera set by the virtual camera data Dg, and updates the non-stereoscopic virtual space image data Dk.

[0159] Next, the processor 81 performs a process of generating a non-stereoscopic user interface image (step S216), and proceeds to the next step. For example, the processor 81 generates a non-stereoscopic user interface image to be superimposed on the single image IMS and displayed on the display 12 (see FIG. 16), and updates the non-stereoscopic UI image data Dm.

[0160] Next, the processor 81 performs a display control process to display the virtual space image in the entire area of ​​the display 12 (step S217), and proceeds to the next step. For example, the processor 81 displays the virtual space image set in the non-stereoscopic virtual space image data Dk as a single image IMS in the entire area of ​​the display 12 (see FIG. 16).

[0161] Next, the processor 81 performs a display control process for superimposing and displaying the non-stereoscopic UI image on the single image IMS (step S218), and proceeds to step S219. For example, the processor 81 superimposes and displays the user interface image set in the non-stereoscopic UI image data Dm as the user interface image IMUb (e.g., two user interface images IMUb1 and IMUb2) on the single image IMS in the upper left corner region and upper right corner region (see FIG. 16) of the display 12. Note that the first display control means executes a process for displaying a first image including a content image, which is a non-stereoscopic image, and a first user interface image on the display screen in the first display mode, and corresponds to the processor 81 performing the processes of steps S213 to S218, for example.

[0162] In step S219, the processor 81 performs a user interface operation process and proceeds to the next step. For example, when a touch operation is performed on the touch panel 13, the processor 81 refers to the operation data Da and sets a user operation instruction corresponding to the touch operation according to a user interface image displayed on the display 12 that overlaps with the position of the touch operation. Then, the processor 81 performs processing according to the set user operation instruction.

[0163] Next, processor 81 determines whether or not to end the game (step S220). Conditions for ending the game in step S220 include, for example, the result of the game being finalized or the user performing an operation to end the game. If processor 81 does not end the game, it returns to step S201 and repeats the process, and if it ends the game, it ends the process according to this flowchart. Thereafter, the series of processes from step S201 to step S220 is repeatedly executed until it is determined in step S220 that the game is to end.

[0164] In this way, in this embodiment, when switching between the normal display mode (non-stereoscopic display mode) and the stereoscopic display mode, the user interface image is displayed at a different position according to the set display mode, so that it is possible to improve the convenience of presenting the user interface image. Also, since the display mode is automatically switched based on the wearing state of the main body device 2 to the goggle device 150, seamless switching of the display mode is possible. Furthermore, in this embodiment, in the stereoscopic display mode, the user interface image is displayed in a third region different from the first region and the second region of the display 12, which respectively displays the right eye image and the left eye image, so that it is possible to display a user interface image that does not interfere with the stereoscopic image display.

[0165] In the above embodiment, the image display system is configured by mounting the main body device 2 having the information processing function on the goggle device 150, but the image display system may be configured in other ways. As a first example, a control unit that performs the above information processing (game processing) to generate an image may be provided in the goggle device 150, and a display device having a function of displaying an image may be mounted on the goggle device 150 to configure an image display system that displays a stereoscopic image. In this case, the control unit provided in the goggle device 150 outputs image data for displaying a stereoscopic image and a user interface image on the display device to the display device, so that the stereoscopic image and the user interface image are displayed on the display device. In the above embodiment of the first example, a mechanism for detecting the mounting state of the display device, a mechanism for detecting the attitude of the image display system, a mechanism for accepting user operations, and the like may be provided in either the display device or the goggle device 150. As a second example, a control device that is wirelessly or wired connected to the display device may be provided separately from the goggle device 150 on which the display device is mounted, and the image display system may be configured by the display device, the goggle device 150, and the control device. In this case, operation data, acceleration data, angular velocity data, illuminance data, etc. are output from the display device to the control device, and a content image and a user interface image in a display mode based on the illuminance data are output from the control device to the display device. In the second example, the mechanism for detecting the wearing state of the display device, the mechanism for detecting the attitude of the image display system, the mechanism for accepting user operations, etc. may be provided in either the display device or the goggle device 150. The control device controls the display range of the content image to be displayed according to the operation data, acceleration data, and angular velocity data acquired from the display device and / or the goggle device 150, and outputs the same to the display device. In this embodiment, the main unit 2 is used as an example of a display device.

[0166] In the above-described embodiment, the method for detecting the attitude of the main unit 2 is merely an example, and the attitude of the main unit 2 may be detected using other methods or other data. As an example, the main unit 2 and / or the goggle device 150 to which the main unit 2 is attached may be imaged from the outside, and the attitude of the main unit 2 and / or the goggle device 150 may be detected using the image thus captured. In addition, the controller for controlling the operation of the operation object may be not only the left controller 3 or the right controller 4, but also another controller.

[0167] Furthermore, the game system 1 and / or main unit 2 may be any type of device, such as a portable game device, any portable electronic device (PDA (Personal Digital Assistant), mobile phone, personal computer, camera, tablet, etc.), etc.

[0168] In the above-described embodiment, an example was used in which a stereoscopic image is displayed by displaying a left-eye image and a right-eye image having a parallax on the left and right screens of the display 12, respectively. However, the left-eye image and the right-eye image may be displayed on different screens. For example, when the display 12 provided on the main unit 2 is composed of a plurality of display screens, the left-eye image is displayed on one of the plurality of display screens, and the right-eye image is displayed on the other of the plurality of display screens. In this case, the user can view a stereoscopic image through the goggle device 150 by viewing the left-eye image displayed on one of the plurality of display screens through the left-eye lens 153L with his left eye and viewing the right-eye image displayed on the other of the plurality of display screens through the right-eye lens 153R with his right eye.

[0169] Moreover, the stereoscopic image and the non-stereoscopic image displayed on the main unit 2 are displayed as game images by the processor 81 executing information processing (game processing) in response to a user operation, or as moving images and still images by the processor 81 performing moving image playback and still image playback in response to a user operation. That is, the stereoscopic image and the non-stereoscopic image displayed on the main unit 2 are generated by the processor 81 of the main unit 2 performing information processing (e.g., game processing, moving image playback processing, still image playback processing), but at least a part of the processing for generating the stereoscopic image and the non-stereoscopic image may be performed by another device. For example, if the main unit 2 is further configured to be able to communicate with another device (e.g., a server, another image display device, another game device, another mobile terminal, another information processing device), the above processing may be further performed by the other device working in cooperation. In this way, by performing at least a part of the above processing by another device, processing similar to the above processing is possible. Moreover, the above information processing can be performed by one processor or by cooperation between multiple processors included in an information processing system configured by at least one information processing device. In addition, in the above embodiment, the processor 81 of the main unit 2 is capable of performing information processing by executing a specified program, but some or all of the above processing may be performed by a dedicated circuit provided in the main unit 2.

[0170] According to the above-mentioned modified example, the present invention can be realized in a so-called cloud computing system form or a distributed wide area network and local network system form. For example, in a distributed local network system form, the above-mentioned processing can be executed by cooperation between a stationary information processing device (stationary game device) and a portable information processing device (portable game device). In these system forms, there is no particular limitation on which device performs the above-mentioned processing, and it goes without saying that the present invention can be realized regardless of the division of processing load.

[0171] Furthermore, the processing sequence, setting values, conditions used for judgment, and the like used in the above-described information processing are merely examples, and it goes without saying that this embodiment can be realized even with other sequences, values, and conditions.

[0172] The program may be supplied to the game system 1 through an external storage medium such as an external memory, or may be supplied to the device through a wired or wireless communication line. The program may be pre-recorded in a non-volatile storage device inside the device. The information storage medium for storing the program may be a non-volatile memory, a CD-ROM, a DVD, or an optical disk-shaped storage medium similar to these, a flexible disk, a hard disk, a magneto-optical disk, a magnetic tape, or the like. The information storage medium for storing the program may be a volatile memory for storing the program. Such a storage medium may be a recording medium that can be read by a computer or the like. For example, the various functions described above can be provided by having a computer or the like read and execute the programs from these recording media.

[0173] Although the present invention has been described in detail above, the above description is merely illustrative of the present invention in all respects and is not intended to limit its scope. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. It is understood that the scope of the present invention should be interpreted only by the claims. It is also understood that a person skilled in the art can implement an equivalent scope based on the description of the present invention and technical common sense from the description of specific examples of the present invention. It should also be understood that the terms used in this specification are used in the sense commonly used in the field unless otherwise specified. Therefore, unless otherwise defined, all technical terms and technical terms used in this specification have the same meaning as commonly understood by a person skilled in the art to which this invention belongs. In the event of a conflict, this specification (including definitions) shall prevail. [Industrial Applicability]

[0174] INDUSTRIAL APPLICABILITY As described above, the present invention can be used as an image display system, an image display program, an image display method, a display device, and the like that can improve the convenience of presenting a user interface image. [Explanation of symbols]

[0175] 1. Game System 2. Main unit 3. Left controller 4…Right controller 11. Housing 12…Display 29…Illuminance sensor 81…Processor 83...Controller communication section 85…DRAM 150…Goggle device 151...Main body 152...Lens frame member 153…Lens 154...Plate-shaped member

Claims

1. An image display system including a display device having a display screen for displaying an image and a goggle-type device on which the display device can be worn, a mode setting means for setting the display device to a first display mode or a second display mode different from the first display mode; a first display control means for causing the display screen to display a first image including a content image, which is a non-stereoscopic image, and a first user interface image in the first display mode; a second display control means for displaying, on the display screen in the second display mode, a second image including a content image including a left-eye image and a right-eye image having parallax therebetween, and a second user interface image which is a non-stereoscopic image at least in a position different from that of the first user interface image; the second display control means, in the second display mode, causes the second user interface image to be displayed in a third area of ​​the display screen excluding an area constituted by a first area on the display screen for displaying the left eye image and a second area on the display screen for displaying the right eye image; the goggle-type device has an opening for exposing to the outside the second user interface image included in the third area, which is a part of the display screen, when the display device is attached to the goggle-type device; The opening is formed at a position corresponding to a user's nose when the user wears the goggle-type device.

2. The present invention further includes a detection means for detecting whether the display device is in a mounted state on the goggle-type device or in a partially mounted state, 2. The image display system according to claim 1, wherein the mode setting means switches the display device to the second display mode based on a detection result of the detection means when the display device is set to the first display mode.

3. The display device includes an illuminance sensor, the goggle-type device includes a light blocking unit that blocks light from reaching the illuminance sensor of the display device when the display device is in a state of being worn or in the process of being worn on the goggle-type device; 3. The image display system according to claim 2, wherein the detection means detects whether the display device is attached to the goggle-type device or is in the process of being attached based on a detection result by the illuminance sensor.

4. the display device includes a touch panel provided on the display screen, the first user interface image displayed on the display screen can provide an operation instruction in response to a touch operation on the touch panel; 4. The image display system according to claim 1, wherein the second user interface image displayed on the display screen is capable of providing operational instructions in response to a touch operation on the touch panel when the display device is attached to the goggle-type device.

5. 5. The image display system according to claim 1, wherein the second display control means displays on the display screen an image corresponding to a content image displayed as a non-stereoscopic image by the first display control means as a content image consisting of an image for the left eye and an image for the right eye.

6. 6. The image display system according to claim 1, wherein the second display control means, in the second display mode, converts a content image that was displayed in the first display mode immediately before being set to the second display mode into a stereoscopic image consisting of an image for the left eye and an image for the right eye, and displays the stereoscopic image on the display screen.

7. 7. The image display system according to claim 1, wherein the second display control means displays, as the second user interface image, a user interface image that is substantially the same in function as the first user interface image but has a different display form.

8. The image display system according to claim 7 , wherein the second display control means adjusts a shape of the second user interface image so as to fit a shape of the third area and displays the second user interface image in the third area.

9. 9. The image display system according to claim 1, wherein the second display control means sets the third area below the display screen between the first area and the second area on the display screen.

10. 10. The image display system according to claim 1, wherein the first display control means, in the first display mode, displays the first user interface image so as to be superimposed on a content image displayed on the display screen.

11. The display device further includes a display device side connection section that can be electrically connected to another device, the goggle-type device includes a goggle-type device-side connection part that is electrically connectable to the display device-side connection part, The image display system of claim 2, wherein the detection means detects whether the display device is attached to the goggle-type device or is in the process of being attached in response to the connection between the display device side connection unit and the goggle-type device side connection unit.

12. 1. An image display method for displaying an image on a display device that has a display screen for displaying an image and is attachable to a goggle-type device, comprising: a mode setting step of setting the display device to a first display mode or a second display mode different from the first display mode; a first display control step of displaying a first image including a content image, which is a non-stereoscopic image, and a first user interface image on the display screen in the first display mode; a second display control step of displaying, on the display screen in the second display mode, a second image including a content image including a left-eye image and a right-eye image having parallax therebetween, and a second user interface image which is a non-stereoscopic image at least in a position different from that of the first user interface image; In the second display control step, in the second display mode, the second user interface image is displayed in a third area of ​​the display screen excluding an area configured by a first area on the display screen for displaying the left eye image and a second area on the display screen for displaying the right eye image, the third region is configured such that, when the display device is attached to the goggle-type device, the second user interface image is exposed to the outside through an opening of the goggle-type device; The image display method, wherein the opening is formed at a position corresponding to a user's nose when the user wears the goggle-type device.

13. A goggle-type device on which a display device can be attached, the display device has a display screen on a front surface for displaying a content image including a left-eye image and a right-eye image having a parallax therebetween, and a user interface image; The goggle-type device comprises: a left-eye lens located in front of the left-eye image when the display device is worn; a right-eye lens located in front of the right-eye image when the display device is worn; an opening portion that is located on a front side of the user interface image when the display device is attached and exposes the user interface image to the outside; A goggle-type device, wherein the opening is formed at a position corresponding to a user's nose when the user wears the goggle-type device.

Images