Head-mounted display

Abstract

To provide a technique that enables appropriate placement of a virtual object in a real space with less effort of a user in an easy-to-handle manner.SOLUTION: A head-mounted display device (HMD device) according to an embodiment offers a function to place and display a virtual object in a space according to a user operation. The HMD device displays grids including multiple points on a display surface to support operations for virtual objects, and is configured to place and display a target virtual object at a first point in response to an operation including specifying the target virtual object and specifying the first point as a placement destination.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to display device technology such as a head mounted display (HMD), and also to technology for placing virtual objects in a scene in real space in relation to virtual reality (VR), augmented reality (AR), mixed reality (MR), and the like. [Background technology]

[0002] The performance of display devices such as HMDs, including smart glasses, is improving. HMDs can display virtual objects (sometimes called virtual images) superimposed on real objects (corresponding real images) in the real-world scene seen from the user's point of view. Images include still images and videos.

[0003] An example of prior art related to the above-mentioned display device is Japanese Patent Application Laid-Open No. 2018-49629 (Patent Document 1). Patent Document 1 describes a method for assisting input in a virtual space, which facilitates object placement, and the following: This method displays a virtual space on the monitor of an HMD, places an object to be placed and a guide object (e.g., a grid) in the virtual space, moves the guide object back and forth in conjunction with the movement of a hand object, and places the object in a specified location. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2018-49629 Summary of the Invention [Problem to be solved by the invention]

[0005] In recent years, HMDs have expanded the space in which virtual objects can be placed and displayed. Therefore, it is desirable for HMDs to be equipped with functions that support user operations related to the placement of virtual objects. Conventional HMDs can require significant or difficult user operations when placing virtual objects in space, leaving room for improvement in terms of usability and support. In particular, conventional HMDs require significant time and effort when placing multiple virtual objects on the display surface seen from the user's viewpoint. They also have limitations on the number of objects that can be placed, and even if multiple objects can be placed, they can be difficult to see and perform tasks.

[0006] The method of Patent Document 1 displays grid lines as guide objects in a virtual space, which are used by the user as a guide for placing virtual objects. This method realizes the placement of virtual objects relative to the grid lines by moving the virtual objects using a hand object (a virtual object for movement operation that resembles a hand). This method is used, for example, as a guide for stacking boxes in a game.

[0007] The present invention provides a technology for display devices such as HMDs that allows users to place virtual objects in real space with minimal effort, ease of use, and optimal placement. Other issues and advantages will be described in the description of the invention. [Means for solving the problem]

[0008] A representative embodiment of the present invention has the following configuration: A head-mounted display device in one embodiment is a head-mounted display device that positions and displays virtual objects in space based on user operation, displays a grid including a plurality of points to support the operation of the virtual objects on a display surface, and positions and displays the target virtual object at the position of the first point in response to an operation that includes designating a target virtual object and designating a first point at which the target virtual object is to be placed. [Effects of the Invention]

[0009] According to a representative embodiment of the present invention, with regard to display device technology such as an HMD, virtual objects can be placed in real space with minimal effort for the user, with good usability, and in an optimal manner. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram showing a configuration of a display system including an HMD device according to a first embodiment of the present invention. [Figure 2] 1 is a diagram showing an example of the external configuration of an HMD device according to a first embodiment. [Figure 3] 1 is a diagram showing a functional block configuration of an HMD device according to a first embodiment. [Figure 4] 3A to 3C are diagrams showing examples of displaying real objects and virtual objects on a display screen in the first embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of the configuration of an OS, applications, etc. in the first embodiment. [Figure 6] 2 is a diagram showing a configuration of processing units and data that constitute the basic functions of the HMD device in the first embodiment. FIG. [Figure 7] FIG. 3 is a diagram showing an example of the configuration of coordinate system information in the first embodiment. [Figure 8] FIG. 3 is a diagram showing an example of the configuration of lattice data in the first embodiment. [Figure 9] FIG. 2 is a diagram showing an example of the configuration of virtual image data in the first embodiment. [Figure 10] FIG. 3 is a diagram showing a processing flow of the HMD device in the first embodiment. [Figure 11] FIG. 2 is an explanatory diagram of a basic method relating to operations for arranging and moving a virtual object with respect to real space on a display screen in the first embodiment. [Figure 12] 3A to 3C are diagrams showing examples of input operations in various operation methods in the first embodiment. [Figure 13] FIG. 3 is a diagram showing an example of display control in the first embodiment. [Figure 14] FIG. 10 is a diagram showing another example of display control in the first embodiment. [Figure 15] FIG. 10 is a diagram showing another example of display control in the first embodiment. [Figure 16] FIG. 10 is a diagram showing another example of display control in the first embodiment. [Figure 17] FIG. 10 is a diagram showing another example of display control in the first embodiment. [Figure 18] FIG. 10 is a diagram showing another example of display control in the first embodiment. [Figure 19] FIG. 10 is a diagram showing another example of display control in the first embodiment. [Figure 20] FIG. 10 is a diagram showing another example of display control in the first embodiment. [Figure 21] FIG. 10 is a diagram showing a case where control is performed to place a virtual object that is not displayed on the display screen, as another example of display control in the first embodiment. [Figure 22] FIG. 2 is an explanatory diagram of a world coordinate system in the first embodiment. [Figure 23] FIG. 2 is an explanatory diagram of a local coordinate system in the first embodiment. [Figure 24] FIG. 2 is an explanatory diagram of an inertial coordinate system in the first embodiment. [Figure 25] 3A to 3C are diagrams illustrating a configuration example when a grid is arranged in an inertial coordinate system in the first embodiment. [Figure 26] FIG. 4 is a diagram showing another example of the configuration of a grid in an inertial coordinate system in the first embodiment. [Figure 27] FIG. 4 is a diagram showing another example of the configuration of a grid in an inertial coordinate system in the first embodiment. [Figure 28] FIG. 2 is a diagram showing an example in which a plurality of virtual objects are handled using a grid of an inertial coordinate system in the first embodiment. [Figure 29] FIG. 2 is a diagram showing an example of a configuration of a three-dimensional lattice as a lattice in the first embodiment. [Figure 30] FIG. 2 is a diagram showing an example of arrangement of a plurality of virtual objects in a three-dimensional lattice in the first embodiment. [Figure 31] FIG. 10 is a diagram showing a display example of a grid or the like in a modified example in the first embodiment. [Figure 32] FIG. 2 is a diagram showing a display example of IDs of points of a three-dimensional lattice in the first embodiment. [Figure 33]FIG. 10 is a diagram showing another example of displaying a three-dimensional lattice in the first embodiment. [Figure 34] FIG. 10 is a diagram showing another example of display control of a three-dimensional grid in the first embodiment. [Figure 35] FIG. 10 is a diagram showing another example of display control of a three-dimensional grid in the first embodiment. [Figure 36] FIG. 10 is a diagram showing another example of display control of a three-dimensional grid in the first embodiment. [Figure 37] FIG. 10 is a diagram showing another example of display control of a three-dimensional grid in the first embodiment. [Figure 38] FIG. 10 is a diagram showing another example of the configuration of a three-dimensional lattice in the first embodiment. [Figure 39] FIG. 10 is a diagram showing another example of display control of a three-dimensional grid in the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0012] [Assignments, etc.] The following is a supplementary explanation of the problems of the HMD of the prior art example. The HMD of the comparative example to the embodiment places a virtual object at a predetermined position in a scene of real space seen on a display screen, and moves the virtual object from one position to another based on a user's operation. Known operation methods for this include a gesture method, a method using a controller, and a voice recognition method. The gesture method is a method in which finger movements in space are detected as gestures based on camera images or the like, and the gestures are associated with commands. The controller method is a method in which beam or button operations using a controller (a so-called remote control) for the HMD are associated with commands. The voice recognition method is a method in which the user's voice is detected and associated with commands.

[0013] When placing or moving a virtual object in space using any of the operation methods, the user must perform the following series of actions in detail: the user must select a target virtual object, move the target virtual object to the desired placement or movement position by dragging or other operation, and then confirm the placement or movement at that position.

[0014] When a user wishes to handle a large number of virtual objects, the user must repeat such operations for each virtual object. Such operations can be time-consuming and laborious, resulting in poor usability. Furthermore, since the placement position of a virtual object is determined at the end of such operations, it can be difficult for the user to accurately or quickly place the virtual object at a desired position in space. Furthermore, particularly when placing or moving a virtual object in front of the HMD and the user's viewpoint in space (in other words, in the depth direction), such operations can be difficult or impossible due to factors such as the long distance to the target position.

[0015] Furthermore, the HMD of the comparative example uses a well-known world coordinate system or local coordinate system as a coordinate system for managing the position of virtual objects relative to space. The world coordinate system is a coordinate system fixed to real space. Because the world coordinate system has a wide space, a large number of virtual objects can be placed. The local coordinate system is a coordinate system fixed to the display surface as seen from the viewpoint of the HMD and the user. A virtual object placed in the local coordinate system has a fixed positional relationship with the display surface. In other words, the virtual object is fixed to a predetermined position within the display surface. When a virtual object is placed in the local coordinate system of the display surface, the position of the virtual object within the display surface is maintained even if the user moves or changes the direction of their head.

[0016] However, with the HMD of the comparative example, it is difficult to arrange a large number of virtual objects using only these two types of coordinate systems, which can make it difficult for users to work. Virtual objects arranged in the world coordinate system are fixed to the location in the space where the user is located. If the user moves away from that location, the virtual objects become invisible or difficult to see. Virtual objects arranged in the local coordinate system have a limited size of the display surface or field of view, so there is a limit to the number of virtual objects that can be arranged. Even if a large number of virtual objects can be arranged on the display surface, it becomes difficult for the user to see both the real and virtual objects, in other words, visibility is reduced, making it difficult for the user to work. For example, if a user wants to switch between virtual objects that they frequently refer to for work on the display surface and arrange them as needed, it requires a lot of effort.

[0017] Taking the above-mentioned problems into consideration, the present invention proposes a new method for user operation related to the placement and display of virtual objects in an HMD, thereby improving operability and usability, and improving the ease and efficiency of tasks using virtual objects.

[0018] (Embodiment 1) An HMD device according to a first embodiment of the present invention will be described with reference to FIGS. 1 to 39. The HMD device according to the first embodiment provides a new method for supporting a user in placing and moving virtual objects, as well as a man-machine interface including a graphical user interface (GUI). The HMD device sets a grid including multiple points that can be used as a reference point for placing virtual objects in a real space (corresponding virtual space) managed using a coordinate system. These points can also be referred to as reference points or grid points. A user can place a virtual object at these points in accordance with a predetermined operation. As a predetermined operation, the user specifies a target virtual object to be placed or moved and specifies a point (first point) corresponding to the placement or movement destination. The HMD device places or moves the target virtual object in space to the first point of the grid in accordance with this operation and displays it on the display screen. This basic operation allows for the placement or movement of a single virtual object with minimal effort. Furthermore, a mechanism described below allows for the placement or movement of multiple virtual objects with minimal effort.

[0019] [Display System] FIG. 1 shows the configuration of a display system including an HMD device according to the first embodiment. This display system includes an HMD 1, which is the HMD device according to the first embodiment, and a server 3, a PC 4, and the like, which are connected to the HMD 1 via communication. The HMD 1 includes an operating device 2. A user wears the HMD 1 on their head. The operating device 2 is a remote control that communicates with the main body, can be operated by the user with their fingers, and serves as an input means. In the field of view in front of the user's eyes, a virtual object (corresponding virtual image) is superimposed on a scene including actual objects in real space via a display surface 5. The HMD 1 can communicate with devices such as a provider's server 3 or a user's home PC 4. The server 3 and the PC 4 may provide the HMD 1 with various application programs and virtual object data, or may store the HMD 1 data. Furthermore, the functions described below may be realized by a system of cooperation between the HMD 1 and the server 3 (e.g., a client-server system).

[0020] The HMD 1 communicates with the controller 2, for example, via short-range wireless communication, to send and receive signals. Note that a configuration without the controller 2 is also possible. The controller 2 includes buttons and sensors. For example, the user moves the controller 2, points to a virtual object or the like using a beam from the controller 2, and presses a button on the controller 2. The HMD 1 recognizes the user's actions in response to signals from the controller 2 and interprets them as a predetermined command, for example, selecting or executing a virtual object. Other input means such as a keyboard or mouse may also be used.

[0021] [HMD] FIG. 2 shows an example of the external configuration of the HMD 1. (A) shows a side view of a user wearing the HMD 1, and (B) shows a front view of the HMD 1. In (A), the HMD 1 has a housing 10 worn on the user's head and a display device 50 connected to the housing 10. The housing 10 has a built-in circuit board and is provided with a microphone 81, a speaker 82, and the like. The display device 50 is disposed in front of the user's eyes. The display device 50 includes a display surface 5, a camera 6, a cover lens 7, and the like. The display surface 5 is, for example, a transmissive display device and includes two lenses disposed close to the eyes. The cover lens 7 is disposed in front of the display surface 5 and includes, for example, a single transmissive lens. The camera 6 includes multiple cameras. The display device 50 can be a projection display device or the like, and is not particularly limited.

[0022] The display method including the display surface 5 of the HMD 1 is a transparent type, but can also be applied to a non-transparent type (VR type). In the transparent type, it is possible to superimpose a virtual object on a real image. In the VR type, it is possible to composite a virtual object with an image captured by a camera. In the VR type, a VR space can be displayed in the field of view based on publicly known technology, and for example, the user can experience moving within the VR space by operating the controller 2 without moving their own body. The user can place virtual objects within the VR space by operating it. The VR space is, for example, a video game space created using 3D CAD. In the transparent type, a world coordinate system is set within the real space. In the VR type, a world coordinate system is set within the VR space.

[0023] The HMD 1 has a control unit built into the housing 10 or the display device 50. The control unit includes a processor, memory, an OS, an application program, a communication interface, etc. The control unit also includes a voice recognition function, etc. The HMD 1 uses the voice recognition function to recognize the user's voice and associates the voice with commands, etc. The HMD 1 also has various built-in sensors. The housing 10 also includes operation buttons, a connector, a battery, etc. Communication interfaces include wireless LAN, mobile network communication standards, USB, Bluetooth (registered trademark), infrared communication methods (e.g., IrDA), Zigbee (registered trademark), HomeRF (registered trademark), RFID methods, etc.

[0024] In (B), the HMD 1 is equipped with multiple microphones 81 and multiple speakers 82 at positions including the left and right sides of the housing 10. The HMD 1 is equipped with multiple cameras 6 at multiple positions relative to the cover lens 7. The multiple cameras 6 include an RGB camera that captures the scene within the field of view, a camera for detecting gestures, a camera that constitutes a distance measurement sensor, and a camera that constitutes a line of sight detection sensor. The HMD 1 performs various detections and controls using images captured by the multiple cameras 6 and detection information from the sensors. The HMD 1 in this example can form a virtual object at a position within a distance range of, for example, 0.5 m to 5 m in front of the user's viewpoint through the display surface 5.

[0025] [HMD-functional block] FIG. 3 shows an example of the functional block configuration of the HMD 1. The HMD 1 includes a processor 101, a memory 102, a display device 50, a camera 6, a sensor 70, a communication device 80, a microphone 81, a speaker 82, operation buttons 83, a battery 84, and the like, all of which are interconnected via a bus or the like. The processor 101 includes a CPU, a GPU, a ROM, a RAM, and the like, and constitutes a controller for the HMD 1, and executes processing for the OS and applications. The processor 101 realizes each function by executing processing in accordance with a program. The processor 101 includes processing units configured based on hardware and software program processing, such as a setting unit 11, a state recognition unit 12, a coordinate calculation unit 13, a grid control unit 14, an instruction recognition unit 15, a display control unit 16, and an application control unit 17.

[0026] The memory 102 is configured with a nonvolatile storage device or the like, and stores various data and information handled by the processor 101. The memory 102 stores, for example, a control program 21, an application program 22, setting information 30, coordinate system information 31, virtual image data 32, grid data 33, etc. The memory 102 also stores data on images captured by the camera 6, detection information from the sensor 70, etc. The control program 21 is a program that realizes the basic functions of the HMD 1 of the first embodiment, which will be described later. The application program 22 includes a known program related to the generation of virtual objects, such as a 3D CAD program.

[0027] The display device 50 includes the display surface 5 shown in FIG. 2 and is driven to display a virtual image corresponding to a virtual object on the display surface 5 under control of the processor 101. The camera 6 includes multiple cameras, each capturing incident light through a lens with an imaging element to obtain an image. The sensor 70 includes various sensors, such as a GPS receiver, a geomagnetic sensor, and an inertial sensor (a gyro sensor and an acceleration sensor). The sensor 70 includes an attitude detection sensor that detects the attitude of the user and the HMD 1. The GPS receiver obtains position information by positioning. The geomagnetic sensor detects orientation and, if a three-axis type is used, can detect head movement. The inertial sensor detects angle, angular velocity, and acceleration corresponding to the direction, movement, and inertial state of the HMD 1. The acceleration sensor detects three-axis acceleration and can determine the vertical direction from changes in the position of the HMD 1 and the direction of gravitational acceleration. The gyro sensor detects the angular velocity of the HMD 1 in the three-axis rotation direction. The gyro sensor detects, from the angular velocity, an angle that represents the attitude (e.g., the direction of the local coordinate system relative to the direction of the world coordinate system) of the HMD 1. This angle can be expressed as the well-known Euler angles (pitch angle, yaw angle, and roll angle) or a normalized quaternion.

[0028] Quaternions are a number system that extends complex numbers. When using quaternions, it is possible to handle the rotation of a vector in three-dimensional space (transformation between corresponding coordinate systems) with a small amount of calculation. Quaternions are represented as four-dimensional vectors. Each vector component stores the result of a predetermined calculation using the rotation axis and rotation angle. If the quaternion is q, it is expressed as q=w+xi+yj+zk. (w,x,y,z) are real numbers. (i,j,k) are the numbers i 2 =j 2 =k 2 =-1, ij=-ij=k, jk=-kj=i, ki=-ik=j. The product of two quaternions is a quaternion. The operation of multiplying a rotation matrix using Euler angles can be expressed as an operation using the product of quaternions.

[0029] The HMD 1 uses the camera 6 and the sensor 70 to detect the user's position, the user's reference direction, head movement and direction, gaze direction, hand and finger positions and gestures, etc. The HMD 1 may detect feature points of the real object from the image captured by the camera 6 to grasp the structure of the real object. The HMD 1 is equipped with a distance measurement sensor and a gaze detection sensor configured using the camera 6 and the sensor 70. The distance measurement sensor is a sensor that measures the distance (in other words, depth) to the position of an object as seen from the user's viewpoint (corresponding position of the HMD 1). The gaze detection sensor is a sensor that measures the user's gaze direction (corresponding position on the display surface). There are no limitations on the type of distance measurement sensor or gaze detection sensor.

[0030] The microphone 81 is an audio input device including multiple microphones. Using multiple input sounds from multiple microphones makes it possible to detect sound directionality in three-dimensional space. The speaker 82 is an audio output device including multiple speakers. Using multiple output sounds from multiple speakers makes it possible to generate stereophonic sound in three-dimensional space. The operation buttons 83 include a power on / off button, a brightness adjustment button, a volume adjustment button, etc. The battery 84 supplies power to each component based on charging. The communication device 80 includes components such as an antenna and an IC corresponding to various communication interfaces, and performs short-range wireless communication with the controller 2 and communication with external base stations, the server 3, the PC 4, etc.

[0031] The setting information 30 is system setting information and user setting information related to basic functions. The coordinate system information 31 is information for managing three types of coordinate systems, which will be described later. The virtual image data 32 is data for displaying a virtual object on the display surface 5. The grid data 33 is data for managing a grid, which will be described later.

[0032] The processor 101 stores data of virtual objects generated by the OS or an application, or data of virtual objects acquired from the server 3, PC 4, etc., in virtual image data 32. The HMD 1 receives input operation information based on the user's operation of the controller 2 from the controller 2, interprets the input operation information, and associates it with commands, etc. Using images from the camera 6 and detection information from the sensor 70, the HMD 1 recognizes the scene in the user's field of view, the posture of the user and the HMD 1, the direction of the user's line of sight, the distance to the object, etc.

[0033] [HMD-display surface] FIG. 4 shows an example of real objects and virtual objects displayed on the display surface 5 of the HMD 1. In this example, an image 400 on the display surface 5 shows a workbench 401 and a whiteboard 402 in front of the user as real objects, and a cubic real object 403 is placed on the workbench 401. In this example, multiple virtual objects are superimposed and displayed on the display surface 5. The virtual objects may be images of two-dimensional or three-dimensional objects, or may be GUI images. Examples of GUI images for virtual objects include system information 411, a menu field 412, an application window 413, and a cursor 414. The system information 411 is an image representing the time, battery status, communication status, volume status, the status of the voice recognition function, etc. The menu field 412 (in other words, a launcher field) displays images such as an application icon 415 and a command button 416. The application icon 415 is an image of an icon representing an application (application program 22) and has, for example, a two-dimensional rectangular shape. When an application icon 415 is selected and executed by the user, the corresponding application is launched, and, for example, an application window 413 is displayed. When a command button 416 is selected and executed by the user, processing of the corresponding command is executed. Display of the menu field 412 and the like can be turned on / off depending on the user's operation and the state of the HMD 1. Other items, such as web browser bookmarks and file icons, may also be placed in the menu field 412. The items placed in the menu field 412 can be set by the user.

[0034] The application window 413 is displayed when an application is running, and has a two-dimensional rectangular shape. The user can adjust the position, size, and display on / off of the application window 413. An image generated by the corresponding application program 22 is displayed within the application window 413. The cursor 414 can be moved in response to a user operation, for example, operation of the controller 2, and allows for operations such as selection of a virtual object. The cursor 414 is shown as an example of a hand-shaped cursor, but is not limited to this, and other shapes such as a point, arrow, or cross are possible.

[0035] In this way, virtual objects constituting the GUI of the HMD 1 can be arranged on the display surface 5. The HMD 1 can arrange such virtual objects of the GUI at predetermined positions on the display surface 5 using a local coordinate system, which will be described later. A predetermined area on the display surface 5 may be set as a fixed area for arranging the GUI. The HMD 1 also controls the positional relationship between real objects and virtual objects based on recognition of the real objects from images captured by the camera 6. For example, the HMD 1 can arrange and display virtual objects at positions that match the surface of the workbench 401 or the surface of the whiteboard 402. For example, if a virtual object is arranged below the workbench 401 or behind the whiteboard 402, the HMD 1 does not display the virtual object on the display surface 5.

[0036] An example of a user's work and application is to display a model created with a 3D CAD application as an image of a virtual object on a workbench 401 in space alongside the real object 403, allowing the user to check the 3D shape of the model from various directions. Another example is to arrange a 2D virtual image on the surface of a whiteboard 402 by the user. The HMD 1 may set a grid, described below, to match the plane of the real object.

[0037] [HMD-OS and Applications] FIG. 5 shows an example configuration of the OS, applications, and the like in the HMD 1. The processor 101 of the HMD 1 executes the processing of a control program 21 within or on the OS 500. The OS 500 includes a program 501 that creates a virtual object corresponding to AR or the like or its original image information (e.g., system information 411). The OS 500 also includes various application programs 22 (e.g., applications A1 to Am) that are launched and executed on the OS 500. Examples of the application programs 22 include an AR application, a photo application, a web browser, a social networking application, and a phone application. Each application program 22 creates a virtual object corresponding to AR or the like or its original image information, and displays an image 503 of the virtual object on the display surface 5. For example, application A1 creates an image B1 of a virtual object and displays it on the display surface 5. When a corresponding virtual object is selected by a user, the program 501 or the application program 22 executes a corresponding predetermined process.

[0038] The control program 21 performs predetermined display control when the program 501 or the application program 22 displays an image 503 of a virtual object on the display surface 5. The control program 21 displays a grid K1 on the display surface 5. The grid K1 includes a plurality of points P1, which are a plurality of grid points, and a plurality of grid lines. The control program 21 displays an ID mark M1 at each point P1. The control program 21 may also display an ID mark on the image 503 of the virtual object. Each grid K1 is associated with a coordinate system or an area in space where it is placed. The control program 21 accepts operations on the virtual object as predetermined input operations by the user. These operations include operations to select a virtual object, as well as operations to place or move a virtual object. Methods of input operations that can be used include voice recognition, finger gestures, cursor operation using the operating device 2, operation using the line of sight, operation using the movement of the head (corresponding HMD 1), and the like.

[0039] When the control program 21 receives a predetermined operation, it places or moves the target virtual object to the position of the specified placement or movement destination point P1. The predetermined operation includes (1) an operation of specifying the target virtual object, and (2) an operation of specifying the placement or movement destination point P1. When the target virtual object is placed or moved, it is automatically set so that it is placed in a coordinate system in which the grid K1 to which the specified point P1 belongs is located.

[0040] The HMD 1 arranges and displays virtual images of GUIs, such as the application icon 415 and application window 413 in FIG. 4, on the display surface 5. The HMD 1 launches an application program 22 designated by the user using the application icon 415 or the like. The processor 101 runs an execution module corresponding to the application program 22. If there is no designation by the user, the HMD 1 arranges and displays an image of the GUI at a default position in the local coordinate system. If there is an instruction operation by the user, the HMD 1 arranges and displays an image of the GUI at a position designated by the user. The application program 22 is classified into two-dimensional applications and three-dimensional applications. In the case of a two-dimensional application, a virtual object is arranged at a position within a two-dimensional plane in the application window 413 or in application mode on the display surface 5. In the case of a three-dimensional application, a virtual object is arranged at a position within a three-dimensional space on the display surface 5.

[0041] The HMD1 manages and controls the position, orientation, display size, etc. of virtual objects relative to the coordinate system and grid K1 in space. When the main body or an application is started, the HMD1 determines whether to apply the coordinate system and grid K1, and determines the virtual object to be placed and its placement position, etc. The HMD1 updates the state of the coordinate system, grid K1, virtual objects, etc. in space according to the user's movements and operations. The HMD1 saves these states as information when the main body is turned off or an application is closed. When the main body or an application is restarted, the HMD1 restores the state of the coordinate system, grid K1, virtual objects, etc. according to the saved information.

[0042] [Operation method] Examples of operation methods in the first embodiment are as follows. The HMD 1 uses at least one operation method. In the voice method, the HMD 1 recognizes a predetermined voice from the user's voice input via the microphone 81 using a voice recognition function and associates the voice with a predetermined command. For example, when "image display on" is input as a voice command, the HMD 1 displays a virtual object on the display surface 5, and when "image display off" is input, the HMD 1 hides the virtual object on the display surface 5. For example, when "grid on" is input, the HMD 1 displays a grid K1 (including the ID mark M1) on the display surface 5, and when "grid off" is input, the HMD 1 hides the grid K1 on the display surface 5. In the gesture method, the HMD 1 detects a hand gesture from the image captured by the camera 6 and associates the gesture with a predetermined command. For example, when the HMD 1 detects a touch or tap gesture on the position of a virtual object on the display surface 5, the HMD 1 associates the gesture with a command indicating the virtual object's designation. For example, when the HMD 1 detects a gesture of opening and closing the hand, the HMD 1 associates the gesture with a command indicating a return to the previous state or a cancellation.

[0043] In the case of a cursor operation method using the controller 2, the HMD 1 moves a cursor displayed on the display surface 5 based on a signal from the controller 2, and when, for example, a button on the controller 2 is pressed, the HMD 1 associates the cursor position at that time with the designation of a virtual object. In the case of an operation method using gaze direction, the HMD 1 detects the position where the user's gaze direction, detected using a gaze detection sensor, intersects with the display surface 5, displays a cursor at that position, and associates it with the designation of a virtual object at that position. In the case of an operation method using head movement, the HMD 1 detects the front direction and movement of the head (corresponding HMD 1) using a sensor, and displays a cursor according to the front direction and movement, and associates it with the designation of a virtual object in the front direction.

[0044] [HMD-Basic Functions] FIG. 6 shows the configuration of processing units and data constituting the basic functions of the HMD 1 based on the configuration of FIG. 3. The basic functions are functions for controlling the display of virtual objects using a grid K1 and the like, and include functions for controlling the placement and movement of virtual objects. The setting unit 11 sets and saves setting information 30 related to the basic functions and the like in advance. As an example, the user can set the application to be applied, the grid K1 to be used, the coordinate system, the spatial domain, the type of ID mark M1 to be displayed, and the like. The state recognition unit 12 recognizes the state of the user and the HMD 1, including their posture, at each point in time, using images from the camera 6 and detection information from the sensor 70.

[0045] The coordinate system calculation unit 13 calculates the state of the coordinate system at each point in time using images from the camera 6 and detection information from the sensor 70. The coordinate system includes three types of coordinate systems, which will be described later, and there are positional relationships between the coordinate systems. The coordinate system calculation unit 13 reads and writes information about the calculated and set coordinate systems to the coordinate system information 31. The coordinate system information 31 includes information about the position of the origin of each coordinate system, information about the front direction, and information about the positional relationships between the coordinate systems. The coordinate system calculation unit 13 calculates the placement of the grid K1 and the placement of virtual objects relative to the coordinate system. Furthermore, when there is a change in the coordinate system or movement between coordinate systems, the coordinate system calculation unit 13 calculates rotation between the coordinate systems using Euler angles or normalized quaternions.

[0046] The grid control unit 14 controls the placement of the grid K1 relative to the display surface 5 and the coordinate system. The grid control unit 14 reads and writes data including the configuration of the grid K1 (including information about the point P1 and the ID mark M1) from the grid data 33. The instruction recognition unit 15 recognizes an input operation (corresponding instruction) by a user using, based on the applied operation method, images from the camera 6, detection information from the sensor 70, signals from the controller 2, and audio input from the microphone 81. The instruction recognition unit 15 associates a predetermined input operation with a predetermined command and controls the display control unit 16 and other components in accordance with the command. The display control unit 16 displays the grid K1 and virtual objects on the display surface 5 based on control from the instruction recognition unit 15, the grid data 33, and the virtual image data 32, and outputs audio (e.g., sound effects associated with the operation) from the speaker 82. The application control unit 17 controls the launch and termination of each application program 22 and stores data of virtual objects generated by each application program 22, such as application icons 415 and application windows 413, in the virtual object data 32.

[0047] Coordinate System Information FIG. 7 shows a table 701 as an example of the configuration of the coordinate system information 31. In the table 701, the origin positions and front directions of three types of coordinate systems are managed as information. The table 701 has columns for coordinate system ID, origin position, and front direction. The three types of coordinate systems are a world coordinate system CS1, a local coordinate system CS2, and an inertial coordinate system CS3. It is also possible to set and use multiple coordinate systems (multiple corresponding regions) for each type of coordinate system.

[0048] [Grid Data] FIG. 8 shows a table 801 as an example of the configuration of the grid data 33. The table 801 has columns such as a grid ID, a point ID, a placement coordinate system, placement position coordinates, a display flag, and an ID mark. The grid ID is an ID for each grid K1. The point ID is an ID for each point P1 among multiple points P1 belonging to that grid K1. The placement coordinate system indicates the coordinate system in which that grid K1 (corresponding point P1) is placed. The placement position coordinates indicate the position coordinates of that point P1 within the placement coordinate system. The display flag is a flag for managing whether that point P1 (or the corresponding ID mark M1) is displayed or hidden on the display surface 5. The ID mark indicates the ID value of the ID mark M1 associated with that point P1.

[0049] [Virtual image data] FIG. 9 shows a table 901 as an example of the configuration of virtual image data 32. Table 901 has columns such as a virtual object ID, shape (file), placement coordinate system, placement grid, placement point, placement direction, and label. The virtual object ID is an ID for each virtual object (corresponding virtual image). The shape (file) is an image file or the like representing the shape of the virtual object. The placement coordinate system indicates the coordinate system in which the virtual object is placed. The placement grid indicates the grid K1 in which the virtual object is placed. The placement point indicates point P1 in which the virtual object is placed. Note that placement position coordinates may be used as the placement position of the virtual object. The placement direction indicates the direction in which the virtual object is placed at point P1. The label indicates the ID value of an ID mark (sometimes referred to as a label to distinguish it from the ID mark M1 at point P1) when the virtual object is assigned and displayed. Note that, although not shown, a placement size, a display flag, and the like may be set for each virtual object.

[0050] [Processing flow] FIG. 10 shows a processing flow of the basic functions of the HMD 1. FIG. 10 includes steps S1 to S13, which will be described in order below. In step S1, the user turns on the main power of the HMD 1 to start it up. Accordingly, the HMD 1 performs a predetermined initialization process. Each sensor of the sensor 70 starts measuring. Each camera of the camera 6 starts capturing images. In step S2, the HMD 1 sets (including resetting) three types of coordinate systems as part of the initialization process. At this time, the user places the HMD 1 in a stationary state. The coordinate system calculation unit 13 of the HMD 1 sets the origin and front direction of each of the three types of coordinate systems according to a predetermined rule. If the coordinate system calculation unit 13 wants to maintain a previously set coordinate system, it reads setting information from the coordinate system information 31 and resets the coordinate system according to the setting information. Alternatively, the coordinate system calculation unit 13 may set a new coordinate system. Examples of setting coordinate systems will be described later.

[0051] In step S3, the HMD 1 displays a virtual image of a GUI such as the example in FIG. 4 on the display surface 5. For example, the HMD 1 displays system information 411 and a menu field 412 in accordance with the local coordinate system CS2, and displays application icons 415 and the like in the menu field 412. In step S4, the HMD 1 selects an application (application program 22) according to an operation or setting by the user. The user can select an application to launch from the application icon 415 in the menu field 412, for example.

[0052] In step S5, the HMD 1 displays virtual objects related to the application selected in step S4 on the display surface 5. At this time, the display control unit 16 references the virtual image data 32 (e.g., table 901 in FIG. 9 ) to confirm the shape, placement coordinate system, placement grid, placement point, etc. of each virtual object to be displayed. The display control unit 16 determines which coordinate system and grid K1 area each virtual object should be placed in. Basically, for virtual objects previously placed, the coordinate system and grid K1 at that time are maintained. If the placement coordinate system or grid K1 is specified by a user's input operation, the HMD 1 may place the virtual object in the specified coordinate system or grid K1. Note that there may be cases where the virtual object is not placed in the grid K1. The display control unit 16 calculates the position, orientation, display size, etc. of each virtual object in the grid K1 of the placement coordinate system. The display control unit 16 converts the placement position and placement direction when the placement coordinate system of the virtual object is the world coordinate system CS1 or the inertial coordinate system CS3 into information in the local coordinate system CS2 and displays it at the corresponding position on the display surface 5.

[0053] In step S6, the display control unit 16 of the HMD 1 displays multiple points P1 of the grid K1 and ID marks M1 for each point P1 on the display surface 5. In step S7, the instruction recognition unit 15 of the HMD 1 accepts an input operation by the user based on the operation method and recognizes it as an instruction. The instruction may be, for example, a known virtual object operation related to a task or application, an operation to place or move a virtual object, or an operation related to setting a coordinate system. A known virtual object operation is an operation for selecting a virtual object and executing a predetermined process using an application program 22 or the like. An operation to place or move a virtual object is a specific operation shown in FIG. 11 (described later) and the like.

[0054] In step S8, the HMD 1 checks whether the input operation (corresponding instruction) in step S7 is an operation (corresponding command) for placing or moving a virtual object. If so (Y), the process proceeds to step S9. In step S9, the HMD 1 updates the display state so that the specified target virtual object is placed or moved to the position of the specified placement or movement destination point P1 on the display surface 5. This update includes updating the display state of the ID mark M1 at point P1, the virtual object label, and the like. When placing or moving a virtual object, the HMD 1 appropriately changes the coordinate system in which the virtual object is placed to match the coordinate system to which the placement destination point P1 belongs. That is, in the first embodiment, virtual objects can be moved between coordinate systems (between corresponding grids K1). For example, if the coordinate system from which the target virtual object is placed is the local coordinate system CS2 or the world coordinate system CS1, the placement destination can be set to the position of point P1 of the corresponding grid K1 in the inertial coordinate system CS3.

[0055] In step S10, the HMD 1 checks whether a coordinate system setting instruction has been entered as the input operation in step S7. If so (Y), the process proceeds to step S11. Examples of coordinate system setting instructions include an instruction to change the coordinate system in which a virtual object is placed between three different coordinate systems, or an instruction to rotate and move the inertial coordinate system CS3 (described later). In step S11, the HMD 1 updates the setting information for the coordinate system, grid K1, and virtual object in response to the coordinate system setting instruction. In step S12, if the user turns off the main power supply to the HMD 1 (Y), the process proceeds to step S13. In step S13, the HMD 1 saves the current state of the coordinate system and other information in the coordinate system information 31, and executes a shutdown process for the HMD 1. This ends the process flow. If the main power supply remains on in step S12 (N), the process returns to step S2, for example, and the same process is repeated at each point in time. The processes of steps S2, S5, S6, etc. are performed so as to update the user's posture including the direction of the head at each time point.

[0056] [Basic method] FIG. 11 is an explanatory diagram of a basic method for operating the placement and movement of virtual objects in real space on the display surface 5 in the basic function of the HMD 1. Image 111 in FIG. 11 shows a scene seen from the user's viewpoint corresponding to the display surface 5. Examples of real images in image 111 include a workbench 401, a whiteboard 402, and a real cubic object 403. Additionally, virtual objects V1, V2, and V3 are displayed as virtual objects 110 based on the processing of an application program 22 selected by the user. For example, virtual object V1 is an image representing a three-dimensional snowman-shaped object. Virtual object V2 is an image representing a three-dimensional cylindrical object. Virtual object V3 is an image representing a three-dimensional cone-shaped object.

[0057] The HMD 1 displays a plurality of points P1 of a grid K1 and ID marks M1 on the display surface 5 to support the user's operations of arranging and moving virtual objects. In this example, the grid K1 has a total of 10 points P1 arranged in 2 rows and 5 columns on a two-dimensional plane (corresponding grid surface). In this example, each point P1 is displayed as a white circular virtual image. An ID mark M1 is also assigned and displayed to each point P1. The ID mark M1 is a virtual image that represents the identification information (point ID) of the point P1. The HMD 1 displays the ID mark M1 at a position near or overlapping the point P1. In this example, the ID mark M1 is integrated with the point P1, and the number of the point ID is displayed within the circular mark. In this example, the 10 points P1 are assigned IDs = 1 to 10. The order of the IDs is not limited.

[0058] The HMD 1 also assigns and displays an ID mark N1 to each virtual object 110 (V1, V2, V3) in the image 111. The ID mark N1 is a virtual image that represents the identification information (in other words, a label) of the virtual object. In this example, the ID mark N1 is a rectangular mark with the alphabetic characters ID displayed within the rectangle. The HMD 1 displays the ID mark N1 at a position near or overlapping with the virtual object 110. In this example, ID=A is displayed for virtual object V1, ID=B for virtual object V2, and ID=C for virtual object V3.

[0059] A user performs a predetermined operation to place or move a virtual object. For example, suppose the user wants to place or move virtual object V1 displayed on the display surface 5 to the position of point P1 indicated by ID=7 in grid K1. In this case, the user performs the following predetermined operations: (1) designating the target virtual object or the origin of the movement, and (2) designating the destination point P1. In response to this operation, the HMD 1 places or moves the designated target virtual object to the designated point P1. This basic operation can easily achieve the placement or movement of a single virtual object. In addition to the above (1) and (2), the basic operation may also include (3) an instruction to place or move. For example, in the case of a method using gestures or the controller 2, the user first selects virtual object V1 by pointing at it with a finger or a cursor, and then selects and points at the destination point P1 indicated by ID=7. In the voice method, the user inputs a voice command such as "Place (or move) object A to number 7." In the voice method, the user can specify the numbers of the ID mark M1 or the letters of the ID mark N1 by voice. In accordance with this operation, the virtual object V1 is placed and displayed at the position of point P1 of ID=7, as shown in the lower image 111b. In this case, it is not necessary for the user to move the virtual object V1 to the position of ID=7 by an operation such as dragging, as in the conventional method.

[0060] The HMD 1 may always display the grid K1 (such as the point P1 or the ID mark M1) on the display surface 5, or may switch the display of the grid K1 on or off in response to a user's instruction or operation. If the user finds the display of the grid K1 on the display surface 5 bothersome, the user can turn the display off. For example, the HMD 1 may not normally display the grid K1, but may display the grid K1 when the user inputs a command to turn the grid on or when the user selects a virtual object. The HMD 1 may also display the grid K1 when the user's finger approaches the location of the grid K1. The HMD 1 may also display only the point P1 of the grid K1 in a partial area of the display surface 5 that corresponds to the user's line of sight. The HMD 1 may also independently control the display of the point P1 and the ID mark M1. For example, the HMD 1 may display only a point image representing the point P1. The HMD 1 may also switch the display of the ID mark M1 on or off in response to a command input by the user. Various command buttons for operating the grid K1, etc. may be provided within the display surface 5. As another example, the grid K1 may be permitted to be displayed only in a certain area within the display surface 5, and the grid K1 may not be permitted to be displayed in other areas. For example, the grid K1 may be displayed only in an area near the center of the display surface 5, or conversely, the grid K1 may be displayed in a peripheral area other than the area near the center.

[0061] In image 111b, virtual object V1 is placed at point P1 with ID=7. In this example, virtual object V1 is displayed superimposed on ID mark M1 with point P1, and the ID mark M1 with point P1 behind it is hidden. It should be noted that ID mark M1 with ID=7 may remain displayed in front so as not to be hidden. Furthermore, in this example, point P1 with ID=0 is located at the original placement position of virtual object V1. Therefore, point P1 is visible. Point P1 with ID=0 may be set as a home region (described later) at a predetermined position within display surface 5 (e.g., the center of the bottom edge region). Setting and display of the home region may be omitted. If this setting is present, point P1 with ID=0 can be specified as the placement or movement destination. For example, virtual object V1 once placed at point P1 with ID=7 can easily be returned to the position of point P1 with ID=0. For example, in the case of a voice command, a user may simply input commands such as "move object A to number 0" or "push object A back."

[0062] [Input operation example] FIG. 12 shows examples of input operations using various operation methods. (A) shows an example in which a beam (e.g., infrared light) emitted from the controller 2 is used as a pointing means. The user moves the controller 2 with their hand to point the tip of the beam at a target virtual object (e.g., virtual object V1) or point P1 on the display surface 5. The HMD 1 may display a cursor at the position of the beam's tip on the display surface 5. (B) shows an example in which a cursor 122 (e.g., a cross shape) at the tip of the beam of the controller 2 is displayed at the position of the virtual object V1 on the display surface 5. (C) shows an example in which a hand-shaped cursor 123 is displayed at the position of the virtual object V1. The HMD 1 may display the cursor so that it is superimposed on the virtual object or point P1, or may display it near them. Using the beam or cursor, the user can specify the target virtual object, the placement point P1, etc. For example, the virtual object V1 is placed in a selected state when the cursor 122, etc., is placed on the virtual object V1 for a certain period of time or more. Similar control is also possible using the line of sight instead of the controller 2. Furthermore, similar control is possible by using a method of displaying a cursor at the center of the display surface 5 in the direction directly in front of the user's head (corresponding HMD 1).

[0063] Furthermore, the operation of specifying the target virtual object or point P1 may be divided into a provisional selection and a confirmed selection. (D) shows an example of changes in the display state associated with, for example, pre-selection (non-selection), provisional selection, and confirmed selection of virtual object V1. For example, the provisional selection operation is performed by directing or overlapping the beam or cursor of the controller 2 on the target virtual object. The HMD 1 changes the display state of the virtual object on which the beam of the controller 2 is directed to a predetermined display state (e.g., a specific color, shape, size, etc.) that indicates provisional selection. In this example, the color is changed, but a frame or the like may be displayed surrounding the provisionally selected virtual object. The selection confirmation operation is performed, for example, by maintaining the beam directed from the provisional selection state for a certain period of time or longer. Other examples of the selection confirmation operation include pressing a button on the controller 2 or inputting a predetermined voice (e.g., "this object," "selected," etc.). The HMD 1 changes the display state of the virtual object that has undergone the selection confirmation operation to a predetermined display state (e.g., a specific color, shape, size, etc.) that indicates confirmed selection. In this example, the color is changed, but a frame or the like may be displayed around the virtual object whose selection has been confirmed. Note that a method of omitting the provisional selection state is also possible.

[0064] In the case of a method in which an instruction (corresponding command) for arranging or moving a virtual object is provided, the operation may be, after specifying the target virtual object and point P1, pressing a button on the controller 2 or a placement button on the display surface 5. Alternatively, the operation may be a gesture representing the placement or movement (for example, a gesture of flicking the target virtual object with a finger), or input of a voice representing the placement or movement (for example, "place", "move", etc.).

[0065] [Display control example (1)] FIG. 13 shows an example of the configuration of a grid K1 in the image 131 on the display surface 5 (FIG. 1) and an example of a virtual object movement operation, as an example of display control. In this example, the grid K1 has 3 rows and 5 columns, with 3×5=15 points P1, and the ID marks M1 for each point P1 have IDs ranging from 1 to 15. In this example, a virtual object V2 is initially placed at point P1 with ID=12. For example, a user may want to move virtual object V2 to point P1 with ID=3. In this case, the user may input, for example, "move object B to number 3" or "move object 12 to number 3" using a voice command. When a virtual object is placed at point P1, the user can specify the target virtual object for movement by specifying the corresponding source point P1. In the case of a gesture or cursor command, the user specifies point P1 or virtual object V2 with ID=12 as the source, and then specifies point P1 with ID=3 as the destination. In accordance with these operations, the HMD 1 moves the virtual object V2 to the position of point P1 with ID = 3. As described above, a method is possible for executing the movement of a virtual object when both the source and destination are specified on the grid K1 within the display surface 5.

[0066] In the example of image 131, a movement button 132 is displayed as a virtual image. This movement button 132 may be used as a movement instruction (corresponding command). For example, the HMD 1 may display the movement button 132 in advance, or may display the movement button 132 after specifying the target virtual object or the point P1. For example, the user presses the movement button 132 after specifying the target virtual object and the destination point P1 (for example, a selection operation using a cursor or the like). The HMD 1 uses this operation as a movement instruction to move the virtual object. As described above, a method using a movement instruction may be used.

[0067] Another method may be a method in which the user first presses the move button 132 and then specifies the source and destination of the movement. Another method may be a method in which the user specifies point P1 as the placement or destination, and then specifies a target virtual object. Another method may be a method in which, when wishing to move multiple virtual objects together, the user specifies multiple target virtual objects and then specifies point P1 as the destination of one of the objects.

[0068] Furthermore, the designation of point P1, which is the placement or movement destination, is not limited to designation of an absolute position, but may also be designation of a position relative to the position of another point P1 or virtual object. For example, in image 131, it is assumed that virtual object V3 is already placed at the position of point P1 with ID=2. When moving virtual object V2 to the position of point P1 with ID=3, the user can designate a position relative to the position of virtual object V3. In the case of a voice-based method, the user can simply input, for example, "Move object B to the right of object C" or "Move object B to the right of number 2."

[0069] The ID mark M1 of point P1 and the ID mark N1 of the virtual object can be represented by numbers, letters, or other characters, or by different colors or shapes. The ID mark M1 and the ID mark N1 are images of different systems to make them easier to distinguish. The ID of point P1 and the ID of the virtual object may be displayed in their entirety from the beginning on the display surface 5, or may not be displayed initially, or only a portion of them may be displayed. For example, when the user's cursor approaches point P1 or a virtual object, the corresponding ID may be displayed. In addition, in this case, as in the example of ID mark M1 with ID=10, only the point P1 or ID near the cursor may be enlarged or the color may be changed to make it stand out.

[0070] In the conventional method, when moving a virtual object within the display screen, the user must move the selected target virtual object to the destination position by an operation such as dragging, which is time-consuming. In the method of the first embodiment, such an operation such as dragging is basically unnecessary, allowing for efficient operation.

[0071] [Display control example (2)] FIG. 14 shows another example of display control. A grid K1 similar to that shown in FIG. 13 is arranged on the image 141 of the display surface 5. In this example, a user places three virtual objects 110, namely, virtual objects V1, V2, and V3, at point P1, which is located at the same position of ID=6. For example, the user first specifies virtual object V3 with ID=C and then moves virtual object V3 by specifying point P1, which is located at ID=6. For example, in the case of a voice-based input, the user might input, for example, "move object C to number 6." Next, the user specifies virtual object V2 with ID=B and then moves virtual object V2 by specifying point P1, which is located at ID=6. Next, the user specifies virtual object V1 with ID=A and then moves virtual object V1 by specifying point P1, which is located at ID=6. As a result, the three virtual objects are arranged so that they overlap at the same point P1, which is located at ID=6. When placing multiple virtual objects at the same point P1, the HMD 1 displays them overlapping in accordance with the order of placement. For example, the last placed virtual object V1 appears to be in the foreground. Alternatively, the HMD 1 may display only the last placed virtual object at the same point P1.

[0072] Furthermore, when multiple virtual objects are placed at the same point P1, the HMD 1 may display multiple corresponding ID marks N1 near the point P1 so that the multiple IDs of the multiple virtual objects can be easily identified. For example, ID marks N1 (ID=A, B, C) corresponding to virtual objects V1, V2, and V3 are displayed in parallel near point P1 with ID=6.

[0073] As another example of operation, a user can specify multiple virtual objects 110 (virtual objects V1, V2, and V3) in order, and then specify the destination point P1 with ID=6, thereby moving these multiple virtual objects together. In the case of a voice-based method, the user can simply input, for example, "Move objects A, B, and C to number 6." As another example, virtual objects V4 and V5 are placed overlapping the position of point P1 with ID=3. Virtual objects V4 and V5 are application icons, and ID mark N1 is a triangle with ID=D and ID=E.

[0074] The user can also move multiple virtual objects that are overlapping and arranged at the same point P1 to another position. For example, the user can first specify the virtual object V1 at the point P1 with ID=6 using the ID mark N1, and then specify the position of another point P1 (e.g., ID=9) to which the virtual object V1 is to be moved, thereby moving the single virtual object V1. If the user wants to move three virtual objects at the position of ID=6 together, for example, the user specifies the point P1 with ID=6 and then specifies the position of another point P1 to which the virtual object V1 is to be moved. In accordance with this operation, the HMD 1 moves the three virtual objects at the position of ID=6 together. In the case of a voice command, the user can simply input, for example, "move object number 6 to number 9."

[0075] Furthermore, when multiple virtual objects are placed at the same point P1, the HMD 1 may display a predetermined image representing that state. For example, a frame image 142 is displayed at the point P1 of ID=10. The frame image 142 represents the state in which multiple virtual objects are placed at that point P1. In response to a user's operation to select the frame image 142, the HMD 1 selects all of the virtual objects placed at that point P1. Alternatively, the HMD 1 may temporarily display multiple virtual objects in parallel so that they can be confirmed in response to a selection operation on the frame image 142 or its internal area. A speech bubble image 143 shows an example in which each virtual object (virtual objects V6, V7, V8) and each ID (F, G, H) are displayed in parallel.

[0076] [Display control example (3)] 15 shows another example of display control. The image 151 on the display surface 5 has two grids K1, a grid K11 and a grid K12, which are arranged in the respective regions. In this example, a first type grid K11 having ten points P11 (IDs 6 to 15) arranged in two rows and five columns is arranged in the upper region of the display surface 5, and a second type grid K12 having five points P12 (IDs 1 to 5) arranged in one row and five columns is arranged in the lower region. In this example, the grid K11 is arranged in the world coordinate system CS1, and the grid K12 is arranged in the inertial coordinate system CS3.

[0077] The two types of grids K1 may be displayed in different ways so that the user can easily distinguish between the different placement coordinate systems. In this example, the shape of the ID mark M11 at point P11 of grid K11 is a shape (e.g., a diamond) that represents the world coordinate system CS1, and the shape of the ID mark M12 at point P12 of grid K12 is a shape (e.g., a circle) that represents the inertial coordinate system CS3. Other display examples for distinguishing between the placement coordinate systems include displaying a frame or boundary line surrounding the area of each grid K1, or displaying an image representing a grid ID (or area ID) in the area of each grid K1. Also, in this example, IDs are assigned to the ID marks M1 on the display surface 5 so that the same ID value is not duplicated across the two grids K11 and K12. Alternatively, the same ID value may be duplicated for each grid K1. However, in this case, since the position cannot be identified by specifying the ID value alone, it is necessary to additionally specify the grid ID, etc.

[0078] This example also illustrates an example of arranging and moving virtual objects between coordinate systems. Assume that initially, virtual object V1 is located at point P12 with ID=3 on grid K12. Assume that the user moves this virtual object V1 to point P11 with ID=7 on grid K11. In the case of a voice-based system, the user inputs, for example, "Move object A to number 7." In response to this operation, the HMD 1 moves virtual object V1, which is currently located at ID=3 on grid K12 in the inertial coordinate system CS3, to ID=7 on grid K11 in the world coordinate system CS1. Accordingly, the coordinate system to which virtual object V1 belongs is automatically changed from the inertial coordinate system CS3 to the world coordinate system CS1. As another example, initially, virtual object V2 is located at point P11 with ID=14 on grid K11. The user moves this virtual object V2 to point P11 with ID=5 on grid K12. In the case of a voice-based system, the user inputs, for example, "move object B to number 5." In response to the operation, the HMD 1 moves the virtual object V2, which is located at ID=14 in grid K11 of the world coordinate system CS1, to point P12, which is located at ID=5 in grid K12 of the inertial coordinate system CS3. The coordinate system to which the virtual object V2 belongs is automatically changed from the world coordinate system CS1 to the inertial coordinate system CS3. However, similar control is possible between the world coordinate system CS1 and the local coordinate system CS2, and between the local coordinate system CS2 and the inertial coordinate system CS3.

[0079] The inertial coordinate system CS3 can change its front direction (direction DIR3) based on a rotational movement operation described below, and accordingly, the area of the grid K12 displayed on the display surface 5 can be changed. For example, the area of the grid K12 shown in the figure may also be continuous on the left and right sides outside the display surface 5. This allows the user to display other virtual objects placed in the grid K12 of the inertial coordinate system CS3 on the display surface 5, or to place a virtual object displayed on the display surface 5 outside the display surface 5, for example.

[0080] Multiple grids K1 for each coordinate system are set in advance as default settings or user settings in the HMD 1. For each grid K1, the placement coordinate system, the number of points P1, the display mode of the ID mark N1, the area within the display surface 5, etc. can be set. The user can perform tasks by using multiple grids K1 appropriately.

[0081] [Display control example (4)] FIG. 16 shows another example of display control. A grid K1 is arranged on an image 161 on the display surface 5. Conceptually, this grid K1 is composed of multiple regions 152. The regions 152 can be referred to as blocks, reference regions, or grid regions. The center point (not shown) of each region 152 corresponds to the aforementioned point P1. In this example, the regions 152 are displayed as virtual images of rectangular dashed-line frames (corresponding to grid lines). For example, in the upper left corner of each region 151, an image of ID 153 corresponding to ID mark M1 is attached and displayed. Even with this grid K1 configuration, virtual objects can be similarly arranged in each region 152. In this example, virtual objects V1, V2, V4, and V5 are arranged in the regions 152 with IDs 3, 7, 13, and 15. The virtual objects V4 and V5 are examples of application icons. For example, in the lower right corner of the region 152, a virtual object ID mark N1 is displayed.

[0082] [Display control example (5)] FIG. 17 shows another example of display control. A predetermined area of the image 171 on the display surface 5, for example, an area near the bottom edge, is set as a home area H0. The home area H0 is used as a home position, work area, etc. for the user's work. The home area H0 may be displayed, for example, as an image with a dashed frame. For example, ID=0 is assigned and displayed in the home area H0. The user can freely place virtual objects in the home area H0. The user can set the home area H0 to a desired area on the display surface 5 by setting or operating it. The operation for setting the home area H0 may be, for example, an operation for specifying the upper left point and the lower right point of the home area H0. The user can also perform operations such as moving and enlarging / reducing the home area H0. In another embodiment, a grid K1 may be placed within the home area H0.

[0083] Image 171 (A) shows a state where nothing is placed within the home area H0. A grid K1 is placed in the upper area of the display surface 5. The grid K1 is placed, for example, in the world coordinate system CS1. Virtual objects V1, V2, and V3 are placed at points P1 of the grid K1, for example, at IDs 7, 8, and 9. Virtual objects V6, V7, and V8 are placed over the point P1 of the grid K1, for example, at ID 5. Image 171b (B) shows a state where virtual objects V1, V2, and V3 are placed within the home area H0. For example, in the state shown in (A), a user can specify and retrieve a desired virtual object using the menu or the like, and place it within the home area H0. A user can also specify a virtual object on the grid K1 and move it within the home area H0. For example, in the case of a voice command, a user can simply input, for example, "Move object A to home (or number 0)." The HMD 1 places virtual object V1 at the center of the home area H0 in accordance with the operation.

[0084] Within the home area H0, the HMD 1 may place virtual objects at positions freely specified by the user, or may place virtual objects in automatically determined aligned positions. For example, if only virtual object V1 is present within the home area H0 and virtual objects V2 and V3 are then moved into the home area H0, the home area H0 will be in the state shown in image 171b in (B). Within the home area H0, the three virtual objects V1, V2, and V3 are aligned at equal intervals, along with their respective ID marks N1. As another example, multiple virtual objects may be placed overlapping each other within the home area H0.

[0085] The user can also move all virtual objects in the home region H0 together onto the grid K1. For example, if the user wants to move virtual objects V1, V2, and V3 together to position ID=1, the user can input, for example, "move the object in home (or number 0) to position ID=1" in the voice mode. If the user wants to move all virtual objects at position ID=5 on grid K1 together to the home region H0, the user can input, for example, "move object number 5 to home (or number 0)." The user can also move all virtual objects in grid K1 together to the home region H0. For example, the user can input, for example, "move all objects to home (or number 0)" in the voice mode. The user can also move multiple virtual objects that are freely positioned within the home region H0 together to aligned positions on grid K1. In this case, the grid ID (or region ID) set for the grid K1 region is used. For example, grid ID=R1. For example, the user can input, for example, "place the object in home (or number 0) in R1" in the voice mode. In response to the operation, the HMD 1 aligns and places a plurality of virtual objects in the home region H0 at the positions of a plurality of vacant points P1 in the grid K1 region.

[0086] [Display control example (6)] FIG. 18 shows another example of display control. This example illustrates an operation for changing the coordinate system. In an image 181 on the display surface 5, a home region H0 is arranged on the lower side and a grid K11 is arranged on the upper side. A virtual object V1 is arranged within the home region H0. The grid K11 is arranged in a world coordinate system CS1. A virtual object V2 is arranged at a point P1 with ID=3 on the grid K11. An image of a grid ID or region ID (in this example, "CS1") indicating that the placement coordinate system is the world coordinate system CS1 may be displayed in the region of the grid K11. In addition, a button 182 for changing the coordinate system is displayed on the display surface 5. This button 182 can be used as a coordinate system setting instruction. The user can change the placement coordinate system of the grid K11 using the button 182. For example, to change the placement coordinate system of the grid K11 to the local coordinate system CS2, the user designates the grid K11 and presses the button 182 for designating the local coordinate system CS2. The grid K11 may be specified by specifying a grid ID or the like, or by selecting a grid line of the grid K11. In response to this operation, the HMD 1 changes the coordinate system for locating the grid K11 from the world coordinate system CS1 to the local coordinate system CS12. In accordance with this change, the virtual object V2 located on the grid K11 is changed to be located in the local coordinate system CS2. In this example, the virtual object V2 is fixed at the position ID=3 on the display surface 5, and is therefore maintained in the same position even if the user moves around or turns their head.

[0087] Similarly, the HMD 1 can easily change the grid K1 of the world coordinate system CS1 to the grid K1 of the inertial coordinate system CS3 in response to a coordinate system setting instruction, including pressing a button 182 specifying the inertial coordinate system CS3. Image 181b in (B) shows a display example in which the grid K11 of the world coordinate system CS1 in (A) is changed to the grid K31 of the inertial coordinate system CS3. The HMD 1 updates the coordinate system information 31, grid data 33, and virtual image data 32 in response to the change. The ID mark M1 is changed from a diamond representing the world coordinate system CS1 to a circle representing the inertial coordinate system CS3. After the change, the number, position, and ID value of the point P1 are maintained. In this case, the area of the grid K31 can be rotated and moved, as described below. In response to this operation, the virtual object V2 can be placed outside the display surface 5.

[0088] [Display control example (7)] FIG. 19 illustrates another example of display control, which allows a user to adjust the position, orientation, display size, and so forth of a virtual object on a grid K1. As described above, a virtual object can be placed at point P1 on the grid K1. However, depending on the task, it may be necessary to adjust the placement of the virtual object more precisely. In such cases, the user can use this function. Image 191 in (A) includes a grid K1. This example illustrates a case in which a virtual object V1 placed at point P1 with ID=0 is moved to the vicinity of point P1 with ID=7 on the grid K1. This movement operation is similar to that described above. Next, the user adjusts the placement position of the virtual object V1. The user performs a predetermined instruction operation for this purpose. This instruction operation may be, for example, a voice input such as "adjust," or may be performed by pressing an adjustment button 195 displayed on the display surface 5. The HMD 1 transitions to an adjustment mode in response to the instruction operation.

[0089] (B) shows an example display in the adjustment mode. This image 191b displays an area 193 for adjusting placement. The area 193 is based on an enlarged copy of the area 192. The area 192 is an area of a predetermined size centered on point P1 with ID=7, where the virtual object V1 was initially placed. In this example, the area roughly encompasses IDs 1, 2, 3, 6, 8, and 0 surrounding ID=7. The area 193 displays an adjustment grid K1b associated with the original grid K1. The grid K1b in the area 193 has a larger number of points P1 than the original grid K1. In this example, the grid K1b has two points P1 added between the points P1 of the original grid K1, doubling the density. However, this is not limiting and multiple points P1 may be added to the grid K1b. In this example, a new ID mark M1b (e.g., a square with a lowercase English ID) is assigned and displayed to the added point P1b. However, the IDs of the entire grid K1b may be reassigned.

[0090] The user can adjust the placement position of the virtual object V1 by moving it within the grid K1b in the area 193 using a predetermined operation. For example, in the case of a voice command, the user can input, for example, "move to position f." This causes the HMD 1 to move the virtual object V1 from point P1 (ID=7) to point P1b (ID=f). The user can also move the target virtual object up, down, left, or right by operating movement buttons 196 indicated by up, down, left, and right arrows displayed on the display surface 5. The area 193 is not limited to displaying the grid K1b; it may also enable pixel-level position adjustment of the target virtual object in response to a predetermined operation. The area 193 also allows the display size of the target virtual object to be changed (enlarged / reduced, etc.) and the orientation of the target virtual object (rotation state in three-dimensional space) to be changed in response to a predetermined operation. The user can exit the adjustment mode and return to the normal state by performing a predetermined operation, such as pressing the end adjustment button 197.

[0091] The following methods can be applied to control the orientation of a target virtual object when it is moved to point P1. One method maintains the orientation of the virtual object before and after the movement. Another method automatically changes the orientation of the virtual object before and after the movement. For example, the HMD1 selects the placement orientation of the virtual object in accordance with the coordinate system of the destination grid K1. For example, the HMD1 changes the front direction of the virtual object to match the vertical direction of the side facing the user (HMD1) on the grid plane to which the destination point P1 belongs.

[0092] [Display control example (8)] FIG. 20 shows another example of display control in which multiple regions are set in space, multiple virtual objects in each region are treated as a group, and each group can be operated collectively. An image 201 on the display surface 5 in FIG. 20 has multiple regions set therein. In this example, the display surface 5 has a region R1 near the center, a region R2 near the top edge, a region R3 near the bottom edge, a region R4 near the left edge, and a region R5 near the right edge. The region R1 near the center is set to a world coordinate system CS1. A real object 403, a virtual object V11, and the like are arranged in the region R1. The region R1 may be set to an inertial coordinate system CS3. The region R2 near the top edge has a menu section set to a local coordinate system CS2. A virtual object V21 (e.g., an application icon) and the like are arranged in the region R2. The region R3 near the bottom edge has a home region set to an inertial coordinate system CS3. A virtual object V31 and the like are arranged in the region R3. Region R4 near the left side and region R5 near the right side are set as inertial coordinate system CS3. Virtual objects V32 and the like are placed in region R4. Virtual objects V33 and the like are placed in region R5. Although not shown, a grid of a corresponding coordinate system is placed in each region. To distinguish between regions, an image of a frame or boundary line may be displayed, or a region ID or the like may be displayed.

[0093] A user can operate multiple virtual objects in an area as a group. This allows the user to work efficiently while using multiple areas appropriately. For example, multiple virtual objects in area R1 are group 1, multiple virtual objects in area R4 are group 2, and multiple virtual objects in area R5 are group 3. A user can move multiple virtual objects in group 1 in area R1 collectively to another area, such as area R4. To do so, the user, for example, in the case of a voice-based system, inputs a command such as "move the objects in the center (or group 1, R1, etc.) to the left (or group 2, R4, etc.)." Similarly, a user can move multiple virtual objects in group 3 in area R5 collectively to another area, such as area R1. To do so, the user, for example, in the case of a voice-based system, inputs a command such as "move the objects on the right (or group 3, R5, etc.) to the center (or group 1, R1, etc.)." When moving virtual objects in groups, the placement coordinate system, etc. of each virtual object is automatically changed, as described above.

[0094] The display size of the placed virtual object may also be different for each region. For example, the display size of the placed virtual object may be made larger in the central region R1 to make it stand out. The display size of the same virtual object is automatically changed depending on the placement region.

[0095] Because regions R3, R4, and R5 are set as inertial coordinate systems CS3, the display content can be switched by rotating and moving the regions as described below. The HMD 1 may also set multiple inertial coordinate systems CS3 relative to the world coordinate system CS1. For example, the three regions R3, R4, and R5 may each be set as a grid region of an independent inertial coordinate system CS3. The HMD 1 manages the grid region of each inertial coordinate system CS3 as a unit such as a group or page. The user can arrange virtual objects by using the groups or pages of each inertial coordinate system CS3 depending on the task, thereby enabling efficient work. For example, the user may operate region R4 with the left hand and region R5 with the right hand. The user can also switch the display of each inertial coordinate system CS3 region on or off. As described above, various coordinate system combinations are possible, or a system using only one type of coordinate system is also possible.

[0096] [Example of placement control] FIG. 21 illustrates another example of display control, in which a virtual object not currently displayed on the display surface 5 is newly placed and displayed at a desired position on a grid K1. As an example of this placement control, a case where a virtual image such as an application icon is placed will be described. In an image 211 on the display surface 5 in (A), a grid K2 is placed. The grid K2 has, for example, 24 points P2 arranged in 4 rows and 6 columns. The grid K2 is placed, for example, in a local coordinate system CS2. An ID mark M2 is displayed at point P2 on the grid K2. The ID mark M2 is, for example, a triangle, and an ID number (1 to 24) is displayed on the ID mark M2. Furthermore, at least one of the following regions on the display surface 5, a region 212 near the top edge, a region 213 near the bottom edge, a region 214 near the left edge, or a region 215 near the right edge, is secured as a predetermined region using the local coordinate system CS2. For example, the aforementioned menu box, system information, etc., are placed in this predetermined region.

[0097] The user performs a predetermined operation including specifying a target virtual object and specifying a placement point P2. In response to the operation, the HMD1 places and displays the target virtual object at the position of the specified point P2 on the grid K2. For example, the user wants to place a virtual object V4 corresponding to the application icon of application X as the target virtual object at the position of point P2 of ID=2. In the case of a voice method, the user inputs, for example, "Place the object (or icon) of X (or application X) at number 2." In the case of a gesture method or cursor method, the user first specifies the target application icon in, for example, a menu field or a pop-up field (not shown), and then specifies the placement point P2. Alternatively, as another example of an operation, the user may specify the placement point P2 and then specify the target virtual object.

[0098] In accordance with this operation, the HMD1 places and displays a virtual object V4 corresponding to the icon of the specified application X at the position of point P1 with the specified ID=2, as shown in image 211b in (B). Furthermore, the HMD1 may proceed with the startup process of application X in the background while placing the icon of application X.

[0099] As another example, a user can launch and place an application window 413 at a desired position of point P2. For example, assume that the user wants to launch and place application window 413 of application Y at the position of point P2 of ID=11 from the state (A). In the voice method, the user inputs, for example, "Place (or launch) window Y (or application Y) at number 11." In the gesture method or cursor method, the user specifies the target application in, for example, a menu field (not shown), and then specifies the placement point P2. In accordance with the operation, HMD 1 executes the launch process of the specified application Y, and places and displays application window 413 of the specified application Y at the position of point P2 of the specified ID=11.

[0100] The user can also move the application icons and application windows arranged on the grid K1 to another position on the grid K1 by the same operation as described above. The predetermined operation for controlling the arrangement may further include an instruction (corresponding command) for arrangement or activation. The instruction may be made possible by a button or the like displayed on the display surface 5, as described above.

[0101] Furthermore, for example, while the user is using the HMD 1, the HMD 1 may start an application (application program 22) in response to an external communication or other trigger. For example, if the HMD 1 is equipped with a telephone application, the HMD 1 may receive an incoming call from the outside. In this case, the HMD 1 displays information about the icon or window of the telephone application, which is the target virtual object, on the display surface 5. At this time, the HMD 1 displays, for example, a GUI image (e.g., a pop-up field) on the display surface 5 that inquires of the user about the placement location of the icon or window of the telephone application, which is the target virtual object. In response to the inquiry, the user performs an operation to specify a desired placement point P2. In response to this operation, the HMD 1 places the icon or window of the telephone application at the position of the specified point P2.

[0102] The placement control function described above allows virtual objects such as icons that were not initially displayed to be easily placed in the user's desired location, such as a suitable location for easy work. For example, real or virtual objects for work are placed in a central area of the display surface 5. The user can place app icons or other objects in a suitable peripheral location so as not to obstruct the view of the real or virtual object near the center, making work easier. Furthermore, when a virtual object for work has already been placed on the display surface 5, the user can call up and place other virtual objects for work in a nearby location near the virtual object. For example, while operating a real device, the user can place a virtual object such as a manual for operating that device next to the device. The user can select a suitable, open location that does not interfere with the device's operation and is not too far away to place the virtual object. Furthermore, for example, the user can place command buttons or other items near the virtual object of a 3D model being created, thereby efficiently facilitating the creation process.

[0103] The above placement control example is for placement in the local coordinate system CS2. Therefore, even if the user rotates their head (the corresponding HMD1), the application icons, etc. are maintained in the same position on the display surface 5. Not only GUI images such as application icons, but also virtual objects within an application can be similarly placed in desired positions. Furthermore, similar control can be applied not only to the local coordinate system CS2, but also to the world coordinate system CS1 and the inertial coordinate system CS3. Furthermore, the placement control of the application icons, etc. can also be performed in advance as a user setting. Note that, as in the above example, if the application icons, etc. can be identified by their image or name, it is not necessary to assign and display the ID mark N1 (label).

[0104] Coordinate System Next, three types of coordinate systems will be described. To manage the placement of virtual objects in three-dimensional space, the HMD 1 uses three types of coordinate systems: a world coordinate system CS1, a local coordinate system CS2, and an inertia coordinate system CS3. A coordinate system calculation unit 13 (see FIG. 6, etc.) of the HMD 1 appropriately sets and calculates the correspondence between each coordinate system. The coordinate system calculation unit 13 calculates the relative relationship between the HMD 1 and the user and the coordinate system for placing virtual objects using the camera 6 and sensor 70. The HMD 1 and the user can place the grid K1 and virtual objects in any coordinate system selected from the three types of coordinate systems. The HMD 1 places the grid K1 and virtual objects in a coordinate system selected according to a predetermined rule. The HMD 1 places the grid K1 and virtual objects in a coordinate system selected by the user, for example.

[0105] The world coordinate system CS1 and the local coordinate system CS2 are coordinate systems according to known techniques. The world coordinate system CS1 is a first coordinate system fixed to real space. The local coordinate system CS2 is a second coordinate system fixed to the display surface 5 as seen from the viewpoint of the HMD 1 and the user. The inertial coordinate system CS3 is a third coordinate system that compensates for the shortcomings of the world coordinate system CS1 and the local coordinate system CS2. The coordinate origin of the world coordinate system CS1 is the origin G1, and the direction is the direction DIR1. The coordinate origin of the local coordinate system CS is the origin G2, and the direction is the direction DIR2. The coordinate origin of the inertial coordinate system CS3 is the origin G3, and the direction is the direction DIR3.

[0106] The origin G3 of the inertial coordinate system CS3 is set to be the same as the origin G2 of the local coordinate system SC2, for example, and follows the positions of the HMD1 and the user's head and viewpoint. The direction DIR3 of the inertial coordinate system CS3 is set to be fixed relative to the direction DIR1 of the world coordinate system CS1. The forward direction (direction DIR1) of the inertial coordinate system CS3 represents the reference direction of the inertial coordinate system CS1. This direction DIR1 remains the user's reference direction even if the user temporarily changes the direction of their head, in other words, even if the orientation of the HMD1 (corresponding rotational state) changes. The user's reference direction is, for example, the direction of their trunk, which is the average direction in which their torso, etc., is facing. The direction DIR3 can be changed as appropriate in response to a predetermined operation by the user (a rotational movement operation, described below) or a predetermined control of the HMD1. A characteristic of the inertial coordinate system CS3 is that the origin G3 moves in response to the movement of the user and the HMD1, but the direction DIR3 is fixed relative to the rotation of the user's head and the HMD1. Another feature of the inertial coordinate system CS3 is that it allows virtual objects to be placed in a space wider than the range of the display surface 5 of the local coordinate system CS2. Another feature of the inertial coordinate system CS3 is that the user can refer to virtual objects in a desired area as needed by changing the direction of their head or the direction DIR3.

[0107] The HMD1 calculates and sets the direction DIR2 of the local coordinate system CS2 and the direction DIR3 of the inertial coordinate system CS3 as directions relative to the world coordinate system CS1, with the direction DIR1 of the world coordinate system CS1 as the reference. The HMD1 expresses the directions DIR2 and DIR3 by a rotation operation when the world coordinate system CS1 is rotated. Calculations of such rotation operations of vectors in three-dimensional space can be realized using the Euler angles or normalized quaternions mentioned above.

[0108] World Coordinate System FIG. 22 is an explanatory diagram of the world coordinate system CS1. The world coordinate system CS1 is a coordinate system having a point fixed in real space as the origin G1, and three coordinate axis directions constituting a three-axis Cartesian coordinate system fixed in real space. The three coordinate axis directions of the world coordinate system CS1 are (X W ,Y W ,Z WThe position of the world coordinate system CS1 is expressed by the position coordinates (x W0 ,y W0 ,z W0 The direction DIR1 of the world coordinate system CS1 is expressed as one coordinate axis direction, for example, axis X W The position of the virtual object placed in the world coordinate system CS1 is expressed by the coordinates (x W ,y W ,z W ) axis Z W is aligned vertically. Axis X W ,Y W are aligned with two orthogonal directions that form a horizontal plane.

[0109] The lower part of FIG. 22 shows an example of the placement of a virtual object in the world coordinate system CS1 and an example of the movement of the user and the HMD 1. For example, the position of point Pw1 (x W1 ,y W1 ,z W1 ), a virtual object V1 is placed at point Pw2. A virtual plane 221 indicates a virtual plane (corresponding to the field of view) on which point Pw1 is placed. A direction 222 indicates the placement direction of the virtual object V1. Initially, the user and the HMD 1 determine the position (x W2 ,y W2 ,z W2 ). Point Pw2 is the center of the HMD1 and the head, and corresponds to the viewpoint. Direction 223 indicates the front direction of the HMD1 and the direction of the user's head. In addition to these directions, there is also the user's line of sight. For example, suppose the user and HMD1 move from point Pw2 to point Pw3. With this movement, the direction of the HMD1 and head changes to direction 224. The position (point Pw1) and direction 222 of virtual object V1 placed in world coordinate system CS1 do not change despite this movement and rotation (corresponding positional and directional changes). From the user's perspective, the way virtual object V1 appears changes. Depending on the state after the movement, the user may not be able to see virtual object V1 well.

[0110] Local Coordinate System FIG. 23 shows an explanatory diagram of the local coordinate system CS2. The local coordinate system CS2 is a three-axis Cartesian coordinate system. The three coordinate axis directions of the local coordinate system CS2 are (X L ,Y L ,Z L The HMD1 calculates the position and direction DIR2 of the origin G2 of the local coordinate system CS2 relative to the world coordinate system CS1 set at initialization. The position of the local coordinate system CS2 is calculated based on the position (x L0 ,y L0 ,z L0 The direction DIR2 of the local coordinate system CS2 is expressed as one axis direction, e.g., axis X L The position of a virtual object placed in the local coordinate system CS2 is expressed by the coordinates (x L ,y L ,z L The origin G2 of the local coordinate system CS2 is set relative to the position of the HMD1, the head position, and the viewpoint position.

[0111] The lower part of FIG. 23 shows an example of movement of the user and HMD1 in the world coordinate system CS1 and an example of placement of a virtual object in the local coordinate system CS2. For example, initially, the user and HMD1 are at the position (x W4 ,y W4 ,z W4 ) axis X of local coordinate system CS2 L is set according to the front direction of the HMD1 and the direction of the head. The left and right direction of the head is the axis Y L The vertical axis is the Z axis. L At this time, on the display surface 5, the position of the point PL1 (x L1 ,y L1 ,z L1 ), and virtual object V1 is placed at point Pw1. For example, the user and HMD1 move from point Pw4 to point Pw5. Furthermore, with this movement, the direction of the HMD1 and head changes from direction 233 to direction 234. For example, the head direction rotates to the right. With this movement and rotation, the state of local coordinate system CS2 is maintained, and the appearance of virtual object V1 on virtual plane 231 as seen by the user is maintained.

[0112] [Inertial coordinate system (1)] FIG. 24 is an explanatory diagram of the inertial coordinate system CS3. The inertial coordinate system CS3 has three coordinate axis directions (X I ,Y I ,Z I The position of the inertial coordinate system CS3 is shown by the position of the origin G3 (x I0 ,y I0 ,z I0 The direction DIR3 of the inertial coordinate system CS3 is the direction of one axis, e.g., the axis X I The position of the virtual object placed in the inertial coordinate system CS3 is expressed by the coordinates (x I ,y I ,z I )

[0113] The lower part of FIG. 24 shows an example of the movement of the user and the HMD1 in the world coordinate system CS1 and an example of the placement of the virtual object in the inertial coordinate system CS3. For example, initially, the user and the HMD1 are at the position (x W6 ,y W6 ,z W6 ) at this position. The origin G2 of the local coordinate system CS2 and the origin G3 of the inertial coordinate system CS3 are set. The front direction of the inertial coordinate system CS3 (axis X I ) is the direction DIR1 (axis X) of the world coordinate system CS1. W ) are set to coincide with the inertial coordinate system CS3. The three axes of the inertial coordinate system CS3 overlap with the three axes of the local coordinate system CS2. At this time, on the display surface 5, on the virtual surface 241 (corresponding to the field of view FOV1) corresponding to the inertial coordinate system CS3, for example, the position (x I1 ,y I1 ,z I1 ), a virtual object V1 is placed on the left side of the virtual surface 241. Another virtual object V1b is placed on the left side of the virtual surface 241. Another virtual object V1c is placed on the extension of the virtual surface 241, but outside the field of view FOV1.

[0114] For example, suppose that the user and HMD1 move from point Pw6 to point Pw7. Also, suppose that the direction of the HMD1 and head changes from direction 243 to direction 244 in accordance with this movement. For example, the head direction rotates about 45 degrees to the left. In response to this movement and rotation, the origin G3 of the inertial coordinate system CS3 moves in accordance with the origin G2. Meanwhile, the direction DIR3 (axis X) of the inertial coordinate system CS3 I ) remains fixed relative to the direction DIR1 of the world coordinate system CS1, just as it was before the movement. As the user rotates their head, the field of view from the user's viewpoint changes to field of view FOV2. On the display surface 5, the virtual surface 242 corresponding to that field of view FOV2 changes the visible virtual objects. On the virtual surface 242, virtual objects V1b and V1c are displayed, and virtual object V1 is no longer visible. In this way, the display area of the virtual objects on the inertial coordinate system CS3 can be changed according to the direction of the HMD 1 and the user's head.

[0115] The virtual surfaces 241 and 242 in FIG. 24 are curved surfaces. The grid K1 arranged in the inertial coordinate system CS3 may be a flat surface or a curved surface. When arranging a virtual object in the inertial coordinate system CS3, the HMD 1 calculates the position of the virtual object on the grid K1 of the inertial coordinate system CS3. If a virtual object arranged on the grid K1 of the inertial coordinate system CS3 is included within the range of the display surface 5, the HMD 1 displays the virtual object.

[0116] [Inertial coordinate system (2)] FIG. 25 shows a configuration example when a grid K1 is arranged in an inertial coordinate system CS3. In the example of FIG. 25, the positions of the user and HMD1 are set as the origin G3 of the inertial coordinate system CS3, and multiple virtual surfaces (virtual surfaces VS1, VS2, VS3, etc.) are arranged around it along an approximately cylindrical surface. The multiple virtual surfaces together form a single area of an approximately cylindrical surface. In this example, each virtual surface is a two-dimensional plane. The direction DIR1 (axis X) of the front of the inertial coordinate system CS3 is I), there is an image 251 on the display surface 5. There is a virtual surface VS1 in the image 251. A grid K31 is arranged on the virtual surface VS1. Adjacent to the left side of the virtual surface VS1 is a virtual surface VS2, and adjacent to the right side is a virtual surface VS3. A grid K32 is arranged on the virtual surface VS2, and a grid K33 is arranged on the virtual surface VS3. The grids K31, K32, K33, etc. constitute one grid K3 of the inertial coordinate system CS3, and the grid K3 is arranged in a region of a roughly cylindrical surface. Virtual objects can be arranged on the grids of each virtual surface. For example, virtual objects v1, v2, and v3 are arranged on the grid K31 of the virtual surface VS1. A virtual object v4 is arranged on the grid K32 of the virtual surface VS2. A virtual object v5 is arranged on the grid K33 of the virtual surface VS3. As in FIG. 24, the user can display the virtual object v4 on the virtual surface VS2 and the virtual object v5 on the virtual surface VS3 within the display surface 5 by changing the direction of his or her head.

[0117] As another operation, the user can also perform a rotational movement operation of the inertial coordinate system CS3. This operation is one of the coordinate system setting instructions in step S10 of FIG. 10 described above. The user can perform, for example, a left rotation operation 252 or a right rotation operation 253. This operation can be defined as a predetermined operation using a voice method, a gesture method, or the like. As an example, it may be a gesture of moving the hand to the left or right like a swipe operation. In the case of a voice method, it may be an input such as "rotate left". When the HMD1 recognizes this rotational movement operation, it determines the direction DIR3 (axis X) of the inertial coordinate system CS3. I ) is rotated. From the state of (A), for example, a left rotation operation 252 is performed. (B) shows the state after rotation. Axis X I and axis Y I is the Z axis I The virtual object VS1 is rotated, for example, by about 45 degrees around the vertical direction. In the image 251b on the display surface 5, the virtual surface VS1 has moved to the left, and the virtual surface VS3 that was on the right is now displayed near the center. As a result, the virtual object v5 is visible on the display surface 5.

[0118] When using this rotational movement operation, the user can change the appearance of virtual objects on the grid K3 of the inertial coordinate system CS3 without having to rotate their head. Real objects, virtual objects on the world coordinate system CS1, and virtual objects on the local coordinate system CS2 are also displayed on the display surface 5. Therefore, the user can switch between virtual objects displayed on the inertial coordinate system CS3 while visually recognizing the real objects, virtual objects on the world coordinate system CS1, and virtual objects on the local coordinate system CS2 in the same positions on the display surface 5. The user can use the large space of the inertial coordinate system CS3 as an extended area of the display surface 5 to handle multiple virtual objects, enabling efficient work. The user can also set or instruct the use of the inertial coordinate system CS3 to be on or off.

[0119] Furthermore, as an example of using the inertial coordinate system CS3, the user can treat each virtual surface as a group. For example, the user can work by using multiple virtual objects placed on virtual surface VS1 as a first group, multiple virtual objects placed on virtual surface VS2 as a second group, etc. The user can also move a specified virtual surface or group to the center of the display surface 5 by specifying a virtual surface or group. The operation of specifying a virtual surface or group may be, for example, an operation of an area frame or a specification of a group ID, etc. The HMD 1 can also move multiple virtual objects collectively between groups of virtual surfaces. The user specifies a source group and a destination group as a predetermined operation. For example, in the case of a voice-based system, the user inputs, for example, "move the first group to the second group." The HMD 1 moves all virtual objects in the virtual surface VS1 collectively into the virtual surface VS2 in accordance with the operation. In addition, at this time, the HMD1 automatically places the multiple virtual objects on the grid K32 of the destination virtual surface VS2 while maintaining as much as possible the positional relationship of the multiple virtual objects on the grid K31 of the source virtual surface VS1. Alternatively, the HMD1 may select multiple vacant points on the destination grid K32 and place the multiple source virtual objects in an automatically aligned state. Such group movement is also possible between different coordinate systems. Moving multiple virtual objects in groups can significantly reduce the effort required to move multiple virtual objects.

[0120] As a modified example, an exchange operation may be possible in units of virtual surfaces or groups. The user specifies two virtual surfaces or groups to be exchanged. For example, the user specifies virtual surface VS1 and virtual surface VS2. In accordance with the operation, the HMD 1 moves the group of all virtual objects on virtual surface VS1 so as to exchange the group of all virtual objects on virtual surface VS2 with the group of all virtual objects on virtual surface VS1, and updates the setting information.

[0121] [Inertial coordinate system (3)] 26 shows another example of the configuration of the grid K3 in the inertial coordinate system CS3. This grid K3 is configured on a cylindrical surface. From the origin G3 of the inertial coordinate system CS3 corresponding to the user's viewpoint position, the axis X I and axis Y I The grid lines are set to extend radially in the direction of the X axis. I and axis Y I The HMD1 has cylindrical surfaces at positions (for example, four positions) at a predetermined distance in the direction (radial direction of the cylinder). The cylindrical surfaces also have grid lines in the circumferential direction. Points P3, which are grid points, are provided on each cylindrical surface. Points P3 are provided in all directions from the origin G3. The HMD1 displays, on the display surface 5, a portion of the grid K3 that corresponds to the field of view FOV1 in the front direction of the user and the HMD1. For example, if the direction of the HMD1 is along the axis X I When the virtual object v1 matches the field of view FOV1, the virtual object v1 is displayed.

[0122] Fig. 27 shows another example of the configuration of the grid K3 in the inertial coordinate system CS3. This grid K3 is configured on a hemispherical surface. From the origin G3 of the inertial coordinate system CS3, which corresponds to the user's viewpoint position, the axis X I and axis Y IGrid lines are provided on the hemisphere at positions at a predetermined distance in the radial direction. Although one hemisphere is shown, grid lines may be provided on multiple hemispheres at similar positions in the radial direction. Point P3 is provided on each hemisphere. The HMD 1 displays, on the display surface 5, a portion of the grid K3 that corresponds to the field of view FOV1 in the forward direction of the user and the HMD 1. With this configuration, for example, even if the user rotates their head up or down, the area of the inertial coordinate system CS3 corresponding to that direction can be used. In the above example, an orthogonal grid is applied to the world coordinate system CS1 and the local coordinate system CS2, and a curved grid is applied to the inertial coordinate system CS3. However, this is not limited to this, and any shape of grid can be applied to any coordinate system.

[0123] [Inertial coordinate system (4)] FIG. 28 shows an example of handling multiple virtual objects using the grid of the inertial coordinate system CS3. Using FIG. 28, it will be explained that multiple virtual objects can be arranged by using the grid of the inertial coordinate system CS3. Initially, in the first state of (A), the user and HMD1 are at position L1. Origins G1, G2, and G3 correspond to position L1. A virtual surface 281 corresponds to a grid on a single cylindrical surface. On the virtual surface 281, multiple virtual objects, for example, virtual objects v1, v2, v3, v4, v5, v6, v7, v8, v9, and v10, are arranged at each position in the circumferential direction. The direction of the user's head and line of sight is determined by the axis X. I At this time, virtual objects v1, v2, and v3 are displayed in the field of view FOV1 on the display surface 5. Next, in the second state of (B), the user rotates his / her head about 45 degrees to the left at the same position L1. At this time, virtual objects v2 and v4 are displayed in the field of view FOV2 on the display surface 5. In this way, the user can see virtual objects such as v4 that are outside the field of view FOV1.

[0124] Next, in the third state (C), the user and HMD 1 move from position L1 to position L2. Position L2 is associated with origins G2 and G3. As this movement occurs, origin G3 moves in parallel, and the multiple virtual objects (virtual objects v1 to v10) in (A) move in tandem while maintaining their positional relationships. At this time, the same virtual objects v1, v2, and v3 are displayed in the field of view FOV1 on the display surface 5. In this way, the user can easily move multiple virtual objects.

[0125] Next, in the fourth state (D), the user performs a rotational movement operation of the inertial coordinate system CS3 at position L2, for example, a left rotation at a rotation angle of 90 degrees. As a result, the direction DIR3 (axis X) of the inertial coordinate system CS3 is changed. I ) is the left-hand direction (axis Y of local coordinate system CS2) L ) As a result of this rotation, the multiple virtual objects on the virtual surface 281 are rotated by a rotation angle of 90 degrees. At this time, virtual object v7 is displayed in the field of view FOV1 on the display surface 5. In this way, the user can place a virtual object in a desired area on the inertial coordinate system CS3 in front of them and view it.

[0126] As described above, by using the grid of the inertial coordinate system CS3, the user can handle a large number of virtual objects with little effort. The direction DIR3 of the inertial coordinate system CS3 can be maintained in accordance with the user's reference direction (for example, the direction of the trunk). For example, the user places virtual objects that the user wants to check or operate frequently in the area of the inertial coordinate system CS3. The user usually performs most of their work in an area near the center of the display surface 5, and can refer to other virtual objects in the inertial coordinate system CS3 as needed by rotating the direction of their head or performing a rotational movement operation.

[0127] As an application, the HMD 1 may set multiple directions in the area of the inertial coordinate system CS3. For example, in a grid arranged on a cylindrical surface as shown in FIG. 26, multiple directions, in other words, multiple positions on the area of the cylindrical surface (for example, positions a1, a2, a3, a4), may be set. The HMD 1 selects the direction (corresponding position) in response to the user's operation, and displays the area of the selected direction in the center of the display surface 5. For example, the user may divide multiple virtual objects into multiple groups depending on the purpose of the work, and set the above direction for each group. For example, in FIG. 26, the front direction (axis X I The virtual object of the first group is placed at position a1 in the left-hand direction (positive direction of axis Y). I The second group is located at position a2 in the right-hand direction (axis Y I The third group is located at position a4 in the rear direction (negative direction of axis X I The fourth group is arranged at position a4 (negative direction of the arrow A4). The user can display on the display surface 5 a group of a direction selected according to the task.

[0128] The rotation of the direction DIR3 of the inertial coordinate system CS3 as shown in (D) of Figure 28, in other words, the change in the display area of the inertial coordinate system CS3, may be temporary. That is, the HMD 1 may maintain, for example, the fourth state (D) for a predetermined time from the time when the user's rotational movement operation is received, and after that time has elapsed, automatically return to the original third state (C). Alternatively, the HMD 1 may maintain the fourth state while the user is performing a predetermined gesture. Furthermore, the HMD 1 may impart inertia (corresponding velocity change) when changing the display state so as to change the direction DIR3 of the inertial coordinate system CS3.

[0129] As another example of a rotational movement operation, the user may specify a target virtual object on the inertial coordinate system CS3 by a predetermined operation, and the HMD 1 may change the direction DIR3 of the inertial coordinate system CS3 so that the target virtual object is displayed at the center position of the display surface 5. As another example of a rotational movement operation, the user may use an action of rotating their head, for example, left or right. For example, in response to the action of the user rotating their head to the left, as in the change from (A) to (B), the HMD 1 changes the direction DIR3 of the inertial coordinate system CS3 as in the example (D).

[0130] As another example, the HMD 1 may automatically change the direction DIR3 of the inertial coordinate system CS3 to match the user's reference direction. The HMD 1 detects the user's reference direction (e.g., the direction of the trunk) using the camera 6 or the sensor 70. In this case, if the user changes the direction of their torso, for example, the direction DIR3 is changed to follow that. The user's reference direction may be limited to the horizontal direction. When the user is moving, the direction of movement may be used as the reference direction. During initialization processing, the HMD 1 may set the direction DIR3 of the inertial coordinate system CS3 to match the user's reference direction. Furthermore, the user can switch between a state in which the direction DIR3 of the inertial coordinate system CS3 can be changed (stationary state) and a state in which it cannot be changed (fixed state) according to settings or instructions.

[0131] Coordinate System Settings An example of the method for resetting the three types of coordinate systems during the initialization process at the start of the HMD1 (step S2 in FIG. 2 described above) is as follows. During the initialization process, the HMD1 sets the world coordinate system CS1 (origin G1 and direction DIR1) based on the position and orientation of the HMD1 at that time. The coordinate system calculation unit 13 detects the direction of gravity based on a three-axis acceleration sensor, and calculates the relationship between this direction of gravity and the axis X of the local coordinate system CS2. Land resets the world coordinate system CS1. At the time of initialization, the HMD 1 aligns the origin G1 of the world coordinate system CS1 with the origin G2 of the local coordinate system CS2. At the time of initialization, the HMD 1 also sets the origin G3 of the inertial coordinate system CS3 to align with the origin G2 of the local coordinate system CS2, and sets the direction DIR3 of the inertial coordinate system CS3 to align with the direction DIR1 of the world coordinate system CS1. The coordinate system calculation unit 13 stores, in the coordinate system information 31, information on the positions (origins G2, G3), directions (directions DIR2, DIR3), and gravitational acceleration vectors at the time of initialization of the local coordinate system CS1 and the inertial coordinate system CS3 relative to the world coordinate system CS1.

[0132] After initialization, the HMD 1 uses the sensors 70 to track changes in the position and orientation of the HMD 1 and updates the settings of each coordinate system as needed in response to those changes. The HMD 1 updates the measurement values of each sensor 70, including the acceleration vector detected by the acceleration sensor and the angular velocity vector detected by the gyro sensor. The coordinate system calculation unit 13 updates the position (origin G2) and direction DIR2 of the local coordinate system CS2 relative to the world coordinate system CS1 based on the updated acceleration vector and angular velocity vector. The coordinate system calculation unit 13 stores the update information in the coordinate system information 31. The coordinate system calculation unit 13 updates the position (origin G3) and direction DIR3 of the inertial coordinate system CS3 relative to the world coordinate system CS1 based on the updated position (origin G2) and direction DIR2 of the local coordinate system CS2 and the rotational movement operation of the inertial coordinate system CS3. The coordinate system calculation unit 13 stores the update information in the coordinate system information 31. The HMD 1 may use position information from a GPS receiver and direction information from a geomagnetic sensor as assistance in calculating each coordinate system.

[0133] In the above example, the origin G3 of the inertial coordinate system CS3 is the same as the origin G2 of the local coordinate system CS2, but this is not limitative and it may be set at a position away from the origin G2. W ,Z L ,Z I ) has been explained, but this is not limited to this, and rotation control is possible for other axes as well. Axis Z of the inertial coordinate system CS3 IFor the setting, axis Z of world coordinate system CS1 W , that is, it may be restricted to align with the vertical direction.

[0134] [3D grid (1)] The grid K1 is not limited to the example shown in FIG. 11, and may have a large number of points P1. Furthermore, the grid K1 is not limited to a two-dimensional grid, and may be a three-dimensional grid. In this case, the number of candidate points P1 within the display surface 5 increases, allowing for more precise position specification within the display surface 5. However, in this case, the number of IDs for points P1 across the entire grid K1 also increases, and the range of ID values becomes larger in order to uniquely identify all IDs. Some ideas for addressing this issue are described below.

[0135] Figure 29 shows an example of a three-dimensional grid configuration as grid K1. In grid K1 (A), points P1 and grid lines are arranged in the directions of three axes, and there are 3 x 3 x 3 = 27 points P1. The X direction is, for example, the axis X of the local coordinate system CS2. L In the depth direction, the corresponding ID is given to all points P1 of the grid K1. When all points P1 of the grid K1 are assigned and displayed with unique IDs using ID marks M1, IDs of 1 to 27 are required, for example. By specifying this ID, the user can specify point P1 where the virtual object will be placed. Image 291 in (B) shows an example of displaying grid K1 in (A) on the display surface 5. In this example, grid K1 is placed in the local coordinate system CS2, and point P1 (ID=5) at the center of the front surface of grid K1 is located on the axis X. L 10 shows a case where the virtual object V1 is placed at the center position of the display surface 5 along the grid K1. For example, the user can place the virtual object V1 at a position on the grid K1 by specifying the target virtual object V1 and specifying the placement point P1 by ID.

[0136] Figure 30 shows the same grid K1 in the depth direction (axis X L) is shown. Virtual objects V1, V2, and V3 are arranged in order from front to back at three points P1 of grid K1, for example, IDs 4, 13, and 22. In this way, by using a three-dimensional grid, a user can also arrange multiple virtual objects in the depth direction. A user can arrange many virtual objects in a limited area within the display surface 5. Note that for virtual objects arranged in the depth direction from the user's viewpoint relative to point P1 of the three-dimensional grid, the display size may be varied depending on the distance from the user to the virtual object. In other words, the farther the virtual object is, the smaller the display size may be. In this way, multiple points P1 and multiple virtual objects may be arranged in the depth direction from the user's viewpoint. Therefore, display techniques described below are effective for making these objects easier for users to see.

[0137] Furthermore, when arranging as in the example of FIG. 30, it is possible to perform the arrangement operation for each individual virtual object, but the following is also possible as an example of an operation. As an example of an operation, the user first specifies the target virtual objects V1, V2, V3 or a group thereof. Next, the user specifies a point P1 (e.g., ID=4) at a desired position on the forefront of the grid K1 (e.g., grid surfaces with ID=1 to 9). Following this operation, the HMD 1 selects three points P1 (ID=4, 13, 22) corresponding to the number of target virtual objects in the depth direction from the specified point P1 with ID=4, and arranges the target virtual objects V1, V2, V3 in a one-to-one correspondence with these points P1.

[0138] The number of points P1 is not limited to the above example, and various settings can be used, such as 8 x 8 x 8 = 512. The HMD 1 provides various configurations of grid K1 (including the number and density of points P1) as default settings, and the user can select and adjust the grid K1 in the settings.

[0139] As a modified example, a configuration is possible in which an ID using an ID mark M1 is not displayed for each point P1 of the grid K1. In this case, an image representing the point P1 of the grid K1 and an image representing the grid lines are displayed on the display surface 5, but an ID image such as a number is not displayed. Although the user cannot specify an ID using voice, the user can specify the placement position of a virtual object by specifying the point P1, grid lines, or grid area using other methods (e.g., gestures such as touch, or pointing with a cursor). Furthermore, when performing the above-mentioned virtual object placement operation, a combination of multiple operation means may be used. For example, the user specifies a target virtual object using a first operation method such as voice or gesture. Next, the user specifies the placement point P1 using a second operation method different from the first operation method.

[0140] FIG. 31 shows an example of displaying a grid K1 or the like on the display surface 5 in a modified example. In this modified example, as shown in image 311 (A), the grid K1 (e.g., a two-dimensional grid) is normally displayed with grid lines, and point P1 is the intersection of the grid lines. The aforementioned circular ID mark or the like is not displayed at this point P1. No ID mark (label) is displayed on the virtual object. For example, when a user wants to move virtual object V1 to point p1 on the grid K1, the user specifies the destination point p1 without using an ID. This operation can be performed using the aforementioned gesture method or the like. Image 311b (B) shows a case in which the user specifies point p1 with their finger or the cursor of the controller 2. When the cursor or the like approaches any point P1 on the grid K1 to a certain extent, the HMD 1 changes the display state of the approaching point P1. In this example, the cursor 312 is shown approaching the vicinity of point p1. The HMD1 displays a circular image 313 representing point P1 at a point p1 located close to the cursor 312 and the four points P1 around it. The HMD1 may also enlarge or change the color of the image 313 for the point p1 closest to the cursor 312 so that it stands out. The HMD1 sets the point p1 that the cursor 312 overlaps to a provisionally selected state or a confirmed selection state, thereby making it easy for the user to select point P1 even if an ID is not displayed in the grid K1.

[0141] [3D grid (2)] In the above example, IDs are displayed using ID marks M1 at all points P1 of the grid K1. However, IDs may be displayed using ID marks M1 at only some of the points P1 of the grid K1. The HMD 1 determines which points P1 will display IDs and which will not, depending on the user's operation, etc. The HMD 1 assigns and displays ID values to some of the points P1 at which IDs will be displayed. In addition, the HMD 1 may continue to use the same ID value consistently for a certain point P1, or may reassign ID values as appropriate. This reduces the number of IDs displayed on the display surface 5 and narrows the range of ID values compared to when IDs are displayed at all points P1 of the grid K1. For the user, the amount of information on the display surface 5 is reduced, making it easier to perform operations such as specifying an ID.

[0142] [Grid Control (1)] An example of display control relating to the ID of point P1 of lattice K1 is shown below. Fig. 32 shows an example in which, in an example of the configuration of a three-dimensional lattice K1, IDs are displayed by ID marks M1 for only some of the points P1, for example, points P1 on a certain lattice surface. The user can perform selection operations on some of the points P1. For ease of explanation, the lattice K1 in Fig. 32 is placed in the world coordinate system CS1, and the case where the lattice K1 is viewed from diagonally above as the user's viewpoint is shown. The following control can also be applied in the case where such a lattice K1 is placed in the local coordinate system CS2 or the inertial coordinate system CS3. The lattice K1 is arranged in the depth direction (axis X L ) and has three grating surfaces, designated as grating surfaces SF1, SF2, and SF3 from the front. When displaying the grating K1 on the display surface 5, the HMD 1 displays an image of an ID mark Q1 representing each grating surface. In this example, the ID mark Q1 is assigned ID=1 for grating surface SF1, ID=2 for grating surface SF2, and ID=3 for grating surface SF3. The ID mark Q1 has, for example, a star shape so that it can be distinguished from other types of ID marks. For example, the ID mark Q1 is displayed at a position connected by a line from point P1 on one of the corresponding grating surfaces, but this is not limiting. The ID mark Q1 may also be displayed connected to the grating line.

[0143] Initially, no ID mark M1 is displayed at point P1 of grid K1. Suppose the user wishes to place, for example, virtual object V1 at point P1, the center of grid K1. After specifying the target virtual object as a predetermined operation, the user specifies the grid surface (e.g., grid surface SF2) that contains the destination point P1. This operation is, for example, an input such as "surface number 2" in the case of a voice method, or an operation of pointing to ID mark Q1 with ID=2 in the case of a cursor method. In accordance with this operation, the HMD1 tentatively selects only the specified grid surface SF2 and deselects the other grid surfaces. The HMD1 displays IDs (e.g., 1 to 9) using the ID marks M1 for multiple points P1 belonging to the specified grid surface SF2. The HMD1 accepts the operation of specifying point P1 for the tentatively selected grid surface SF2, but does not accept the operation for the unselected grid surfaces SF1 and SF3.

[0144] Next, the user specifies the placement destination point P1, for example, point p1 with ID=5, from the provisionally selected lattice surface SF2. For example, the user inputs "number 5" using the voice method, or points to point p1 with ID=5 using the cursor method or the like. The HMD1 follows this operation and confirms the selection of the specified point p1 with ID=5. The HMD1 then places the virtual object V1 at the position of point p1 with ID=5. As described above, even if there are many points P1 in the three-dimensional lattice K1, the user can easily specify one point p1 from a small range of ID values.

[0145] Furthermore, after a user has provisionally selected a lattice surface, the user can easily select another lattice surface. For example, if the user selects lattice surface SF3 from the provisionally selected state of lattice surface SF2, the user can input, for example, "surface number 3" using the voice method. The HMD 1 provisionally selects lattice surface SF3 and displays ID marks M1 for multiple points P1 belonging to that surface. In this case, the HMD 1 may reuse the same IDs (1 to 9) used for lattice surface SF2 for the nine points P1 on lattice surface SF3, or may display different IDs (for example, the ID originally uniquely assigned to lattice surface K1). If the same ID is used for each lattice surface, the range of ID values displayed on the display surface 5 can be limited, for example, to 1 to 9.

[0146] As another control example, a button for specifying a lattice surface may be provided on the display surface 5, and this button may be used instead of the ID mark Q1. As another control example, the HMD 1 does not display the ID mark Q1 or the like representing the lattice surface on the display surface 5. When the user provisionally selects or confirms the selection of a certain point P1, or when the HMD 1 moves a cursor or the like close to the point P1, the HMD 1 provisionally selects the lattice surface in the depth direction to which the point P1 belongs, and displays the ID mark M1.

[0147] FIG. 33 shows another example of displaying a three-dimensional grid. A grid K1 is displayed in an image 331 on the display surface 5. Initially, no point image or ID mark M1 is displayed at point P1 of this grid K1. The user selects a grid surface, for example, grid surface SF2. The user may select a grid surface using button 332 for specifying a grid surface. In response to the operation, the HMD 1 displays a point image (for example, a black circle) for point P1 belonging to the specified grid surface SF2. In addition to the point image, an ID mark M1 may also be displayed.

[0148] Another control example may be as follows. The HMD 1 moves a virtual object within a three-dimensional grid K1 based on a user's operation. First, if the grid plane to which the source point P1 on which the target virtual object is placed belongs and the destination grid plane are the same, the HMD 1 displays ID marks M1 for each point P1 on the grid plane to which both belong and accepts the operation. If the grid plane to which the source point P1 belongs and the destination grid plane are different, the HMD 1 displays ID marks M1 for each point P1 on the destination grid plane and accepts the operation.

[0149] As another control example, a grid line of grid K1 may be selected to select the corresponding grid surface. For example, in FIG. 33, when grid line 333 is selected, grid line 333 belongs to grid surface SF2, and therefore can be associated with the selection of grid surface SF2.

[0150] [Grid Control (2)] FIG. 34 shows another example of display control related to the ID of point P1 of a three-dimensional grid K1. Similar to FIG. 32, the three-dimensional grid K1 in FIG. 34 has a configuration in which each grid plane can be designated by an ID mark Q1. With this configuration, a user can move or exchange virtual objects between grid planes, or even move or exchange grid planes themselves, by manipulating the ID mark Q1 of a grid plane. For example, in the first state, based on a user's operation, multiple virtual objects 341 (e.g., virtual objects V1, V2, and V3) are placed on the front grid plane SF1 (e.g., the center row) represented by the ID mark Q1 with ID=1. Next, the user wants to move the multiple virtual objects 341 on grid plane SF1 to grid plane SF2. In this case, the user might input, for example, "move the object on plane 1 to plane 2" using a voice command. In the cursor method, the user first designates lattice surface SF1 as the target lattice surface with ID mark Q1 of ID=1, and then designates lattice surface SF2 as the destination lattice surface with ID mark Q1 of ID=2.

[0151] In response to the operation, the HMD1 moves multiple virtual objects 341 arranged on the grid surface SF1 to the grid surface SF2. At this time, the HMD1 maintains the positional relationship of points P1 at which the virtual objects 341 are arranged between the grid surfaces before and after the movement as much as possible. For example, if all points P1 on the grid surface SF2 are empty, multiple virtual objects 341 are arranged at points P1 with IDs 4, 5, and 6 in the center row of the grid surface SF2. If virtual objects have already been arranged at points P1 at corresponding positions on the grid surface SF2, the HMD1 may select other empty points P1 on the grid surface SF2 and arrange multiple virtual objects 341 there. If there are not enough empty points P1 on the grid surface SF2 to arrange multiple virtual objects 341, the HMD1 notifies the user of this.

[0152] It is also possible to move a single virtual object between lattice planes. For example, suppose you want to move only virtual object V1 on lattice plane SF1 to lattice plane SF2. In that case, the operation can be performed by inputting, for example, "move object A to plane number 2" or "move object A to the back (or rear)" in a voice-based manner. Even if you do not specify the ID of the plane, you can move it by specifying the relative positional relationship (for example, "back").

[0153] As another control example, virtual objects may be exchanged between grid surfaces. This exchange may be considered as moving or swapping grid surfaces. For example, when exchanging a virtual object on grid surface SF1 with a virtual object on grid surface SF2, the process is as follows. The user uses ID mark Q1 to specify ID=1 for the source grid surface SF1 and ID=2 for the destination grid surface SF2. Following this operation, the HMD1 moves grid surface SF1 together with virtual objects V1, V2, and V3 to the position of grid surface SF2, and moves grid surface SF2 to the position of grid surface SF1. The movement of the grid surfaces may also be a circular movement across the entire grid K1. For example, when moving grid surface SF1 to the position of grid surface SF2, the HMD1 also moves grid surface SF2 to the position of grid surface SF3, and moves grid surface SF3 to the position of grid surface SF1.

[0154] The above control example is for the depth direction (axis X L ), but the left-right direction (axis Y L ) and vertical direction (axis Z L ) is also possible. According to the above control example, it is easy to arrange and move virtual objects even at positions that are far away in the depth direction from the user's viewpoint. Furthermore, even when multiple virtual objects are arranged in the depth direction and appear to overlap, the above operation allows the desired virtual object to be brought to the front for easy viewing.

[0155] [Grid Control (3)] FIG. 35 shows another example of display control related to a three-dimensional grid. Image 351 (A) shows a three-dimensional grid K1 displayed on the display surface 5. This grid K1 also has three grid planes (grid planes SF1, SF2, and SF3) in the depth direction. For example, virtual objects are arranged on the front grid plane SF1 and the middle grid plane SF2. Virtual objects v11, v12, and V13 are arranged in the center row of grid plane SF1. Virtual objects V1, V2, and V3 are arranged in the center row of grid plane SF2. The ID mark M1 for point P1 is not displayed on grid K1. In this example, multiple virtual objects overlap in the depth direction from the user's viewpoint. For example, virtual object v11 and virtual object V1 overlap, making it difficult to recognize point P1. In this case, it may be difficult to select point P1. Therefore, the HMD 1 switches the display of the virtual objects arranged in grid K1 on and off in response to a specific instruction operation by the user. Alternatively, the HMD 1 switches the display of the virtual object on the grid K1 to a transparent state in response to a predetermined instruction operation.

[0156] In the state of image 351 (A), virtual objects on the grid K1 are displayed normally. If the user wants to make all points P1 easier to recognize, the user can input an instruction operation, for example, "object transparency (or object display off)" using a voice command. Alternatively, an object transparency button 352 or the like displayed on the display surface 5 can be used instead. In response to the operation, the HMD 1 makes all virtual objects arranged on the grid K1 transparent (for example, a state in which only the outlines are displayed with dashed lines), as in the state of image 351b (B). This makes it easier for the user to recognize each point P1 on the grid K1, making it easier to specify the destination point P1, etc.

[0157] As another control example, when a user selects a lattice plane, the HMD 1 may display only the virtual objects on that lattice plane in a normal state and display the virtual objects on other lattice planes in a transparent state. The HMD 1 may also hide all of the points P1 and grid lines on the other lattice planes. Alternatively, the HMD 1 may display the virtual objects on all lattice planes in front of the selected lattice plane in a transparent state, or may hide all of the points P1 and the like.

[0158] [Grid Control (4)] FIG. 36 shows another example of display control related to a three-dimensional grid. In this control example, operating the grid lines of the grid K1 enables operation of a group of points P1. The HMD 1 displays the grid lines (e.g., solid straight lines) of the grid K1 within the display surface 5. There are grid lines between each point P1. For example, the front grid surface SF1 has grid lines KL11, KL12, etc. In this example, the grid K1 does not initially display the ID mark M1 for point P1. When moving a virtual object, the user performs a predetermined operation to specify the grid line. For example, in the case of a method using the cursor of the controller 2, the predetermined operation is an operation of pointing to the grid line with the cursor. This example shows the case of specifying the grid line KL11. The grid line KL11 is a line extending from one end (point p1) to the other end (point p3) of the grid surface SF1.

[0159] For example, when a grid line KL11 is specified, the HMD1 highlights the grid line KL11 (for example, by making it thicker or changing its color). The HMD1 changes the display state of all points P1 (for example, points p1, p2, and p3) on the grid line KL11. For example, the HMD1 displays circular ID marks M1 (for example, ID = 1, 2, and 3) for all points P1 on the grid line KL11. The HMD1 places these points P1 (points p1, p2, and p3) in a provisionally selected state. This allows the user to provisionally select a row of multiple points P1 corresponding to the specified grid line KL11. The user can then specify a desired point P1 from among the points P1 on the provisionally selected grid line KL11. For example, the user can specify point p2 by aligning the cursor with point p2 to confirm the selection. During this operation, points P1 on other grid lines of the grid K1 cannot be specified. As another example of control, an ID mark having grid line identification information for each grid line may be displayed and operated.

[0160] [Grid Control (5)] FIG. 37 shows another example of display control related to a three-dimensional grid. This control example is a variation of the control example shown in FIG. 36. Initially, the grid K1 in FIG. 37 does not display an ID mark M1 at point P1. Assume that the user points to a certain point P1, for example, the central point p1, in the grid K1 using, for example, the operating device 2. In this case, the HMD 1 highlights the point p1 and highlights grid lines (for example, grid lines KL31, KL32, and KL33) in three directions (X, Y, and Z) relative to the point p1. These grid lines may extend from one end of the grid K1 to the other end, or may be lines between adjacent points P1. The HMD 1 also highlights all points P1 belonging to the three grid lines, for example, points p2, p3, p4, p5, p6, and p7 adjacent to point p1. The HMD 1 then displays circular ID marks M1 at, for example, a total of seven points p1 to p7. For example, IDs 1 to 7 are displayed at points p1 to p7. The IDs are not limited to numbers, and may be "left," "right," "up," "down," "front," or "back." The HMD 1 temporarily selects the three grid lines and seven points P1. The user can specify a desired point P1 from the temporarily selected portion.

[0161] [Grid Control (6)] FIG. 38 shows another example of a three-dimensional grid configuration. The concept of blocks can be applied to the three-dimensional grid K1, as in FIG. 16 above. The user can select each block. In this example, a cubic portion of the grid K1 consisting of eight vertices is one block. The HMD 1 displays an ID mark 381 representing the block ID for each block. The user can specify a target virtual object by performing a predetermined operation, specify the block where the target virtual object will be placed, and place the target virtual object within the area of that block. In this example, when the cursor approaches a certain block, the block (corresponding point P1 and grid line) is highlighted, and an ID mark 381 (e.g., ID=B1) is displayed connected to the block by a line. A direction 382, etc., can also be set for a virtual object placed in a block.

[0162] In the various control examples described above, the selection state of the target virtual object, point P1, group, etc., when specified, can be canceled by a predetermined operation by the user. This operation may be, for example, a voice input of "cancel" or an operation of a cancel button, or an operation of pointing to an empty space outside the grid K1.

[0163] [Grid Control (7)] FIG. 39 shows another example of display control related to a three-dimensional grid. When multiple points P1 of a grid K1 or multiple virtual objects overlap in the depth direction, it may be difficult for the user to see or operate them. Therefore, the HMD 1 performs the following control to make them easier to see and operate. In the example of image 391 in FIG. 39, near the center of the grid K1, the depth direction (axis X L ), three points P1 (points p1, p2, and p3) are lined up. The three points P1 are close together on the display surface 5, making them difficult for the user to see. Therefore, the HMD 1 changes the display state of the three points P1 (p1, p2, and p3) so that they are easier to see, for example, automatically or when pointed at with a cursor or the like.

[0164] The image 391b in (B) shows the display state after the change. The HMD 1 calculates the position of the three points P1 (points p1, p2, and p3) along the axis X in (A). L In this example, the HMD1 displays a straight line 392 (for example, a dotted line) starting from point p1, which is the center point P1 of the front grating surface SF1, and arranges three points p1, p2, and p3 on the straight line 392, and displays the corresponding ID mark M1. The direction of the straight line 392 is different from the direction of the axis X. L The straight line 392 is set to a direction other than the direction of the grid lines of the grid K1, and is arranged in an area where there are as few other grid lines or points P1 as possible. In particular, the direction of the straight line 392 may be set to a direction that is as close as possible to the direction of the user's finger, the direction of the beam of the controller 2, the line of sight, or the like. Furthermore, the display sizes of the ID marks M1 for the multiple points P1 arranged on the straight line 392 may be varied to suit the sense of perspective.

[0165] The above control example can also be applied to the overlapping of virtual objects. In image 391 (A), three virtual objects, for example, virtual objects V1, V2, and V3, appear to be positioned and overlapped in the depth direction at point P1 (e.g., point p4) to the right of point p1. Therefore, the HMD 1 displays virtual objects V1, V2, and V3 and their corresponding labels side by side on a line 393 connecting from point p4, as in image 391b (B). The direction of line 393 is shown as being, for example, approximately 90 degrees relative to the direction of the beam of the controller 2. Line 392 and line 393 may be curved, for example.

[0166] [Effects, etc.] As described above, the HMD 1 of the first embodiment allows a user to place virtual objects in real space with minimal effort, ease of use, and favorable placement. According to the first embodiment, by using grid and coordinate system control, multiple virtual objects can be favorably placed and moved with little effort and in a short time. The user can perform efficient work using multiple virtual objects. According to the first embodiment, support can be provided for various applications, improving the usability of the applications. The user can use the grid to align multiple virtual objects for easy viewing. According to the first embodiment, even when placing virtual objects in the depth direction from the user's viewpoint, positioning and other operations are easily possible, and they can also be placed far away. While the present invention has been specifically described based on the above embodiments, the present invention is not limited to the above embodiments and can be modified in various ways without departing from the spirit and scope of the present invention. The present invention is applicable not only to HMDs but also to other display devices. [Explanation of symbols]

[0167] 1...HMD, 2...operator, 3...server, 4...PC, 5...display surface, K1...grid, P1...reference point, M1...ID mark, N1...ID mark, 111, 111b...image, 110, V1, V2, V3...virtual object, 403...real object.

Claims

1. A head-mounted display, A display surface; a communication interface that communicates with an operating device used by a user of the head mounted display; A gyro sensor, a processor, The processor: When a movement operation of a virtual object is received by the controller via the communication interface, a plurality of areas for supporting placement of the virtual object are displayed on the display surface; when receiving an instruction to place the virtual object in one of the plurality of areas from the operation device via the communication interface, placing the virtual object so as to be displayed in one of the plurality of areas; when a button operation by the controller is received via the communication interface, a position of the displayed virtual object is changed from a world coordinate system to a local coordinate system based on an angle representing an attitude of the head mounted display detected by the gyro sensor. A head-mounted display characterized by:

2. 2. The head-mounted display according to claim 1, the processor does not display the plurality of regions on the display surface when no movement operation of the virtual object is received. A head-mounted display characterized by:

3. 3. The head-mounted display according to claim 1, The virtual object is an object representing an application. A head-mounted display characterized by:

4. 3. The head-mounted display according to claim 1, The orientation of the head-mounted display is the direction of the local coordinate system relative to the direction of the world coordinate system. A head-mounted display characterized by:

5. A head-mounted display, A display surface; a communication interface that communicates with an operating device used by a user of the head mounted display; a processor, The processor: displaying a menu field including a plurality of application icons on the display surface; when a move operation of one of the plurality of application icons is received by the operation device via the communication interface, a plurality of areas for supporting arrangement of an application window corresponding to one of the plurality of application icons is displayed on the display surface; When an instruction to place the application window in one of the plurality of areas is received from the controller via the communication interface, the application window is displayed in one of the plurality of areas. A head-mounted display characterized by:

6. 6. The head-mounted display according to claim 5, When the processor does not accept a move operation of the application icon, the processor does not display the plurality of areas on the display surface. A head-mounted display characterized by:

7. A head-mounted display, A display surface; a communication interface that communicates with an operating device used by a user of the head mounted display; a processor, The processor: displaying a menu field including a plurality of application icons on the display surface; when a selection operation of one of the plurality of application icons is received by the operation device via the communication interface, a plurality of areas for supporting arrangement of application windows corresponding to the selected application icon are displayed on the display surface; When an instruction to place the application window in one of the plurality of areas is received from the controller via the communication interface, the application window is displayed in one of the plurality of areas. A head-mounted display characterized by:

8. 8. The head-mounted display according to claim 7, When the processor has not received a selection operation of the application icon, the processor does not display the plurality of areas on the display surface. A head-mounted display characterized by:

9. The head-mounted display according to any one of claims 5 to 8, the menu section is displayed near the bottom edge of the display surface; A head-mounted display characterized by:

10. The head-mounted display according to any one of claims 5 to 8, the head mounted display further includes a gyro sensor; When a button operation by the controller is received via the communication interface, the position of the displayed application window is changed from a world coordinate system to a local coordinate system based on an angle representing an attitude of the head mounted display detected by the gyro sensor. A head-mounted display characterized by:

11. The head-mounted display according to claim 10, The orientation of the head-mounted display is the direction of the local coordinate system relative to the direction of the world coordinate system. A head-mounted display characterized by:

Images