Head-mounted information processing device and head-mounted display system

Abstract

To easily grasp the position of a virtual object in a real space different from the real space. [Solution] In a head-mounted information processing device (100), a control unit (125) has a virtual object generation processing unit (155) that generates virtual objects to be displayed by a display unit. The virtual object generation processing unit (155) generates a first virtual object that is associated with and placed in a first real space in which a user exists, and a second virtual object that is associated with and placed in a second real space different from the first real space. The control unit (125) then displays the first virtual object and the second virtual object on a display (122) in response to virtual object display instruction information that is input from an operation input interface (151) and instructs the display of the first virtual object and the second virtual object.

Description

[Technical Field]

[0001] The present invention relates to a head-mounted information processing device and a head-mounted display system, and in particular to a technique that is effective for grasping the position of a virtual object. [Background technology]

[0002] In recent years, virtual reality (VR) technology, augmented reality (AR) technology, and mixed reality (MR) technology have been widely used.

[0003] Virtual reality is a technology that creates a virtual world that resembles reality, allowing users to experience the sensation of being there. Augmented reality is a technology that adds digital information to the real world and reflects and extends virtual spaces (virtual objects) created using CG (Computer Graphics) into the real world. Mixed reality is a technology that combines and fuses information from the real world with virtual worlds artificially created using CG.

[0004] As a tool for realizing these technologies, head-mounted information processing devices that are worn on the head and have a display, camera, etc. In head-mounted information processing devices, in order to increase the realism of virtual objects, a display method has been put into practical use in which virtual objects are represented by being linked to spatial coordinates in real space, making it appear as if the real object is actually there.

[0005] This type of display method allows for an intuitive operation system by allowing the user to view the desired virtual object by going to the real space where the desired virtual object is linked and placed, but it has the problem that the user cannot view or operate the desired virtual object unless he or she goes to the real space to which the desired virtual object is linked.

[0006] One technique to solve this problem is to keep at least a part of an augmented reality object within the real world space and make it easily accessible even when the wearer is moving around in the real world space (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Special Publication No. 2018-505472 Summary of the Invention [Problem to be solved by the invention]

[0008] Although the technology of Patent Document 1 above describes that a virtual object is displayed while remaining within the user's field of view even when the user moves their field of view, it does not take into consideration the display of virtual objects in other real spaces. As a result, there is a problem in that it is difficult to conveniently view and operate virtual objects in other real spaces.

[0009] An object of the present invention is to provide a technology that enables easy recognition of the location of a virtual object in a real space that is different from the real space.

[0010] The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings. [Means for solving the problem]

[0011] Among the inventions disclosed in this application, the outline of representative inventions will be briefly explained as follows.

[0012] That is, a typical head-mounted information processing device includes an operation input interface, a camera unit, a display unit, and a control unit. The operation input interface inputs information. The camera unit captures images of real space. The display unit displays the real-life images captured by the camera unit. The control unit controls the display unit.

[0013] The control unit also has a virtual object generation processing unit that generates virtual objects to be displayed by the display unit. The virtual object generation processing unit generates a first virtual object that is associated with and placed in a first real space in which the user exists, and a second virtual object that is associated with and placed in a second real space different from the first real space.

[0014] Then, the control unit causes the display unit to display the first virtual object and the second virtual object in accordance with virtual object display instruction information that is input from the operation input interface and instructs the display of the first virtual object and the second virtual object. [Effects of the Invention]

[0015] The effects obtained by the representative inventions disclosed in this application will be briefly explained as follows.

[0016] The location of virtual objects placed in different real spaces can be accurately visually confirmed, thereby improving convenience. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a block diagram showing an example of the configuration of a head-mounted information processing device according to a first embodiment. [Figure 2] 2 is an explanatory diagram showing an example of a general view of the surroundings in a usage situation of the head-mounted information processing device of FIG. 1. FIG. [Figure 3] 2 is an explanatory diagram showing an example of a list display of a group of virtual objects by the head-mounted information processing device of FIG. 1; [Figure 4]10 is an explanatory diagram showing another example of the list display of the virtual objects in FIG. 3. FIG. [Figure 5] 5 is an explanatory diagram showing another example of the list display of the virtual objects in FIG. 4. FIG. [Figure 6] FIG. 2 is an explanatory diagram showing an example of a usage situation of the head-mounted information processing device of FIG. [Figure 7] 7 is an explanatory diagram showing an example of a display screen of a group of virtual objects displayed as a list in the example of the panoramic surroundings of FIG. 6; FIG. [Figure 8] 7 is an explanatory diagram showing another example of the usage state of FIG. 6. FIG. [Figure 9] 9 is an explanatory diagram showing an example of a display screen of a group of virtual objects displayed as a list in the example of the panoramic surroundings of FIG. 8. FIG. [Figure 10] 2 is an explanatory diagram showing an example of display of a group of virtual objects by the head-mounted information processing device of FIG. 1. FIG. [Figure 11] 11 is an explanatory diagram showing another example of the display of the virtual object group in FIG. 10. FIG. [Figure 12] 2 is an explanatory diagram showing an example of switching display of a group of virtual objects by the head-mounted information processing device of FIG. 1. FIG. [Figure 13] 2A to 2C are explanatory diagrams showing an example of enlarging / reducing and attitude manipulation of a virtual object by the head-mounted information processing device of FIG. 1. [Figure 14] FIG. 14 is an explanatory diagram showing another example of FIG. 13. [Figure 15] FIG. 10 is an explanatory diagram showing an example of a panoramic view of the surroundings when viewing all virtual objects in a plurality of real spaces. [Figure 16] 16 is an explanatory diagram showing an example of a display of a virtual object when viewed from the opposite direction to the rear entrance in FIG. 15. FIG. [Figure 17] FIG. 17 is an explanatory diagram showing another example of the display of the virtual object in FIG. 16. [Figure 18] FIG. 17 is an explanatory diagram showing another example of the list display of the virtual objects in FIG. 16. [Figure 19] 2 is an explanatory diagram showing an example of a multi-display screen of the head-mounted information processing device of FIG. 1. FIG. [Figure 20] FIG. 20 is an explanatory diagram showing another example of the display shown in FIG. 19. [Figure 21] FIG. 10 is a block diagram showing an example of the configuration of a head-mounted display system according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] In all the drawings for explaining the embodiments, the same components are generally designated by the same reference numerals, and repeated explanations thereof will be omitted.

[0019] (Embodiment 1) The embodiments will be described in detail below.

[0020] <Configuration example of head-mounted information processing device> FIG. 1 is a block diagram showing an example of the configuration of a head-mounted information processing device according to the first embodiment.

[0021] 1, the head-mounted information processing device 100 is composed of a camera unit 111, a right eye gaze detection unit 112, a left eye gaze detection unit 113, a vibration generation unit 117, an external sound microphone 118, a speech sound microphone 119, headphones 120, an operation input interface 121, a display 122, a control unit 125, a memory 124, a depth sensor 142, an acceleration sensor 143, a gyro sensor 144, a geomagnetic sensor 145, and a stimulus generation unit 146. These functional blocks are connected to each other via a bus 140.

[0022] Camera unit 111 captures an image of the scenery in front of the user. Display 122, which is a display unit, displays an actual image of the real space captured by camera unit 111. Camera unit 111 may be configured with multiple cameras, or may be a 360-degree omnidirectional camera that can capture an image of the entire celestial sphere by combining one or multiple cameras.

[0023] The control unit 125 controls each functional block by executing a program 126 (described later) stored in the memory 124 , and controls the operation of the head-mounted information processing device 100 as a whole.

[0024] The control unit 125 has a display control unit 151 , a data management unit 152 , an image processing unit 153 , a virtual object attitude operation processing unit 154 , a virtual object generation processing unit 155 , and a virtual object deformation operation processing unit 156 .

[0025] The virtual object generation processing unit 155 generates a group of virtual objects, each of which is made up of at least one virtual object in a virtual space different from the real space. The virtual object generation processing unit 155 also links the generated group of virtual objects to the real space and places them therein.

[0026] Here, a group of virtual objects that are linked to and arranged in a first real space that is visually recognized or displayed on display 122 is referred to as a first virtual object group. Also, a group of virtual objects that are linked to and arranged in a second real space that is a real space different from the first real space is referred to as a second virtual object group.

[0027] Similarly, a group of virtual objects that are linked to and placed in a third real space that is a real space different from the first and second real spaces is referred to as a third virtual object group, and a group of virtual objects that are linked to and placed in a fourth real space that is a real space different from the first to third real spaces is referred to as a fourth virtual object group.

[0028] The virtual object generation processing unit 155 generates a virtual object based on template data of the virtual object read from the memory 124 in response to a user operation input from the operation input interface 121 .

[0029] Note that template data for a virtual object is not necessarily required, and virtual object data may be generated directly by a user operation without a template. For example, to generate a rectangular parallelepiped virtual object, a user operates operation input interface 121 to specify eight points in real space that will become the vertices of the virtual object.

[0030] Virtual object attitude manipulation processing unit 154 rotates, enlarges, and reduces the size of the virtual object displayed on display 122, manipulating it so that it assumes an attitude that is easy to see. This is called attitude manipulation. The results of the attitude manipulation are not reflected in the attitude, shape, orientation, etc. of the original virtual object.

[0031] The virtual object deformation operation processing unit 156 performs a deformation operation on the virtual object displayed on the display 122. The deformation operation may, for example, change the orientation of the virtual object, change the size or shape, delete a part of the virtual object, or delete the entire object. The result of the deformation operation performed by the virtual object deformation operation processing unit 156 is also reflected in the posture, shape, orientation, etc. of the original virtual object.

[0032] Video processing unit 153 processes the video data captured by camera unit 111 and stores the processed video data in memory 124 as information data 127. Video processing unit 153 simplifies the video data captured by camera unit 111, mainly for the purposes of reducing the amount of display data and improving visibility. For example, if the video data includes a rectangular bookshelf, the video data is simplified to a shape such as a rectangular parallelepiped with the same external dimensions.

[0033] Furthermore, the image processing unit 153 processes images to make it easier to recognize each space. For example, based on image data taken by the camera unit 111 from the user's point of view, the image processing unit 153 generates image data that looks down on the space in which the user was present at the time of taking the image.

[0034] The display control unit 151 generates display data by appropriately combining information data 127 stored in the memory 124, and displays the data on the display 122. In this case, the information data 127 includes virtual object data, video data captured by the camera unit 111, and processed display data generated by the video processing unit 153.

[0035] The data management unit 152 manages the actual image data captured by the camera unit 111, the virtual object data, and the processed display data generated by the image processing unit 153, etc.

[0036] The control unit 125 is composed of a CPU (Central Processing Unit) including a dedicated processor for each arithmetic process such as a GPU (Graphics Processing Unit), and controls each functional block by executing a program 126 stored in the memory 124, thereby controlling the operation of the entire head-mounted information processing device 100. The program 126 is a program such as the OS (Operating System) of the head-mounted information processing device 100 or an application for operation control.

[0037] Control unit 125 controls display control unit 151 to arrange and display, in the first real space, a group of virtual objects that are linked to and arranged in a real space other than the first real space, such as a group of second virtual objects that are linked to and arranged in a second real space, in accordance with virtual object display instruction information input from operation input interface 121. This makes it possible to view and operate the group of virtual objects that are linked to and arranged in a real space other than the first real space.

[0038] When the group of virtual objects generated by the virtual object generation processing unit 155 is displayed within the display field of view of the display 122, for example, a celestial sphere image showing the entire surrounding scenery from the head-mounted information processing device 100 is projected and reflected on the display 122, and the group of virtual objects is arranged at predetermined positions of the reflected celestial sphere image.

[0039] In this case, the control unit 125 controls the display control unit 151 to arrange and display a first group of virtual objects and a group of virtual objects that are linked to and placed in a real space other than the first real space, such as a second group of virtual objects that are linked to and placed in a second real space, within the first real space.

[0040] Furthermore, the control unit 125 may sequentially switch between the first virtual object group and the second virtual object group that is linked to and arranged in the second real space, and display them on the display screen of the display 122.

[0041] Furthermore, the control unit 125 may arrange and display the first virtual object group and the virtual object group associated with and placed in a real space other than the first real space on the display screen of the display from the field of view position of the first real space.

[0042] In addition, the control unit 125 may reduce and arrange the display screens showing the virtual objects associated with and arranged in each real space to display them in a multi-screen, and may return the display screen of the virtual objects in the selected real space to its normal size, thereby enabling the viewing and operation of the desired virtual objects arranged in the selected real space.

[0043] Alternatively, for a virtual object that is difficult to see and is placed within the display field of view, the control unit 125 may control the virtual object attitude operation processing unit 154 to operate the scaling and attitude of the virtual object so that the overall shape of the virtual object is easy to see, and may use the display control unit 151 to display the virtual object after the attitude operation.

[0044] Memory 124 is a non-volatile memory such as a flash memory, and stores various programs 126 and information data 127 used by control unit 125. Information data 127 is data on virtual objects, coordinate position information of the virtual objects, and data on real-life images.

[0045] The display 122 is made up of a liquid crystal panel or the like, and displays virtual objects, live images of real space, etc. The display 122 also displays on the screen display contents such as presentation notification information and operation status to the user.

[0046] For example, when displaying a real-life image captured by the camera unit 111 or a virtual object, the virtual object is displayed by being arranged at a predetermined position on a celestial sphere image showing the scenery all around the head-mounted information processing device 100. Furthermore, the display 122 displays a multi-display of groups of virtual objects linked to multiple real spaces on the display screen.

[0047] The right eye gaze detection unit 112 detects the gaze of the user's right eye. The left eye gaze detection unit 113 detects the gaze of the user's left eye. Note that the gaze detection process can utilize well-known technology that is commonly used as eye tracking processing.

[0048] For example, a known method using corneal reflex involves shining an infrared LED (Light Emitting Diode) onto the face, photographing it with an infrared camera, and using the position on the cornea of ​​the reflected light from the infrared LED (corneal reflex) as a reference point to detect the gaze based on the position of the pupil relative to the position of the corneal reflex.

[0049] The acceleration sensor 143 is a sensor that detects acceleration, which is a change in speed per unit time, and can detect movement, vibration, shock, etc. The gyro sensor 144 is a sensor that detects angular velocity in the rotational direction, and can detect vertical, horizontal, and diagonal posture states. Therefore, the acceleration sensor 143 and the gyro sensor 144 can be used to detect the movement of the head of a user wearing the head-mounted information processing device main unit 100.

[0050] The geomagnetic sensor 145 is a sensor that detects the magnetic force of the Earth and detects the direction in which the head-mounted information processing device main body 100 is facing. The geomagnetic sensor 145 is a three-axis type that detects geomagnetism in the up and down directions in addition to the front-back and left-right directions, and is capable of detecting head movement by capturing changes in geomagnetism in response to head movement.

[0051] These sensors enable detailed detection of movements and fluctuations of the head-mounted information processing device 100 worn by the user.

[0052] The depth sensor 142 measures the distance to an object on a surface. The depth sensor 142 may use, for example, infrared or laser reflection, but may also be implemented using other methods, such as obtaining distance information from the parallax of images captured by multiple cameras attached at different positions.

[0053] The control unit 125 can detect hand and body movements by analyzing distance information acquired by the depth sensor 142. Information obtained from images captured by the camera unit 111 may also be used to analyze hand and body movements.

[0054] The stimulus generating unit 146 generates a stimulus that can be perceived by the skin under the control of the control unit 125. The stimulus generating unit 146 converts notification information for the user transmitted by the head-mounted information processing device 100 into a stimulus that can be perceived by the skin.

[0055] The stimuli that can be perceived by the skin include pressure, a warm sensation, a cold sensation, an electric stimulation, etc. The stimulation generating unit 146 can reliably convey a notification to the user by generating a stimulation that can be perceived by the skin on the head of the user who is wearing the device closely.

[0056] The vibration generating unit 117 generates vibrations under the control of the control unit 125, and is composed of, for example, a vibrator, haptics, or force feedback. The vibration generating unit 117 converts notification information for the user into vibrations. The vibration generating unit 117 can reliably convey notifications to the user by generating vibrations on the user's head, which is closely fitted to the device.

[0057] The ambient sound microphone 118 and the speech sound microphone 119 collect sounds from the outside and the user's own speech. The speech sound microphone 119 may be a sound input device such as a bone conduction microphone.

[0058] The headphones 120 are worn on the user's ears to listen to audio and can notify the user of notification information by audio. The headphones 120 may be an audio output device such as a speaker or a bone conduction earphone.

[0059] The operation input interface 121 is made up of, for example, a keyboard, key buttons, or a touch pad, and is used by the user to set and input information that the user wishes to input. The operation input interface 121 may be provided in any position that allows the user to easily perform input operations.

[0060] The operation input interface 121 may be separated from the main body of the head-mounted information processing device 100 and connected by wire or wirelessly. Examples of an input operation device separated from the head-mounted information processing device 100 include a spatial mouse or a controller device.

[0061] A spatial mouse is a 3D spatial position input device that uses a gyro sensor, acceleration sensor, etc. The controller device detects and inputs the spatial position of the controller itself, which is worn on the body, from camera images of the body and information from various sensors built into the controller device.

[0062] The operation input interface 121 may display an input operation screen on the display screen of the display 122 and acquire input operation information based on the position on the input operation screen to which the gaze detected by the right eye gaze detection unit 112 and the left eye gaze detection unit 113 is directed.

[0063] The operation input interface 121 may capture input operation information by displaying a pointer on the input operation screen and operating the pointer using the operation input interface 121. Alternatively, the operation input interface 121 may capture input operation information by having the user utter a voice indicating an input operation and collecting the voice using the voice microphone 119.

[0064] By using voice and display for input operations in this way, it is possible to further improve the usability of a head-mounted information processing device that is worn on the head.

[0065] With the above configuration, in response to a virtual object display request instruction input via operation input interface 121 instructing the display of virtual objects, a group of virtual objects that are associated with and placed in a real space different from the first real space, for example, a group of second virtual objects that are associated with and placed in a second real space, can be displayed superimposed on the first real space or switched to be displayed.

[0066] In addition, by displaying all virtual objects, including those in other real spaces, at the field of view position of the first real space, it is possible to easily view and operate virtual objects in other real spaces. Furthermore, when there are many virtual objects, it is possible to eliminate the problem of not knowing which real space a desired virtual object is in.

[0067] It also makes it possible to operate a virtual object placed in another real space without moving from the real space you are viewing. For example, if you have placed a calendar as a virtual object on the wall in real space A and want to check or write an appointment in another real space B, you can view and operate the calendar as a virtual object without moving to real space A.

[0068] This example is a transformation operation on a calendar, which is a virtual object, and is processed by the virtual object transformation operation processing unit 156. The result of the transformation operation by the virtual object transformation operation processing unit 156 is reflected in the original object, so for example, if schedule Z is written from real space B onto a calendar on a wall in real space A as a transformation operation, the written schedule Z can be seen when looking at the calendar on the wall in the actual real space A.

[0069] <Example of operation of head-mounted information processing device> Next, the operation of the head-mounted information processing device 100 will be described.

[0070] Fig. 2 is an explanatory diagram showing an example of a general view of the surroundings in a usage situation in the head-mounted information processing device 100 of Fig. 1. Fig. 3 is an explanatory diagram showing an example of a list display of a group of virtual objects by the head-mounted information processing device 100 of Fig. 1. Fig. 3 shows a display example when a group of virtual objects displayed as a list is placed within the display screen of the display 122 in the general view of the surroundings in the usage situation shown in Fig. 2.

[0071] In FIG. 2, a user 200 wearing the head-mounted information processing device 100 is positioned in the center of a first room 201, and is looking in a direction 203 opposite to the rear entrance door 202.

[0072] A desk 204 and a personal computer 205 are placed in front of the user 200, and a bookshelf 206 is placed behind the user 200. Virtual objects 211 to 213 are a first virtual object group, and are generated by the virtual object generation processing unit 155.

[0073] Virtual object 211 is placed directly in front of user 200. Virtual object 212 is placed to the right of desk 204. Virtual object 213 is placed to the right of bookshelf 206 behind the user.

[0074] In the panoramic view of the surroundings of the first room 201 shown in Figure 2, the user 200 looks directly at the first real space projected in the direction 203 of the first room 201, or looks at the real image captured by the camera unit 111 displayed on the display 122.

[0075] For visual recognition of the first virtual object group, each of virtual objects 211 to 213 is displayed in a list as shown in Fig. 3. This list display is achieved by projecting and reflecting the omnidirectional image captured by camera unit 111 onto the display screen of display 122, and displaying all virtual objects in the projected and reflected omnidirectional image. At this time, virtual objects 211 to 213 are arranged at predetermined positions, respectively.

[0076] Display control unit 151 displays virtual objects 211 to 213 on display 122 based on the data read by data management unit 152. Data management unit 152 reads the shape data and placement coordinate data of virtual objects 211 to 213 recorded as information data 127 in memory 124, and outputs the data to display control unit 151. At this time, virtual object attitude operation processing unit 154 performs attitude operation of the virtual objects as necessary.

[0077] This allows the user to visually recognize all virtual objects present in the panoramic surroundings, along with their locations.

[0078] 3 are real-life objects, and are intended to clearly show the positional relationship between the real-life objects and the virtual object. Therefore, the real-life objects indicated by the dotted lines do not need to be displayed.

[0079] Fig. 4 is an explanatory diagram showing another example of a list display of the virtual objects of Fig. 3. Fig. 4 shows an example in which only virtual objects 211 to 213 are displayed without displaying the real objects indicated by dotted lines.

[0080] Fig. 5 is an explanatory diagram showing another example of a list display of the virtual objects of Fig. 4. When displaying all virtual objects present in the panoramic surroundings, as shown in Fig. 5, real-life objects, namely, desk 204, personal computer 205, and bookshelf 206, shown by dotted lines in Fig. 3, may be displayed as a background image. This makes it easier to recognize the positional relationships between desk 204, personal computer 205, and bookshelf 206 in real space and virtual objects 211 to 213.

[0081] For example, if virtual object 213 is located above and to the right of bookshelf 206 on the back as shown in FIG. 2, the position of the virtual object can be easily recognized if bookshelf 206 is displayed below and to the left of virtual object 213 as a background image.

[0082] In the above, the display control unit 151 generates display data such as virtual objects and displays it on the display 122. The display control unit 151 reads data such as the shape and display position of objects included in the information data 127 from the memory 124, as well as background video captured by the camera unit 111 and data generated by the video processing unit 153, and generates display data from the read data.

[0083] Furthermore, virtual object attitude manipulation processing unit 154 manipulates the attitude of the virtual object as necessary, and adjusts the display position, size, shape, etc. so that the virtual object is displayed at a corresponding position on the screen. This adjustment is performed by virtual object attitude manipulation processing unit 154 based on instructions from program 126 recorded in memory 124.

[0084] The omnidirectional image showing the entire surrounding scenery may be acquired using a full-angle omnidirectional camera that can capture an image of the entire celestial sphere at once, or may be generated by stitching together multiple images captured by a camera with a normal angle of view using image processing unit 153.

[0085] Furthermore, the scenery image may be displayed using only a partial image that can be acquired, for example, a hemispherical image of the upper half of the entire solid angle, in which almost all virtual objects can be viewed in an image of a partial range rather than the entire celestial sphere.

[0086] Fig. 6 is an explanatory diagram showing an example of a usage situation of the head-mounted information processing device 100 of Fig. 1. Fig. 7 is an explanatory diagram showing an example of a display screen of a group of virtual objects displayed as a list in the example of the panoramic surroundings of Fig. 6.

[0087] 6 shows a state in which a user 200 wearing a head-mounted information processing device 100 is positioned near an entrance door 402 of a second room 401 and is looking toward a direction 403 inside the second room 401. A television stand 404 and a television 405 are installed in front of the user 200. A shelf 406 is installed on the wall to the right of the user 200.

[0088] The second virtual object group generated by virtual object generation processing unit 155 is made up of virtual objects 411 to 413. In Fig. 7, virtual object 411 is located above and behind television 405. Virtual object 412 is located on the right side and behind television 405. Virtual object 413 is located near the left wall of entrance door 402.

[0089] In the panoramic view of the surroundings of the second room 401 shown in Figure 6, the user 200 directly looks at the second real space projected in the direction of the room 403, or views an image of the real space captured by the camera unit 111 displayed on the display 122.

[0090] Regarding the visual recognition of the second virtual object group, as shown in FIG. 7, a celestial sphere image showing the entire surrounding scenery is projected and reflected on the display screen of display 122, and virtual objects 411 to 413 are arranged at predetermined positions in the projected and reflected celestial sphere image and displayed in a list.

[0091] This allows the presence of all virtual objects present in the panoramic surroundings to be visually confirmed along with their positions, as in the case of Fig. 3. When displaying all virtual objects present in the panoramic surroundings, real-life objects such as a TV stand 404, a TV 405, and a shelf 406, which are indicated by dotted lines in Fig. 7, may be displayed as background images.

[0092] This makes it easier to recognize the positional relationship between the real space and the virtual object. Note that the display process on the display 122 is the same as that shown in FIG.

[0093] Fig. 8 is an explanatory diagram showing another example of the usage situation shown in Fig. 6. Fig. 9 is an explanatory diagram showing an example of a display screen showing a group of virtual objects displayed as a list in the example of the panoramic surroundings of Fig. 8. Fig. 8 shows a state in which a user 200 wearing a head-mounted information processing device 100 is positioned in the center of a third room 601, looking in a direction 603 to the left with an entrance door 602 behind him. A board 604 is located directly in front of the user 200, and a window 605 is located to the right of the user 200. In addition, a clock 606 is located to the right of the window 605.

[0094] The third virtual object group generated by virtual object generation processing unit 155 is made up of virtual objects 611 to 613. In Fig. 8, virtual object 611 is located on the left side of board 604, and virtual object 612 is located above window 605. Virtual object 613 is located behind user 200.

[0095] In the panoramic view of the surroundings in the third room 601 shown in Figure 8, the user 200 looks directly at the third real space projected in the direction 603, or views an image of the real space captured by the camera unit 111 displayed on the display 122.

[0096] Regarding the visual recognition of the third virtual object group, as shown in FIG. 9 , a celestial sphere image showing the entire surrounding scenery is projected and reflected on the display screen of display 122, and each of virtual objects 611 to 613 is arranged at a position where it exists in the projected and reflected celestial sphere image, thereby displaying all of the virtual objects 611 to 613 in a list.

[0097] This allows the presence of all virtual objects present in the panoramic surroundings to be visually confirmed along with their positions, as in the cases of Figures 3 and 7. When displaying all virtual objects present in the panoramic surroundings, real-life objects such as a board 604, a window 605, and a clock 606, which are shown by dotted lines in Figure 9, may be displayed as background images.

[0098] This makes it easier to recognize the positional relationship between the real space and the virtual object. Note that the display process on the display 122 is the same as that shown in FIG.

[0099] Fig. 10 is an explanatory diagram showing an example of display of a group of virtual objects by the head-mounted information processing device of Fig. 1. In the examples shown in Fig. 3, Fig. 7, and Fig. 9, a group of virtual objects associated with and arranged in each real space are visually recognized in each real space. However, in these examples, a group of virtual objects associated with and arranged in a real space other than the currently displayed real space cannot be visually recognized in the currently displayed real space.

[0100] FIG. 10 illustrates a display example in which virtual objects associated with and arranged in each real space are superimposed on the display screen of the display 122 while maintaining the field of view positions of each real space.

[0101] In Figure 10, parts shown in Figures 3, 7 and 9 and assigned the same reference numerals have the same operations as those already explained in Figures 3, 7 and 9, so detailed explanations of them will be omitted.

[0102] Figure 10 shows a state in which, in the currently displayed first real space, in addition to a first group of virtual objects that are linked to and placed in the first real space, a second group of virtual objects that are linked to and placed in the second real space, and a third group of virtual objects that are linked to and placed in the third real space are superimposed and displayed on the display screen of display 122.

[0103] The first virtual object group consists of virtual objects 211, 212, and 213. The second virtual object group consists of virtual objects 411, 412, and 413. The third virtual object group consists of virtual objects 611, 612, and 613.

[0104] As shown in FIG. 10, by arranging and displaying virtual objects 411-413, 611-613 arranged in another real space in the current real space, the virtual objects associated with and arranged in the other real space can all be viewed without switching real spaces.

[0105] This allows users to easily view a desired virtual object from the display screen on which all virtual objects are displayed, even if there are a large number of real spaces and virtual objects. Furthermore, desired operations such as modifying a selected virtual object can be easily performed. As a result, usability can be improved.

[0106] The above operations are displayed on the display 122 by the display control unit 151. Furthermore, the virtual object attitude manipulation processing unit 154 manipulates the attitude of the virtual object as necessary.

[0107] Virtual object attitude operation processing unit 154 adjusts the display position, size, shape, etc. of the virtual object so that the virtual object is displayed at the corresponding position on the screen in accordance with instructions from program 126 stored in memory 124. Display control unit 151 generates display data from the data adjusted by virtual object attitude operation processing unit 154 and displays the data on display 122.

[0108] Fig. 11 is an explanatory diagram showing another example of the display of the virtual object group of Fig. 10. Fig. 11 shows an example in which virtual objects displayed on the display screen of display 122 are arranged at approximately the same coordinate position, and the virtual objects are displayed overlapping each other.

[0109] In Figure 11, parts shown in Figures 2, 3, 6, 7, 8, 9, and 10 and assigned the same reference numerals have the same operations as those already explained in Figures 2, 3, 6, 7, 8, 9, and 10, and therefore detailed explanations thereof will be omitted.

[0110] 10 are virtual objects placed in overlapping positions at approximately the same coordinate positions, and overlapping reduces visibility. Therefore, the overlapping virtual objects 212 and 412 are displayed by shifting the virtual objects 901 and 902 to coordinate positions where they do not overlap, as shown in FIG.

[0111] Furthermore, a mark 903 is displayed at the coordinate position where the original virtual objects 212 and 412 were placed, and virtual lines 904 and 905 are displayed connecting the mark 903 and the virtual objects 901 and 902 that are placed and displayed at shifted coordinate positions.

[0112] In this way, by shifting the virtual objects that are displayed overlappingly at approximately the same position, the virtual objects can be displayed without overlapping. Also, by displaying mark 903, the original position of the virtual object can be easily recognized. Furthermore, by displaying virtual lines 904 and 905, the visibility of the virtual object can be further improved.

[0113] Furthermore, instead of shifting the display position of the virtual object, the virtual object may be displayed overlappingly at the same position and displayed semi-transparently, that is, as a transparent image.

[0114] The above operations are displayed on the display 122 by the display control unit 151. Furthermore, the virtual object attitude manipulation processing unit 154 manipulates the attitude of the virtual object as necessary.

[0115] The attitude manipulation by the virtual object attitude manipulation processing unit 154 adjusts the display position, size, shape, etc. based on the program 126 stored in the memory 124 so that the virtual object is displayed at the corresponding position.

[0116] Hereinafter, unless otherwise specified, the same processing is performed to display a virtual object on the display 122.

[0117] Furthermore, the display of the virtual objects may be such that the user selects a desired virtual object from the superimposed virtual objects 212, 412 (FIG. 10), and only the virtual objects similarly linked to the selected virtual object are displayed in the real space linked to the selected virtual object. In this case, the virtual objects linked to and placed in another virtual space are not displayed.

[0118] This makes it possible to easily select a next desired virtual object from among a group of virtual objects that are linked to and arranged in the same real space as the desired virtual object.

[0119] <Example of switching between virtual objects> Fig. 12 is an explanatory diagram showing an example of switching display of a group of virtual objects by the head-mounted information processing device 100 of Fig. 1. In Fig. 12, parts shown in Fig. 2, Fig. 3, Fig. 6, Fig. 7, Fig. 8, Fig. 9, and Fig. 10 and assigned the same reference numerals have the same operations as those already explained in Fig. 2, Fig. 3, Fig. 6, Fig. 7, Fig. 8, Fig. 9, and Fig. 10, and therefore detailed explanations thereof will be omitted.

[0120] FIG. 12 shows an example in which, instead of displaying all virtual objects in an overlapping manner as shown in FIG. 11, a group of virtual objects that are linked to and arranged in each real space are displayed in a sequentially switched manner.

[0121] Display screen 1001 is a display screen that initially displays a first group of virtual objects that are linked to and placed in a first real space. Display screen 1002 is a display screen that initially displays a second group of virtual objects that are linked to and placed in a second real space.

[0122] Display screen 1003 is a display screen that displays a third group of virtual objects that are linked to and arranged in a third real space. Display screen 1004 is a display screen that is displayed when there is a group of virtual objects that are linked to and arranged in a real space different from the first to third real spaces. These display screens 1001 to 1004 are displayed on display 122 by switching sequentially.

[0123] This allows the user to sequentially view only the virtual objects associated with and arranged in each real space, rather than viewing all of the virtual objects at once, which allows the user to efficiently view a desired virtual object from each virtual object group, further improving visibility.

[0124] Furthermore, the display of the virtual objects may be switched at regular intervals to make them easier to see, for example, by an input operation such as a swipe from the operation input interface 121. This can further improve the visibility of the virtual objects.

[0125] When switching between virtual objects at regular intervals, if you want to take more time to view them in detail, or conversely, if you want to switch to the next screen in a shorter time, you can increase or decrease the viewing time for the group of virtual objects you are viewing by operating the operation input interface 121.

[0126] When displaying the display screens 1001, 1002, and 1003, not only the virtual objects but also actual images of real space corresponding to the virtual objects may be displayed as a background.

[0127] In this case, displaying a real-life image of the background makes it easier to recognize the virtual object. The real-life background can be captured by the camera unit 111 in Fig. 1 and stored in memory 124. If the angle of view of the camera unit 111 is narrow, individually captured images can be stitched together for use.

[0128] Scaling and Orientation Control of Virtual Objects Next, we will explain the operation of enlarging / reducing and manipulating the attitude of a virtual object by the head-mounted information processing device 100. Fig. 13 is an explanatory diagram showing an example of enlarging / reducing and manipulating the attitude of a virtual object by the head-mounted information processing device of Fig. 1. Fig. 14 is an explanatory diagram showing another example of Fig. 13.

[0129] In Figures 13 and 14, parts shown in Figures 2, 3, 6, 7, 8, 9, and 10 and assigned the same reference numerals have the same operations as those already explained in Figures 2, 3, 6, 7, 8, 9, and 10, and therefore detailed explanations thereof will be omitted.

[0130] FIG. 13 shows an example in which a small and hard-to-see virtual object 612 is selected from among the displayed list of virtual objects 211 to 213, 411 to 413, and 611 to 613, and the visibility of the selected virtual object 612 is improved.

[0131] In this case, virtual object attitude operation processing unit 154 performs an operation to enlarge the shape of virtual object 612, moves the enlarged virtual object to a predetermined position in the foreground, and displays it as virtual object 1101. At this time, the list display of all virtual objects 211 to 213, 411 to 413, and 611 to 613 remains in the background part of the display screen.

[0132] Alternatively, the virtual object attitude operation processing unit 154 may perform an enlargement operation by moving the virtual object 612 to the front without leaving any remaining virtual object 612. The predetermined position is set as an initial value to a position that is easy to view. This initial value is stored in advance in the memory 124, for example.

[0133] Alternatively, the initial value may be set by the control unit 125 writing setting information input by the user via the operation input interface 121 into the memory 124. For example, by setting the range of movement of the hands in front of the body as the initial value, which is a predetermined position, it becomes easier to view the virtual object and to perform posture manipulation and deformation manipulation.

[0134] The operation of moving a virtual object to a predetermined position may be performed automatically by the user selecting an arbitrary virtual object using the operation input interface 121. Alternatively, the selected object may be manually placed using a natural operation such as pulling it towards the user. Furthermore, when placing the object, the user may use the operation input interface 121 to perform an operation to determine the magnification ratio, and change the magnification ratio.

[0135] In the above operations, the control unit 125 controls the virtual object attitude operation processing unit 154 and the like in accordance with user operations input from the operation input interface 121. The virtual object attitude operation processing unit 154 changes information about the shape and display position of the selected object.

[0136] The display control unit 151 reads information data 127 such as the shape and display position of the object stored in the memory 124 and displays it on the display 122 .

[0137] This allows virtual objects that are small and difficult to see in the list display to be more clearly visible. After visual confirmation, the selected virtual object is returned to its original position in the background part of the display screen before the placement operation, under the control of the control unit 125. This operation is performed automatically after the visual confirmation end operation.

[0138] If another virtual object is subsequently selected and moved to the front, the previously selected virtual object returns to its original position, and the list display image of the original virtual object remains as the background, making it easier to place and visually confirm the next virtual object.

[0139] In a display in which virtual objects are arranged on a spherical image, the presence of all virtual objects can be visually recognized, but it may be difficult to visually recognize the overall shape of the virtual objects.

[0140] Therefore, for a virtual object that is difficult to view and is placed on the display screen of the display 122, the virtual object orientation manipulation processing unit 154 manipulates the orientation of the virtual object so that the overall shape of the virtual object is easy to view.

[0141] 14 is originally a cube, but when displayed in a spherical image, the display shape is such that it cannot be visually identified as a cube. The virtual object orientation operation processing unit 154 first moves the virtual object 411 to a display position closer to the user where orientation operation is easier, while enlarging the virtual object 411.

[0142] Thereafter, the virtual object attitude manipulation processing unit 154 rotates the moved virtual object 411, including a three-dimensional rotation motion, and manipulates the attitude of the object 411 to a display shape that makes the overall shape easier to see, thereby converting the object 411 into the display shape shown by the virtual object 1201 and displaying it.

[0143] After the virtual object 1201 is visually confirmed, the virtual object 1201 may be returned as the virtual object 411 to its original position before the placement operation.

[0144] In this way, the virtual object attitude manipulation processing unit 154 manipulates the attitude of a virtual object whose overall shape is difficult to see, and converts it into a virtual object whose overall shape is a display shape that can be visually identified, thereby making it possible to accurately visually grasp the overall shape and overall appearance of the virtual object.

[0145] The operation of displaying the overall shape in an easily visible display shape may be performed without the user performing an orientation operation, for example, by displaying the shape in a display shape stored in advance in memory 124. The display shape in which the overall shape is easily visible may be obtained by storing information such as the orientation, size, and color of the object as easily visible orientation information in memory 124 in shape data that serves as a template when generating a virtual object, and then passing this orientation information on to the generated virtual object. Alternatively, the user may specify orientation information for each virtual object, store it in memory 124, and use it when displaying the object.

[0146] Transforming virtual objects Furthermore, virtual object deformation operation processing unit 156 can perform a deformation operation on a virtual object. Virtual object deformation operation processing unit 156 reads the shape and display position of a virtual object stored in memory 124, changes information on the shape and display position of a selected virtual object, and writes the changed information to memory 124. The shape of a virtual object includes orientation, size, angle, etc.

[0147] The display control unit 151 reads the information written in the memory 124, and displays the virtual object that has been transformed based on the information on the display 122.

[0148] The result of the transformation operation by the virtual object transformation operation processing unit 156 is also reflected in the display state of the original virtual object. When the orientation of the object is changed by the transformation operation, the orientation of the virtual object itself changes, and the virtual object is displayed superimposed on the scenery in real space. For example, even in the normal display state shown in FIGS. 2, 6, and 8, the virtual object is displayed in the orientation after the transformation operation.

[0149] During a transformation operation, the shape of the original virtual object before transformation, including its orientation, size, etc., may be displayed semi-transparently, or in a location within the field of view that is not used for the transformation operation, so that the difference between the shape of the original virtual object before transformation and the shape of the virtual object after transformation, including its orientation, size, etc., may be clearly displayed.

[0150] Whether to perform a posture manipulation or a deformation manipulation on the virtual object is specified before the manipulation by, for example, an manipulation mode switching button (not shown) provided on the head-mounted information processing device 100.

[0151] The posture manipulation and the transformation manipulation may be combined, and the transformation manipulation may be performed after enlarging the virtual object by the posture manipulation to make it easier to see. Furthermore, the rotation, enlargement, reduction, and other manipulations applied to the virtual object by the posture manipulation may be applied to the transformation manipulation.

[0152] <Example of displaying virtual objects in other real spaces> Next, an operation for displaying all virtual objects linked and placed in another real space from the current real space will be described with reference to FIGS.

[0153] In Figures 15 to 18, parts shown in Figures 2, 3, 6, 7, 8, 9, and 10 and assigned the same symbols have the same operations as those already explained in Figures 2, 3, 6, 7, 8, 9, and 10, so detailed explanations of them will be omitted.

[0154] FIG. 15 is an explanatory diagram showing an example of a panoramic view of the surroundings when viewing all virtual objects in a plurality of real spaces.

[0155] In Figure 15, a user 200 wearing a head-mounted information processing device 100 is positioned in the center of a first room 201, as in Figure 2, and is looking in the direction 203 opposite the entrance door 202.

[0156] Also placed in first room 201 are desk 204, personal computer 205, and bookshelf 206. In first room 201, virtual objects 211, 212, and 213 are placed as a first virtual object group generated by virtual object generation processing unit 155, similar to FIG.

[0157] A second room 401 is located to the left of the first room 201, and a third room 601 is located to the right of the first room 201. In the second room 401, a TV stand 404, a TV 405, and a shelf 406 are placed.

[0158] In the second room 401, virtual objects 411, 412, and 413 are arranged as a second virtual object group generated by the virtual object generation processing unit 155, similarly to FIG.

[0159] In the third room 601, a board 604, a window 605, and a clock 606 are arranged, and virtual objects 611, 612, and 613 are arranged as a third virtual object group generated by the virtual object generation processing unit 155 as in FIG. 8.

[0160] Fig. 16 is an explanatory diagram showing an example of the display of virtual objects as viewed from a direction 203 opposite to the rear entrance door 202 in Fig. 15. As shown in Fig. 16, virtual objects 411 to 413, 611 to 613 that are linked to and placed in another real space are displayed within the display screen, with the object that separates the real space, such as a room wall, visible through the display screen.

[0161] This makes it possible to easily view all of the virtual objects 211 to 213, 411 to 413, and 611 to 613 regardless of the real space, thereby improving usability when selecting a desired virtual object.

[0162] The virtual objects are displayed in such a way that the real space to which the virtual objects are associated and arranged can be easily recognized based on their display positions.

[0163] For example, in FIG. 16, virtual objects 611 to 613 are displayed at the right side of the display screen, which makes it easy to recognize that they are a group of virtual objects associated with and placed in the third room 601 in FIG. 15.

[0164] Similarly, virtual objects 411 to 413 are displayed on the left side of the display screen, which makes it easy to recognize that they are a group of virtual objects that are associated with and placed in the second room 401 in FIG. 15.

[0165] Fig. 17 is an explanatory diagram showing another example of the display of the virtual objects in Fig. 16. Here, in the display example shown in Fig. 16, if another room, i.e., another real space, is located far away, the group of virtual objects linked to and placed in that real space may be displayed small, which may make them difficult to view.

[0166] In such a case, the virtual objects 411 to 413 are enlarged and displayed as virtual objects 1514 to 1516. Similarly, the virtual objects 611 to 613 are enlarged and displayed as virtual objects 1511 to 1513.

[0167] The virtual object attitude operation processing unit 154 performs shape enlargement operations on the virtual objects 411 to 413 and 611 to 613, respectively, and generates enlarged virtual objects 1511 to 1516.

[0168] This allows a small, hard-to-see virtual object to be more clearly visible. Note that when the virtual object orientation operation processing unit 154 operates to enlarge the shape of the virtual object, it is sufficient to enlarge it to a size that is easy to handle and easy to see.

[0169] 16 is displayed in a display shape that makes it easy to see the overall shape based on the previously specified orientation information. Depending on the user's designation, some or all of the virtual objects may be displayed in a display shape that makes it easy to see the overall shape.

[0170] As shown in Figure 16, by displaying all virtual objects through room walls or other walls that separate the current real space from another real space, it becomes easy to identify the real space to which the selected virtual object is associated and located.

[0171] Fig. 18 is an explanatory diagram showing another example of a list display of the virtual objects of Fig. 16. Fig. 18 shows a display example when transitioning from the display of Fig. 10 to the display of Fig. 16. Virtual objects 611a, 612a, and 613a are the virtual objects shown in Fig. 10. Virtual objects 611b, 612b, and 613b are virtual objects when transitioning from the display example of Fig. 10 to the display example of Fig. 16.

[0172] That is, when the display of FIG. 10 is changed to the display of FIG. 16, virtual objects 611a, 612a, and 613a move to gather on the right side of the display screen as virtual objects 611b, 612b, and 613b, as shown in FIG.

[0173] This makes it easy to recognize that these virtual objects 611b, 612b, and 613b are virtual objects that are linked to and placed in the third real space of the third room 601 located on the right side of the screen of the first room 201.

[0174] During this display transition, the transition is made slowly and gradually, at a speed that the user can follow with their eyes, so that the user can reliably see which real space the group of moving virtual objects are associated with and placed in.

[0175] 18 illustrates an example in which an arbitrary group of virtual objects is moved, but all virtual objects may be moved simultaneously. Alternatively, only one or more selected virtual objects may be moved and displayed, while other unselected virtual objects may be hidden.

[0176] Next, a case where a group of virtual objects linked to and arranged in each real space is multi-displayed will be described.

[0177] Fig. 19 is an explanatory diagram showing an example of a multi-display screen of the head-mounted information processing device of Fig. 1. Fig. 19 shows an example in which the display screens shown in Fig. 3, Fig. 7, and Fig. 9 are reduced and multi-displayed on the display screen of display 122.

[0178] In Figure 17, parts shown in Figures 2, 3, 6, 7, 8, 9, and 10 and assigned the same symbols have the same operations as those already explained in Figures 2, 3, 6, 7, 8, 9, and 10, so detailed explanations of them will be omitted.

[0179] In FIG. 19, a display screen 1701, a display screen 1702, a display screen 1703, and a display screen 1704 are displayed on the display screen of the display 122 in this order from the upper left.

[0180] Display screen 1701 displays a first group of virtual objects that are linked to and arranged in a first real space. Display screen 1702 displays a second group of virtual objects that are linked to and arranged in a second real space. Display screen 1703 displays a third group of virtual objects that are linked to and arranged in a third real space. Display screen 1704 displays a fourth group of virtual objects that are linked to and arranged in a fourth real space.

[0181] On a display screen 1704 displaying a fourth virtual object group, virtual objects 1705 and 1706 are linked and arranged in a fourth real space consisting of a landscape of buildings, cars, people, and the like.

[0182] When selecting a desired virtual object, the user searches display screens 1701 to 1704, which display a reduced and multi-displayed group of virtual objects associated with and arranged in each real space, as shown in Fig. 19. After that, the display screen on which the selected virtual object exists is changed to a normal full display screen, making it easier to view the desired virtual object.

[0183] This allows a user to easily select a real space in which a desired virtual object is located from a multi-display screen that displays groups of virtual objects linked to and arranged in each real space.Furthermore, by displaying only the group of virtual objects linked to and arranged with the selected virtual object, the user can easily view the desired virtual object.

[0184] Although FIG. 19 shows a 4-multi display in which four display screens are displayed, the multi display method may have various variations, such as a 9-multi display or a 16-multi display.

[0185] 20, a plurality of spaces may be displayed side by side in a bird's-eye view. In this case, the image processing unit 153 generates an image of the bird's-eye view, and the display control unit 151 displays the image generated by the image processing unit 153 on the display 122. This makes it easier to imagine the space, and makes it easier to select a space that includes a desired virtual object.

[0186] When generating a bird's-eye view image of a space, the user's own avatar can be displayed at the user's current position in the image, making it easier for the user to understand the space in which they are located.

[0187] In addition, if other users are present in multiple spaces being displayed, coordinate information in the spaces where the other users are present may be obtained via wireless communication or the like, and displayed at the corresponding coordinate positions in the overhead space.

[0188] Similarly, the presence of the user himself or herself or other users may be displayed on the multi-display image using avatars, marks, etc. This allows the user to easily grasp the positional relationship between the space and each user.

[0189] The user's spatial position can be identified using various sensor information, such as the distance to a wall obtained by a depth sensor and an image captured by the camera unit 111. The image of the avatar to be displayed is recorded in advance in the memory 124 as information data 127. Then, the image processing unit 153 uses the information data 127 stored in the memory 124 to synthesize it into an image of the space.

[0190] The coordinate information of other users is acquired from various sensors, cameras, etc. of the information terminals worn by the other users and transmitted from the communication interface of the information terminals. The head-mounted information processing device 100 receives the coordinate information directly via the communication interface 1804. Alternatively, the head-mounted information processing device 100 may receive the coordinate information via a server (not shown).

[0191] In Figures 2 to 19, an example is explained in which a virtual object is placed in the real space in which the user exists, but similar operations are possible even when the space that the user deals with, such as VR, is itself a virtual space and a virtual object is placed in the virtual space.

[0192] Furthermore, the virtual object can be used as a reminder, etc. The virtual object generation processing unit 155 can generate a virtual object in a space other than the current space.

[0193] This function can be used as a reminder. For example, if the virtual object generation processing unit 155 generates and places a virtual umbrella object at the entrance of the living room, the virtual object can serve as a reminder not to forget your umbrella when you go out.

[0194] The display 122 may display information indicating a request instruction from the user via the operation input interface 121, an operation operation performed by the head-mounted information processing device 100, and a display operation.

[0195] Alternatively, the above-mentioned information may be notified to the user by vocalizing it from headphones 120, by generating vibrations from vibration generating unit 117 that is in close contact with the user, or by generating stimuli from stimulus generating unit 146.

[0196] This allows the user to be surely informed of and aware of the operating status of the head-mounted information processing device.

[0197] Furthermore, input operations for operations and display actions performed by the head-mounted information processing device 100 may not only be input operations using the operation input interface 121, but may also be input operations by detecting, for example, the movement of the user's hand or the like using the camera unit 111, and capturing the input operations from that movement.

[0198] As a result, even if the virtual object is located in another real space, the virtual object can be easily viewed and operated.

[0199] (Embodiment 2) <Example of head-mounted display system configuration> FIG. 21 is a block diagram showing an example of the configuration of a head-mounted display system 1801 according to the second embodiment.

[0200] 21, the head-mounted display system 1801 is composed of a head-mounted information processing device 100 and a virtual object generation server device 1802. The head-mounted information processing device 100 and the virtual object generation server device 1802 are each connected to a network 1803.

[0201] The head-mounted information processing device 100 shown in Fig. 21 is newly provided with a communication interface 1804 and a transmitting / receiving antenna 1805 in addition to the configuration of each functional block with the same reference numerals as in Fig. 1. On the other hand, the head-mounted information processing device 100 in Fig. 21 is not provided with a virtual object generation processing unit 155.

[0202] Virtual object generation server device 1802 is composed of a virtual object generation processing unit 1811, memory 1812, control unit 1813, communication interface 1814, and transmitting / receiving antenna 1815. The functional blocks in virtual object generation server device 1802 are connected to each other via a bus 1820. In Fig. 21, the same processing parts as those in the embodiment in Fig. 1 are denoted by the same reference numerals, and their description will be omitted.

[0203] In a head-mounted display system 1801, a virtual object generation processing unit 1811 included in a virtual object generation server device 1802 generates a virtual object.

[0204] Memory 1812 stores virtual objects generated by virtual object generation processing unit 1811. Communication interface 1814 transmits the virtual objects stored in memory 1812 from transmitting / receiving antenna 1815 to head-mounted information processing device 100 via network 1803, which is a communication network. Head-mounted information processing device 100 receives the virtual objects transmitted via network 1803.

[0205] Also, in FIG. 21, the display process of the virtual object on the head-mounted information processing device 100 is the same as that of the first embodiment, but differs from the first embodiment in that the virtual object is generated by a virtual object generation server device 1802, which is a device separate from the head-mounted information processing device 100.

[0206] In the virtual object generation server device 1802, the memory 1812 is a non-volatile semiconductor memory such as a flash memory, similar to the memory 124 of the head-mounted information processing device 100.

[0207] The memory 1812 stores various programs used by the control unit 1813 of the virtual object generation server device 1802, generated virtual objects, etc. The communication interface 1814 is a communication interface that communicates with the head-mounted information processing device 100 via the network 1803, and transmits and receives information to and from the head-mounted information processing device 100.

[0208] The control unit 1813 is composed of, for example, a CPU, and controls each functional block by executing programs such as an OS and operation control applications stored in the memory 1812, thereby controlling the entire virtual object generation server device 1802.

[0209] The control unit 1813 controls the generation of virtual objects by the virtual object generation processing unit 1811 and the saving of the generated virtual objects in the memory 1812. Furthermore, the control unit 1813 controls the transmission of the generated virtual objects to the head-mounted information processing device 100 in response to a request from the head-mounted information processing device 100 to transmit the virtual objects.

[0210] This allows the virtual object to be generated not by the head-mounted information processing device 100 but by the virtual object generation server device 1802 separate from the head-mounted information processing device 100 .

[0211] As a result, it is possible to handle a large amount of virtual object information, and it is possible to simultaneously generate and distribute requested virtual objects to multiple head-mounted information processing devices 100 at multiple locations.

[0212] As a result, it is possible to easily view and operate virtual objects placed in different real spaces using a plurality of head-mounted information processing devices 100 at the same time.

[0213] The invention made by the inventor has been specifically described above based on an embodiment, but it goes without saying that the present invention is not limited to the above embodiment and can be modified in various ways without departing from the gist of the invention.

[0214] It should be noted that the present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to an embodiment having all of the described configurations.

[0215] It is also possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment, and it is also possible to add, delete, or replace part of the configuration of each embodiment with another configuration.

[0216] Furthermore, the above-described configurations, functions, processing units, processing functions, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0217] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0218] 100 Head-mounted information processing device 111 Camera Club 112 Right eye gaze detection unit 113 Left eye gaze detection unit 117 Vibration generating unit 118 Surrounding Sound Microphone 119 Speech Microphone 120 headphones 121 Operation Input Interface 122 Display 124 memory 125 Control Unit 140 Bus 142 Depth Sensor 143 Acceleration Sensor 144 Gyro Sensor 145 Geomagnetic Sensor 146 Stimulus Generator 151 Display control unit 152 Data Management Department 153 Video Processing Department 154 Virtual object attitude operation processing unit 155 Virtual object generation processing unit 156 Virtual object transformation operation processing unit 1801 Head-Mounted Display System 1802 Virtual object generation server device 1803 Network 1804 Communication Interface 1805 Transmitting and receiving antenna 1811 Virtual object generation processing unit 1812 memory 1813 Control Unit 1814 Communication Interface 1815 Transmitting and receiving antenna

Claims

1. an operation input interface for inputting information; a camera unit that captures images of real space; a display unit that displays an actual image captured by the camera unit; a control unit that controls the display unit, the control unit includes a virtual object generation processing unit that generates a virtual object to be displayed by the display unit, The virtual object generation processing unit generating a first virtual object that is linked to and placed in a first real space in which a user exists, and a second virtual object that is linked to and placed in a second real space different from the first real space; The control unit control to detect position coordinates in the first real space corresponding to position coordinates in the second real space of the second virtual object; Controlling the processing of the display shape of the second virtual object; Controlling the display of the second virtual object based on the detected position coordinates in the first real space; When the first virtual object and the second virtual object overlap, control is performed so that the first virtual object and the second virtual object are displayed in an overlapping manner. Head-mounted information processing device.

2. An operation input interface for inputting information; a camera unit that captures images of real space; a display unit that displays an actual image captured by the camera unit; a control unit that controls the display unit, the control unit includes a virtual object generation processing unit that generates a virtual object to be displayed by the display unit, The virtual object generation processing unit generating a first virtual object that is linked to and placed in a first real space in which a user exists, and a second virtual object that is linked to and placed in a second real space different from the first real space; The control unit control to detect position coordinates in the first real space corresponding to position coordinates in the second real space of the second virtual object; Controlling the processing of the display shape of the second virtual object; Controlling the display of the second virtual object based on the detected position coordinates in the first real space; When the first virtual object and the second virtual object overlap, control is performed so that the second virtual object is displayed in a specific area of ​​the display screen of the display unit. Head-mounted information processing device.

3. An operation input interface for inputting information; a camera unit that captures images of real space; a display unit that displays an actual image captured by the camera unit; a control unit that controls the display unit, the control unit includes a virtual object generation processing unit that generates a virtual object to be displayed by the display unit, The virtual object generation processing unit generating a first virtual object that is linked to and placed in a first real space in which a user exists, and a second virtual object that is linked to and placed in a second real space different from the first real space; The control unit control to detect position coordinates in the first real space corresponding to position coordinates in the second real space of the second virtual object; Controlling the processing of the display shape of the second virtual object; Controlling the display of the second virtual object based on the detected position coordinates in the first real space; When the first virtual object and the second virtual object overlap, the first virtual object and the second virtual object are displayed in a shifted manner. Head-mounted information processing device.

4. 2. The head-mounted information processing device according to claim 1, the control unit controls the display unit to display the first virtual object or the second virtual object as a transparent image. Head-mounted information processing device.

5. 4. The head-mounted information processing device according to claim 3, the control unit controls to generate a display mark indicating a display position where the first virtual object and the second virtual object overlap, and a virtual line connecting the display mark with the first virtual object and the second virtual object, and controls to display the display mark and the virtual line on the display unit. Head-mounted information processing device.

6. An operation input interface for inputting information; a camera unit that captures images of real space; a display unit that displays an actual image captured by the camera unit; a control unit that controls the display unit, the control unit includes a virtual object generation processing unit that generates a virtual object to be displayed by the display unit, The virtual object generation processing unit generating a first virtual object that is linked to and placed in a first real space in which a user exists, and a second virtual object that is linked to and placed in a second real space different from the first real space; The control unit control to detect position coordinates in the first real space corresponding to position coordinates in the second real space of the second virtual object; Controlling the processing of the display shape of the second virtual object; Controlling the display of the second virtual object based on the detected position coordinates in the first real space; When the first virtual object and the second virtual object overlap, control is performed so that the first virtual object and the second virtual object are switched and displayed on the display unit. Head-mounted information processing device.

7. 2. The head-mounted information processing device according to claim 1, the control unit controls the display unit to display an actual image of the first real space captured by the camera unit when displaying the first virtual object and the second virtual object. Head-mounted information processing device.

8. 2. The head-mounted information processing device according to claim 1, the control unit includes a virtual object attitude operation processing unit that operates an attitude of the first virtual object or the second virtual object selected by the operation input interface. Head-mounted information processing device.

9. 2. The head-mounted information processing device according to claim 1, the control unit includes a virtual object deformation operation processing unit that performs a deformation operation to deform the first virtual object or the second virtual object selected by the operation input interface. Head-mounted information processing device.

10. 3. The head-mounted information processing device according to claim 2, the control unit controls the second virtual object to move to a right portion of the display screen as the specific area. Head-mounted information processing device.

11. a head-mounted information processing device connected to a communication network and displaying real-space objects and virtual objects; a virtual object generation server device connected to the communication network; and The head-mounted information processing device includes: an operation input interface for inputting information; a display unit that displays the virtual object; a control unit that controls the display unit; Equipped with the virtual object generation server device, a virtual object generation processing unit that generates the virtual object; a communication interface for transmitting and receiving information to the communication network; Equipped with The virtual object generation processing unit generating a first virtual object that is linked to and placed in a first real space in which a user exists, and a second virtual object that is linked to and placed in a second real space different from the first real space; The communication interface includes: transmitting the first virtual object and the second virtual object generated by the virtual object generation processing unit to the communication network; The control unit control to detect position coordinates in the first real space corresponding to position coordinates in the second real space of the second virtual object; Controlling the processing of the display shape of the second virtual object; Controlling the display of the second virtual object based on the detected position coordinates in the first real space; When the first virtual object and the second virtual object overlap, control is performed so that the first virtual object and the second virtual object are displayed in an overlapping manner. Head-mounted display system.

12. A head-mounted information processing device connected to a communications network and displaying real-space objects and virtual objects; a virtual object generation server device connected to the communication network; and The head-mounted information processing device includes: an operation input interface for inputting information; a display unit that displays the virtual object; a control unit that controls the display unit; Equipped with the virtual object generation server device, a virtual object generation processing unit that generates the virtual object; a communication interface for transmitting and receiving information to the communication network; Equipped with The virtual object generation processing unit generating a first virtual object that is linked to and placed in a first real space in which a user exists, and a second virtual object that is linked to and placed in a second real space different from the first real space; The communication interface includes: transmitting the first virtual object and the second virtual object generated by the virtual object generation processing unit to the communication network; The control unit control to detect position coordinates in the first real space corresponding to position coordinates in the second real space of the second virtual object; Controlling the processing of the display shape of the second virtual object; Controlling the display of the second virtual object based on the detected position coordinates in the first real space; When the first virtual object and the second virtual object overlap, control is performed so that the second virtual object is displayed in a specific area of ​​the display screen of the display unit. Head-mounted display system.

13. A head-mounted information processing device connected to a communication network and displaying real-space objects and virtual objects; a virtual object generation server device connected to the communication network; and The head-mounted information processing device includes: an operation input interface for inputting information; a display unit that displays the virtual object; a control unit that controls the display unit; Equipped with the virtual object generation server device, a virtual object generation processing unit that generates the virtual object; a communication interface for transmitting and receiving information to the communication network; Equipped with The virtual object generation processing unit generating a first virtual object that is linked to and placed in a first real space in which a user exists, and a second virtual object that is linked to and placed in a second real space different from the first real space; The communication interface includes: transmitting the first virtual object and the second virtual object generated by the virtual object generation processing unit to the communication network; The control unit control to detect position coordinates in the first real space corresponding to position coordinates in the second real space of the second virtual object; Controlling the processing of the display shape of the second virtual object; Controlling the display of the second virtual object based on the detected position coordinates in the first real space; When the first virtual object and the second virtual object overlap, the first virtual object and the second virtual object are displayed in a shifted manner. Head-mounted display system.

14. The head-mounted display system according to claim 11, the control unit controls the display unit to display the first virtual object or the second virtual object as a transparent image. Head-mounted display system.

15. The head-mounted display system according to claim 12, the control unit controls the second virtual object to move to a right portion of the display screen as the specific area. Head-mounted display system.

16. The head-mounted display system according to claim 13, the control unit performs control to generate a display mark indicating a display position where the first virtual object and the second virtual object overlap, and virtual lines connecting the display mark to the first virtual object and the second virtual object; controlling the display unit to display the display mark and the virtual line; Head-mounted display system.

Images