Information Processing Apparatus, Display System, Information Processing Method, and Program

Abstract

To provide an apparatus, a system, a method, and a program that can improve usability in a three-dimensional virtual space.SOLUTION: An information processing apparatus includes: acquisition means that acquires, from detection means that detects the position of display means worn on a user, the inclination of the display means with respect to a reference direction, and the position of operation means operated by the user or the user's hand, position information and inclination information on the display means and position information on the operation means or the hand; first creation means that, based on the acquired position information on the display means and position information on the operation means or the hand, creates a user object on a three-dimensional virtual space; and second creation means that, based on the acquired inclination information on the display means and virtual space data, creates an image in an inclination direction of the display means in the three-dimensional virtual space.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a display system, a method for generating an image, and a program for causing a computer to execute a process for generating an image.

Background Art

[0002] In VR (Virtual Reality) technology that enables a user to experience content represented in a three-dimensional virtual space, a head-mounted display (HMD) worn on a person's head is used as a display device for displaying an image of the three-dimensional virtual space.

[0003] Even in a three-dimensional virtual space, there may be a case where user input such as writing characters on a whiteboard is required. However, there is still no standard input method using a mouse or a keyboard like a PC (Personal Computer) for user input in a three-dimensional virtual space.

[0004] As a method for performing user input in a three-dimensional virtual space, a method is known in which an avatar object corresponding to the user is arranged at the user's line of sight, and a user interface for receiving an operation from the user is displayed on the avatar object (see, for example, Patent Document 1).

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the above conventional method, since the user interface is displayed on one arm of the avatar object, it is necessary to tilt the head to visually recognize the one arm and operate it with the other hand, which is not easy to operate and cannot be said to have high usability.

[0006] The present invention has been made in view of the above problems, and an object thereof is to provide an information processing apparatus, a display system, an information processing method, and a program that can improve usability in a three-dimensional virtual space.

Means for Solving the Problem

[0007] In order to solve the above-described problems, in one embodiment of the invention, there is provided an information processing apparatus that generates an image based on virtual space data indicating a three-dimensional virtual space, the apparatus including: an acquisition unit that acquires position information and inclination information of a display unit and position information of an operation unit or a hand from a position of the display unit worn by a user, an inclination of the display unit with respect to a reference direction, and a detection unit that detects the operation unit operated by the user or a position of the hand of the user; a first generation unit that generates a user object on the three-dimensional virtual space based on the acquired position information of the display unit and the position information of the operation unit or the hand; and a second generation unit that generates an image in the inclination direction of the display unit in the three-dimensional virtual space based on the acquired inclination information of the display unit and the virtual space data.

Advantages of the Invention

[0008] According to the present invention, user usability in a three-dimensional virtual space can be improved.

Brief Description of the Drawings

[0009] [Figure 1] A diagram showing a first configuration example of a display system. [Figure 2] A diagram showing a first hardware configuration of an HMD, a controller, and a PC. [Figure 3] A block diagram showing an example of the functional configuration of a PC. [Figure 4] A sequence diagram showing the flow of processing executed by the display system shown in FIG. 1. [Figure 5] A diagram for explaining a first method of estimating the position of a user's shoulder. [Figure 6] A flowchart showing the flow of processing for estimating the position of a user's shoulder by the first method shown in FIG. 5. [Figure 7] A diagram for explaining a second method of estimating the position of a user's shoulder. [Figure 8]A flowchart showing the process of estimating the position of the user's shoulder by the second method shown in FIG. 7. [Figure 9] As a first example of a user object for assisting user input in a three-dimensional virtual space, a flowchart showing the process of generating an assistant object and displaying an image. [Figure 10] A diagram showing an example of a display when an assistant object is generated in a three-dimensional virtual space and an image is displayed. [Figure 11] A diagram showing where an assistant object was touched and a function was called. [Figure 12] As a second example of a user object for assisting user input in a three-dimensional virtual space, a flowchart showing the process of generating a laser object and displaying an image. [Figure 13] A diagram showing an example of selecting an object with a laser object. [Figure 14] A diagram showing the relationship between the length of the laser object and the distances between the position of the user's shoulder and the position of the controller. [Figure 15] A diagram exemplifying the state of moving an object selected by a laser object. [Figure 16] A flowchart showing the process of moving an object selected by a laser object. [Figure 17] As an example of user input, a flowchart showing the flow of voice input. [Figure 18] A diagram showing an example of a display when voice input is being performed. [Figure 19] A diagram showing a second configuration example of the display system. [Figure 20] A diagram showing a second hardware configuration of the HMD and the controller.

Modes for Carrying Out the Invention

[0010] FIG. 1 is a diagram showing a first configuration example of a display system. The display system includes an HMD 10 that functions as a display means to be worn on the user's head, a controller 11 that functions as an operation means to be held or worn on the user's hand for operation, and a PC 12 that functions as an information processing device. The display system also includes a position detection sensor 13 that functions as a detection means for detecting the position of the HMD 10 and the inclination of the HMD 10 with respect to a reference direction, and the position and inclination of the controller 11 with respect to the reference direction. In the example shown in FIG. 1, the display system includes a server 14 that manages the position information, operation information, etc. of a plurality of users.

[0011] The PC 12 and the server 14 are communicably connected by a network 15. The HMD 10 and the position detection sensor 13 are connected to the PC 12 by a cable or the like. The controller 11 is wirelessly connected to the PC 12 by Bluetooth (registered trademark), WiFi (registered trademark), or the like. Note that the HMD 10 and the position detection sensor 13 may also be wirelessly connected by WiFi (registered trademark) or the like.

[0012] The HMD 10 has a display for displaying an image to the user, and displays an image corresponding to the position of the HMD 10 and the inclination with respect to a reference direction on the display. The image is two images corresponding to each of the user's left and right eyes in order to show the image stereoscopically using the parallax of the user's left and right eyes. For this reason, the HMD 10 includes two displays for displaying images corresponding to each of the left and right eyes. The reference direction is, for example, an arbitrary direction parallel to the floor. The HMD 10 has a light source such as an infrared LED (Light Emitting Diode) and emits infrared rays.

[0013] The controller 11 is an operation means that the user holds in the hand or wears on the hand position, has buttons, a wheel, a touch sensor, etc., receives an input from the user, and transmits the received information to the PC 12. The controller 11 also has a light source such as an infrared LED and emits infrared rays.

[0014] The position detection sensor 13 is arranged at an arbitrary position in front of the user, and detects the positions and inclinations of the HMD 10 and the controller 11 from the infrared rays radiated from the HMD 10 and the controller 11, and outputs their position information and inclination information. The position detection sensor 13 is, for example, an infrared camera or the like, and can detect the positions and inclinations of the HMD 10 and the controller 11 based on the captured image. Note that a plurality of light sources provided in the HMD 10 and the controller 11 are provided to detect the positions and inclinations of the HMD 10 and the controller 11 with high accuracy. The position detection sensor 13 is composed of one or more sensors, and when using a plurality of sensors, they can also be provided on the side, rear, etc.

[0015] The PC 12 generates user objects for assisting user input in the three-dimensional virtual space displayed on the display of the HMD 10 based on the position information and inclination information of the HMD 10 output from the position detection sensor 13, the position information of the controller 11, and, if necessary, the inclination information of the controller 11. Then, the PC 12 generates an image in the viewing direction of the user (accurately, the inclination direction of the HMD 10) in the three-dimensional virtual space based on the position information and inclination information of the HMD 10 and the three-dimensional virtual space data, an image corresponding to the left and right eyes, and executes a process of displaying it on the display of the HMD 10.

[0016] The PC 12 can communicate with the server 14 via the network 15, acquire the position information etc. of other users in the same three-dimensional virtual space, and execute a process of displaying avatar objects representing the avatars of other users on the display of the HMD 10.

[0017] The display system can be used, for example, to gather avatar objects of each user in a virtual conference room as a three-dimensional virtual space and hold a meeting using a whiteboard or the like. Since the display system allows the participants of the meeting to actively participate in the meeting using a whiteboard or the like, it can be used when holding an interactive meeting.

[0018] In a meeting using the display system, the user can operate the controller 11, call the pen input function by touching the user object in the displayed image, pick up the displayed pen, move the pen, and input characters on the whiteboard. Note that since this is one form of use, it is not limited to this form of use.

[0019] In the example shown in FIG. 1, it was described that the HMD 10 and the controller 11 have light sources and the position detection sensor 13 is arranged at an arbitrary position. However, the HMD 10 and the controller 11 may be configured to include the position detection sensor 13 and have a light source or a marker that reflects infrared rays arranged at an arbitrary position. When using a marker, the HMD 10 and the controller 11 are provided with a light source and the position detection sensor 13, and the infrared rays emitted from the light source are reflected by the marker, and the reflected infrared rays are detected by the position detection sensor 13, whereby the position and inclination of the HMD 10 and the controller 11 can be detected.

[0020] When there is some object between the position detection sensor 13 and the HMD 10 and the controller 11, the infrared rays are blocked and the position and inclination cannot be accurately detected. Therefore, when performing operations and displays using the HMD 10 and the controller 11, it is desirable to execute them in an open space.

[0021] In the example shown in FIG. 1, a space is provided where the user can wear the HMD 10, hold the controller 11 in the hand, and stretch or spread the arm, and the PC 12 and the position detection sensor 13 are arranged outside that space.

[0022] 2 is a diagram showing an example of the hardware configuration of the HMD 10, the controller 11, and the PC 12. The HMD 10 includes an external I / F 20, a CPU 21, a display 22, a memory 23, an HDD 24, a light source 25, and a microphone 26. The CPU 21 controls the entire HMD 10 and executes processes such as light emission by the light source 25, communication with the outside, and display on the display 22. The external I / F 20 is an interface for communicating with the PC 12. The display 22 may be a liquid crystal display or an organic EL (Electro Luminescence) display.

[0023] The memory 23 provides a working area for the CPU 21. The HDD 24 stores image data to be displayed in the three-dimensional virtual space. The light source 25 is an infrared LED or the like, which emits infrared rays. The infrared rays can be emitted in a flashing manner in a predetermined pattern. The microphone 26 is a voice input device that allows the user to input information by voice.

[0024] The controller 11 includes an operation I / F 30, an external I / F 31, and a light source 32. The operation I / F 30 is a button, a wheel, a touch sensor, etc., and is arranged on the outer surface of the controller 11 to enable operation by the user and to accept input of operation information. The external I / F 31 is wirelessly connected to the PC 12 and transmits operation information accepted by the operation I / F 30 to the PC 12. The light source 32 emits infrared light that flashes in a predetermined pattern. The light source 32 can be distinguished from the HMD 10 by flashing in a pattern different from the light source of the HMD 10.

[0025] The PC 12 includes a CPU 40 , a ROM 41 , a RAM 42 , a HDD 43 , an external I / F 44 , an input / output I / F 45 , an input device 46 , and a display device 47 .

[0026] The CPU 40 controls the entire PC 12, generates the above-mentioned user objects, and executes processing to generate an image in the user's line of sight in the three-dimensional virtual space and display it on the display of the HMD 10. The ROM 41 stores a boot program for starting up the PC 12, firmware for controlling the HDD 43, the external I / F 44, etc. The RAM 42 provides a working area for the CPU 40.

[0027] The HDD 43 stores an OS (Operating System), programs for executing the above processes, image data, etc. The external I / F 44 is connected to the network 15 shown in Fig. 1 and communicates with the server 14 via the network 15. The external I / F 44 is also connected to the HMD 10 and the position detection sensor 13 via a cable or the like, and is wirelessly connected to the controller 11, and communicates with the HMD 10, the controller 11, and the position detection sensor 13.

[0028] The input device 46 is a mouse, keyboard, etc., and is used by the user to input information and accept operations. The display device 47 provides a display screen for the user and displays input information, processing results, etc. The input / output I / F 45 is an interface that controls the input of information from the input device 46 and the output of information to the display device 47.

[0029] 3 is a block diagram showing an example of the functional configuration of the PC 12. Here, since the PC 12 functions as an information processing device, the functional configuration of the PC 12 will be described. The CPU 40 executes a program stored in the HDD 43 to generate functional units for realizing each function, and the PC 12 can be equipped with these functional units. Note that each functional unit is not limited to being realized by a program, and may be realized by a device such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or a conventional circuit module designed to execute the function.

[0030] The PC 12 includes at least an acquisition unit 50, a first generation unit 51, and a second generation unit 52. The PC 12 may also include other functional units. The acquisition unit 50 acquires position information and tilt information of the HMD 10 and the controller 11 detected by the position detection sensor 13. The acquisition unit 50 also acquires information (operation information) input by the user from the controller 11. The acquisition unit 50 also acquires three-dimensional virtual space data from the HDD 43 and the server 14.

[0031] The three-dimensional virtual space data is data on a virtual three-dimensional space represented by x-, y-, and z-axes, and includes object data and image data of one or more objects placed in that space. Taking the above-mentioned virtual conference room as an example, the image data is image data including a whiteboard, floor, walls, ceiling, entrances, and exits. Objects are physical objects placed in the three-dimensional virtual space, and include input objects such as pens for realizing user input, objects such as sticky notes for text input, and user objects for supporting user input.

[0032] The first generation unit 51 generates a user object in a three-dimensional virtual space based on the position information and tilt information of the HMD 10 and the controller 11 acquired by the acquisition unit 50. The user object may be an assistant object that can call functions within an application, a laser object that can select objects, or the like. When the display system is used for a conference, the application may be a conference application, and an example of a function within the application may be pen input.

[0033] The second generation unit 52 generates image data to be displayed on the display of the HMD 10 based on the position information, tilt information, three-dimensional virtual space data, and operation information acquired by the acquisition unit 50, and transmits the image data to the HMD 10. The second generation unit 52 uses the three-dimensional virtual space data based on the position information and tilt information of the HMD in the three-dimensional virtual space to generate image data of an image in the field of view to which a tilt is applied, with the position of the HMD in the three-dimensional virtual space as the starting point.

[0034] FIG. 4 is a sequence diagram showing the overall processing flow executed by the display system. When the power is turned on for the HMD 10, the controller 11, the PC 12, and the position detection sensor 13, the PC 12 acquires three-dimensional virtual space data from its own HDD 43 or the server 14 (S1). The HMD 10 and the controller 11 emit infrared rays from their respective light sources, and the position detection sensor 13 detects the positions and inclinations of the HMD 10 and the controller 11 based on the light sources emitting infrared rays (S2). The position detection sensor 13 transmits the detected positions and inclinations to the PC 12 as respective position information and inclination information (S3).

[0035] The controller 11 receives an operation from the user and transmits operation information of the received operation to the PC 12 (S4). The PC 12 generates image data based on the acquired three-dimensional virtual space data, the position information and inclination information of the HMD 10 and the controller 11 received from the position detection sensor 13, and the operation information received from the controller 11 (S5). Then, the PC 12 transmits the generated image data to the HMD 10 (S6). The HMD 10 displays the received image data on the display (S7). Thereafter, until the power of the HMD 10 or the like is turned off, the detection of the position and inclination in S2 to the transmission of the image data in S5 is repeated, and the image data displayed on the display of the HMD 10 is updated.

[0036] The overall processing flow in the display system is as described above. Hereinafter, a method for generating a user object to be displayed together with image data on the three-dimensional virtual space will be described in detail.

[0037] The user object must be easy for the user to touch or select in order to call a function within the application.

[0038] Since the image data is generated by applying an inclination starting from the position of the HMD 10, the user object can also be arranged in the three-dimensional virtual space based on the position and inclination of the HMD 10.

[0039] However, when the user's head moves, the user object moves accordingly. Therefore, when only the head moves, depending on the position and orientation of the head, it may be difficult to touch the user object. For example, when the user object moves to the side opposite the dominant hand. This is the same when using the user object to select an object. This does not mean high usability.

[0040] When a person moves their arm, they move the arm up, down, left, and right with the shoulder as a pivot. The shoulder is one of the physical parts existing between the head and the hand where the controller 11 is held. When the user touches the user object, they move the arm to change the position of the controller 11. Therefore, if the user object is arranged based on the position of the shoulder, even if only the head moves, the user object will not move accordingly, and even if the shoulder moves, the user object can be arranged within the range easily reachable by the dominant hand.

[0041] When arranging the user object based on the position of the shoulder, it is necessary to estimate the position of the shoulder based on the position information and inclination information of the HMD 10 and the controller 11 obtained from the position detection sensor 13. To estimate the position of the shoulder, the relative positional relationship between the HMD 10 and the position of the shoulder must be known. The relative positional relationship is the position of the shoulder as seen from the HMD 10 or the position of the HMD 10 as seen from the position of the shoulder.

[0042] Referring to FIG. 5, a first method for obtaining the relative positional relationship between the position of the user's shoulder and the HMD 10 will be described. FIG. 5 is a view of the user seen from above. The user holds or wears the controller 11 in the hand, extends the elbow horizontally with respect to the floor, and in the state where the elbow is extended, repeats the operations of closing the arms by bringing the left and right hands closer and opening the arms by separating the left and right hands.

[0043] In such an operation, the left and right hands move in an arc with the left and right shoulders as respective fulcrums. The position information of the controller 11 at this time is acquired by the position detection sensor 13. The position information acquired by the position detection sensor 13 is the position information of points on the arc. The position of the shoulder is a point equidistant from each point on the arc. From this, the shoulder position information can be calculated from the position information acquired by the position detection sensor 13.

[0044] The position information of the HMD 10 has been acquired by the position detection sensor 13 before starting the operation of opening or closing the arm. Therefore, the position of the shoulder based on the position of the HMD 10 can be calculated as the relative positional relationship between the HMD 10 and the shoulder position.

[0045] FIG. 6 is a flowchart showing the flow of processing for obtaining the relative positional relationship between the user's shoulder position and the HMD 10 by the first method shown in FIG. 5. This processing is started from step 100 at the timing of starting the use of the HMD 10 and the controller 11, or at the timing when it is felt that the positional deviation detected by the position detection sensor 13 has become large.

[0046] In step 101, with the elbows extended, hold the arms horizontally. In step 102, with the elbows extended, open and close the arms. In step 103, continuously acquire the position information of the controller 11. Here, the position information may be the position information in one operation from the closed state to the open state, but in order to improve the accuracy of the position information, it is desirable to repeat it several times.

[0047] In step 104, the acquired position information is averaged. Because the arm is closed and opened repeatedly, multiple pieces of position information are obtained, such as when the arm is closed and when it is opened. For this reason, the position information for the same closed position or the same open position is averaged. This averaging makes it possible to estimate the shoulder position with high accuracy. In step 105, the relative positional relationship between the HMD 10 and the shoulder position is calculated from the averaged position information, and in step 106, this process ends.

[0048] The shoulder positions are calculated as shoulder positions based on the position of the HMD 10. The calculated shoulder position data is shoulder position data that indicates the relative positional relationship with the HMD 10, and is coordinate data expressed by, for example, the x-axis, y-axis, and z-axis. The shoulder position data is calculated for each of the left and right shoulders.

[0049] A second method for determining the relative positional relationship between the user's shoulder position and the HMD 10 will be described with reference to Fig. 7. Fig. 7 is a view of the user as seen from the front. In the second method, to directly determine the shoulder position, controllers 11 worn on the left and right hands are placed on the left and right shoulders, and position information of the controllers 11 is detected by position detection sensors 13.

[0050] The controller 11 worn on either hand may be placed at either shoulder position, and the controller 11 may be placed at the shoulder position in either pose shown in FIGS. 7(a) and 7(b).

[0051] If the controller 11 is on the left side of the position and orientation of the HMD 10, the position information of the controller 11 is estimated to be left shoulder position information, and if it is on the right side, the position information of the controller 11 is estimated to be right shoulder position information, and each position information is used as shoulder position data.

[0052] FIG. 8 is a flowchart showing the flow of a process for obtaining the relative positional relationship between the position of the user's shoulder and the HMD 10 by the second method shown in FIG. 7. This process can be started from step 200 at the timing when the use of the HMD 10 and the controller 11 is started, or at the timing when it is felt that the positional deviation detected by the position detection sensor 13 has become large.

[0053] In step 201, the position information of each of the left and right controllers 11 is acquired by the position detection sensor 13. In step 202, the position information and tilt information of the HMD 10 are acquired by the position detection sensor 13. In step 203, it is estimated from the position information and tilt information of the HMD 10 which of the left and right shoulders each controller 11 corresponds to. In step 204, the relative positional relationship between the position of the HMD 10 and each shoulder is obtained, and in step 205, this process is terminated.

[0054] When the shoulder position data, which is the relative position data with respect to the HMD 10, is created, the PC 12 executes a process of generating a user object based on the position of the shoulder. The user object is an object for assisting user input, and examples include an assistant object that calls a function within an application and a laser object as a selection object for selecting one of the objects arranged in the three-dimensional virtual space. The process of estimating these shoulder positions is also executed by the first generation unit 51.

[0055] Referring to FIG. 9, a process of generating an assistant object in a three-dimensional virtual space and generating an image to be displayed on the HMD 10 will be described. This process is executed after obtaining the relative positional relationship between the position of the user's shoulder and the HMD 10. This process starts from step 300. In step 301, the position information and tilt information of the HMD 10, and the position information and tilt information of the controller 11 are acquired from the position detection sensor 13. The position information and tilt information of the controller 11 are the position information and tilt information of the two controllers 11 worn on both the left and right hands. Hereinafter, for the sake of simplicity of explanation, it will be simply described as the controller 11, but actually it means the two controllers 11 of both hands.

[0056] In step 302, with respect to the position information of the HMD 10 (coordinates represented by the x-axis, y-axis, and z-axis), shoulder position data, which is relative position data with respect to the HMD 10, is applied to estimate the position of the shoulder. The shoulder position data is data indicating the position of the shoulder in coordinates with respect to the HMD 10 as a reference. In step 303, based on the estimated position of the shoulder, the position information (coordinates) of the assistant object in the three-dimensional virtual space is calculated. The assistant object can be an object of any shape, and its shape, size, etc. are set in advance.

[0057] The PC 12 holds data for calculating the position of the assistant object indicating the relative positional relationship between the position of the shoulder and the assistant object, and applies the data for calculating the position of the assistant object (data in the uvw coordinate system obtained by applying the tilt of the HMD 10 to the xyz coordinate system based on the reference direction of the HMD 10) to the position of the shoulder to calculate the coordinates of the assistant object in the three-dimensional virtual space. The data for calculating the position of the assistant object is data that has been pre-adjusted and stored for the optimal position for the user.

[0058] In step 304, based on the position information of the controller 11 and the calculated position information of the assistant object, the distance between the controller 11 and the assistant object is created as distance information. The PC 12 can be provided with an arithmetic unit that calculates the distance in order to create this distance information. The arithmetic unit also executes other arithmetic processes such as the above-mentioned weighted average. In step 305, it is determined whether the created distance information is longer than a predetermined distance. If it is less than or equal to the predetermined distance, the process proceeds to step 306, where the application of the assistant object position calculation data to the shoulder position is aborted, and the coordinates of the assistant object in the three-dimensional virtual space at the time when the application of the assistant object position calculation data is aborted (coordinates represented in the xyz coordinate system) are fixed. The PC 12 can be provided with a determination unit that determines whether the distance information is longer than a predetermined distance, and if it is less than or equal to the predetermined distance, fixes the coordinates. The determination unit also executes other determination processes.

[0059] The coordinates of the assistant object fixed based on the shoulder position do not change every time the HMD 10 and the controller 11 are powered on and started up. Therefore, the coordinates of the assistant object to be fixed can use the coordinates of the assistant object determined in the process of displaying the previous image.

[0060] If the coordinates of the assistant object are not fixed, when trying to call a function by touching the assistant object and the shoulder moves, the assistant object will move and it will be difficult to touch. This is because the assistant object moves along with the movement of the shoulder. Therefore, when trying to touch the assistant object, the coordinates of the assistant object are fixed so that the assistant object does not move along with the movement of the shoulder.

[0061] If it is determined in step 305 that the distance is longer than a predetermined distance, directly proceed to step 307. In step 307, based on the coordinates of the assistant object, the assistant object is placed in the three-dimensional virtual space. In step 308, based on the position information and tilt information of the HMD 10, image data of an image in the viewing direction centered on the position of the HMD 10 and applying the tilt of the HMD 10 is generated. The generated image data is displayed on the display of the HMD 10, and in step 309, an assistant object is generated, and the process of generating an image to be displayed on the HMD 10 is terminated.

[0062] FIG. 10 is a diagram showing an example of the display of the generated assistant object. The image for displaying the three-dimensional virtual space includes a whiteboard 60, and a virtual user's hand 61 is shown in front of the whiteboard 60. The assistant object 62 is a polyhedron having a surface that can be touched by the finger of the hand 61. Since the assistant object 62 is generated and arranged based on the position of the shoulder, it is easy to touch the assistant object 62 with a finger and easy to call a function. Information such as characters representing functions is shown on the surface of the polyhedron, and since the assistant object 62 is arranged based on the position of the shoulder, the information is easy to see.

[0063] In FIG. 10, the user operates the controller 11 and touches the assistant object 62 with the finger of the hand 61 on the three-dimensional virtual space. When the assistant object 62 is composed of a polyhedron, information such as characters representing different functions may be shown on each surface, and a predetermined function can be called by touching a predetermined surface. The user can send an instruction to call a function to the PC 12 by touching the assistant object 62 with the finger of the hand 61 on the three-dimensional virtual space and pressing a predetermined button of the controller 11.

[0064] FIG. 11 is a diagram showing an example of a display when the assistant object 62 is touched with the finger of the hand 61 to call a function in the application. The function is displayed as a pop-up (a screen that appears to jump out to the front surface) when the PC 12 generates an object representing the function for the HMD 10 and calls the function by touching the object. For this reason, the PC 12 can include a third generation unit for generating an object representing the function. In FIG. 11, a marker 63 for writing on the whiteboard 60, an eraser 64, etc. are generated and displayed in the image. In addition, as functions, a function for moving the whiteboard 60, a function for copying the content written on the whiteboard 60, etc. may be provided.

[0065] The user can operate the controller 11 to pick up the marker 63 with the hand 61 in the three-dimensional virtual space, move it to the whiteboard 60, and move the hand 61 to write characters on the whiteboard 60. Also, the user can operate the controller 11 to pick up the eraser 64 with the hand 61 in the three-dimensional virtual space and move the hand 61 to erase the characters written on the whiteboard 60.

[0066] With reference to FIG. 12, the process of generating a laser object on the three-dimensional virtual space and generating an image to be displayed on the HMD 10 will be described. This process is also executed after obtaining the relative positional relationship between the position of the user's shoulder and the HMD 10. This process starts from step 400, and in step 401, the position information and tilt information of the HMD 10 and the position information and tilt information of the controller 11 are acquired from the position detection sensor 13.

[0067] In step 402, shoulder position data is applied to the position information (coordinates represented by the x-axis, y-axis, and z-axis) of the HMD 10 to estimate the position of the shoulder. In step 403, based on the estimated shoulder position and the position information of the controller 11, distance information indicating the distance between the shoulder position and the controller 11 is created. In step 404, based on the created distance information, the length (number of pixels) of the laser object is determined. The laser object is a rod-shaped object, and the longer the distance between the shoulder and the controller 11, the longer the length of the laser object.

[0068] Here, FIG. 13 shows the relationship between the length of the laser object and the distance between the shoulder and the controller 11. FIG. 13(a) is a graph showing the relationship in which the length of the laser object increases quadratically according to the length of the distance. FIG. 13(b) is a graph showing the relationship in which the length increases at a constant rate up to a certain distance, and when exceeding the certain distance, the length of the laser object increases quadratically.

[0069] As shown in FIG. 13(c), when the length of the laser object increases linearly according to the length of the distance between the shoulder and the controller 11, the slope of the graph is larger than the slope of the graph up to the certain distance shown in FIG. 13(b). In this case, when the target object is close, just by slightly extending the arm, the laser object becomes too long, and when the target object is far, even if the arm is greatly extended, it may not reach the target object.

[0070] However, as shown in FIGS. 13(a) and 13(b), by changing the length of the laser object so that the rate of change of the length of the laser object increases as the distance increases, it becomes possible to appropriately select the target object whether the target object is close or far. Note that the length of the laser object is not limited to the examples shown in FIGS. 13(a) and 13(b) as long as the length can be changed so that the rate of change of the length of the laser object increases as the distance increases.

[0071] Referring to FIG. 12 again, when the length of the laser object is determined from the created distance information, the laser object is arranged in the three-dimensional virtual space in a specific direction starting from the position of the controller 11. The specific direction is, for example, a predetermined inclination direction of the controller 11 or a direction connecting the controller 11 and the position of the shoulder. When the controller 11 has a grip portion held by hand, the grip portion has a shape extending in a certain direction. The predetermined inclination direction is the direction (longitudinal direction) in which the grip portion extends.

[0072] In step 405, it is determined whether the laser object arranged in the three-dimensional virtual space passes through a specific object in the three-dimensional virtual space. The specific object is an object for which an operation such as movement becomes possible when a specific input (for example, a movement instruction by button input, etc.) is made after selection. When it is determined that the laser object passes through a specific object, the process proceeds to step 406, and a selection process for the passed specific object is executed. Specifically, a selection flag is given to the passed specific object, the laser object corresponding to the part that has passed through the specific object is deleted, and the process proceeds to step 407. The PC 12 can include a selection processing unit that executes this selection process. On the other hand, when it is determined in step 405 that the laser object does not pass through a specific object, the process proceeds to step 408.

[0073] In step 407, an operation by the user is received. The operation by the user is, for example, the movement of an object, and the movement of the object includes the input of a movement instruction for the object. The movement of the object will be described later. In step 408, in the three-dimensional virtual space, based on the position information (coordinates) of the laser object, the position information and inclination information of the HMD 10, image data in the viewing direction centered on the position of the HMD 10 and applying the inclination of the HMD 10 is generated and displayed on the display. Then, in step 409, the process of generating the laser object and generating the image to be displayed on the HMD 10 is terminated.

[0074] FIG. 14 is a diagram showing an example of selecting an object by a laser object. The laser object 65 is generated as an object that extends in a rod shape in the direction in which the virtual user's hand 61 extends on the three-dimensional virtual space in step 404 of FIG. 12, and is arranged in the three-dimensional virtual space. In FIG. 14(a), the laser object 65 is arranged so as to extend from the hand 61 on the three-dimensional virtual space. In FIG. 14(b), in step 406 of FIG. 12, the object 66 on the three-dimensional virtual space is selected, and the laser object 65 corresponding to the passed part is deleted.

[0075] FIG. 15 is a diagram showing a state of moving the object 66 as an example of a user's operation on the selected object 66 in step 407 of FIG. 12. By changing the position and inclination of the controller 11, as shown in FIG. 15(a), the direction in which the object 66 is arranged can be changed. Further, when the user wears the controller 11 on the hand and pulls the arm, as shown in FIG. 15(b), the object 66 can be pulled toward the user.

[0076] FIG. 16 is a flowchart showing a process flow of moving the object 66 selected by the laser object 65 as an example of a specific process in step 407 of FIG. 12. Here, the moving process will be described, but the operation on the object 66 selected by the laser object 65 is not limited to moving the object 66.

[0077] The process shown in FIG. 16 is performed after arranging the laser object 65 on the three-dimensional virtual space and completing the selection of a specific object. This process starts from step 500, and in step 501, it is confirmed whether a selection flag is given to a specific object 66. If it is given, the process proceeds to step 502, and it is determined whether there is a movement instruction from the controller 11. The controller 11 includes an operation I / F 30 such as a button, and determines whether there is a movement instruction based on whether a button is pressed or not.

[0078] When there is a movement instruction, proceed to step 503 and execute the movement process of the object 66 with a selection flag based on the operation of the user's controller 11. The PC 12 can include a movement processing unit that executes the movement process of the object 66. When the movement process is a pulling process, the coordinates of the object 66 with a selection flag are instantaneously moved to the user's hand. Also, when the movement process is other than the pulling process, the object 66 with a selection flag moves up and down, left and right, forward and backward in the three-dimensional virtual space. Whether to execute the pulling process or other movement processes can be selected by changing the button pressed among a plurality of buttons provided on the controller 11. In addition to changing the button, it may also be selected by a gesture of pulling the hand (the position and inclination of the controller 11).

[0079] If the selection flag is not given in step 501, if there is no movement instruction in step 502, or if the execution of the movement process ends in step 503, proceed to step 504 and determine whether there is an input instruction from the user. The input instruction from the user is, for example, voice input by the user or character input in the three-dimensional virtual space. Whether there is an input instruction can also be determined by the presence or absence of pressing a button on the controller 11, similar to the presence or absence of a movement instruction. Also, the presence or absence of an input instruction can be determined by the presence or absence of an operation by a gesture of bringing the object 66 close to the user's mouth.

[0080] If there is an input instruction from the user in step 504, proceed to step 505 and execute the input process for the object 66 with a selection flag. A specific explanation of the input process will be described later. If there is no input instruction in step 504 or if the input process ends in step 506, end the movement process of the object 66. Furthermore, when moving the object 66 again, the process can be started from step 500.

[0081] Figure 17 is a flowchart showing the flow of voice input as an example of user input. The process starts from step 600. In step 601, the position information of the HMD 10, the position information of the controller 11, and the position information of the laser object 65 are acquired. The position information of the laser object 65 is the position of the tip of the laser object 65, that is, the object 66 selected by the laser object 65.

[0082] In step 602, distance information indicating the distance between the HMD 10 and the laser object 65 is created from the position information of the HMD 10 and the position information of the laser object 65 in the three-dimensional virtual space. The distance information is constantly created after the movement process is executed.

[0083] In step 603, the position of the user's mouth is estimated based on the position of the HMD 10, and it is determined whether the distance between the user's mouth and the laser object 65 in the three-dimensional virtual space is shorter than a predetermined distance. When the user pulls the object 66 closer and the distance becomes shorter than the predetermined distance, the process proceeds to step 604 to execute voice input. The HMD 10 receives voice input through the microphone 26 that functions as an input reception unit, and sends the input voice data to the PC 12. The PC 12 executes voice recognition processing on the voice data acquired from the HMD 10. In the voice recognition processing, the content spoken by the user is converted into image data including characters by voice recognition, and the converted image data is pasted onto the object 66 on which the movement process has been executed. The PC 12 can be provided with a data processing unit that executes predetermined processing such as voice recognition processing on the input data. Note that the microphone for receiving voice input is not limited to the microphone 26 of the HMD 10, and may be a microphone separately connected to the PC 12.

[0084] If it is longer than the predetermined distance in step 603, the process of voice input is terminated in step 605 without executing voice input.

[0085] FIG. 18 is a diagram showing a display example when voice input is being performed. An object 66 is being moved to the vicinity of the mouth of a user 67 wearing the HMD 10 on the head, showing a state where voice input is being performed on the object 66.

[0086] So far, the display system has been described as including the HMD 10, the controller 11, the PC 12, and the position detection sensor 13 shown in FIG. 1, but the configuration of the display system is not limited to this configuration. As shown in FIG. 19, the functions of the PC 12 and the position detection sensor 13 may be mounted on the HMD 10, and it may be composed only of the HMD 10 and the controller 11.

[0087] When detecting the position of the HMD 10 or the controller 11, the HMD 10 can be equipped with an imaging device (camera) for detecting the position of the HMD 10 or the controller 11. The HMD 10 and the controller 11 can be provided with sensors for detecting the inclination of the HMD 10 and the controller 11.

[0088] The position and inclination of the HMD 10 can be calculated from the size, direction, inclination, etc. of the image of a marker or the like arranged at a reference position by the camera. The position and inclination of the controller 11 can also be calculated from the image captured by the camera. When the HMD 10 is equipped with a camera, the controller 11 may not be used. A marker can be provided on the hand, and an image including the marker can be captured using the camera of the HMD 10, and the position and inclination of the hand can be calculated from the captured image.

[0089] FIG. 20 is a diagram showing the hardware configuration of the HMD 10 and the controller 11 when the configuration shown in FIG. 19 is adopted. The HMD 10 includes an external I / F 20, a CPU 21, a display 22, a memory 23, an HDD 24, and a microphone 26, similar to the configuration shown in FIG. 2. The HMD 10 further includes a sensor 27 and a camera 28.

[0090] The sensor 27 is a gyro sensor that detects angular velocity, an acceleration sensor that detects acceleration, or the like, and detects the inclination and orientation of the HMD 10. The camera 28 recognizes markers provided on the three-dimensional virtual space or the controller 11, and detects the position and inclination of the HMD 10 and the controller 11.

[0091] Similar to the configuration shown in FIG. 2, the controller 11 also includes an operation I / F 30 and an external I / F 31. Instead of the light source 32, the controller 11 includes a sensor 33. The sensor 33 is a gyro sensor, an acceleration sensor, or the like, similar to the sensor 27 of the HMD 10, and detects the inclination and orientation of the controller 11.

[0092] The inclination information of the HMD 10 and the controller 11 may use information from the sensors 27 and 33, or may use information detected by the camera 28 of the HMD 10. The position and inclination of the HMD 10 and the controller 11 may be detected from an image captured by the camera 28 of the HMD 10. Further, the information detected by the sensors 27 and 28 of the HMD 10 and the information detected by the controller 11 may be collected in the HMD 10, and based on the obtained information, the position and inclination of the HMD 10 and the controller 11 may be detected.

[0093] Without using the controller 11, the shape and position of the hand may be detected by the camera 28, and the inclination may be detected from the shape of the hand.

[0094] As shown in FIG. 20, the HMD 10 may include the sensor 27 and the camera 28, the controller 11 may include the sensor 33, and further, as an external sensor, the position detection sensor 13 shown in FIG. 1 may be further provided. In this case, it is possible to select and switch between using the sensors and cameras of the HMD 10 and the controller 11 or using the external position detection sensor 13. For example, when high-precision information is desired in the position information and inclination information, the position detection sensor 13 can be used, and when reactivity is required, the sensors 27, 33 and the camera 28 can be used.

[0095] By providing the apparatus, system, method, and program of the present invention, when arranging user objects, there is no need to check one's hands, nor is it necessary to arrange them using both hands. Therefore, the usability of the user in the three-dimensional virtual space can be improved. Further, by using the shoulder position as the starting point of the arm to determine the arrangement position of the user object, the accuracy of the arrangement position can be improved. Furthermore, by determining the arrangement position based on the shoulder position, it becomes possible to handle the user object with a feeling close to human intuition.

[0096] So far, the present invention has been described with the above-described embodiments as an information processing apparatus, a display system, an information processing method, and a program. However, the present invention is not limited to the above-described embodiments, and can be changed within the scope that those skilled in the art can conceive, such as other embodiments, additions, changes, deletions, etc. Also, as long as the functions and effects of the present invention are exhibited in any aspect, it is included in the scope of the present invention.

Explanation of Reference Numerals

[0097] 10…HMD, 11…Controller, 12…PC, 13…Position Detection Sensor, 14…Server, 15…Network, 20…External I / F, 21…CPU, 22…Display, 23…Memory, 24…HDD, 25…Light Source, 26…Microphone, 30…Operation I / F, 31…External I / F, 32…Light Source, 40…CPU, 41…ROM, 42…RAM, 43…HDD, 44…External I / F, 45…Input / Output I / F, 46…Input Device, 47…Display Device, 50…Acquisition Unit, 51…First Generation Unit, 52…Second Generation Unit, 60…Whiteboard, 61…Hand, 62…Assistant Object, 63…Marker, 64…Blackboard Eraser, 65…Laser Object, 66…Object, 67…User

Prior Art Documents

Patent Documents

[0098]

Patent Document 1

Claims

1. An information processing apparatus for generating an image based on virtual space data indicating a three-dimensional virtual space, comprising: an acquisition unit that acquires position information and inclination information of the display unit and position information of the operation unit or the hand from a position of the display unit worn by the user, an inclination of the display unit with respect to a reference direction, and a detection unit that detects the operation unit operated by the user or a position of the user's hand; an estimation unit that estimates a position of a part of the user existing between the display unit and the operation unit or the hand based on the acquired position information of the display unit and the position information of the operation unit or the hand; a first generation unit that generates a user object on the three-dimensional virtual space based on the estimated position information of the part; and the first generation unit changes a length of the user object based on the estimated position information of the part and the position information of the operation unit or the hand, the information processing apparatus.

2. The information processing apparatus according to claim 1, wherein the part is a shoulder.

3. The information processing apparatus according to claim 1 or 2, further comprising a second generation unit that generates an input object for realizing a user input on the three-dimensional virtual space in response to an operation on the user object.

4. The information processing apparatus according to claim 1 or 2, further comprising a selection processing unit that executes a process of selecting an object arranged on the three-dimensional virtual space with the user object based on the position information of the part and the position information of the operation unit or the hand.

5. including an arithmetic unit that calculates a distance between the position of the part and the position of the operation unit or the hand, wherein the selection processing unit selects the object arranged on the three-dimensional virtual space by changing the length of the user object by the first generation unit according to the distance calculated by the arithmetic unit, the information processing apparatus according to claim 4.

6. The information processing apparatus according to claim 5, wherein the first generation unit changes the length of the user object such that a rate of change of the length of the user object increases as the distance increases.

7. The information processing apparatus according to any one of claims 4 to 6, further comprising movement processing means for executing a process of moving the selected object to a position designated by the user object based on the position information of the part and the position information of the operation means or the hand.

8. The information processing apparatus according to claim 7, further comprising input reception means for receiving voice input from a user based on the position information of the display means, the position information of the operation means or the hand, and the position information of the selected object in the three-dimensional virtual space after the movement process by the movement processing means.

9. A display system including an information processing apparatus, wherein the information processing apparatus acquisition means for acquiring the position information and tilt information of the display means and the position information of the operation means or the hand from a detection means for detecting the position of the display means worn by the user, the tilt of the display means with respect to a reference direction, and the operation means operated by the user or the position of the hand of the user; estimation means for estimating the position of the part of the user existing between the display means and the operation means or the hand based on the acquired position information of the display means and the position information of the operation means or the hand; generation means for generating a user object in a three-dimensional virtual space based on the estimated position information of the part; and the generation means changes the length of the user object based on the estimated position information of the part and the position information of the operation means or the hand. A display system.

10. The display means that is worn by the user, inside the information processing apparatus or separately from the information processing apparatus, and displays the user object generated by the information processing apparatus and an image in the tilt direction of the display means in the three-dimensional virtual space, The display system according to claim 9, further comprising, inside the information processing apparatus or separately from the information processing apparatus, detection means for detecting the position of the display means and the tilt of the display means with respect to a reference direction and the operation means operated by the user or the position of the hand of the user.

11. A method executed by an information processing apparatus for generating an image based on virtual space data indicating a three-dimensional virtual space, A step of obtaining position information and inclination information of the display means and position information of the operation means or the hand from a position of the display means worn by the user, an inclination of the display means with respect to a reference direction, and a detection means for detecting a position of the operation means operated by the user or a position of the user's hand. A step of estimating a position of a part of the user existing between the display means and the operation means or the hand based on the obtained position information of the display means and the position information of the operation means or the hand. A step of generating a user object on the three-dimensional virtual space based on the estimated position information of the part. Including In the step of generating the user object, a method of changing the length of the user object based on the estimated position information of the part and the position information of the operation means or the hand.

12. A program for causing a computer to execute a process of generating an image based on virtual space data indicating a three-dimensional virtual space, A step of obtaining position information and inclination information of the display means and position information of the operation means or the hand from a position of the display means worn by the user, an inclination of the display means with respect to a reference direction, and a detection means for detecting a position of the operation means operated by the user or a position of the user's hand. A step of estimating a position of a part of the user existing between the display means and the operation means or the hand based on the obtained position information of the display means and the position information of the operation means or the hand. A step of generating a user object on the three-dimensional virtual space based on the estimated position information of the part. Executing In the step of generating the user object, a program of changing the length of the user object based on the estimated position information of the part and the position information of the operation means or the hand.

Images