Information processing apparatus, display system, information processing method, and program

Abstract

To provide an apparatus, a system, a method, and a program that can improve usability in a three-dimensional virtual space.SOLUTION: An information processing apparatus includes: acquisition means that acquires, from detection means that detects the position of display means worn on a user, the inclination of the display means with respect to a reference direction, and the position of operation means operated by the user or the user's hand, position information and inclination information on the display means and position information on the operation means or the hand; first creation means that, based on the acquired position information on the display means and position information on the operation means or the hand, creates a user object on a three-dimensional virtual space; and second creation means that, based on the acquired inclination information on the display means and virtual space data, creates an image in an inclination direction of the display means in the three-dimensional virtual space.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an information processing device, a display system, and a method for generating an image, and a program for causing a computer to execute a process for generating an image. [Background technology]

[0002] VR (Virtual Reality) technology, which allows users to experience content expressed in a three-dimensional virtual space, uses a head-mounted display (HMD) worn on a person's head as a display device for displaying images of the three-dimensional virtual space.

[0003] Even in a 3D virtual space, there are cases where user input, such as writing on a whiteboard, is required. However, there is no standard input method for user input in a 3D virtual space, such as using a mouse and keyboard as in a personal computer (PC).

[0004] One method of allowing user input in a three-dimensional virtual space is known in which an avatar object corresponding to a user is placed in the user's field of view and a user interface for accepting operations from the user is displayed on the avatar object (see, for example, Patent Document 1). Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the above-mentioned conventional method, the user interface is displayed on one of the arms of the avatar object, so that the user needs to tilt his / her head to view that arm and operate it with the other hand, which makes operation difficult and does not have high usability.

[0006] The present invention has been made in consideration of the above problems, and has an object to provide an information processing device, a display system, an information processing method, and a program that can improve usability in a three-dimensional virtual space. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, one embodiment of the invention provides an information processing device that generates an image based on virtual space data indicating a three-dimensional virtual space, comprising: an acquisition means for acquiring position information and tilt information of the display means and position information of the operating means or hand from a detection means that detects the position of a display means worn by a user, the tilt of the display means relative to a reference direction, and the position of an operating means operated by the user or the user's hand; a first generation means for generating a user object in the three-dimensional virtual space based on the acquired position information of the display means and position information of the operating means or hand; and a second generation means for generating an image in the tilt direction of the display means in the three-dimensional virtual space based on the acquired tilt information of the display means and virtual space data. Effect of the Invention

[0008] According to the present invention, it is possible to improve the usability in a three-dimensional virtual space. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing a first example of the configuration of a display system. [Diagram 2] A diagram showing the first hardware configuration of an HMD, a controller, and a PC. [Diagram 3] FIG. 2 is a block diagram showing an example of the functional configuration of a PC. [Figure 4] 2 is a sequence diagram showing the flow of processing executed by the display system shown in FIG. 1. [Diagram 5] 4A to 4C are diagrams illustrating a first method for estimating the position of a user's shoulder. [Figure 6] 6 is a flowchart showing a process for estimating the position of the user's shoulder by the first method shown in FIG. 5. [Figure 7] 11A to 11C are diagrams illustrating a second method for estimating the position of a user's shoulder. [Figure 8]8 is a flowchart showing the flow of processing for estimating the position of the user's shoulder by the second method shown in FIG. 7. [Figure 9] 11 is a flowchart showing a process for generating an assistant object and displaying an image as a first example of a user object for assisting a user in inputting data into a three-dimensional virtual space. [Figure 10] FIG. 13 is a diagram showing a display example in which an assistant object is generated in a three-dimensional virtual space and an image is displayed. [Figure 11] A diagram showing the assistant object being touched and its function being invoked. [Figure 12] 13 is a flowchart showing a process of generating a laser object and displaying an image as a second example of a user object for supporting user input into a three-dimensional virtual space. [Figure 13] FIG. 13 is a diagram showing an example of selecting an object by a laser object. [Figure 14] A diagram showing the relationship between the length of a laser object and the distance between the position of a user's shoulder and the position of a controller. [Figure 15] 13A and 13B are diagrams illustrating an example of how a selected object is moved by a laser object. [Figure 16] 11 is a flowchart showing a process for moving an object selected by a laser object. [Figure 17] 11 is a flowchart showing a flow of voice input as an example of user input. [Figure 18] FIG. 13 is a diagram showing a display example when voice input is being performed. [Figure 19] FIG. 13 is a diagram showing a second configuration example of the display system. [Figure 20] A diagram showing a second hardware configuration of an HMD and a controller. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Fig. 1 is a diagram showing a first configuration example of a display system. The display system includes an HMD 10 functioning as a display means worn on a user's head, a controller 11 functioning as an operation means that a user holds in his / her hand or wears in his / her hand and operates, and a PC 12 functioning as an information processing device. The display system also includes a position detection sensor 13 functioning as a detection means for detecting the position of the HMD 10 and the inclination of the HMD 10 relative to a reference direction, and the position of the controller 11 and the inclination of the controller 11 relative to a reference direction. In the example shown in Fig. 1, the display system includes a server 14 that manages position information, operation information, etc. of multiple users.

[0011] The PC 12 and the server 14 are communicatively connected via a network 15. The HMD 10 and the position detection sensor 13 are connected to the PC 12 via a cable or the like. The controller 11 is wirelessly connected to the PC 12 via Bluetooth (registered trademark), WiFi (registered trademark), or the like. The HMD 10 and the position detection sensor 13 may also be wirelessly connected via WiFi (registered trademark) or the like.

[0012] The HMD 10 has a display for displaying images to the user, and displays images on the display corresponding to the position of the HMD 10 and the inclination relative to a reference direction. The images are generated as two images corresponding to the left and right eyes, respectively, to make the images appear three-dimensional using the parallax between the left and right eyes of the user. For this reason, the HMD 10 has two displays for displaying images corresponding to the left and right eyes, respectively. The reference direction is, for example, any direction parallel to the floor. The HMD 10 has a light source such as an infrared LED (Light Emitting Diode) and emits infrared rays.

[0013] The controller 11 is an operating means that the user holds in his / her hand or wears in the position of the hand, and has buttons, a wheel, a touch sensor, etc., receives input from the user, and transmits the received information to the PC 12. The controller 11 also has a light source such as an infrared LED, and emits infrared rays.

[0014] The position detection sensor 13 is disposed at an arbitrary position in front of the user, detects the position and tilt of the HMD 10 and the controller 11 from infrared rays emitted from the HMD 10 and the controller 11, and outputs the position information and tilt information. The position detection sensor 13 is, for example, an infrared camera, and can detect the position and tilt of the HMD 10 and the controller 11 based on the captured image. Note that the HMD 10 and the controller 11 are provided with a plurality of light sources in order to detect the position and tilt of the HMD 10 and the controller 11 with high accuracy. The position detection sensor 13 is composed of one or more sensors, and when a plurality of sensors are used, they can be provided on the side, rear, etc.

[0015] The PC 12 generates a user object for supporting user input in the three-dimensional virtual space displayed on the display of the HMD 10, based on the position information and tilt information of the HMD 10 output from the position detection sensor 13, the position information of the controller 11, and, if necessary, the tilt information of the controller 11. Then, based on the position information and tilt information of the HMD 10 and the three-dimensional virtual space data, the PC 12 generates images corresponding to the left and right eyes, which are images in the user's viewing direction (more precisely, the tilt direction of the HMD 10) in the three-dimensional virtual space, and executes processing to display the images on the display of the HMD 10.

[0016] The PC 12 communicates with the server 14 via the network 15, obtains position information, etc. of other users who are in the same three-dimensional virtual space, and can execute a process of displaying avatar objects representing the other users on the display of the HMD 10.

[0017] The display system can be used, for example, to gather the avatar objects of each user in a virtual conference room as a three-dimensional virtual space and hold a conference using a whiteboard, etc. The display system can be used when holding an interactive conference because the participants in the conference can proactively participate in the conference using a whiteboard, etc.

[0018] In a conference using the display system, a user can operate the controller 11, touch a user object in a displayed image to call up a pen input function, pick up the displayed pen, move the pen, and input characters on the whiteboard. Note that this is one form of use, and the use is not limited to this form.

[0019] 1, the HMD 10 and the controller 11 have a light source and the position detection sensor 13 is disposed at an arbitrary position, but the HMD 10 and the controller 11 may have the position detection sensor 13 and a light source or a marker that reflects infrared light may be disposed at an arbitrary position. When a marker is used, the HMD 10 and the controller 11 have a light source and the position detection sensor 13, and infrared light emitted from the light source is reflected by the marker and the reflected infrared light is detected by the position detection sensor 13, thereby making it possible to detect the position and inclination of the HMD 10 and the controller 11.

[0020] If there is an object between the position detection sensor 13 and the HMD 10 and controller 11, the infrared rays will be blocked, making it impossible to accurately detect the position and inclination. For this reason, when using the HMD 10 and controller 11 to perform operations or displays, it is desirable to do so in an open space.

[0021] In the example shown in Figure 1, a space is provided in which a user can wear an HMD 10, hold a controller 11 in their hand, and stretch or unstretch their arms, and a PC 12 and a position detection sensor 13 are placed outside the space.

[0022] 2 is a diagram showing an example of the hardware configuration of the HMD 10, the controller 11, and the PC 12. The HMD 10 includes an external I / F 20, a CPU 21, a display 22, a memory 23, a HDD 24, a light source 25, and a microphone 26. The CPU 21 controls the entire HMD 10 and executes processes such as emission of light from the light source 25, communication with the outside, and display on the display 22. The external I / F 20 is an interface for communicating with the PC 12. The display 22 may be a liquid crystal display or an organic EL (Electro Luminescence) display.

[0023] The memory 23 provides a working area for the CPU 21. The HDD 24 stores image data to be displayed in the three-dimensional virtual space. The light source 25 is an infrared LED or the like, which emits infrared rays. The infrared rays can be emitted in a flashing manner in a predetermined pattern. The microphone 26 is a voice input device that allows the user to input by voice.

[0024] The controller 11 includes an operation I / F 30, an external I / F 31, and a light source 32. The operation I / F 30 is a button, a wheel, a touch sensor, etc., and is arranged on the outer surface of the controller 11 to enable operation by a user and to accept input of operation information. The external I / F 31 is wirelessly connected to the PC 12, and transmits operation information accepted by the operation I / F 30 to the PC 12. The light source 32 emits infrared rays so as to blink in a predetermined pattern. The light source 32 can be distinguished from the HMD 10 by blinking in a pattern different from the light source of the HMD 10.

[0025] The PC 12 includes a CPU 40 , a ROM 41 , a RAM 42 , a HDD 43 , an external I / F 44 , an input / output I / F 45 , an input device 46 , and a display device 47 .

[0026] The CPU 40 executes the process of controlling the entire PC 12, generating the above-mentioned user object, and generating an image in the user's viewing direction in the three-dimensional virtual space and displaying it on the display of the HMD 10. The ROM 41 stores a boot program for starting up the PC 12, firmware for controlling the HDD 43, the external I / F 44, etc. The RAM 42 provides a working area for the CPU 40.

[0027] The HDD 43 stores an OS (Operating System), programs for executing the above processes, image data, etc. The external I / F 44 is connected to the network 15 shown in Fig. 1 and communicates with the server 14 via the network 15. The external I / F 44 is also connected to the HMD 10 and the position detection sensor 13 by cables or the like, and is wirelessly connected to the controller 11, and communicates with the HMD 10, the controller 11, and the position detection sensor 13.

[0028] The input device 46 is a mouse, a keyboard, etc., and is used by a user to input information and accept operations. The display device 47 provides a display screen for the user and displays the input information, processing results, etc. The input / output I / F 45 is an interface that controls the input of information from the input device 46 and the output of information to the display device 47.

[0029] 3 is a block diagram showing an example of the functional configuration of the PC 12. Here, since the PC 12 functions as an information processing device, the functional configuration of the PC 12 will be described. The PC 12 can be provided with functional units that are generated to realize each function by the CPU 40 executing a program stored in the HDD 43. Note that each functional unit is not limited to being realized by a program, and may be realized by a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function.

[0030] The PC 12 includes at least an acquisition unit 50, a first generation unit 51, and a second generation unit 52. The PC 12 may include other functional units. The acquisition unit 50 acquires position information and tilt information of the HMD 10 and the controller 11 detected by the position detection sensor 13. The acquisition unit 50 also acquires information (operation information) input by the user from the controller 11. The acquisition unit 50 also acquires three-dimensional virtual space data from the HDD 43 and the server 14.

[0031] The three-dimensional virtual space data is data of a virtual three-dimensional space represented by an x-axis, a y-axis, and a z-axis, and includes object data and image data of one or more objects placed in the space. Taking the above-mentioned virtual conference room as an example, the image data is image data including a whiteboard, a floor, walls, a ceiling, an entrance and an exit. The objects are objects placed in the three-dimensional virtual space, and include input objects such as a pen for implementing user input, objects such as sticky notes for inputting characters, and user objects for supporting user input.

[0032] The first generating unit 51 generates a user object in a three-dimensional virtual space based on the position information and tilt information of the HMD 10 and the controller 11 acquired by the acquiring unit 50. The user object is an assistant object capable of calling a function in an application, a laser object capable of selecting an object, etc. When the display system is used for a conference, the application is a conference application, and an example of a function in the application is pen input, etc.

[0033] The second generating unit 52 generates image data to be displayed on the display of the HMD 10 based on the position information, tilt information, three-dimensional virtual space data, and operation information acquired by the acquiring unit 50, and transmits the image data to the HMD 10. The second generating unit 52 generates image data of an image in the field of view direction to which a tilt is applied with the position of the HMD in the three-dimensional virtual space as the origin, using the three-dimensional virtual space data, based on the position information and tilt information of the HMD in the three-dimensional virtual space.

[0034] 4 is a sequence diagram showing the flow of the entire process executed by the display system. When the HMD 10, controller 11, PC 12, and position detection sensor 13 are powered on, the PC 12 acquires three-dimensional virtual space data from its own HDD 43 or server 14 (S1). The HMD 10 and controller 11 emit infrared rays from their respective light sources, and the position detection sensor 13 detects the positions and inclinations of the HMD 10 and controller 11 based on the light sources emitting the infrared rays (S2). The position detection sensor 13 transmits the detected positions and inclinations to the PC 12 as position information and inclination information (S3).

[0035] The controller 11 accepts an operation from the user and transmits operation information of the accepted operation to the PC 12 (S4). The PC 12 generates image data based on the acquired three-dimensional virtual space data, the position information and tilt information of the HMD 10 and the controller 11 received from the position detection sensor 13, and the operation information received from the controller 11 (S5). The PC 12 then transmits the generated image data to the HMD 10 (S6). The HMD 10 displays the received image data on the display (S7). From this point on, the detection of the position and tilt in S2 through the transmission of image data in S5 are repeated until the power of the HMD 10, etc. is turned off, and the image data displayed on the display of the HMD 10 is updated.

[0036] The overall processing flow in the display system has been described above. Below, a detailed description will be given of a method for generating a user object that is displayed together with image data in a three-dimensional virtual space.

[0037] A user object must be easy for a user to touch to invoke functionality within an application, or to select an object.

[0038] Since the image data is generated with the position of the HMD 10 as the origin and with the tilt applied, the user object can also be placed in the three-dimensional virtual space based on the position and tilt of the HMD 10.

[0039] However, when the user's head moves, the user object moves accordingly. Therefore, if only the head moves, it may be difficult to touch the user object depending on the position and orientation of the head. For example, this may occur when the user object moves to the side opposite to the dominant hand. This is also true when using the user object to select an object. This is not considered to be high usability.

[0040] When moving an arm, a person moves the arm up, down, left and right with the shoulder as the fulcrum. The shoulder is one of the parts of the body that exists between the head and the hand on which the controller 11 is worn. When a user touches a user object, the user moves the arm to change the position of the controller 11. Therefore, if the user object is positioned based on the position of the shoulder, the user object will not move following the head movement, and the user object can be positioned within easy reach of the dominant hand even if the shoulder moves.

[0041] When locating a user object based on the shoulder position, it is necessary to estimate the shoulder position based on the position information and tilt information of the HMD 10 and the controller 11 acquired from the position detection sensor 13. To estimate the shoulder position, it is necessary to know the relative positional relationship between the HMD 10 and the shoulder position. The relative positional relationship is the shoulder position as seen from the HMD 10 or the position of the HMD 10 as seen from the shoulder position.

[0042] A first method for determining the relative positional relationship between the position of the user's shoulder and the HMD 10 will be described with reference to Fig. 5. Fig. 5 is a diagram of the user as viewed from above. The user holds or wears the controller 11 in his or her hand, stretches his or her elbow horizontally toward the floor, and repeats the motion of bringing the left and right hands close together to close the arms while the elbows are stretched, and then moving the left and right hands apart to open the arms.

[0043] In such a movement, the left and right hands move in a circular arc with the left and right shoulders as their respective fulcrums. At this time, position information of the controller 11 is acquired by the position detection sensor 13. The position information acquired by the position detection sensor 13 is position information of a point on the circular arc. The position of the shoulders is a point equidistant from each point on the circular arc. Therefore, it is possible to calculate the position information of the shoulders from the position information acquired by the position detection sensor 13.

[0044] The position information of the HMD 10 is acquired by the position detection sensor 13 before starting the movement of opening and closing the arms. Therefore, the shoulder position based on the position of the HMD 10 can be calculated as the relative positional relationship between the HMD 10 and the shoulder position.

[0045] Fig. 6 is a flowchart showing the flow of processing for determining the relative positional relationship between the user's shoulder position and the HMD 10 by the first method shown in Fig. 5. This processing starts from step 100 when the use of the HMD 10 and the controller 11 is started, or when it is felt that the positional deviation detected by the position detection sensor 13 has become large.

[0046] In step 101, the arm is held horizontally with the elbow extended. In step 102, the arm is opened and closed with the elbow extended. In step 103, position information of the controller 11 is continuously obtained. Here, the position information may be position information for one movement from the closed state to the open state, but in order to increase the accuracy of the position information, it is preferable to repeat the movement several times.

[0047] In step 104, the acquired position information is averaged. Because the movement of closing and opening the arms is repeated several times, multiple pieces of position information, such as when the arms are closed and when they are opened, are obtained. For this reason, the position information when the arms are closed and when they are opened are averaged. This averaging makes it possible to estimate the shoulder position with high accuracy. In step 105, the relative positional relationship between the HMD 10 and the shoulder position is found from the averaged position information, and in step 106, this process ends.

[0048] The shoulder position is calculated as the shoulder position based on the position of the HMD 10. The calculated shoulder position data is shoulder position data indicating the relative positional relationship with the HMD 10, and is, for example, coordinate data represented by the x-axis, y-axis, and z-axis. The shoulder position data is calculated for each of the left and right shoulders.

[0049] A second method for determining the relative positional relationship between the user's shoulder position and the HMD 10 will be described with reference to Fig. 7. Fig. 7 is a diagram showing the user from the front. In the second method, in order to directly determine the shoulder position, controllers 11 worn on the left and right hands are placed on the left and right shoulders, and position information of the controllers 11 is detected by a position detection sensor 13.

[0050] The controller 11 worn on either hand may be placed at either shoulder position, and the controller 11 may be placed at the shoulder position in either pose shown in FIG. 7(a) or (b).

[0051] If the controller 11 is on the left side relative to the position and orientation of the HMD 10, the position information of the controller 11 is estimated to be left shoulder position information, and if it is on the right side, the position information of the controller 11 is estimated to be right shoulder position information, and each position information is treated as shoulder position data.

[0052] Fig. 8 is a flowchart showing the flow of processing for determining the relative positional relationship between the user's shoulder position and the HMD 10 by the second method shown in Fig. 7. This processing can be started from step 200 when the use of the HMD 10 and the controller 11 is started, or when it is felt that the positional deviation detected by the position detection sensor 13 has become large.

[0053] In step 201, position information of the left and right controllers 11 is obtained by the position detection sensor 13. In step 202, position information and tilt information of the HMD 10 are obtained by the position detection sensor 13. In step 203, it is estimated which shoulder (left or right) each controller 11 corresponds to, based on the position information and tilt information of the HMD 10. In step 204, the relative positional relationship between the HMD 10 and the positions of each shoulder is obtained, and in step 205, this process ends.

[0054] After creating shoulder position data, which is position data relative to the HMD 10, the PC 12 executes a process of generating a user object based on the shoulder position. A user object is an object for assisting a user input, such as an assistant object that calls a function in an application, or a laser object as a selection object that selects one of the objects placed in the three-dimensional virtual space. The process of estimating these shoulder positions is also executed by the first generation unit 51.

[0055] With reference to FIG. 9, a process of generating an assistant object in a three-dimensional virtual space and generating an image to be displayed on the HMD 10 will be described. This process is executed after determining the relative positional relationship between the position of the user's shoulder and the HMD 10. This process starts from step 300, and in step 301, position information and tilt information of the HMD 10 and position information and tilt information of the controller 11 are acquired from the position detection sensor 13. The position information and tilt information of the controller 11 are the position information and tilt information of the two controllers 11 worn on both the left and right hands. Hereinafter, for ease of explanation, the controller 11 will be simply described as a controller 11, but in reality, it means the two controllers 11 on both hands.

[0056] In step 302, shoulder position data, which is position data relative to the HMD 10, is applied to position information of the HMD 10 (coordinates represented by the x-axis, y-axis, and z-axis) to estimate the position of the shoulder. The shoulder position data is data indicating the position of the shoulder with respect to the HMD 10 as a reference in coordinates. In step 303, position information (coordinates) of an assistant object in a three-dimensional virtual space is calculated with reference to the estimated shoulder position. The assistant object may be an object of any shape, and the shape, size, etc. are set in advance.

[0057] The PC 12 holds assistant object position calculation data that indicates the relative positional relationship between the shoulder position and the assistant object, and calculates the coordinates of the assistant object in the three-dimensional virtual space by applying the assistant object position calculation data (UVW coordinate system data in which the inclination of the HMD 10 is applied to the xyz coordinate system based on the reference direction of the HMD 10) to the shoulder position. The assistant object position calculation data is data that is adjusted and stored in advance to be the optimal position for the user.

[0058] In step 304, the distance between the controller 11 and the assistant object is created as distance information from the position information of the controller 11 and the calculated position information of the assistant object. The PC 12 can be provided with a calculation unit that calculates the distance in order to create this distance information. The calculation unit also executes other calculation processes such as the arithmetic mean described above. In step 305, it is determined whether the created distance information is longer than a predetermined distance. If it is equal to or shorter than the predetermined distance, the process proceeds to step 306, where application of assistant object position calculation data to the shoulder position is stopped, and the coordinates (coordinates represented by the xyz coordinate system) of the assistant object in the three-dimensional virtual space at the time when application of assistant object position calculation data is stopped are fixed. The PC 12 can be provided with a determination unit that determines whether the distance information is longer than a predetermined distance, and fixes the coordinates if it is equal to or shorter than the predetermined distance. The determination unit also executes other determination processes.

[0059] The coordinates of the assistant object to be fixed based on the position of the shoulder do not change every time the power is turned on and the HMD 10 and the controller 11 are started. Therefore, the coordinates of the assistant object to be fixed can use the coordinates of the assistant object determined in the process of displaying the previous image.

[0060] If the coordinates of the assistant object are not fixed, when you try to call a function by touching the assistant object and your shoulder moves, the assistant object moves and it becomes difficult to touch. This is because the assistant object moves with the movement of the shoulder. Therefore, the coordinates of the assistant object are fixed so that the assistant object does not move with the movement of the shoulder when you try to touch the assistant object.

[0061] If it is determined in step 305 that the distance is longer than the predetermined distance, the process proceeds directly to step 307. In step 307, the assistant object is placed in the three-dimensional virtual space based on the coordinates of the assistant object. In step 308, image data of an image in a viewing direction centered on the position of the HMD 10 and applying the tilt of the HMD 10 is generated based on the position information and tilt information of the HMD 10. The generated image data is displayed on the display of the HMD 10, and in step 309, an assistant object is generated, and the process of generating an image to be displayed on the HMD 10 is completed.

[0062] FIG. 10 is a diagram showing a display example of a generated assistant object. A whiteboard 60 is included in an image displaying a three-dimensional virtual space, and a virtual user's hand 61 is shown in front of the whiteboard 60. The assistant object 62 is a polyhedron having a surface that can be touched by the fingers of the hand 61. Since the assistant object 62 is generated and placed based on the position of the shoulder, it is easy to touch the assistant object 62 with a finger and to call up a function. Information such as characters representing a function is displayed on the surface of the polyhedron, and since the assistant object 62 is placed based on the position of the shoulder, the information is easy to see.

[0063] In Fig. 10, the user operates the controller 11 and touches the assistant object 62 with the fingers of the hand 61 in the three-dimensional virtual space. When the assistant object 62 is composed of a polyhedron, information such as characters representing different functions may be displayed on each face, and a predetermined function may be called by touching a predetermined face. The user can send an instruction to call a function to the PC 12 by touching the assistant object 62 with the fingers of the hand 61 in the three-dimensional virtual space and pressing a predetermined button on the controller 11.

[0064] FIG. 11 is a diagram showing a display example when an assistant object 62 is touched with the fingers of a hand 61 to call a function in an application. The function is displayed as a pop-up (a screen that appears to jump out in front) by the PC 12 generating an object representing the function for the HMD 10, for example, and invoking the function by touching the object, etc. For this reason, the PC 12 can be provided with a third generating unit for generating objects representing the function. In FIG. 11, a marker 63 for writing on the whiteboard 60, a blackboard eraser 64, etc. are generated and displayed in the image. Other functions may include a function for moving the whiteboard 60, a function for copying the contents written on the whiteboard 60, etc.

[0065] A user can write characters on the whiteboard 60 by operating the controller 11, picking up a marker 63 in the hand 61 in the three-dimensional virtual space, moving the marker 63 to the whiteboard 60, and moving the hand 61. A user can also erase characters written on the whiteboard 60 by operating the controller 11, picking up a blackboard eraser 64 in the hand 61 in the three-dimensional virtual space, and moving the hand 61.

[0066] 12, a process of generating a laser object in a three-dimensional virtual space and generating an image to be displayed on the HMD 10 will be described. This process is also executed after determining the relative positional relationship between the position of the user's shoulder and the HMD 10. This process starts from step 400, and in step 401, position information and tilt information of the HMD 10 and position information and tilt information of the controller 11 are obtained from the position detection sensor 13.

[0067] In step 402, the shoulder position data is applied to the position information of the HMD 10 (coordinates represented by the x-axis, y-axis, and z-axis) to estimate the shoulder position. In step 403, distance information indicating the distance between the shoulder position and the controller 11 is created based on the estimated shoulder position and the position information of the controller 11. In step 404, the length (number of pixels) of the laser object is determined based on the created distance information. The laser object is a rod-shaped object, and the longer the distance between the shoulder and the controller 11, the longer the length of the laser object.

[0068] Here, Fig. 13 shows the relationship between the length of the laser object and the distance between the shoulder and the controller 11. Fig. 13(a) is a graph showing the relationship in which the length of the laser object increases quadratically according to the distance. Fig. 13(b) is a graph showing the relationship in which the length increases at a constant rate up to a certain distance, and once the certain distance is exceeded, the length of the laser object increases quadratically.

[0069] As shown in Fig. 13(c), when the length of the laser object increases linearly according to the distance between the shoulder and the controller 11, the slope of the graph becomes steeper than the slope of the graph up to a certain distance shown in Fig. 13(b). In this case, when the target object is nearby, even if the arm is stretched out a little, the laser object becomes too long, and when the target object is far away, even if the arm is stretched out a long way, the target object may not be reached.

[0070] However, as shown in Figures 13(a) and (b), by changing the length of the laser object so that the rate at which the length of the laser object changes increases as the distance increases, it becomes possible to appropriately select a target object whether the target object is close or far away. Note that the length of the laser object is not limited to the examples shown in Figures 13(a) and (b) as long as it can be changed so that the rate at which the length of the laser object changes increases as the distance increases.

[0071] 12 again, once the length of the laser object is determined from the created distance information, the laser object is placed in the three-dimensional virtual space in a specific direction starting from the position of the controller 11. The specific direction is, for example, a specific tilt direction of the controller 11 or a direction connecting the controller 11 and the position of the shoulder. If the controller 11 has a grip portion to be held by the hand, the grip portion has a shape that extends in a certain direction. The specific tilt direction is the direction in which the grip portion extends (longitudinal direction).

[0072] In step 405, it is determined whether the laser object arranged in the three-dimensional virtual space has passed through a specific object in the three-dimensional virtual space. The specific object is an object that can be moved or otherwise operated when a specific input (e.g., a movement instruction by a button input) is made after selection. If it is determined that the laser object has passed through the specific object, the process proceeds to step 406, where a selection process is performed for the specific object that has passed. Specifically, a selection flag is assigned to the specific object that has passed, the laser object that has passed through the specific object is deleted, and the process proceeds to step 407. The PC 12 may include a selection processing unit that performs this selection process. On the other hand, if it is determined in step 405 that the laser object has not passed through the specific object, the process proceeds to step 408.

[0073] In step 407, a user operation is accepted. The user operation is, for example, the movement of an object, which includes input of an instruction to move the object. The movement of the object will be described later. In step 408, in the three-dimensional virtual space, image data is generated in the field of view direction centered on the position of HMD 10 and applying the tilt of HMD 10 based on the position information (coordinates) of the laser object and the position information and tilt information of HMD 10, and the image is displayed on the display. Then, in step 409, a laser object is generated, and the process of generating an image to be displayed on HMD 10 is completed.

[0074] Fig. 14 is a diagram showing an example of selecting an object by a laser object. In step 404 of Fig. 12, a laser object 65 is generated as an object extending in a rod shape in the direction of the extension of a virtual user's hand 61 in the three-dimensional virtual space, and is placed in the three-dimensional virtual space. In Fig. 14(a), a laser object 65 is placed in the three-dimensional virtual space so as to extend from the hand 61. In Fig. 14(b), in step 406 of Fig. 12, an object 66 in the three-dimensional virtual space is selected, and the laser object 65 that has passed through is deleted.

[0075] Fig. 15 is a diagram showing a state in which the object 66 is moved, as an example of a user operation on the object 66 selected in step 407 in Fig. 12. The user can change the direction in which the object 66 is placed by changing the position or tilt of the controller 11, as shown in Fig. 15(a). Also, the user can pull back their arm while wearing the controller 11, thereby pulling the object 66 towards the user, as shown in Fig. 15(b).

[0076] Fig. 16 is a flowchart showing a process flow for moving an object 66 selected by a laser object 65, as an example of a specific process in step 407 in Fig. 12. Here, the moving process will be described, but the operation on the object 66 selected by the laser object 65 is not limited to moving the object 66.

[0077] 16 is performed after placing the laser object 65 in the three-dimensional virtual space and completing selection of the specific object. This process starts from step 500, and in step 501, it is confirmed whether or not a selection flag has been assigned to the specific object 66. If assigned, the process proceeds to step 502, where it is determined whether or not a movement instruction has been issued by the controller 11. The controller 11 is provided with an operation I / F 30 such as a button, and determines whether or not a movement instruction has been issued based on whether or not a button has been pressed.

[0078] If there is a movement instruction, the process proceeds to step 503, where movement processing of the object 66 to which a selection flag has been assigned is executed based on the user's operation of the controller 11. The PC 12 can be equipped with a movement processing unit that executes movement processing of the object 66. If the movement processing is an attracting process, the coordinates of the object 66 to which a selection flag has been assigned are instantaneously moved to the user's hand. If the movement processing is other than an attracting process, the object 66 to which a selection flag has been assigned moves up, down, left, right, forward, backward in the three-dimensional virtual space. Whether to execute the attracting process or other movement processing can be selected by changing the button to be pressed among multiple buttons provided on the controller 11. Note that, in addition to changing the button, the selection may also be selected by a gesture of pulling the hand toward the user (the position or tilt of the controller 11).

[0079] If the selection flag is not assigned in step 501, if there is no movement instruction in step 502, or if the execution of the movement process is completed in step 503, the process proceeds to step 504, where it is determined whether or not there is an input instruction from the user. An input instruction from the user is, for example, a voice input by the user or a character input in the three-dimensional virtual space. The presence or absence of an input instruction can be determined based on the presence or absence of a button on the controller 11 being pressed, as with the presence or absence of a movement instruction. The presence or absence of an input instruction can also be determined based on the presence or absence of an operation using a gesture to bring the object 66 closer to the user's mouth.

[0080] If an input instruction is given by the user in step 504, the process proceeds to step 505, where input processing is executed for the object 66 to which the selection flag has been assigned. A detailed description of the input processing will be given later. If there is no input instruction in step 504, or if the input processing ends in step 506, the movement processing of the object 66 ends. If the object 66 is to be moved further, the processing can be started from step 500.

[0081] 17 is a flowchart showing the flow of voice input as an example of user input. Processing starts from step 600, and in step 601, position information of the HMD 10, position information of the controller 11, and position information of the laser object 65 are acquired. The position information of the laser object 65 is the position of the tip of the laser object 65, that is, the object 66 selected by the laser object 65.

[0082] In step 602, distance information indicating the distance between the HMD 10 and the laser object 65 is generated from the position information of the HMD 10 and the position information of the laser object 65 in the three-dimensional virtual space. The distance information is constantly generated after the movement process is executed.

[0083] In step 603, the position of the user's mouth is estimated based on the position of the HMD 10, and it is determined whether the distance between the user's mouth and the laser object 65 in the three-dimensional virtual space is shorter than a predetermined distance. If the user pulls the object 66 closer and the distance becomes shorter than the predetermined distance, the process proceeds to step 604 and executes voice input. The HMD 10 receives voice input through the microphone 26 functioning as an input receiving unit, and sends the input voice data to the PC 12. The PC 12 executes voice recognition processing on the voice data acquired from the HMD 10. In the voice recognition processing, the contents of the user's speech are converted into image data including characters by voice recognition, and the converted image data is pasted on the object 66 for which the movement processing has been executed. The PC 12 may include a data processing unit that executes a predetermined process such as voice recognition processing on the input data. Note that the microphone that receives the voice input is not limited to the microphone 26 of the HMD 10, and may be a microphone separately connected to the PC 12.

[0084] If it is determined in step 603 that the distance is longer than the predetermined distance, then in step 605 the voice input process ends without executing the voice input.

[0085] 18 is a diagram showing a display example when voice input is being performed. The figure shows a state in which an object 66 is moved to the mouth of a user 67 wearing the HMD 10 on his / her head, and voice input is being performed to the object 66.

[0086] Although the display system has been described so far as including the HMD 10, controller 11, PC 12, and position detection sensor 13 shown in Fig. 1, the configuration of the display system is not limited to this configuration. As shown in Fig. 19, the functions of the PC 12 and the position detection sensor 13 may be incorporated in the HMD 10, and the display system may be composed of only the HMD 10 and the controller 11.

[0087] When detecting the positions of the HMD 10 and the controller 11, the HMD 10 may be equipped with an imaging device (camera) in order to detect the positions of the HMD 10 and the controller 11. The HMD 10 and the controller 11 may be equipped with a sensor for detecting the inclination of the HMD 10 and the controller 11.

[0088] The position and inclination of the HMD 10 can be calculated from the size, direction, inclination, etc. of an image of a marker or the like placed at a reference position captured by a camera. The position and inclination of the controller 11 can also be calculated from an image captured by a camera. Note that if the HMD 10 is equipped with a camera, it is not necessary to use the controller 11, and a marker can be placed on the hand, an image including the marker can be captured using the camera of the HMD 10, and the position and inclination of the hand can be calculated from the captured image.

[0089] Fig. 20 is a diagram showing the hardware configuration of the HMD 10 and the controller 11 when the configuration shown in Fig. 19 is adopted. The HMD 10 includes an external I / F 20, a CPU 21, a display 22, a memory 23, a HDD 24, and a microphone 26, similar to the configuration shown in Fig. 2. The HMD 10 further includes a sensor 27 and a camera 28.

[0090] The sensor 27 is a gyro sensor that detects angular velocity, an acceleration sensor that detects acceleration, or the like, and detects the tilt and orientation of the HMD 10. The camera 28 recognizes markers in the three-dimensional virtual space and attached to the controller 11, and detects the positions and tilts of the HMD 10 and the controller 11.

[0091] 2, the controller 11 also includes an operation I / F 30 and an external I / F 31. The controller 11 includes a sensor 33 instead of the light source 32. The sensor 33 is a gyro sensor, an acceleration sensor, or the like, like the sensor 27 of the HMD 10, and detects the inclination and orientation of the controller 11.

[0092] The tilt information of the HMD 10 and the controller 11 may use information from the sensor 27 and the sensor 33, or may use information detected by the camera 28 of the HMD 10. The position and tilt of the HMD 10 and the controller 11 may be detected from an image captured by the camera 28 of the HMD 10. Furthermore, the information detected by the sensor 27 and the camera 28 of the HMD 10 and the information detected by the controller 11 may be compiled in the HMD 10, and the position and tilt of the HMD 10 and the controller 11 may be detected based on the obtained information.

[0093] Instead of using controller 11, the shape and position of the hand may be detected by camera 28, and the inclination may be detected from the shape of the hand.

[0094] 20, the HMD 10 may include a sensor 27 and a camera 28, the controller 11 may include a sensor 33, and may further include the position detection sensor 13 shown in FIG. 1 as an external sensor. In this case, it may be possible to select and switch between using the sensor and camera of the HMD 10 and the controller 11, or using the external position detection sensor 13. For example, when it is desired to use highly accurate information in position information or tilt information, the position detection sensor 13 can be used, and when responsiveness is required, the sensors 27, 33 and the camera 28 can be used.

[0095] By providing the device, system, method, and program of the present invention, it is not necessary to check the hand when placing a user object, and it is not necessary to use both hands to place the user object, so that usability in a three-dimensional virtual space can be improved. In addition, since the position of the shoulder is set as the starting point of the arm and the position of the user object is determined, the accuracy of the position can be improved. Furthermore, by determining the position of the shoulder as a reference, it is possible to handle the user object with a feeling close to human intuition.

[0096] The present invention has been described so far with respect to the above-mentioned embodiments as an information processing device, a display system, an information processing method, and a program. However, the present invention is not limited to the above-mentioned embodiments, and can be modified within the scope of what a person skilled in the art can conceive, such as other embodiments, additions, modifications, deletions, etc. Furthermore, any embodiment is included in the scope of the present invention as long as it provides the functions and effects of the present invention. [Explanation of symbols]

[0097] 10...HMD, 11...controller, 12...PC, 13...position detection sensor, 14...server, 15...network, 20...external I / F, 21...CPU, 22...display, 23...memory, 24...HDD, 25...light source, 26...microphone, 30...operation I / F, 31...external I / F, 32...light source, 40...CPU, 41...ROM, 42...RAM, 43...HDD, 44...external I / F, 45...input / output I / F, 46...input device, 47...display device, 50...acquisition unit, 51...first generation unit, 52...second generation unit, 60...whiteboard, 61...hand, 62...assistant object, 63...marker, 64...blackboard eraser, 65...laser object, 66...object, 67...user [Prior art documents] [Patent documents]

[0098] [Patent Document 1] JP 2018-190395 A

Claims

1. An information processing device that generates an image based on virtual space data representing a three-dimensional virtual space, an acquisition means for acquiring position information and tilt information of the display means and position information of the operation means or the hand from a detection means for detecting a position of a display means worn by a user, a tilt of the display means with respect to a reference direction, and a position of an operation means operated by the user or a hand of the user; a first generation means for generating a user object in the three-dimensional virtual space based on the acquired position information of the display means and the position information of the operation means or the hand; Including, The first generation means estimates a position of a part of the user that is between the display means and the operating means or the hand based on position information of the display means and position information of the operating means or the hand, and generates the user object based on the position information of the estimated part.

2. The information processing device according to claim 1 , wherein the part is a shoulder.

3. 3. The information processing apparatus according to claim 1, further comprising: second generating means for generating an input object for implementing user input in said three-dimensional virtual space in response to an operation on said user object.

4. 3. The information processing device according to claim 1, further comprising a selection processing means for executing a process of selecting an object arranged in the three-dimensional virtual space by the user object based on the position information of the body part and the position information of the operation means or the hand.

5. a calculation means for calculating a distance between the position of the part and the position of the operation means or the hand, The information processing apparatus according to claim 4 , wherein the selection processing means changes a length of the user object in accordance with the distance calculated by the calculation means, and selects the object placed in the three-dimensional virtual space.

6. The information processing apparatus according to claim 5 , wherein the selection processing means changes the length of the user object such that the rate at which the length of the user object changes increases as the distance increases.

7. The information processing device according to any one of claims 4 to 6, further comprising a movement processing means for executing a process of moving the selected object to a position specified by the user object based on the position information of the part and the position information of the operation means or the hand.

8. 8. The information processing device according to claim 7, further comprising an input receiving means for receiving user input based on position information of the display means, position information of the operation means or the hand, and position information of the selected object in the three-dimensional virtual space after movement processing by the movement processing means.

9. A display system including an information processing device, a display means that is attached to a user within the information processing device or separately from the information processing device and displays an image of a user object generated by the information processing device and an image of a tilt direction of the display means in a three-dimensional virtual space; The information processing device, an acquisition means for acquiring position information and tilt information of the display means and position information of the operation means or the hand from a detection means for detecting a position of the display means worn by the user, a tilt of the display means with respect to a reference direction, and a position of the operation means operated by the user or the hand of the user; a generating means for generating a user object in the three-dimensional virtual space based on the acquired position information of the display means and the position information of the operation means or the hand; Including, The generation means estimates the position of a part of the user that is between the display means and the operating means or the hand based on position information of the display means and position information of the operating means or the hand, and generates the user object based on the position information of the estimated part.

10. The display system according to claim 9 , further comprising a detection means for detecting the position of the display means and the inclination of the display means relative to a reference direction, and the position of an operating means operated by the user or the position of the user's hand, within the information processing device or separately from the information processing device.

11. A method executed by an information processing device that generates an image based on virtual space data representing a three-dimensional virtual space, acquiring position information and tilt information of the display means and position information of the operation means or the hand from a detection means for detecting a position of a display means worn by a user, a tilt of the display means with respect to a reference direction, and a position of an operation means operated by the user or a hand of the user; generating a user object in the three-dimensional virtual space based on the acquired position information of the display means and the position information of the operation means or the hand; Including, A method in which, in the step of generating the user object, a position of a part of the user that is between the display means and the operating means or the hand is estimated based on position information of the display means and position information of the operating means or the hand, and the user object is generated based on the estimated position information of the part.

12. A program for causing a computer to execute a process for generating an image based on virtual space data representing a three-dimensional virtual space, comprising: acquiring position information and tilt information of the display means and position information of the operation means or the hand from a detection means for detecting a position of a display means worn by a user, a tilt of the display means with respect to a reference direction, and a position of an operation means operated by the user or a hand of the user; generating a user object in the three-dimensional virtual space based on the acquired position information of the display means and the position information of the operation means or the hand; Run the command, In the step of generating the user object, a program estimates the position of a part of the user that is between the display means and the operating means or the hand based on position information of the display means and position information of the operating means or the hand, and generates the user object based on the estimated position information of the part.

Images