Information processing device, information processing system, information processing method, and program

Abstract

To provide a user interface with high convenience as a user interface for a VR system, an MR system, or the like.SOLUTION: An information processing device includes: acquisition means for acquiring information about the position or orientation of a controller; setting means for setting a criterion for displaying a virtual object based on a predetermined area and the information acquired by the acquisition means at a predetermined time; and control means for controlling display means to display the virtual object based on the criterion set by the setting means and the information acquired by the acquisition means.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] In conventional cross reality (XR) systems that allow users to experience virtual reality, hand controllers are used to convert hand movements into actions in the virtual space when controlling the display of a head mounted display (HMD). HMDs are glasses-type devices equipped with a small display that the user wears on the head. When operating a user interface (UI) through an HMD, hand controllers are used to convert hand movements into actions in the virtual space. Based on the position and orientation of the hand controller, CG (Computer Graphics) such as virtual light rays called rays and virtual pointers are displayed on the screen, allowing the user to understand the position to point to and freely operate virtual objects.

[0003] For example, Patent Document 1 discloses a technology related to a hand controller that can detect the position and posture of the hand by emitting a plurality of infrared lights (IR lights) from the hand controller and receiving the infrared lights with a camera mounted on the HMD. Patent Document 2 discloses a technology related to a device that reflects the position and posture of a user in a virtual space by comparing the body parts of the user captured in an image captured by a camera mounted on the HMD with a skeletal model stored in a memory. By using such a technology, it is possible to display a ray or a pointer, which is a CG (Computer Graphics) that indicates what the user is pointing to, according to the movement of the controller. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2020-519992 [Patent Document 2] Patent Publication No. 2021-60627 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the above-mentioned conventional technology, when an operation is performed on a virtual object using a ray or a pointer, the direction in which the ray or the pointer comes out from the controller is predetermined. Depending on the virtual object to be pointed to, there is a problem that the arm may get tired if the arm is held out in front of the user for a long time. Therefore, the present invention aims to provide a user interface for a VR system, an MR system, or the like, which is highly convenient and allows the user to point to a virtual object by displaying a ray or a pointer in a desired direction. [Means for solving the problem]

[0006] One aspect of the present invention is an information processing device comprising: an acquisition means for acquiring information relating to a position or attitude of a controller; a setting means for setting a criterion for displaying a virtual object based on a predetermined area and the information acquired by the acquisition means at a predetermined time; and a control means for controlling a display means to display the virtual object based on the criterion set by the setting means and the information acquired by the acquisition means. Effect of the Invention

[0007] According to the present invention, it is possible to provide a highly convenient user interface as a user interface for a VR system or an MR system, which is capable of displaying a ray or a pointer in a direction desired by the user to point to a virtual object. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an information processing system according to a first embodiment. [Diagram 2] FIG. 2 is a diagram showing the internal configuration of an HMD and the like in the first embodiment. [Diagram 3] 11 is a flowchart illustrating a process for determining an output direction of a pointer according to the first embodiment. [Figure 4] 4 is a diagram for explaining a predetermined output direction of a pointer in the first embodiment. FIG. [Diagram 5] 5A to 5C are diagrams illustrating the pointer output direction before and after calibration in the first embodiment. [Figure 6] FIG. 13 is a diagram illustrating the contents displayed during calibration in the first embodiment. [Figure 7] FIG. 2 is a diagram for explaining object display in the first embodiment. [Figure 8] 13 is a flowchart illustrating a process for determining the output direction of a pointer in the second embodiment. [Figure 9] 13 is a diagram illustrating an object display and a determination region in the second embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0010] (Embodiment 1) An information processing system 1 according to the first embodiment will be described with reference to Fig. 1. The information processing system 1 includes an HMD 100, a PC (personal computer) 110, and a controller 120.

[0011] The HMD 100 is a head-mounted display device (electronic device) that can be worn on the head of a user. The HMD 100 displays a composite image that combines a captured image of the area in front of the user captured by the HMD 100 with content such as CG in a form that corresponds to the posture of the HMD 100.

[0012] The PC 110 controls the HMD 100. The PC 110 is connected to the HMD 100 by a wired connection such as a USB cable, or wirelessly by a connection such as Bluetooth (registered trademark) or Wi-Fi (Wireless Fidelity) (registered trademark). The PC 110 generates a composite image by combining a captured image with CG, and transmits the composite image to the HMD 100. Note that, although a PC will be described here as an example of an information processing device, the information processing device is not limited to this. For example, the information processing device may be a smartphone or a tablet terminal, and each component of the PC 110 may be included in the HMD 100.

[0013] The controller 120 performs various controls of the HMD 100. If the PC 110 is in a specific control mode, when a user operates the controller 120, the HMD 100 is controlled in response to the user's operation. As shown in FIG. 1, the controller 120 may be in the shape of a finger ring that can be worn and supported by the user, or in the shape of a handheld device that can be held in the hand. The controller 120 also has physical buttons for performing decision and selection operations on the display. The controller 120 communicates wirelessly with the PC 110 via Bluetooth.

[0014] By moving the controller 120, the user can change the indicated position on the display according to the movement of the controller. The indicated position may be represented by a point, or may be represented by a virtual ray (ray) connecting the point of the indicated position and the controller with a straight line (line segment) or a dotted line. By pressing a physical button, a menu determination operation or a selection operation can be performed. Although the shape of the controller 120 is assumed to be a ring type or a handheld type, it is not limited thereto as long as it can be supported by a finger, a hand, or an arm. Also, although the button is a physical button, it may be operable like a track pad, a touch panel, a wheel, or a track ball, and in addition to pressing the button, a slide operation, a flick operation, or a touch-on operation may be used.

[0015] Note that the controller may be worn on at least one of a finger, a hand, or an arm.

[0016] <Internal Configuration of HMD> Referring to FIG. 2, the internal configuration of the HMD 100 will be described. The HMD 100 includes an HMD control unit 201, an imaging unit 202, an image display unit 203, an attitude sensor unit 204, a non-volatile memory 205, and a working memory 206.

[0017] The HMD control unit 201 is a CPU that controls each component of the HMD 100. When the HMD control unit 201 acquires a composite image (an image obtained by synthesizing an imaging image captured by the imaging unit 202 of the space in front of the user and a CG) from the PC 110, the HMD control unit 201 displays the composite image on the image display unit 203. Note that instead of the HMD control unit 201 controlling the entire device, a plurality of hardware components may share the processing to control the entire device.

[0018] The imaging unit 202 includes two cameras (imaging devices). The two cameras are disposed near the positions of the left and right eyes of the user when wearing the HMD 100 in order to capture video or images of a space similar to the space the user normally sees. Images of a subject (the range in front of the user) captured by the two cameras are output to the PC 110 and the control unit 201. The two cameras in the imaging unit 202 can also acquire information on the distance from the two cameras to the subject as distance information by measuring distances with a stereo camera. The imaging unit 202 may capture and output video.

[0019] The image display unit 203 displays the composite image. The image display unit 203 has a liquid crystal panel or an organic EL panel. When the user wears the HMD 100, an organic EL panel is disposed in front of each eye of the user. Note that a device using a semi-transmissive half mirror can also be used for the image display unit 203. In this case, for example, the image display unit 203 may display an image by using a technology generally called AR (Augmented Reality) so that the CG is superimposed directly on the real space seen through the half mirror. Also, the image display unit 203 may display an image of a complete virtual space without using a captured image by using a technology generally called VR (Virtual Reality).

[0020] The attitude sensor unit 204 acquires attitude (and position) information of the HMD 100. Then, the attitude sensor unit 204 acquires attitude information of the user (user wearing the HMD 100) corresponding to the attitude (and position) of the HMD 100. The attitude sensor unit 204 has an inertial measurement unit (IMU) composed of an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The attitude sensor unit 204 is used when acquiring information on the user's attitude (attitude information), and the HMD control unit 201 outputs the information on the user's attitude (attitude information) to the PC 110.

[0021] The HMD control unit 201 estimates the position or posture of each joint point of the user's hand and fingers from the two camera images obtained by the imaging unit 202. The joint points include points that are characteristic of parts such as finger joints, fingertips, the back of the hand (palm), and the arm. Each joint point indicates a coordinate position, and the posture can be estimated from information on a plurality of joint points. For example, a known object recognition or pose estimation method of machine learning using a convolutional neural network can be used as a method for estimating the position or posture of the hand and each joint point of the hand. In addition, the position information of each joint point of the hand in the depth direction can be obtained by calculating the distance from the imaging unit 202 to each joint point by triangulation by stereo matching using the two camera images obtained by the imaging unit 202. The estimated coordinate information of each joint point of the hand is output from the control unit 201 to the PC 110. In addition, the HMD control unit 201 may estimate the position and posture of each joint point of the user's hand and fingers.

[0022] The non-volatile memory 205 is an electrically erasable and recordable non-volatile memory, and stores programs executed by the control unit 211, which will be described later.

[0023] The volatile memory 206 is used as a buffer memory for temporarily storing image data captured by the imaging unit 202, an image display memory for the image display unit 203, a working area for the control unit 201, and the like.

[0024] The gaze imaging unit 207 is a camera that acquires an image for detecting the gaze of the user, and is attached inside the HMD to capture an image of the user's eyes when the user wears the HMD 100. The image of the subject (user's eyes) captured by the camera is output to the control unit 211 of the PC 110 via the HMD control unit 201. The control unit 211 detects the gaze of the user wearing the HMD 100 from the image captured by the gaze imaging unit 207, and identifies the location on the image display unit 203 that the user is gazing at.

[0025] <Internal configuration of the controller> Referring to FIG. 2, the internal configuration of the controller 120 will be described. The controller 120 includes a controller control unit 221, an operation unit 222, a communication unit 223, and a controller attitude sensor unit 224.

[0026] The controller control unit 221 is a CPU that controls each component of the controller 120. Note that instead of the controller control unit 221 controlling the entire device, a plurality of hardware components may share the processing to control the entire device.

[0027] The operation unit 222 includes buttons. The operation unit 222 detects whether a button has been pressed and transmits detection information to the PC 110 via the communication unit 223.

[0028] The communication unit 223 performs wireless communication with the PC 110 via Bluetooth.

[0029] The controller attitude sensor unit 224 has an inertial measurement unit (IMU) composed of an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The inertial measurement unit detects changes in the position or attitude of the controller 120. The detected position and attitude change information is communicated from the communication unit 223 to the PC 110 via the controller control unit 221. Note that the inertial measurement unit may detect changes in the position and attitude of the controller 120.

[0030] <Internal Configuration of the PC> Referring to FIG. 2, the internal configuration of the PC 110 will be described. The PC 110 includes a control unit 211, a non-volatile memory 212, a volatile memory 213, a communication unit 214, and a recording medium 215.

[0031] The control unit 211 is a CPU that controls each unit of the PC 110 according to an input signal or a program described later. Instead of the control unit 211 controlling the entire device, the entire device may be controlled by a plurality of hardware devices sharing the processing. The control unit 211 receives an image (captured image) acquired by the imaging unit 202 and orientation information acquired by the orientation sensor unit 204 from the HMD 100. The control unit 211 performs image processing on the captured image to cancel aberrations in the optical system of the imaging unit 202 and the optical system of the image display unit 203. Then, the control unit 211 composites the captured image with any CG to generate a composite image. The control unit 211 transmits the composite image to the HMD control unit 201 in the HMD 100. The control unit 211 also acquires coordinates in the real space of the HMD 100 based on the captured image acquired from the imaging unit 202. The control unit 211 may acquire the coordinates of the HMD 100 in the real space based on the captured image acquired from the imaging unit 202 and the information acquired by the HMD 100 (distance information and orientation information).

[0032] The control unit 211 controls the position, orientation, and size of the CG in the composite image based on the information (distance information and orientation information) acquired by the HMD 100. For example, when placing a virtual object represented by the CG near a specific object existing in real space in the space represented by the composite image, the control unit 211 makes the virtual object (CG) larger as the distance between the specific object and the imaging unit 202 is closer. By controlling the position, orientation, and size of the CG in this manner, the control unit 211 can generate a composite image in which a CG object that is not placed in real space appears as if it were placed in real space.

[0033] In addition, the control unit 211 receives information estimated by the control unit 201 of the HMD 100. The received information is temporarily stored in the volatile memory 213.

[0034] Furthermore, in the control unit 211, the communication unit 214 receives information about a change in the position or attitude of the controller 120 from the communication unit 223 of the controller 120. The control unit 211 superimposes and displays an indication position corresponding to the change information about the position or attitude of the controller 120 on the combined image. Note that the communication unit 214 of the control unit 211 may receive the information about the change in the position and attitude of the controller 120 from the communication unit 223 of the controller 120. Note that the control unit 211 may superimpose and display an indication position corresponding to the change information about the position and attitude of the controller 120 on the combined image.

[0035] The non-volatile memory 212 is an electrically erasable and recordable non-volatile memory, and stores information such as programs to be described later and CG executed by the control unit 211. The control unit 211 can switch the CG read from the non-volatile memory 212 (i.e., the CG used to generate a composite image).

[0036] The volatile memory 213 is used as a buffer memory for temporarily storing image data captured by the imaging unit 202 and time-series information on the estimated coordinate positions of each joint point of the hand, as an image display memory for the image display unit 203, a working area for the control unit 211, etc.

[0037] The wrist joints may be estimated in the PC 110. In this case, after the captured image is output from the imaging unit 202 to the PC 110, the control unit 211 of the PC 110 estimates the position or posture of each joint point of the hand, processes the image using the information, and outputs the image to the HMD 100. Note that the control unit 211 may estimate the position and posture of each joint point of the hand, processes the image using the information, and outputs the image to the HMD 100.

[0038] <About Ray Display> Referring to FIG. 4, pointer manipulation using the controller 120 will be described.

[0039] 4 is the ring-type controller described above, and a three-dimensional controller coordinate system 401 (X, Y, Z) is defined based on the position and orientation of the controller. The HMD control unit 211 receives sensor data acquired by the controller orientation sensor unit 224 from the controller 120 via the communication unit 214 and the communication unit 223, and calculates the position and orientation of the controller 120 based on the sensor data. A known technique can be used to calculate the position and orientation of the controller 120.

[0040] The position and orientation of the controller 120 are also identified by a method such as identifying the position and orientation of the controller by image recognition. Known techniques such as machine learning can be used as the method of identifying the position and orientation of the controller by image recognition.

[0041] The pointer is set to face a predetermined direction relative to the controller 120. That is, the pointer is set to face a predetermined direction from the controller based on a predetermined standard. In the initial state before the process of adjusting the pointer direction (hereinafter, calibration) is executed, the direction of the pointer is set to face forward relative to the controller 120, for example, as shown at 402. By doing as described above, the pointer can be operated so that it moves in accordance with the movement of the hand wearing the controller. In the first embodiment, in order to make it easier for the user to recognize the position of the pointer, the image processing unit 103 synthesizes, as the pointer, a CG image that expresses a ray (hereinafter, ray) extending from the controller 120 in the direction of 402.

[0042] <About the calibration process> The calibration process will be described with reference to the flowchart of Fig. 3. Note that it is assumed that the operation mode of the HMD 100 is set to the calibration mode before the start of this flowchart.

[0043] In step S301, the control unit 211 acquires the position and orientation of the HMD 100. The control unit 211 performs control so as to estimate the position and orientation of the HMD from an image captured by the imaging unit 202, and acquires the estimated position information and orientation information.

[0044] In step S302, the control unit 211 acquires the position and orientation of the controller. The control unit 211 acquires sensor information acquired by the controller orientation sensor unit 224 via wireless communication by the communication unit 214 and the communication unit 223, and acquires the position and orientation of the controller 120. Furthermore, the control unit 211 detects the controller 120 from an image captured by the imaging unit 202, and acquires the position and orientation by converting it into a spatial coordinate system in the virtual space.

[0045] In step S303, the control unit 211 detects the line of sight and acquires the position in real space where the user is gazing. The control unit 211 may acquire the area where the user is gazing. The control unit 211 captures an image of the eye of the user wearing the HMD by the line of sight imaging unit 207. Furthermore, the control unit 211 identifies the gaze point in the virtual space by detecting the gaze point on the image display unit 203 from the captured image. Any known technology may be used as a method of capturing an image of the user's eye and detecting the gaze.

[0046] In step S304, the control unit 211 determines whether or not the calibration is completed. The control unit 211 compares the position information and posture information of the controller acquired in step S302 and the gaze information acquired in step S303 with previously acquired information, and obtains the amount of change during a predetermined time. If the amount of change is equal to or less than the predetermined amount, the control unit 211 determines that the calibration is completed, and proceeds to the next step S304. If the amount of change is not equal to or less than the predetermined amount, the control unit 211 proceeds to step S301. The determination of the completion of the calibration may be based on whether or not the operation unit 222 of the controller 120 has been operated. The determination of the completion of the calibration may be based on the blinking of the user. Here, when the determination of the completion of the calibration is made by a method other than the operation of the buttons of the controller, there is no deviation in the position and posture of the controller due to the button operation, and therefore a unique effect is achieved in that the output direction of the ray desired by the user can be set.

[0047] In step S305, the control unit 211 determines the output direction of the ray from the position information and orientation information of the controller 120 acquired in step S302, so that the ray is output toward the line of sight position acquired in step S303.

[0048] In step S306, the control unit 211 calculates the amount of correction. The control unit 211 determines the pointer output direction and calculates the amount of correction from the information acquired in steps S301, S302, and S303, and then completes the process.

[0049] With reference to FIG. 5(a) and FIG. 5(b), the processing in step S305 and step S306 in FIG. 3 will be described. A new controller coordinate system 502 (X', Y', Z') is set so as to conform to the ray output direction determined in step S305 for the controller coordinate system 501 (X, Y, Z) in the initial state before calibration shown in FIG. 5(a). A transformation matrix from the coordinate system 501 to 502 is calculated as a correction amount and recorded in the memory 108. Note that the correction amount may be a correction value expressed by an angle by which the coordinate system is rotated, or a correction value expressed by a length by which the coordinate system is translated. In FIG. 5(a) and FIG. 5(b), the origin is drawn as the same position before and after calibration, but the origin after calibration may be adjusted to a different position according to the user's preference. In other words, the starting point of the ray may be adjusted to a different position.

[0050] In the flowchart of Fig. 3, the gaze information is acquired to display the gaze area, and the direction of the ray is determined assuming that the position and attitude of the controller at a certain point in time are pointing to that area. However, instead of being limited to the gaze area, the ray may be displayed by setting an area that the controller is pointing to based on the feature points of a real object, or the ray may be displayed as if the controller is pointing to an area of ​​a virtual object. In addition, when a gaze pointer indicating the location of the gaze is displayed, the gaze area may be the same area as the gaze pointer, may be an area having a larger area than the gaze pointer, or may be an area having a smaller area than the gaze pointer.

[0051] <Ray display during correction> The display of the image display unit 203 during calibration will be described with reference to FIG. 6(a) and FIG. 6(b). FIG. 6(a) and FIG. 6(b) are diagrams showing a scene in which the position on the screen to be adjusted by calibration is displayed so that the user can understand it. The display screen 601 corresponds to the image display unit 203. The area 602 on the screen that the user is gazing at is represented by a virtual object, and the ray output is adjusted so as to be directed in this direction. On the display screen 601, a ray is displayed so as to be directed from the controller 120 worn on the hand at 603 to the area 602, making it easier for the user to recognize the direction of the pointer. When the calibration is not completed, a ray is always displayed so as to be directed from the controller 120 to the area 602. For example, when the user's hand is at the position 603 as shown in FIG. 6(a), a ray 604 is displayed so as to be directed from the controller 120 worn at the position 603 to the area 602. 6(b), when the user's hand is at position 613, a ray 614 is displayed pointing from the controller 120 worn at position 613 to the area 602. In this way, the position to which the ray is to be adjusted is shown to the user, making calibration easier.

[0052] In step S303, the position of the user's gaze is detected, and the ray is adjusted so as to face the direction of the gaze. However, in order to prevent the user from being confused about the pointing position, the user may indicate a predetermined position on the screen and specify the posture of pointing to that position, that is, the position and posture of the controller that indicates the predetermined position. For example, as shown in FIG. 7, a CG object indicated by 702 is displayed on a display screen 701 corresponding to the image display unit 203, and the user is notified to assume a posture of pointing to 702. In this case, in determining whether the calibration is complete in step S304, the calibration is determined to be complete when the gaze position is on the CG object 702 for a predetermined time and the amount of change in the position information and the posture information of the controller 120 is equal to or less than a predetermined amount. Note that the predetermined time may be set in any manner, for example, 5 seconds. For example, in the case of 5 seconds, the calibration is determined to be complete when the gaze position is on the CG object 702 for a predetermined time and the amount of change in the position information and the posture information of the controller 120 is equal to or less than a predetermined amount for 5 seconds. Note that the predetermined time may be determined in advance, or may be set by the user. Although the region 602 is a gaze region, i.e., a region based on the line of sight, it may be any region, for example, a region of a virtual object different from a virtual object that responds to the movement of a controller, such as a ray or a pointer. Note that a region based on a real object may be set instead of the region 602. For example, a region that is characteristic of a real object may be determined and selected. Note that the calibration completion may be determined at the time when a button input of the controller is accepted, or a specific operation may be prompted to be performed with the controller.

[0053] When the operation mode is set to the calibration mode, the color of a virtual object such as a ray or a pointer may be changed so that the user can know that the operation mode is set to the calibration mode. For example, the color of the virtual object may be changed until the processing of the flowchart in Fig. 3 is completed. That is, the color of the virtual object may be changed while the virtual object is displayed so as to indicate the area 602 even when the controller is moved, as shown in Fig. 6(a) and Fig. 6(b).

[0054] <About the controller coordinate system> In the first embodiment, a ring-type controller is described as an example, but a grip-type, i.e., handheld, controller may be used. In this case, a coordinate system for expressing the position of the controller may be set near the grip position, and a reference coordinate system for a pointer (hereinafter, pointer coordinate system) may be set near the tip of the controller for ray operation, and multiple coordinate systems may be provided in the controller. In this case, the reference coordinates for ray operation are corrected in the calibration. In an environment in which an application can be installed and executed on the PC 110, the position and orientation information of the controller 120 transmitted from the control unit 211 to the application is a pointer coordinate system corrected based on the correction information stored in the working memory 213. In this way, the application can execute ray processing without being aware of calibration.

[0055] In the first embodiment, the method of adjusting the ray to match the gaze position has been described, but the output direction of the ray may be adjusted by a user operation using OTP or the like.

[0056] Although the position and the posture of the HMD and the controller are acquired, respectively, it is acceptable to acquire only the position or only the posture. When only the position is acquired and the process described in the first embodiment is performed, even if the posture of the controller is changed, the virtual object does not change, and the operation is performed by only translation. For example, when an area that can be designated while standing is to be designated while sitting, it may be difficult to designate depending on the area to be designated because the heights are different. In such a case, by changing the output direction of the ray, it is possible to designate any place on the screen within the range of the movable range of the user's hand or arm without difficulty. Also, when only the posture is acquired and the process described in the first embodiment is performed, even if the controller is translated, the ray is output with a predetermined position as the starting point, and the direction of the ray changes by changing the posture of the controller. When it is desired to designate the front direction of the user, the controller must be kept facing the front direction, which may cause fatigue. In such a case, by changing the output direction of the ray, it is possible to designate the front direction without necessarily keeping the controller facing the front direction. For example, it is possible to change the output direction of the ray so that the ray is output in the front direction of the user while the controller is facing the ground.

[0057] As described above, in the first embodiment, the ray output direction is calibrated and corrected according to the user's gaze position. This makes it possible to easily realize ray operations suited to the user, and to provide a highly convenient user interface.

[0058] (Embodiment 2) In the second embodiment, a system that automatically executes the calibration mode and automatically aligns the rays will be described.

[0059] Although the calibration is performed as shown in the first embodiment, after a while, the surrounding situation may change, and it may become difficult to operate the device in the posture in which the calibration was performed previously. In the second embodiment, in such a situation, the device automatically judges the situation and performs the calibration. The basic configuration of the second embodiment is the same as that of the first embodiment, so the description of the configuration is omitted.

[0060] The calibration process of the second embodiment will be described with reference to the flowchart of FIG. 8. Note that it is assumed that calibration has already been performed once before the start of the flowchart. Also, with reference to FIG. 9(a) and FIG. 9(b), a scene in which the calibration mode in the second embodiment is automatically executed will be described. FIG. 9(a) is a diagram showing a ray after calibration has already been performed. It is assumed that a virtual object 902 that the user wishes to point to is present within the screen of a display screen 901 corresponding to the image display unit 203. The user is wearing a controller 120 on his hand 903, and a ray is displayed extending from the controller. This is a scene in which the ray points to a pointing position 905.

[0061] In FIG. 8, in step S801, the control unit 211 checks the selection status of the virtual object 902. The control unit 211 checks the positional relationship between the position of the virtual object 902 and the position 905 pointed to by the user with a ray. If there is little change in the surroundings from the previous calibration situation, it is assumed that the ray points to the periphery of the virtual object 902 when the user takes a posture to select the virtual object 902 with the controller 120. However, there may be a change in the surrounding situation of the user from the previous calibration situation, making it difficult to take a posture to point to the virtual object 902. In such a case, the ray points to a position away from the virtual object 902. For this reason, in step S801, if the control unit 211 has pointed to a position away from the virtual object 902, it determines that the surrounding situation of the user has changed and that re-calibration is necessary, and the process proceeds to step S802. To determine whether a position away from the virtual object 902 is being pointed to, for example, an area is defined like the area 906 in Fig. 9(a), and if the ray pointing position 905 is outside the area 906, it is determined that re-calibration is necessary. In step S801, the control unit 211 determines that calibration is not necessary if the ray pointing position 905 is inside the area 906, and completes the process without performing calibration.

[0062] In step S802, the control unit 211 starts the calibration mode, and the process proceeds to step S803. Steps S803 to S808 correspond to steps S301 to S306 in the first embodiment, and therefore a description thereof will be omitted.

[0063] In step S809, the control unit 211 ends the calibration mode, and completes the process.

[0064] Since the second embodiment performs calibration automatically, in order to avoid confusing the user during calibration, the gaze position and ray display during calibration described in the first embodiment may not be performed, and the corrected ray may be displayed after the calibration is completed.

[0065] Furthermore, since automatic calibration may cause inconvenience to the user, the user may be notified after automatic calibration so that the user can choose whether or not to apply the calibration results.

[0066] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0067] (Other embodiments) The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-mentioned embodiments are supplied to a system or device via a network or various storage media, and the computer (or control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0068] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0069] Each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0070] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0071] [Configuration 1] An acquisition means for acquiring information regarding a position or an attitude of a controller; a setting means for setting a criterion for displaying a virtual object based on a predetermined area and the information acquired by the acquisition means at a predetermined time point; a control means for controlling a display means to display the virtual object based on the criteria set by the setting means and the information acquired by the acquisition means; 13. An information processing device comprising:

[0072] [Configuration 2] An acquisition means for acquiring information regarding a position or an attitude of a controller; a control means for controlling a display means to display the virtual object based on a first criterion, which is a predetermined criterion for displaying the virtual object, and the information acquired by the acquisition means; a setting means for setting a second criterion, which is different from the first criterion, based on a predetermined area and the information acquired by the acquisition means at a predetermined time point; when the second criterion is set by the setting means, the control means controls the display means to display the virtual object based on the second criterion and the information acquired by the acquisition means. 23. An information processing apparatus comprising:

[0073] [Configuration 3] the acquiring means acquires information regarding a position and an attitude of a controller; 3. The information processing device according to configuration 1 or 2.

[0074] [Configuration 4] the criterion set by the setting means is a criterion regarding a relationship between a position or an orientation of the controller and the predetermined area; 4. The information processing device according to any one of configurations 1 to 3.

[0075] [Configuration 5] the reference set by the setting means is a reference regarding a direction indicated by the controller; 5. The information processing device according to any one of configurations 1 to 4.

[0076] [Configuration 6] The virtual object is a ray extending from a controller or the hand of the user, The criterion set by the setting means is a criterion related to the starting point. 6. The information processing device according to any one of configurations 1 to 5.

[0077] [Configuration 7] The virtual object is a pointer, the criterion set by the setting means is a criterion relating to a position or an orientation of the controller and a position of the pointer; 6. The information processing device according to any one of configurations 1 to 5.

[0078] [Configuration 8] the reference set by the setting means is a reference with respect to a coordinate system having the controller as its origin; 6. The information processing device according to any one of configurations 1 to 5.

[0079] [Configuration 9] the setting means sets the second reference by setting a correction value related to an angle for rotating a coordinate system having an origin at the controller serving as the first reference. 9. The information processing device according to configuration 8.

[0080] [Configuration 10] the setting means sets the second reference by setting a correction value relating to a distance by which a coordinate system having an origin at the controller serving as the first reference is translated; 9. The information processing device according to configuration 8.

[0081] [Configuration 11] Further comprising a second acquisition means for acquiring information on the user's line of sight. 11. The information processing device according to any one of configurations 1 to 10.

[0082] [Configuration 12] The line of sight information is information indicating a position or a range, 12. The information processing device according to configuration 11.

[0083] [Configuration 13] The predetermined area is an area based on information of the user's line of sight. 13. The information processing device according to configuration 11 or 12.

[0084] [Configuration 14] The predetermined time point is a time point at which an input based on the user's gaze is received. 14. The information processing device according to any one of configurations 11 to 13.

[0085] [Configuration 15] the predetermined time point is a time point when an input is received via a button of the controller; 14. The information processing device according to any one of configurations 1 to 13.

[0086] [Configuration 16] The predetermined time point is a time point when the virtual object is in the predetermined area for a predetermined period of time. 14. The information processing device according to any one of configurations 1 to 13.

[0087] [Configuration 17] The predetermined area is an area including points that are characteristic of a real object. 17. The information processing device according to any one of configurations 1 to 12 and 14 to 16.

[0088] [Configuration 18] the predetermined area is an area represented by a second virtual object different from the virtual object; 17. The information processing device according to any one of configurations 1 to 12 and 14 to 16.

[0089] [Configuration 19] a recording means for recording the criteria set by the setting means; A selection means for selecting one of the criteria recorded in the recording means, The setting means sets the criteria selected by the selection means. 19. The information processing device according to any one of configurations 1 to 18.

[0090] [method] An acquisition step of acquiring information regarding a position or an attitude of a controller; a setting step of setting a criterion for displaying a virtual object based on a predetermined area and the information acquired by the acquisition means at a predetermined time point; a control step of controlling a display means to display the virtual object based on the criteria set by the setting means and the information acquired by the acquisition means; 13. A method for controlling an information processing apparatus comprising the steps of:

[0091] [program] 20. A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 19.

[0092] [system] A controller; an acquisition device for acquiring information regarding a position or an attitude of the controller; A setting device that sets a criterion for displaying a virtual object based on a predetermined area and the information acquired by the acquisition device at a predetermined time point; a control device that controls a display device to display the virtual object based on the criteria set by the setting device and the information acquired by the acquisition device; An information processing system comprising:

Claims

1. An acquisition means for acquiring information regarding the position or orientation of the controller, A control means for controlling the display unit to display the virtual object based on a first criterion, which is a predetermined criterion for displaying a virtual object, and the information acquired by the acquisition means, It has a setting means for setting a second criterion, which is a substitute for the first criterion, to indicate a predetermined area specified by the user and the position or orientation of the controller at a predetermined time, The control means is When the second criterion is set by the setting means, the display unit is controlled to display the virtual object based on the second criterion and the information acquired by the acquisition means. If the second criterion is not set, the controller receives a first operation based on the position of the virtual object displayed based on the first criterion, and if the second criterion is set, the controller receives the first operation based on the position of the virtual object displayed based on the second criterion. An information processing device characterized by the following:

2. The virtual object is a ray of light extending from the controller or the user's hand, The first and second criteria mentioned above are criteria relating to the starting point. The information processing apparatus according to feature 1.

3. The aforementioned virtual object is a pointer, The first and second criteria mentioned above are criteria relating to the position or orientation of the controller and the position of the pointer. The information processing apparatus according to feature 1.

4. The first and second criteria mentioned above are criteria relating to a coordinate system with the controller as the origin. The information processing apparatus according to feature 1.

5. The second criterion is set by setting a correction value related to the angle of rotation of the coordinate system with the controller, which is the origin of the first criterion. The information processing apparatus according to feature 4.

6. The second criterion is set by setting a correction value relating to the distance by which the coordinate system with the controller that serves as the first criterion is translated. The information processing apparatus according to feature 4.

7. The system further includes a second means for acquiring information about the user's gaze, The information processing apparatus according to feature 1.

8. The aforementioned line-of-sight information is information indicating position or range. The information processing apparatus according to feature 7.

9. The aforementioned predetermined area is an area based on the user's gaze information. The information processing apparatus according to feature 7.

10. The aforementioned predetermined time is the time when the user's gaze input is received. The information processing apparatus according to feature 7.

11. The aforementioned predetermined time is the time when the controller receives input via a button. The information processing apparatus according to feature 1.

12. The predetermined time point is the time when the virtual object has been in the predetermined area for a predetermined period of time. The information processing apparatus according to feature 1.

13. The aforementioned predetermined region is a region that includes points that represent the characteristics of the real object. The information processing apparatus according to feature 1.

14. The aforementioned predetermined region is a region represented by a second virtual object different from the aforementioned virtual object. The information processing apparatus according to feature 1.

15. A recording means for recording the criteria set by the setting means, The recording means further comprises a selection means for selecting one of the criteria recorded in the recording means, The setting means sets the criteria selected by the selection means. The information processing apparatus according to feature 1.

16. Control the appearance of a virtual object while it is operating in the first mode so that it differs from the appearance of a virtual object displayed according to the first or second criterion. The information processing apparatus according to feature 1.

17. An acquisition step to obtain information regarding the position or orientation of the controller, A control step controls the display unit to display the virtual object based on a first criterion, which is a predetermined criterion for displaying a virtual object, and the information acquired in the acquisition step. The system includes a setting step of setting a predetermined area specified by the user and a second criterion that replaces the first criterion, so as to indicate the position or orientation of the controller at a predetermined time, In the control step described above, If the second criterion is set by the setting step, the display unit is controlled to display the virtual object based on the second criterion and the information acquired by the acquisition step. If the second criterion is not set, the controller receives a first operation based on the position of the virtual object displayed based on the first criterion, and if the second criterion is set, the controller receives the first operation based on the position of the virtual object displayed based on the second criterion. A control method for an information processing device characterized by the following features.

18. A program for causing a computer to function as each of the means of the information processing apparatus described in claim 1.

19. Controller and Display device and An acquisition device that acquires information regarding the position or orientation of the controller, A control device that controls the display device to display the virtual object based on a first criterion, which is a predetermined criterion for displaying a virtual object, and the information acquired by the acquisition device. The system includes a setting device that sets a second criterion, which replaces the first criterion, to indicate a predetermined area specified by the user and the position or orientation of the controller at a predetermined time, The control device is When the setting device has set the second criterion, the display device is controlled to display the virtual object based on the second criterion and the information acquired by the acquisition device. If the second criterion is not set, the controller receives a first operation based on the position of the virtual object displayed based on the first criterion, and if the second criterion is set, the controller receives the first operation based on the position of the virtual object displayed based on the second criterion. An information processing system characterized by the following:

Images