Information processing apparatus, system of information processing apparatus, control method of information processing apparatus, and program

Abstract

To improve operability of a display device which is supported by a user.SOLUTION: An information processing apparatus communicates with a display device, the information processing apparatus including: first acquisition means for acquiring operation information of a first operation member of the display device; second acquisition means for acquiring operation information of a second operation member of a controller; and control means configured to perform control so as to execute first processing, when a user does not support the display device, based on the operation information of the first operation member acquired by the first acquisition means, and execute second processing based on the operation information of the second operation member acquired by the second acquisition means, and perform control so as to execute the first processing, when the user supports the display device, based on the operation information of the second operation member acquired by the second acquisition means.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a display device and an information processing device. [Background technology]

[0002] Conventionally, in a cross reality (XR) system that allows a user to experience virtual reality, a controller is used to convert hand movements into actions in a virtual space when controlling the display of a head mounted display (HMD). An HMD is a head-mounted display device equipped with a small display that a user wears on his / her head. Also, by attaching a handle to the display device and holding the handle in the hand, it becomes possible to support and view the display device in the hand rather than wearing it on the head. Also, depending on the shape of the controller, it becomes possible to support the display device in the hand wearing the controller. Patent Document 1 is an example of a prior art in which a handle is attached to such a display device and held in the hand, making it possible to support and view the display device in the hand rather than wearing it on the head. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-114756 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, when a display device is attached with a handle and the handle is held by hand, it is difficult to press the button on the display device because the hand is occupied, and when one hand is taken off to press the button on the display device, the display device is held in an unstable position. In addition, when the button is pressed, the display device may move, and the position of the display unit may become misaligned with the position of the eyes. When the display device is pointed in this way, the operability of the display device may be impaired.

[0005] Therefore, an object of the present invention is to improve the operability of a display device even when the user is holding the display device. [Means for solving the problem]

[0006] One aspect of the present invention is an information processing device that communicates with a display device, comprising: a first acquisition means for acquiring operation information of a first operating member of the display device; a second acquisition means for acquiring operation information of a second operating member of a controller; and a control means for, when the display device is not being supported by a user, controlling to execute a first process based on the operation information of the first operating member acquired by the first acquisition means, and to execute a second process based on the operation information of the second operating member acquired by the second acquisition means, and, when the display device is being supported by the user, controlling to execute the first process based on the operation information of the second operating member acquired by the second acquisition means. Effect of the Invention

[0007] According to the present invention, even when the display device is supported, it is possible to improve the operability of the display device. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating the internal configuration of an information processing system according to a first embodiment, and an HMD and a controller which are examples of an information processing device. [Diagram 2] FIG. 1 is a schematic diagram of a display device 101 and a controller 200 according to a first embodiment. [Diagram 3] FIG. 2 is a schematic diagram showing a state in which a display device 101 with a handle attached is supported in the first embodiment. [Figure 4] 3 is a diagram showing an example of operation buttons of the display device 101 in the first embodiment. FIG. [Diagram 5] 3A to 3C are diagrams illustrating examples of operation buttons of a controller 200 in the first embodiment. [Figure 6] 10 is a flowchart for detecting that the display device 101 is being supported and replacing buttons on the controller in the first embodiment. [Figure 7] 4 is a table showing an example of replacement of operation buttons on a controller in the first embodiment. [Figure 8] 11 is a flowchart for comparing movement trajectories and determining that the display device 101 is supported in the first embodiment. [Figure 9] 10 is a flowchart for determining that the display device 101 is being supported by detecting that a handle is attached in the first embodiment. [Figure 10] 1 is a display example of a ray in the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that the embodiment described below is one example of a means for realizing the present invention, and may be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions. Also, each embodiment may be appropriately combined.

[0010] (First embodiment) <System configuration> An information processing system according to a first embodiment will be described with reference to Fig. 1(a). The information processing system includes an HMD 100 including a display device 101 and a control device 102, and a controller 200. The controller 200 is a device for performing various operations of the HMD 100.

[0011] The internal configuration of an HMD and a controller, which are examples of an information processing device, will be described with reference to FIG.

[0012] <Configuration of the display device 101> The display device 101 includes an imaging unit 111, a display unit 112, a position and orientation detection unit 113, an operation unit 114, an eyepiece detection unit 115, and a handle detection unit .

[0013] The display device 101 is configured, for example, as a glasses-type display portion of an HMD, and detects the position and orientation of a user wearing the display device 101. Then, a composite image is displayed in which a captured image capturing an area in front of the user and a virtual object shown in CG (computer graphics) in a form corresponding to the detected position and orientation are combined.

[0014] This allows a user wearing the display device 101 to observe a virtual reality image in which CG in a virtual space is superimposed according to the line of sight in real space. In order to generate a stereo image, two display units for the right eye and the left eye may be implemented. In the following description, the display device 101 is assumed to be a glasses-type display part of an HMD, but it may also be a display device such as a tablet terminal or a smartphone. In other words, any display device that is portable and can display an image corresponding to the user's field of view can be used.

[0015] The imaging unit 111 is composed of an objective optical system that captures a real image of the outside world as light, and an image sensor that converts an optical signal into an electrical signal. The imaging unit 111 includes two cameras (imaging devices). The two cameras are for capturing captured images used for compositing with an image of a virtual space and generating position and orientation information, and have an imaging unit for a left eye and an imaging unit for a right eye. The imaging unit for the left eye captures a moving image of the real space corresponding to the left eye of the person wearing the display device 101, and an image (captured image) of each frame in the moving image is output from the imaging unit for the left eye. The imaging unit for the right eye captures a moving image of the real space corresponding to the right eye of the person wearing the display device 101, and an image (captured image) of each frame in the moving image is output from the imaging unit for the right eye. In other words, the imaging unit 111 acquires captured images as stereo images having parallax that approximately matches the positions of the left eye and right eye of the person wearing the display device 101. By measuring distances using the stereo cameras, information on the distances from the two cameras to the subject can be acquired as distance information. In an HMD for an MR (Mixed Reality) system, the imaging unit is preferably disposed so that the central optical axis of the imaging range of the imaging unit is approximately aligned with the line of sight of the wearer of the HMD. Alternatively, the imaging unit may be disposed at a position where the central optical axis of the imaging range of the imaging unit does not match the line of sight of the wearer of the HMD, and the position of the viewpoint may be converted to match the line of sight of the wearer of the HMD (a so-called view conversion method).

[0016] Each of the left eye imaging section and the right eye imaging section has an optical system and an imaging device. Light entering from the outside world enters the imaging device via the optical system, and the imaging device outputs an image corresponding to the entering light as a captured image. The images of a subject (the area in front of the user) captured by the two cameras are output to the control device 102. Note that the imaging section 202 may capture and output a video instead of a captured image.

[0017] The display unit 112 displays an image generated by the control device 102. The display unit 112 has a liquid crystal panel or an organic EL panel. When the user wears the display device 101, the display unit 112 is disposed in front of each eye of the user. It is also possible to use a device using a semi-transmissive half mirror for the image display unit 203. In this case, for example, the display unit 112 may display an image by using a technology generally called AR (Augmented Reality) so that the CG is superimposed directly on the real space seen through the half mirror. The image display unit 203 may also display an image of a complete virtual space without using a captured image by using a technology generally called VR (Virtual Reality).

[0018] The position and orientation detection unit 113 is a functional unit for acquiring the position and orientation of the display device 101, and can calculate the change in orientation, relative direction, and position in real space of the display device 101. For example, an IMU (inertial measurement unit; inertial sensor) capable of detecting inertial information (amount of spatial movement and angle), a direction sensor using geomagnetism, and an orientation detection sensor using GPS can be used.

[0019] It is assumed that the position and orientation detection unit 113 is a functional unit that acquires at least one of the position and orientation of the display device 101 .

[0020] The operation unit 114 is a device for the user to operate the HMD 100. For example, it may be an operation member such as a button as shown in Fig. 4, or a mouse or a keyboard. By operating the operation unit 114, the user can switch between displaying and not displaying the display unit 112 and set the interpupillary distance, which will be described later.

[0021] The eye proximity detection unit 115 detects the user's actions of looking into the display unit 112 and moving away from the display unit 112. One method for detecting this is to provide an infrared sensor on the side of the display unit 112 and detect whether the user's face is in the vicinity of the display unit 112 based on a change in infrared detection.

[0022] The eye may be detected using a contact sensor, a non-contact sensor, or a proximity sensor.

[0023] A camera that detects the user's line of sight may be attached inside the HMD. An image of a subject (user's eyes) captured by the camera is output to the control unit 123 of the control device 102. The control unit 123 detects the line of sight of the user wearing the HMD 100 from the image captured by the camera that detects the user's line of sight, and acquires the detection result of the user's line of sight as detection information. The control unit 123 may also be configured to identify the location on the image display unit 203 that the user is gazing at.

[0024] The handle detection unit 116 is a mechanical mechanism that detects that the handle 300 is attached to the control device 101, and detects that the handle 300 is attached to the handle coupling portion 405 described later in Figure 4, and outputs the detection result to the control unit 123.

[0025] <Configuration of the control device 102> The control device 102 includes a CPU 120, a ROM 130, a RAM 140, and a communication unit 150. An image acquisition unit 121, a position and orientation extraction unit 122, and a control unit 123 are control blocks that operate on the CPU 120.

[0026] It is assumed here that the control device 102 is a device having high-performance arithmetic processing functions and graphic display functions, such as a smartphone, a PC (personal computer), or a workstation.

[0027] By connecting the control device 102 and the display device 101, a user wearing the display device 101 can view VR images, which are images of a virtual space. Note that instead of images of a virtual space, MR images, which are images of a mixed reality world in which the real world and the virtual world are seamlessly integrated in real time, may be used.

[0028] The image acquisition unit 121 acquires an image of the real space (real image) captured by the imaging unit 111 .

[0029] The position and orientation extraction unit 122 acquires the position and orientation information detected by the position and orientation detection unit 113. Furthermore, a configuration may be adopted in which a marker arranged in the real space is detected from the real image acquired by the image acquisition unit 121, and the position and orientation are calculated.

[0030] The control unit 123 generates an image in which the real image acquired by the image acquisition unit 121 is combined with CG, and transmits the combined image to the display unit 112. Therefore, the user can view the combined image displayed on the display unit 112 by wearing the display device 101. The user can experience various mixed realities, such as CG blended into real space. Note that instead of the control unit 211 controlling the entire device, the entire device may be controlled by multiple hardware devices sharing the processing.

[0031] The control unit 123 controls the position, orientation, and size of the CG in the composite image based on the information (distance information and orientation information) acquired by the display device 101. For example, when the control unit 123 places a virtual object represented by the CG near a specific object existing in real space in the space represented by the composite image, the control unit 123 makes the virtual object (CG) larger as the distance between the specific object and the imaging unit 111 is closer. By controlling the position, orientation, and size of the CG in this manner, the control unit 123 can generate a composite image in which a CG object that is not placed in real space appears as if it were placed in real space.

[0032] Furthermore, in the control unit 123, the communication unit 150 receives information about a change in the position or attitude of the controller 200 from the communication unit 205 of the controller 200. The control unit 123 superimposes on the combined image an indication position according to the information about the change in the position or attitude of the controller 200. Note that the control unit 123 may also superimpose on the combined image an indication position according to the information about the change in the position and attitude of the controller 200.

[0033] The ROM 130 is an electrically erasable and recordable non-volatile memory that stores information such as CG, etc. The control unit 123 can switch the CG read from the ROM 130 (i.e., the CG used to generate a composite image).

[0034] The control unit 123 controls the HMD 100 based on values ​​input from the controller 200 acquired via the communication unit 150 .

[0035] The communication unit 150 is an interface for communicating with an external device via so-called wireless LAN or Bluetooth (registered trademark) in accordance with the IEEE802.11 standard. However, the communication method is not limited to wireless LAN or Bluetooth, and any communication protocol may be used, regardless of whether it is wireless or wired, as long as communication can be performed. The HMD 100 communicates with the controller 200 via the communication unit 150. The communication unit 150 can also communicate with devices other than the controller 200, such as a smartphone or a tablet terminal.

[0036] The display device 101 and the control device 102 are connected to each other so that they can communicate data with each other. Therefore, the connection between the display device 101 and the control device 102 may be wired or wireless. The display device 101 and the control device 102 may be integrated and configured so as to be portable by the user. For example, they may be a so-called stand-alone HMD.

[0037] <Configuration of controller 200> 1(b), the internal configuration of the controller 200 will be described. The controller 200 includes a CPU 201, a position and orientation detection unit 202, an operation unit 203, a vibration unit 204, and a communication unit 205.

[0038] The CPU 201 is a control unit that controls each component of the controller 200. Note that instead of the CPU 201 controlling the entire device, the entire device may be controlled by having a plurality of hardware devices share the processing.

[0039] The position and orientation detection unit 202 is a functional unit for acquiring the position and orientation of the controller 200, and can calculate the orientation change, relative direction, and position of the controller 200 in real space. For example, an IMU (inertial measurement unit; inertial sensor) capable of detecting inertial information (spatial movement amount and angle), a direction sensor using geomagnetism, and an orientation detection sensor using GPS can be used. The position and orientation detection unit 202 may include any device that does not hinder the miniaturization of the controller 200 and can detect inertial information (information such as positional displacement, speed, or acceleration).

[0040] It is assumed that the position and orientation detection unit 202 is a functional unit that acquires at least one of the position and orientation of the controller 200.

[0041] The operation unit 203 may include any of a button, a touch pad, a touch panel, a cross key, a joystick, and a track pad device. For example, the user presses and holds a button to display a menu including a pointer on the HMD 100. After that, the user can move the pointer to a desired item by pressing the cross key in any direction. Then, the user can perform a decision action such as deciding the selection of the item by pressing the button. For example, it is possible to switch between displaying and not displaying a ray 1000, which will be described later in FIG. 10, by displaying and selecting the ray 1000 in a menu. Operation information in the operation unit 203 is transmitted to the HMD 100 via the communication unit 205.

[0042] The vibration unit 204 vibrates the controller 200. For example, when the ray 1000 touches the CG 1001, the CPU 201 may receive a vibration instruction from the HMD 100 via the communication unit 205, and control the vibration unit 204 to vibrate the controller 200. When the controller 200 vibrates, the user knows that the ray 1000 has touched the CG 1001.

[0043] The communication unit 205 performs wireless communication with the communication unit 105 of the HMD 100. When there are multiple controllers, each performs wireless communication with the communication unit 105.

[0044] Note that the controller 200 may have an output unit. The output unit is composed of, for example, an LED light source, a speaker, or the like.

[0045] <Method of wearing> FIG. 2 is a schematic diagram of the HMD, hand, and controller. As shown in FIG. 2, the display device 101 is worn on the user's head, and the controller 200 is worn on the user's hand 210. The controller 200 has a ring shape, and if the controller 200 can be worn on the user's finger, the user can move the hand freely while supporting the controller 200.

[0046] Note that although the shape of the controller 200 is assumed to be a ring shape, it is not limited thereto.

[0047] For example, the shape of the controller 200 may be a shape that can be worn on the hand, such as a glove shape. In this way, the controller 200 is preferably in a form that can be held or worn on the user's hand so that the user can easily use it. In this specification, holding the controller and wearing the controller are collectively referred to as supporting the controller.

[0048] <Method of attaching a handle to the HMD and supporting it with the hand> FIG. 3 is a schematic diagram when the display device 101 is supported by the hand. As shown in FIG. 3, a handle 300 is attached to the display device 101 of the HMD 100, and the user can hold the handle 300 with the hand to support and view the HMD with the hand instead of wearing it on the head. In this specification, holding the display device 101 and placing and holding the display device 101 on the hand or finger are collectively referred to as supporting the display device 101.

[0049] Depending on the shape of the controller 200, it may be possible to support the display device 101 with the hand holding the controller 200.

[0050] <Operation Buttons of HMD and Controller> An example of the operation buttons of the HMD is shown in FIG. 4. Here, an example is shown when viewing the display device 101 of the HMD from above. As the operation unit 114 of the display device 101 of the HMD, there are a button A (401), a button B (402), a button C (403), and a button D (404). Also, the handle coupling portion 405 is a coupling portion for attaching the handle 300.

[0051] For example, when adjusting the interpupillary distance, the adjustment is started with the button B (402), the distance between the lenses is widened with the button C (403), and the distance between the lenses is narrowed with the button D (404).

[0052] It is assumed that the button A (401) is used for other operations and is not used when the display device 101 is being supported.

[0053] An example of the operation buttons of the controller 200 is shown in FIG. 5. Here, as the operation unit 203 of the controller 200, there are a button X (501), a button Y (502), and a button Z (503). For example, it is possible to wear the controller 200 on the index finger and operate these buttons with the thumb.

[0054] By holding the handle 300 with the hand wearing the ring-shaped controller 200 in this way, it becomes possible to operate the operation buttons of the operation unit 203 of the controller 200 while supporting the display device 101 of the HMD by hand.

[0055] <Judgment of Support by Hand> With reference to FIG. 6, a flowchart for detecting that the user is supporting the display device 101 by hand and replacing the buttons of the controller will be described.

[0056] In step S601, the control unit 123 determines whether or not the user is supporting the display device 101. If the control unit 123 determines that the user is supporting the display device 101, the process proceeds to step S602, and if the control unit 123 determines that the user is not supporting the display device 101, the process proceeds to step S603.

[0057] In step S602, the control unit 123 performs a replacement so that the process that is executed when the operation button of the display device 101 is operated is executed by operating the operation button of the controller, and the process proceeds to step S604.

[0058] In step S603, the control unit 123 does nothing and the process proceeds to step S601.

[0059] In step S604, the control unit 123 determines whether or not the user is supporting the display device 101. If the control unit 123 determines that the user is supporting the display device 101, the process proceeds to step S604, and if the control unit 123 determines that the user is not supporting the display device 101, the process proceeds to step S605.

[0060] In step S605, the control unit 123 ends the process replaced in step S602. That is, the control unit 123 controls so that the replaced process is executed by operating the operation button of the display device 101. Also, the control unit 123 controls so that the process that is executed when the operation button of the display device 101 is operated is not executed by operating the operation button of the controller. A specific example of the replacement process will be described later.

[0061] In step S604, it may be determined that the user is not supporting the display device 101 the moment the user no longer supports it, or it may be determined that the user is not supporting the display device 101 if the state of not supporting it continues for a predetermined period of time. When it is determined that the user is not supporting the display device 101 if the state of not supporting it continues for a predetermined period of time, for example, when the user touches the body or changes the hand holding the display device 101, replacement of the operation buttons is not immediately completed, improving operability. Note that such a waiting time may be individually set by the user.

[0062] <Button replacement example> FIG. 7 shows an example in which the operation buttons of the display device 101 are replaced with operation buttons of a controller in step S602 in FIG.

[0063] Button B of the display device 101 is replaced with button Z of the controller 200. That is, when adjusting the interpupillary distance, the adjustment is started by operating button Z.

[0064] The button C of the display device 101 is replaced with the button X of the controller 200. That is, when the button X is operated, the button is replaced so that the distance between the lenses is increased when the interpupillary distance is adjusted.

[0065] The button D of the display device 101 is replaced with the button Y of the controller 200. That is, the button Y is replaced so that the distance between the lenses is narrowed when the button Y is operated.

[0066] In this way, by assigning processes to be executed by button operations on the display device 101 to the buttons on the controller 200, operating the operation members of the controller 200 enables operations equivalent to button operations on the display device 101. In other words, the operation functions of the display device 101 can be assigned to the controller.

[0067] In addition, at this time, the processes executed by operating some of the operation members of the display device 101 may not be replaced by the operation members of the controller 200. For example, button A of the display device 101 is not used for adjusting the interpupillary distance. Therefore, when the display device 101 is supported, the processes executed by operating button A are not executed even if any button on the controller is operated.

[0068] <Support determination based on movement trajectory> A case will be described where, in step S601 of FIG. 6, it is determined whether the user is holding the display device 101 with his / her hand based on whether the movement trajectories of the display device 101 and the controller 200 match.

[0069] FIG. 8 is a flowchart for determining that the display device 101 is being held by the hand by comparing the movement trajectories of the display device 101 and the controller 200. In FIG.

[0070] In step S801, the control unit 123 calculates a movement trajectory indicating how the position or orientation of the display device 101 has changed in real space from the position or orientation information acquired by the position and orientation extraction unit 122, and proceeds to step S802.

[0071] In step S802, the control unit 123 receives the position and orientation information of the controller 200 via the communication unit 150, and the process proceeds to step S803.

[0072] In step S803, a movement trajectory indicating how the position or orientation of the controller 200 has changed in real space is calculated from the received position or orientation information, and the process proceeds to step S804.

[0073] In step S804, the control unit 123 determines whether the movement trajectory of the display device 101 and the movement trajectory of the controller 200 match. When the control unit 123 determines that the movement trajectory of the display device 101 and the movement trajectory of the controller 200 match, it determines that the user is supporting the display device 101. When the control unit 123 determines that the movement trajectory of the display device 101 and the movement trajectory of the controller 200 do not match, it determines that the user is not supporting the display device 101. Note that a threshold value may be set to determine whether or not they match, taking into consideration an error when acquiring a position or attitude. When the difference between the movement trajectory of the display device 101 and the movement trajectory of the controller 200 matches by a threshold value or more, it is determined that the display device 101 is supported, and when the difference between the movement trajectory of the display device 101 and the movement trajectory of the controller 200 matches by less than the threshold, it is determined that the display device 101 is not supported.

[0074] Incidentally, the agreement between the movement trajectory of the display device 101 and the movement trajectory of the controller 200 may be expressed by the degree of agreement. Here, it is defined that the closer the agreement between the movement trajectory of the display device 101 and the movement trajectory of the controller 200 is, the higher the degree of agreement is, that is, the larger the value of the degree of agreement is.

[0075] In this way, when the movement trajectories of the HMD and the controller match, it is determined that the user is supporting the display device 101, and it becomes possible to replace the operation buttons of the controller. In this case, the handle detection unit 116 is not necessary.

[0076] <Method of determining support other than movement trajectory> A method for determining support other than along a movement trajectory will be described with reference to Fig. 9. Fig. 9 is a flowchart for determining that the display device 101 is being supported by detecting that a handle is attached and that the user is looking into the display device.

[0077] In step S901, the control unit 123 determines whether or not the handle 300 is attached based on the detection result of the handle detection unit 116. If the control unit 123 determines that the handle 300 is attached, the process proceeds to step S902, and if the control unit 123 determines that the handle 300 is not attached, the control unit 123 determines that the user is not supporting the display device 101 and completes the process.

[0078] In step S902, control unit 123 determines whether the user is looking at display unit 112 based on the detection result of eyepiece detection unit 115. When control unit 123 obtains a detection result detecting eyepiece by eyepiece detection unit 115, it determines that the user is looking at display unit 112, and when control unit 123 does not obtain a detection result detecting eyepiece by eyepiece detection unit 115, it determines that the user is not looking at display unit 112. When control unit 123 determines that the user is looking at display unit 112, it proceeds to step S903, and when control unit 123 determines that the user is not looking at display unit 112, it determines that the user is not supporting display device 101 and completes the process.

[0079] In step S903, the control unit 123 determines whether or not it is communicating with the controller 200. When it is determined that it is communicating with the controller 200, the control unit 123 determines that the user is supporting the display device 101, and proceeds to step S602 in FIG. 6. When it is determined that it is not communicating with the controller 200, the control unit 123 determines that the user is not supporting the display device 101, and completes the process. Note that here, even if the user is actually supporting the display device 101, it is determined that the user is not supporting the display device 101 because there is no controller to which a process to be executed by operating an operating member of the display device 101 can be assigned.

[0080] 9, although the handle 300 is attached, there is a possibility that the user is not holding the display device 101 in his / her hands, so after the determination in step S901, a determination is also made in step S902. Note that, if the handle is attached, it may be determined that the user is supporting the display device 101, and the determination in step S902 may not be made.

[0081] In this way, when the handle 300 is attached and the display device 100 is being held in the hand while communicating with the controller 200, it is possible to determine that the user is supporting the display device 101 and to replace it with the operation buttons on the controller.

[0082] Thus, the method of determining that the display device 101 is being supported by comparing the movement trajectories of the HMD and the controller or by confirming that the handle is attached has been described with reference to FIG. 8 and FIG. 9, but the determination may be made by other methods. For example, a camera may be prepared near the user, and the user wearing the HMD may be captured by the camera, and it may be determined whether the user is supporting the display device 101 at that time. In this case, the handle detection unit 116 is not required. Also, for example, the display device 101 may be configured to include at least one of a contact sensor, a non-contact sensor, and a proximity sensor to detect hands or eyes, or the user's line of sight may be detected, and if detected, it may be determined that the display device 101 is being supported.

[0083] In addition, the distance between the display device 101 and the controller 200 may be calculated from the position information of the display device 101 and the position information of the controller 200, and if the distance is less than a threshold, it may be determined that there is support, and if the distance is equal to or greater than the threshold, it may be determined that there is no support.

[0084] The support determination methods described in this specification may be appropriately combined. By combining a plurality of processes, it becomes possible to more accurately determine whether the user is supporting the display device or using the controller.

[0085] <Prevention of erroneous operation> Operation methods using the controller 200 include displaying a ray or a pointer for operating CG. Here, a ray is a display item of a light beam that points to a position. An example of a ray display is shown in FIG. 10. A ray 1000 is displayed from the controller 200 worn on the user's hand 210 toward a CG 1001. The tip position and light beam of the ray 1000 are determined based on at least one of information regarding the position and orientation of the controller 200 worn on the user's hand 210.

[0086] When it is detected that the display device 101 is being supported and the buttons on the controller are replaced, normal operations of the controller may be invalidated. The control unit 123 hides the ray 1000, and ignores operation information and position and orientation information even if it receives them from the controller 200.

[0087] Also, a vibration instruction may not be sent to the controller 200, that is, the vibration function may be disabled.

[0088] In such a case, when supporting the display device 101 with the hand holding the controller 200, the movement of holding the display device 101 is detected as position or posture information, making it possible to prevent unintended instructions from being issued from the controller 200.

[0089] <Cancel> In the flowchart of Fig. 6, the movement trajectories of the display device 101 and the controller 200 are compared to detect that the display device 101 is being supported. Here, if it is detected that the movement trajectories of the display device 101 and the controller 200 do not match for a predetermined period after a process is assigned to a button on the controller, the assignment to the button on the controller may be stopped. For example, when the user changes his grip on the handlebars or touches a part of his body, there may be a period during which the movement trajectories of the display device 101 and the controller 200 do not match, so it is possible to prevent the assignment to the button on the controller from ending at a timing different from the user's intention.

[0090] Note that, after detecting that the display device 101 is supported and replacing the buttons on the controller, if the user operates a button on the HMD, the assignment to the buttons on the controller may be stopped. At this time, if the process of replacing the buttons on the controller is being executed, the control unit 123 detects that the operation unit 114 has been operated, and performs control to stop replacing the buttons on the controller.

[0091] <Notification> When the replacement process to the buttons of the controller is being executed, the user may be notified that the replacement process is being executed. As a method of notifying the user, an icon or text indicating that the replacement is being executed may be displayed on the display unit 112 of the display device 101, and if the controller is provided with an LED, the LED may be controlled to emit light during the replacement process.

[0092] (Other embodiments) The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-mentioned embodiments are supplied to a system or device via a network or various storage media, and the computer (or control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0093] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0094] Each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0095] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0096] [Configuration 1] An information processing device that communicates with a display device, a first acquisition means for acquiring operation information of a first operation member of the display device; a second acquisition means for acquiring operation information of a second operation member included in the controller; a control means for controlling, when the display device is not supported by a user, to execute a first process based on operation information of the first operating member acquired by the first acquisition means, and for controlling, when the display device is supported by the user, to execute a second process based on operation information of the second operating member acquired by the second acquisition means, and for controlling, when the display device is supported by the user, to execute the first process based on the operation information of the second operating member acquired by the second acquisition means. 23. An information processing apparatus comprising:

[0097] [Configuration 2] The information processing device is provided in the display device. 2. The information processing device according to configuration 1.

[0098] [Configuration 3] the control means, when the user is holding the display device, does not execute the first process even if the first acquisition means acquires operation information of the first operation member. 3. The information processing device according to configuration 1 or 2.

[0099] [Configuration 4] further comprising an assignment means for assigning an operation function to the first operation member and the second operation member, the assignment means assigns the operation function assigned to the first operation member to the second operation member when the user is holding the display device; the control means executes a process corresponding to the operation function assigned to the second operating member by the assignment means, based on operation information of the first operating member and the second operating member. 4. The information processing device according to any one of configurations 1 to 3.

[0100] [Configuration 5] The display device further includes a determination unit for determining whether the user is supporting the display device, When the determination means determines that the user is holding the display device, the control means executes the first process based on operation information of the second operation member of the controller. 5. The display device according to any one of configurations 1 to 4.

[0101] [Configuration 6] The determination means determines that the user is supporting the display device when a predetermined member is attached to the display device. 6. The information processing device according to configuration 5.

[0102] [Configuration 7] The predetermined member is a member for the user to support the display device. 7. The information processing device according to configuration 6.

[0103] [Configuration 8] The present invention further includes a gaze detection means for detecting a gaze of the user, The determination means determines that the user is supporting the display device when the gaze detection means detects the gaze of the user. 8. The information processing device according to any one of configurations 5 to 7.

[0104] [Configuration 9] a third acquiring means for acquiring detection information of a contact with the display device from a contact sensor of the display device; The determination means determines that the user is holding the display device when detection information of contact with the display device is acquired by the third acquisition means. 9. The information processing device according to any one of configurations 5 to 8.

[0105] [Configuration 10] A fourth acquisition means for acquiring a detection result of the inertial sensor of the controller; A fifth acquisition means for acquiring a detection result of the inertial sensor of the display device, The determination means determines whether or not the user is holding the display device based on the detection result acquired by the fourth acquisition means and the detection result acquired by the fifth acquisition means. 10. The information processing device according to any one of configurations 5 to 9.

[0106] [Configuration 11] the determination means determines that the user is holding the display device when a difference between a movement trajectory based on a detection result of the fourth acquisition means and a movement trajectory based on a detection result of the fifth acquisition means is equal to or greater than a threshold, and determines that the user is not holding the display device when the difference is less than the threshold. 11. The information processing device according to configuration 10.

[0107] [Configuration 12] A sixth acquisition means for acquiring a distance between the display device and the controller, The determination means determines that the user is holding the display device when the distance acquired by the sixth acquisition means is less than a predetermined value. 12. The information processing device according to any one of configurations 5 to 11.

[0108] [Configuration 13] A seventh acquisition means for acquiring a captured image from an external device, The determination means determines whether or not the user is holding the display device based on the captured image acquired by the seventh acquisition means. 13. The information processing device according to any one of configurations 5 to 12.

[0109] [Configuration 14] Further comprising a fourth acquisition means for acquiring a detection result of the inertial sensor of the controller, the control means, when supporting the display device, does not execute a process based on the detection result of the inertial sensor of the controller acquired by the fourth acquisition means. 14. The information processing device according to any one of configurations 1 to 13.

[0110] [Configuration 15] the control means disables a vibration function of the controller when the display device is supported. 15. The information processing device according to any one of configurations 1 to 14.

[0111] [Configuration 16] Further comprising an eighth acquisition means for acquiring operation information of an eighth operation member of the display device, when the user is not supporting the display device, executes an eighth process based on operation information of the eighth operation member acquired by the eighth acquisition means, and when the user is supporting the display device, does not execute the eighth process based on the operation information of the eighth operation member acquired by the eighth acquisition means and the operation information of the second operation member acquired by the second acquisition means. 16. The information processing device according to any one of configurations 1 to 15.

[0112] [Configuration 17] the assignment means, when the user is holding the display device, assigns the operation function assigned to the second operation member to the first operation member based on the operation information of the first operation member acquired by the first acquisition means. 17. The information processing device according to any one of configurations 4 to 16.

[0113] [Configuration 18] The display device is a head-mounted display. 18. The display device according to any one of configurations 1 to 17.

[0114] [Control method] A first acquisition step of acquiring operation information of a first operation member of the display device; a second acquisition step of acquiring operation information of a second operation member of the controller; a control step of controlling to execute a first process based on the operation information of the first operation member acquired in the first acquisition step when the display device is not being supported by a user, and to execute a second process based on the operation information of the second operation member acquired in the second acquisition step when the display device is being supported by the user, 23. A method for controlling an information processing apparatus comprising:

[0115] [program] A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 18.

[0116] [system] A display device; A controller; a first acquisition device that acquires operation information of a first operation member included in the display device; a second acquisition device that acquires operation information of a second operation member included in the controller; a control device that, when the display device is not being held by a user, controls to execute a first process based on operation information of the first operation member acquired by the first acquisition device, and controls to execute a second process based on operation information of the second operation member acquired by the second acquisition device, and, when the display device is being held by the user, controls to execute the first process based on the operation information of the second operation member acquired by the second acquisition device; An information processing system comprising:

Claims

1. An information processing device that communicates with a display device, a first acquisition means for acquiring operation information of a first operation member of the display device; a second acquisition means for acquiring operation information of a second operation member of the controller; a control means for controlling, when the display device is not being held by a user, to execute a first process based on operation information of the first operating member acquired by the first acquisition means, and for controlling, when the display device is being held by the user, to execute a second process based on operation information of the second operating member acquired by the second acquisition means, and for controlling, when the display device is being held by the user, to execute the first process based on the operation information of the second operating member acquired by the second acquisition means.

23. An information processing apparatus comprising:

2. The information processing device is provided in the display device.

2. The information processing apparatus according to claim 1,

3. the control means, when the user is holding the display device, does not execute the first process even if the first acquisition means acquires operation information of the first operation member.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. The device further includes an assignment unit that assigns an operation function to the first operation member and the second operation member, the assignment means assigns the operation function assigned to the first operation member to the second operation member when the user is holding the display device; the control means executes a process corresponding to the operation function assigned to the second operating member by the assignment means, based on operation information of the first operating member and the second operating member.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

5. The display device further includes a determination unit for determining whether the user is supporting the display device, and when the determination means determines that the user is holding the display device, the control means executes the first process based on operation information of the second operation member of the controller.

3. The display device according to claim 1 or 2.

6. The determination means determines that the user is supporting the display device when a predetermined member is attached to the display device.

6. The information processing apparatus according to claim 5,

7. The predetermined member is a member for the user to support the display device.

7. The information processing apparatus according to claim 6,

8. The present invention further includes a gaze detection means for detecting a gaze of the user, The determination means determines that the user is supporting the display device when the gaze detection means detects the gaze of the user.

6. The information processing apparatus according to claim 5,

9. a third acquiring means for acquiring detection information of a contact with the display device from a contact sensor of the display device, The determination means determines that the user is holding the display device when detection information of contact with the display device is acquired by the third acquisition means.

6. The information processing apparatus according to claim 5,

10. A fourth acquisition means for acquiring a detection result of an inertial sensor of the controller; and a fifth acquisition means for acquiring a detection result of the inertial sensor of the display device, The determination means determines whether or not the user is holding the display device based on the detection result acquired by the fourth acquisition means and the detection result acquired by the fifth acquisition means.

6. The information processing apparatus according to claim 5,

11. the determination means determines that the user is holding the display device when a difference between a movement trajectory based on a detection result of the fourth acquisition means and a movement trajectory based on a detection result of the fifth acquisition means is equal to or greater than a threshold, and determines that the user is not holding the display device when the difference is less than the threshold.

11. The information processing apparatus according to claim 10,

12. a sixth acquisition means for acquiring a distance between the display device and the controller, The determination means determines that the user is holding the display device when the distance acquired by the sixth acquisition means is less than a predetermined value.

6. The information processing apparatus according to claim 5,

13. A seventh acquisition means for acquiring a captured image from an external device, The determination means determines whether or not the user is holding the display device based on the captured image acquired by the seventh acquisition means.

6. The information processing apparatus according to claim 5,

14. The controller further includes a fourth acquisition means for acquiring a detection result of the inertial sensor of the controller, the control means, when supporting the display device, does not execute a process based on the detection result of the inertial sensor of the controller acquired by the fourth acquisition means.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

15. the control means disables a vibration function of the controller when the display device is supported; 3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

16. The display device further includes an eighth acquisition means for acquiring operation information of an eighth operation member, when the user is not supporting the display device, an eighth process is executed based on operation information of the eighth operation member acquired by the eighth acquisition means, and when the user is supporting the display device, the eighth process is not executed based on the operation information of the eighth operation member acquired by the eighth acquisition means and the operation information of the second operation member acquired by the second acquisition means.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

17. the assigning means, when the user is holding the display device, assigns the operation function assigned to the second operating member to the first operating member based on the operation information of the first operating member acquired by the first acquisition means.

5. The information processing apparatus according to claim 4.

18. The display device is a head-mounted display.

3. The display device according to claim 1 or 2.

19. A first acquisition step of acquiring operation information of a first operation member of the display device; a second acquisition step of acquiring operation information of a second operation member of the controller; a control step of controlling to execute a first process based on operation information of the first operation member acquired in the first acquisition step when the display device is not being held by a user, and to execute a second process based on operation information of the second operation member acquired in the second acquisition step when the display device is being held by the user, and a control step of controlling to execute the first process based on the operation information of the second operation member acquired in the second acquisition step.

23. A method for controlling an information processing apparatus comprising:

20. A program for causing a computer to function as each of the means of the information processing apparatus according to claim 1.

21. A display device; A controller; a first acquisition device that acquires operation information of a first operation member included in the display device; a second acquisition device that acquires operation information of a second operation member of the controller; a control device that, when the display device is not being held by a user, controls to execute a first process based on operation information of the first operating member acquired by the first acquisition device, and controls to execute a second process based on operation information of the second operating member acquired by the second acquisition device, and, when the display device is being held by the user, controls to execute the first process based on the operation information of the second operating member acquired by the second acquisition device; An information processing system comprising:

Images