Controller, information processing device, and information processing method

The integration of mouse and 6DoF functions in a single controller with automatic mode switching addresses the limitations of existing controllers, allowing for seamless transition between 2D and 3D tasks without hand switching, thus reducing user burden and improving operational efficiency.

WO2024237020A1PCT designated stage expired Publication Date: 2024-11-21SONY GROUP CORP
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/015698
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-15
Filing Date
2024-04-22
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Conventional 6DoF controllers are inadequate for both aerial operations and precise desk tasks due to lack of accuracy and ease of grip, while mice are unsuitable for aerial operations, leading to user burden from frequent hand switching between the two.

Method used

A controller integrating a mouse function unit for 2D coordinate output, a 6DoF function unit for position and orientation output, and a switching signal unit to seamlessly switch between 2D and 3D display modes, allowing for continuous operation without hand switching.

Benefits of technology

Enables users to perform both 2D and 3D tasks with reduced user load by automatically switching between modes based on the controller's position, enhancing operational efficiency and comfort.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024015698_21112024_PF_FP_ABST
    Figure JP2024015698_21112024_PF_FP_ABST
Patent Text Reader

Abstract

A controller according to the present disclosure comprises: a mouse function unit that outputs coordinate information on a plane; a 6DoF function unit that outputs position / orientation information; and a switching signal output unit that outputs a switching signal for switching between a mode for displaying a two-dimensional image to which the coordinate information output from the mouse function unit is applied and a mode for displaying a three-dimensional image to which the position / orientation information output from the 6DoF function unit is applied.
Need to check novelty before this filing date? Find Prior Art

Description

Controller, information processing device, and information processing method

[0001] The present disclosure relates to a controller, an information processing device, and an information processing method.

[0002] 2. Description of the Related Art Systems that use VR (Virtual Reality) HMDs (Head Mounted Displays) to perform design reviews of CAD (Computer Aided Design) data are becoming commonplace.

[0003] Design work and review work are often performed separately, with a mouse being used for design work and a 6DoF (6 degree of freedom) controller being used for review work.

[0004] Special table 2019-532396 publication

[0005] Conventional 6DoF controllers for VR are designed for mid-air operation, making them suitable for manipulating stereoscopically displayed three-dimensional (3D) objects in mid-air, but are not suitable for natural precision work on a desk in terms of accuracy or ease of grip. On the other hand, a mouse, which is a suitable and common means for performing precision work on a desk, does not have a means for mid-air operation, and attempting to perform both operations using the appropriate means requires the user to switch between holding the mouse and the VR controller, which places a burden on the user.

[0006] Conventionally, techniques have been disclosed that provide a more natural user input mode by converting inputs from multiple 6DoF controllers into a common coordinate system, but no controllers have been disclosed that enable both three-dimensional tasks and tasks on a two-dimensional (2D) plane (display).

[0007] The present disclosure has been made in consideration of the above-mentioned problems, and aims to provide a controller that can reduce the burden on the user by eliminating the need to switch between a controller for performing three-dimensional tasks and a mouse for tasks on a two-dimensional plane, as well as an information processing device and information processing method that processes signals from the controller.

[0008] The controller of the present disclosure has a mouse function unit that outputs coordinate information on a plane, a 6DoF function unit that outputs position and orientation information, and a switching signal output unit that outputs a switching signal that switches between a display mode of a two-dimensional image to which the coordinate information output from the mouse function unit is applied and a display mode of a three-dimensional image to which the position and orientation information output from the 6DoF function unit is applied.

[0009] The information processing device of the present disclosure includes a receiving unit that receives a switching signal for switching between display modes of two-dimensional images and three-dimensional images, and a display mode switching unit that switches from a first display mode that displays the two-dimensional images to a second display mode that displays the three-dimensional images based on the received switching signal.

[0010] The information processing method disclosed herein receives a switching signal for switching between display modes of two-dimensional images and three-dimensional images, and switches from a first display mode displaying the two-dimensional images to a second display mode displaying the three-dimensional images based on the received switching signal.

[0011] 1 is a diagram for explaining an image of 2D display and 3D display in design using a VR HMD according to an embodiment. FIG. 1 is a diagram showing an operation of moving and changing the posture of an object displayed in 3D using a controller according to an embodiment. FIG. 2 is a diagram showing an operation of specifying a three-dimensional point on an object displayed in 3D using a controller according to an embodiment. FIG. 3 is a diagram showing an operation of pointing (Ray control) at an object displayed in 3D using a controller according to an embodiment. FIG. 4 is a diagram showing a rear view of a controller according to an embodiment. FIG. 5 is a side view of a controller according to an embodiment. FIG. 6 is a diagram showing a state in which a controller according to an embodiment is used as a normal mouse. FIG. 7 is a diagram showing an external appearance of a controller equipped with a magnetic tracking mechanism according to an embodiment. FIG. 8 is a diagram showing an external appearance of a controller equipped with a bright spot tracking mechanism according to an embodiment. FIG. 9 is a diagram for explaining switching between 2D operation and 6DoF operation of a controller equipped with a bright spot tracking mechanism according to an embodiment. FIG. 10 is a diagram showing fingers used when operating a controller according to an embodiment. FIG. 11 is a diagram showing a rear view of a controller according to an embodiment when held in the same manner as when operating a normal mouse. FIG. 12 is a diagram showing a side view of a controller according to an embodiment when held in the same manner as when operating a normal mouse. 1 is a diagram showing a case where 6 DoF operations are performed with the controller when the controller is held in the same manner as when operating a normal mouse. FIG. 2 is a front view of the controller according to the embodiment. FIG. 3 is a diagram showing the position of the center of gravity of the controller according to the embodiment. FIG. 4 is a diagram for explaining the structure of the controller according to the embodiment. FIG. 5 is a diagram for explaining the behavior of the controller due to gravity when the controller according to the embodiment is lifted. FIG. 6 is a diagram for explaining the relationship between the inner part and the upper part when the controller according to the embodiment is lifted. FIG. 7 is a diagram showing the case where the tip of the controller is pointed downward for 3D operations. FIG. 8 is a diagram for explaining the relationship between the inner part and the upper part when the tip of the controller according to the embodiment is pointed downward. FIG. 9 is a diagram showing the case where the tip of the controller is pointed upward for 3D operations. FIG. 10 is a diagram for explaining the relationship between the inner part when the tip of the controller according to the embodiment is pointed upward.1 is a diagram showing the relationship between a distance from a plane measured by a distance sensor of a controller according to an embodiment and an operation. FIG. 2 is a diagram showing a first example of switching an operation screen according to an embodiment. FIG. 3 is a diagram showing a second example of switching an operation screen according to an embodiment. FIG. 4 is a diagram showing a first example of switching button operations of a controller according to an embodiment between 2D operation (normal mouse operation) and 6DoF operation (air operation). FIG. 5 is a diagram showing a second example of switching button operations of a controller according to an embodiment between 2D operation (normal mouse operation) and 6DoF operation (air operation). FIG. 6 is a diagram showing a first pattern in which the display screen is a real monitor when a controller according to an embodiment is "operated in 2D". FIG. 7 is a diagram showing a second pattern in which the display screen is a virtual screen when a controller according to an embodiment is "operated in 2D". FIG. 8 is a diagram showing a third pattern in which the display screen is a virtual screen and a 3D displayed object when a controller according to an embodiment is "operated in 2D". FIG. 9 is a diagram showing a fourth pattern in which the display screen is a virtual screen and a 3D displayed object or a 3D displayed object when a controller according to an embodiment is "operated in 6DoF". FIG. 10 is a diagram showing the connection relationship between a controller and an HMD according to an embodiment. FIG. 1 is a diagram showing a state in which a controller according to an embodiment is connected to an HMD via a PC. FIG. 2 is a functional block diagram of a controller according to an embodiment. FIG. 3 is a flowchart for explaining operations of a mouse function unit and a 6DoF function unit in response to a switching signal of a controller according to an embodiment. FIG. 4 is a flowchart for explaining operations of a mouse function unit and a 6DoF function unit in response to a switching signal of a controller 6 according to an embodiment. FIG. 5 is a flowchart for explaining a first example of operations in response to a switching signal of an HMD according to an embodiment. FIG. 6 is a flowchart for explaining a second example of operations in response to a switching signal of an HMD according to an embodiment. FIG. 7 is a flowchart for explaining a third example of operations in response to a switching signal of an HMD according to an embodiment. FIG. 8 is a flowchart for explaining a fourth example of operations in response to a flip-down operation of an HMD according to an embodiment. FIG. 9 is a diagram showing a state in which a pen-type mouse-shaped controller according to a modified example of the shape of the controller according to an embodiment is in contact with the ground.1 is a diagram showing a state in which a pen-type mouse-shaped controller according to a modified example of the shape of the controller in accordance with the embodiment is lifted up; FIG. 2 is a diagram showing a configuration of a controller according to a modified example of the embodiment; FIG. 3 is a diagram showing a state in which a grip-type mouse-shaped controller according to a modified example of the shape of the controller in accordance with the embodiment is placed on the ground; FIG. 4 is a diagram showing a state in which a grip-type mouse-shaped controller according to a modified example of the shape of the controller in accordance with the embodiment is lifted up; FIG. 5 is a diagram showing a configuration of a controller according to a modified example of the embodiment; FIG. 6 is a diagram showing a state in which a bright spot tracking mouse-shaped controller according to a modified example of the shape of the controller in accordance with the embodiment is placed on the ground; FIG. 7 is a diagram showing a state in which a bright spot tracking mouse-shaped controller according to a modified example of the shape of the controller in accordance with the embodiment is lifted up; FIG. 8 is a diagram showing a state in which a bright spot tracking mouse-shaped controller according to a modified example of the shape of the controller in accordance with the embodiment is placed on the ground; FIG. 9 is a diagram showing a state in which a bright spot tracking mouse-shaped controller according to a modified example of the shape of the controller in accordance with the embodiment is covered by a hand in a different manner; FIG. 10 is a hardware configuration diagram showing an example of a computer that realizes an information processing device for an HMD according to the embodiment.

[0012] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted. The description will be given in the following order.

[0013] 1. Image of the coexistence of 2D and 3D displays displayed on a VR HMD 2. Definition of a mouse 3. Main 6DoF operations 4. Means of 6DoF detection 5. Example of a controller 6 integrating a mouse structure and a 6DoF detection mechanism 6. Structural requirements of the controller 6 according to the embodiment 6-1. First feature of the controller 6 according to the embodiment 6-2. Second feature of the controller 6 6-3. Third feature of the controller 6 6-4. Fourth feature of the controller 6 6-5. Fifth feature of the controller 6 7. Switching between 2D display and 3D display according to the embodiment 8. Functional block diagram 8-1. Connection configuration between the controller 6 and the HMD 5 8-2. Functional block diagram of the controller 6 according to the embodiment 8-3. Functional block diagram of the HMD 5 according to the embodiment 9. Explanation of operation 9-1. Operation of the controller 6 according to the embodiment 9-2. Operation of the HMD 5 according to the embodiment 9-2-1. 9. First Operation of HMD 5 According to Embodiment 9-2-2. Second Operation of HMD 5 According to Embodiment 9-2-3. Third Operation of HMD 5 According to Embodiment 9-2-4. Operation in Response to Flip-Down Operation of HMD 5 According to Embodiment 10. Modified Examples of the Embodiment 10-1. Shape of Controller 6 10-1-1. Pen-Type Mouse 10-1-1-1. Third Feature of Controller 306 According to Embodiment 10-1-2. Grip-Type Mouse 10-1-2-1. Third Feature of Controller 406 According to Embodiment 10-2. Shape of Controller 6 (In Case of Bright Spot Tracking Method) 10-3. Example of Switching Between 2D Operation and 6DoF Operation in Case of Other Controller Structures 11. Hardware Configuration

[0014] 1. Coexistence of 2D and 3D displays displayed on a VR HMD> Figure 1 is a diagram for explaining an image of 2D and 3D displays in design using a VR HMD according to an embodiment. Figure 1 shows an example in which a virtual screen 1 (virtual monitor) and a 3D display object 3, which is a camera to be reviewed, are displayed on the VR HMD.

[0015] A 2D representation of an object 2 of a camera, which is the design target, is displayed on a virtual screen 1. In Fig. 1, an example is shown in which a 3D representation of an object 3 is displayed in front of the 2D representation of the object 2.

[0016] For the object 2 displayed in 2D, mouse operation is preferable due to the high accuracy of position detection and ease of position retention. For the object 3 displayed in 3D, 6DoF operation is preferable due to the intuitiveness of operation and ease of spatial indication.

[0017] In the embodiment, when the VR HMD is flipped up instead of the virtual screen 1, the user can also view a real monitor.

[0018] 2. Definition of Mouse In the embodiment, a mouse is a controller having the following configuration, and normal mouse operation is called "2D operation."

[0019] 2-1. It is configured to be operated on a flat surface such as a desk. 2-2. It has a sensor that detects two-dimensional relative movement with respect to the flat surface. 2-3. It is configured to maintain its position on the flat surface even without operation. 2-4. It has an operation button.

[0020] <3. Main 6DoF Operations> Next, the main 6DoF operations will be described. The main 6DoF operations include moving the object 3 displayed in 3D, changing its posture (directly grabbing or releasing, etc.), specifying a point in three dimensions (3D modeling, wiring work, etc.), and pointing at an object arranged in space (ray control).

[0021] 2 is a diagram showing the operation of moving and changing the posture of a 3D-displayed object 3 using the controller 6 of the embodiment. In Fig. 2, the user wearing the HMD 5 performs the operation of "grabbing and releasing" the 3D-displayed object 3 using a button (not shown) on the controller 6 as a trigger.

[0022] 3 is a diagram showing the operation of specifying a three-dimensional point on a 3D display object 3, which is a car, using the controller 6 of the embodiment. In this example, the user wearing the HMD 5 performs a "line drawing" operation of drawing a line 7 on the 3D display object 3 using a button (not shown) on the controller 6 as a line drawing trigger.

[0023] 4 is a diagram showing the operation of pointing (controlling Ray) at a 3D displayed object 3 (not shown) using the controller 6 of the embodiment. In Fig. 4, the user wearing the HMD 5 performs the operation of using a button (not shown) as a trigger to "grab" or "release" the 3D displayed object 3 shown by Ray 8.

[0024] <4. 6DoF Detection Means> 6DoF detection means include a bright spot marker method, a laser light receiving method, a SLAM (Simultaneous Localization and Mapping) camera method, a magnetic tracking method, and an ultrasonic tracking method.

[0025] Examples of the bright point marker system include the Oculus Touch controller, examples of the laser light receiving system include the Vive controller, examples of the magnetic tracking system include Magic Leap One, and examples of the ultrasonic tracking system include Magic Leap One.

[0026] 5. Example of a controller 6 integrating a mouse structure and a 6DoF detection mechanism A controller 6 according to an embodiment integrating a mouse structure and a 6DoF detection mechanism has a generally V-shaped configuration with the sides curved upward. Fig. 5 is a rear view of the controller 6 according to the embodiment, and Fig. 6 is a side view of the controller 6 according to the embodiment. Fig. 7 is a diagram illustrating the controller 6 according to the embodiment in use as a normal mouse.

[0027] Operation buttons are provided on the top of the controller 6. When the controller 6 is operated with the right hand, as shown in Fig. 6, a left button 41 operated with the index finger and a right button (not shown) operated with the middle finger are provided on the top of the controller 6. A thumb button 42 operated with the thumb is also provided on the left side of the controller 6. Note that the left button 41 and thumb button 42 are not shown in Fig. 7.

[0028] As shown in FIG. 6, the upper and lower portions of the controller 6 that come into contact with the flat surface 9 are formed parallel to the flat surface 9 .

[0029] The controller 6 includes a movement detection unit 43 such as an optical sensor that outputs movement information relative to the plane 9, and a mouse function unit 62 (not shown) that outputs coordinate information on the plane 9 based on the movement information output from the movement detection unit.

[0030] The controller 6 satisfies the requirements of a mouse explained in <2. Definition of a Mouse>. The upper and lower portions of the controller 6 that come into contact with the flat surface 9 are formed parallel to the flat surface, a left button 41 and a right button are provided on the upper portion of the controller 6, and the controller 6 is configured to be operated on the flat surface 9. Therefore, the requirement of <2-1. Configured to be operated on a flat surface such as a desk> is met.

[0031] The controller 6 has a movement detection unit 43 such as an optical sensor, and therefore satisfies the requirement of <2-2. Having a sensor that detects two-dimensional relative movement with respect to a plane>.

[0032] The upper and lower portions of the controller 6 that come into contact with the flat surface 9 are formed parallel to the horizontal direction, and therefore satisfy the requirement of <2-3. Configured to maintain its position on the flat surface even when no operation is performed>.

[0033] A left button 41 and a right button are provided on the top of the controller 6, and a thumb button 42 is provided on the left side of the controller 6. Therefore, the requirement of <2-4. Having operation buttons> is met.

[0034] As shown in FIG. 8, the controller 6 may have the appearance of a mouse-type controller 106 incorporating a magnetic tracking mechanism similar to that of a normal mouse.

[0035] 9 is a diagram showing the appearance of a controller 206 equipped with a bright spot tracking mechanism according to an embodiment. The appearance of the controller 6 may be that of the controller 206 equipped with a bright spot tracking mechanism, as shown in Fig. 9. As shown in Fig. 9, a circular ring 107 for arranging bright spots is provided at the tip of the controller 206.

[0036] 6. Structural Requirements of Controller 6 According to the Embodiment Next, structural requirements for switching between 2D operation and 6DoF operation (3D operation) of controller 6 according to the embodiment will be described.

[0037] The controller 6 of the embodiment can freely and easily switch between "2D operation" using a mouse on a flat surface and "6DoF operation" by lifting the controller 6 in the air with little strain. Figure 10 is a diagram for explaining switching between 2D operation and 6DoF operation of the controller 206 equipped with the bright spot tracking mechanism according to the embodiment.

[0038] 10, 2D operation is possible when the controller 206 is placed on a flat surface, and 6DoF operation is possible when the controller 206 is lifted from the flat surface. A structure that allows the controller 206 to be lifted with a low load will be described later.

[0039] In addition to a horizontal surface such as a desk, a wall, the foot (around the knee), the surface of the left hand resting on the surface, etc. may also be recognized as a "plane" and operations may be switched accordingly.

[0040] <6-1. First feature of controller 6> Fig. 11 is a diagram illustrating the fingers used when operating the controller 6 according to this embodiment. When using the fingers as shown in Fig. 12, the thumb, ring finger, and little finger of the right hand move the controller 6 in 2D operation and grip the controller 6 in 6DoF operation. The index finger and middle finger of the right hand are used to operate the operation buttons on the controller 6.

[0041] Fig. 12 is a diagram showing the back of the controller 6 when the controller 6 according to the embodiment is held in the same manner as when operating a normal mouse. Fig. 13 is a diagram showing the side of the controller 6 when the controller 6 according to the embodiment is held in the same manner as when operating a normal mouse. Fig. 14 is a diagram showing a case where 6DoF operations are being performed with the controller 6 when the controller 6 according to the embodiment is held in the same manner as when operating a normal mouse.

[0042] 15 is a front view of the controller 6 according to the embodiment. As shown in FIG. 15 , the controller 6 according to the embodiment has a shape that protrudes outward above the operator's thumb and ring finger / little finger when the controller 6 is used for normal mouse operation (2D operation).

[0043] According to the embodiment, because the controller 6 has such a shape, when lifting the controller 6, it is possible to lift the controller 6 naturally with a low load without applying force with the fingers in the direction of grasping the controller 6.

[0044] Specifically, the controller 6 has a first side 12 and a second side 13 opposite the first side 12, and the first side 12 has a third side 14 and a fourth side 15 formed on the top of the third side 14 when the controller 6 is viewed from the front, and formed further outward from the controller 6 than the third side 14 and continuing from the third side 14.

[0045] The second side 13 has a fifth side 16 and a sixth side 17 that is continuous with the fifth side 16 and is formed at the top of the fifth side 16 and further outside the controller 6 than the fifth side 16 when the controller 6 is viewed from the front.

[0046] <6-2. Second Feature of Controller 6> A second feature of the controller 6 according to this embodiment is that, when using the controller 6 in a normal mouse operation (2D operation), the center of gravity 21 of the controller 6 is located within a triangle connecting the contact points of the operator's thumb and ring finger / little finger with the controller 6. Fig. 16 is a diagram showing the position of the center of gravity 21 of the controller 6 according to this embodiment.

[0047] 16 , the center of gravity 21 is located at the top of the generally V-shaped controller 6, between the first side surface 12 and the second side surface 13. Specifically, the center of gravity 21 is located between the fourth side surface 15 of the first side surface 12 and the sixth side surface 17 of the second side surface 13.

[0048] By positioning the center of gravity 21 of the controller 6 in this manner, when lifting the controller 6, it becomes possible to lift the controller 6 naturally with little load and without tilting the controller 6, without applying force with the fingers in the direction of grasping the controller 6.

[0049] <6-3. Third feature of controller 6> The controller 6 according to the embodiment has a structure for preventing the controller 6 from moving longitudinally within the palm of the hand due to the influence of gravity when performing 6 DoF operations with the controller 6.

[0050] FIG. 17 is a diagram illustrating the structure of a controller 6 according to an embodiment. As shown in FIG. 17 , the controller 6 has a roughly dogleg shape. The center of gravity 21 of the controller 6 is located in the center of the interior of the controller 6. The controller 6 has an inner portion 25 near the center of gravity 21, an inner portion 26 adjacent to the inner portion 25 and in the center, an inner portion 27 at the lower portion of the controller 6, and an upper portion 28 at the lower portion of the controller 6. Below, it will be described how the inner portions 25, 26, 27, and upper portion 28 are used to prevent the controller 6 from moving.

[0051] Mouse operation is performed by holding the controller 6 as shown in Fig. 5. Fig. 18 is a diagram for explaining the behavior of the controller 6 due to gravity when the controller 6 according to the embodiment is lifted. As shown in Fig. 18, when the controller 6 is lifted to perform 3D operation, gravity causes the controller 6 to fall.

[0052] 19 is a diagram illustrating the relationship between the inner portion 26 and the upper portion 28 when the controller 6 according to this embodiment is lifted. As shown in FIG. 19 , when the controller 6 according to this embodiment is lifted for 3D operation, the inner portion 26 mainly comes into contact with the side of the thumb, generating a reaction force that prevents the controller 6 from falling due to gravity. Furthermore, the controller 6 tends to rotate due to gravity acting on the center of gravity 21, but the rotation is prevented by the palm of the hand coming into contact with the upper portion 28, and the position of the controller 6 in the hand is maintained without having to increase the grip strength of the hand or change the position of the hand.

[0053] 20 is a diagram showing the state when the tip of the controller 6 is pointed downward for 3D operation. As shown in FIG. 20, when the tip of the controller 6 is pointed downward, the controller 6 tends to fall due to gravity.

[0054] 21 is a diagram illustrating the relationship between the inner portion 27 and the upper portion 28 when the tip of the controller 6 according to the embodiment is pointed downward. As shown in FIG. 21 , when pointing the mouse selection downward during 3D operation, the controller 6 according to the embodiment prevents the mouse from falling due to gravity by mainly contacting the inner portion 27 with the ring finger and little finger, generating a reaction force that prevents the mouse from falling. Furthermore, contact between the palm of the hand and the upper portion 28 prevents rotation, and the position of the controller 6 relative to the hand is maintained without having to strengthen the grip or change the position of the hand.

[0055] 22 is a diagram showing the state when the tip of the controller 6 is pointed upward for 3D operation. As shown in FIG. 22, when the tip of the controller 6 is pointed upward, the controller 6 tends to fall due to gravity.

[0056] 23 is a diagram illustrating the relationship between the controller 6 according to the embodiment and the inner portion 25 when the tip of the controller 6 is pointed upward. As shown in FIG. 23 , when the controller 6 according to the embodiment is pointed downward during 3D operation, the controller 6 tends to fall due to gravity, but the inner portion 25 mainly comes into contact with the thumb, generating a reaction force that prevents the controller 6 from falling. Furthermore, since the inner portion 25 is located below the center of gravity 21, no rotational moment is generated. In this way, the position of the controller 6 in the hand is maintained without having to increase the grip strength or change the hand position.

[0057] With this structure, the user can maintain the position of the controller 6 in the palm of his / her hand without applying force to grip the controller 6.

[0058] <6-4. Fourth feature of controller 6> The controller 6 according to the embodiment has a function of determining, by the built-in distance sensor 31, whether a 2D operation or a 6DoF operation is being performed, and outputting a switching signal for switching between a display mode for a 2D image to which the coordinate information output from the mouse function unit 62 is applied and a display mode for a 3D image to which the position and orientation information output from the 6DoF function unit 63 is applied.

[0059] 24 is a diagram showing the relationship between the distance from the plane measured by the distance sensor 31 of the controller 6 according to the embodiment and the operation. As shown in Fig. 24, the controller 6 performs a 6DoF operation when the distance Dm from the plane 9 measured by the distance sensor 31 is greater than the distance threshold Dt (Dm>Dt), and performs a 2D operation when the distance Dm is equal to or less than the distance threshold Dt (Dm≦Dt). The controller 6 outputs a switching signal when switching from the 2D operation to the 6DoF operation.

[0060] The distance threshold Dt is set to a value greater than the distance Dm that occurs when the controller 6 is lifted up during normal mouse operation.

[0061] The controller 6 may determine that the controller 6 is not in a horizontal position by an IMU (Inertial Measurement Unit), and output a switching signal for switching the operation screen.

[0062] A pressure sensor may be placed on the side of the controller 6, and the lifting operation may be detected from the change in the side pressure that increases when the controller 6 is lifted, and a switching signal may be output.

[0063] Alternatively, it may be possible to determine whether the controller 6 is on the work plane based on the position information of the 6DoF controller 6 and the position information of the work plane obtained by a camera or the like mounted on the HMD 5, and output a switching signal.

[0064] Alternatively, if the movement detection unit 43 (for example, in FIG. 17) for outputting movement information relative to a plane is an optical sensor type, the output information of this optical sensor may be used to determine whether the controller 6 is on the working plane or not, and a switching signal may be output.

[0065] The above explanation has been given of an example in which the controller 6 outputs a switching signal, but the position information obtained from the controller 6 may be received on the main body (PC 52, HMD 5, etc.), the distance between the controller 6 and the plane on which the controller 6 is placed may be calculated from the received position information of the controller 6, and if the calculated distance is longer than a threshold distance, the switching process may be performed on the main body.

[0066] Furthermore, a camera on the main unit acquires the light emission position of an infrared light-emitting diode (LED) on the controller 6, and the main unit calculates position information of the controller 6 from the light emission position of the controller 6 acquired from the camera. The main unit may then calculate the distance between the controller 6 and the plane on which the controller 6 is placed from the position information of the controller 6 calculated by the main unit, and if the calculated distance is longer than a threshold distance, the main unit may perform switching processing.

[0067] It may also be possible to make it possible to perform 2D operations on the XY plane in the air by holding the controller 6 in the normal position (parallel to the plane 9) in the air for n seconds. At this time, a virtual plane is presented in the 3D virtual space. When operating on the virtual plane, haptic feedback is used to notify the user that the operation is being performed on a plane.

[0068] At the moment when the controller 6 lands or is lifted, the position of the 6DoF operation / 2D operation will be shaken, reducing operability, so the 6DoF operation / 2D operation function may be turned off during that period.

[0069] When returning from 6DoF operation to 2D operation, the initial position of the controller 6 is often lost, so gaze detection may be performed and a mouse cursor may be displayed at the gaze position on the HMD 5. Alternatively, when returning to 2D operation, the mouse cursor may be highlighted on the HMD 5 by enlarging it, for example.

[0070] The controller 6 itself may not have the above-described function of detecting switching, but the hand operating the controller 6 may be detected by hand tracking using the HMD 5 or the like, and the above-described function switching may be performed based on the position of the hand.

[0071] The status of 2D operation / 6DoF operation may be indicated by an LED display or the like on the controller 6 or HMD 5.

[0072] 25 is a diagram showing a first example of switching of the operation screen according to the embodiment. As shown in Fig. 25, in the first example of switching, the virtual screen 1 is displayed during 2D operation, and only the object 3 in 3D display is displayed during 6DoF operation. In other words, in the first example of switching, the 3D display is turned ON only in the 6DoF operation mode.

[0073] 26 is a diagram showing a second example of switching of the operation screen according to the embodiment. As shown in FIG. 26, in the second example of switching, a virtual screen 1 is displayed during 2D operation, and the virtual screen 1 and a 3D display object 3 are displayed during 6DoF operation. The 3D display object 3 is displayed in front of the virtual screen 1.

[0074] In the above explanation, a method for switching the display method in accordance with the state of the controller 6 has been described, but it is also possible to switch the method for reflecting operations on the controller 6 in the system without switching the display method. For example, the following implementation is possible.

[0075] In both 2D operation and 6DoF operation, the display state shows both a 2D virtual screen and a 3D solid object, as shown in the right diagram of Fig. 26. In contrast to this state, during 2D operation, movement operations and button operations of the controller 6 are reflected as operations on the 2D virtual screen (for example, movement operations of the controller move the mouse cursor, and button operations of the controller perform a confirm operation on the cursor).

[0076] On the other hand, during 6DoF operation, the movement and button operations of the controller 6 are reflected as operations on a 3D solid object (for example, as shown in the right diagram of Figure 10, movement of the controller 6 operates Ray 8, which controls the solid object, and button operation on the controller performs a confirmation operation on the Ray. Alternatively, button operation on the controller performs an operation to grab a solid object as shown in 3 in the right diagram of Figure 10, and movement of the controller moves the grabbed solid object).

[0077] As described above, it is conceivable to implement an implementation in which the target on which the button operations of the controller are reflected changes by switching the method of reflecting the operations on the controller in the system without switching the display method.

[0078] Note that the above is based on the assumption that one controller 6 is used for both 2D mouse operation and 6DoF operation, but as an alternative configuration, even when a 6DoF controller 6 separate from the mouse is used, it is also possible to automatically switch the display mode by detecting when the 6DoF controller 6 is lifted from a resting state. In this case, 2D operation and 6DoF operation may be mutually exclusive, and 2D operation using the mouse may be disabled (moving the mouse will not function) during operation using the 6DoF controller 6.

[0079] It is also possible that a user may operate the mouse with the same hand while holding the 6DoF controller 6, which is separate from the mouse. In such a case, the system may detect that a mouse operation has been performed from the output information of the mouse, and switch the state in which the operation of the 6DoF controller 6 is reflected in the system, without performing the above-described disabling process. Specifically, the Ray that is displayed when the 6DoF controller 6 is being operated may be turned off when a mouse operation is performed.

[0080] Furthermore, in the above description, it is assumed that a functional component, such as the movement detection unit 43, such as an optical sensor shown in FIG. 17, is mounted as a separate component from a component for detecting the 6DoF position and orientation of the controller in order to output movement information relative to a plane. On the other hand, movement information of the controller 6 on a plane can also be obtained from position and orientation information generated by the 6DoF function of the controller 6. Therefore, it is possible to perform the above-described display and function switching control without providing a separate mouse functional component, such as the movement detection unit 43. Specifically, for example, it is conceivable to detect that the controller 6 is positioned on a plane based on planar information obtained by a camera on an HMD and 6DoF position information of the controller, and in that case, switch control to 2D operation mode, and, for example, move a mouse cursor on a 2D display using the 6DoF position information.

[0081] Furthermore, in such an implementation, if the controller 6 is a grip-type controller, suitable operation can be achieved by providing a flat surface (regardless of whether or not the movement detection unit 43 is present) as shown in Figures 44 and 47 so that the controller 6 can stand upright without tipping over when placed on a flat surface.

[0082] <6-5. Fifth Feature of Controller 6> The controller 6 according to the embodiment switches the operation of the buttons on the controller 6 between 2D operation (normal mouse operation) and 6DoF operation (air operation).

[0083] FIG. 27 is a diagram showing a first example of switching the button operation of the controller 6 according to the embodiment between the case of 2D operation (normal mouse operation) and the case of 6DoF operation (air operation).

[0084] 27 , during 2D operation, the main selection operation, which is the function of the left button 41 (equivalent to the left button on a normal mouse) of the controller 6, is assigned. During 6DoF operation, the main selection operation, which is the function of the left button, is assigned to the thumb button 42 of the controller 6.

[0085] In the controller 6 according to the embodiment, when the thumb button 42 is pressed with the thumb, the reaction force can be received by the ring finger and little finger in the air, and therefore the thumb button 42 is used as the primary selection operation (the left click operation on a normal mouse) during 6DoF operation.

[0086] FIG. 28 is a diagram showing a second example of switching the button operation of the controller 6 according to the embodiment between the case of 2D operation (normal mouse operation) and the case of 6DoF operation (air operation).

[0087] As shown in FIG. 28, in 2D operation, left and right clicks are assigned as in a normal mouse, but in 6DoF operation, clicking when there is no ground requires reaction support from the finger on the opposite side, making operation difficult. Therefore, instead of "clicking," the button is turned on by "touching" the area where the finger is placed (for example, the left button 41).

[0088] 7. Switching Between 2D Display and 3D Display According to the Embodiment The following patterns shown in FIGS. 29 to 32 are possible as patterns of operations on the controller 6 and displays visible to the user.

[0089] 29 is a diagram showing a first pattern in which the display screen is a real monitor when the controller 6 according to the embodiment is operated in 2D. In this first pattern, the user flips up the HMD 5 and views the screen displayed on the real monitor 51.

[0090] 30 is a diagram showing a second pattern in which the display screen is the virtual screen 1 when the controller 6 according to the embodiment is operated in 2D. In this second pattern, the user views the virtual screen 1 displayed on the HMD 5.

[0091] 31 is a diagram showing a third pattern in which, when the controller 6 according to the embodiment is operated in 2D, the display screen shows the virtual screen 1 and the 3D display of the object 3. In this third pattern, the user sees the virtual screen 1 and the 3D display of the object 3 displayed on the HMD 5.

[0092] Figure 32 is a diagram showing a fourth pattern in which, when the controller 6 in the embodiment is operated in a "6DoF manner," the display screen displayed on the HMD 5 is a virtual screen 1 and a 3D display object 3 or a 3D display object 3.

[0093] As a method of switching the display, the display switching from Fig. 30 to Fig. 32 or the display switching from Fig. 31 to Fig. 32 may be performed automatically by detecting the operation of lifting the controller 6 having the 6DoF function according to the embodiment (or lifting a 6DoF controller separate from the mouse) (also described in "<6-4. Fourth feature of controller 6>"). Furthermore, the display switching from Fig. 52 to Fig. 50 and the reverse display switching from Fig. 52 to Fig. 51 may be performed automatically by detecting the operation of placing the controller 6 on a desk.

[0094] The display switching from Fig. 29 to Fig. 30, from Fig. 29 to Fig. 31, and from Fig. 29 to Fig. 32 may be performed automatically by detecting a flip-down operation of the HMD 5 having a flip-up structure. Also, the display switching in the reverse direction from Fig. 50 to Fig. 49, from Fig. 51 to Fig. 49, and from Fig. 52 to Fig. 49 may be performed automatically by detecting a flip-up operation.

[0095] The display switching from Fig. 29 to Fig. 30, from Fig. 29 to Fig. 31, and from Fig. 29 to Fig. 32 may be performed by a UI operation (for example, selecting and confirming the "VR mode" button) using a GUI operation on the real monitor 51. Similarly, the display switching in the reverse direction, from Fig. 50 to Fig. 49, from Fig. 51 to Fig. 49, and from Fig. 52 to Fig. 49, may be performed by operating the controller 6 (or using a normal mouse or a 6DoF mouse) on the GUI displayed on the HMD 5.

[0096] The display switching may be performed by voice input by the user, or by operating buttons on the controller 6, a mouse, or a keyboard.

[0097] As a display method for the virtual screen 1, when the virtual screen 1 is displayed in the VR space, the position, orientation, size, etc. may be matched to those of the real monitor 51. The virtual screen 1 may also be displayed with the same image quality, cursor position, and display content as the real monitor 51. This makes it possible to reduce the operational load and cognitive load on the user associated with switching the display.

[0098] 30, 31, and 32, whether to generate the virtual screen 1 and at what position to display it may be switched based on the type and size of the 3D display object 3. For example, when the size of the display object is large, such as the inside of a car, the 2D monitor screen may be covered by the 3D display object 3, and in that case the virtual screen 1 is not displayed. Also, for example, when the display area of ​​the 3D display object 3 overlaps with the virtual screen 1 due to a user operation, the display of the virtual screen 1 may be turned ON / OFF or the display form (transparency, etc.) may be switched.

[0099] <8. Functional Block Diagram> <8-1. Connection Configuration Between Controller 6 and HMD 5> Fig. 33 is a diagram showing the connection relationship between the controller 6 and HMD 5 according to the embodiment. In this case, output signals including switching signals etc. from the controller 6 are input to the HMD 5. The HMD 5 performs processing such as switching the display mode of the display screen based on the output signals output from the controller 6.

[0100] As shown in FIG. 33, the controller 6 is directly connected to the HMD 5, but the controller 6 may also be connected to the HMD 5 via a PC (Personal Computer) 52 as shown in FIG.

[0101] In this case, output signals including a switching signal and the like from the controller 6 are input to the PC 54. The signals exchanged between the PC 52 and the HMD 5 are determined by the role of the PC 52 and the role of the HMD 5.

[0102] The PC 52 performs processing such as switching the display mode of the display screen based on an output signal output from the controller 6. The PC 52 also performs processing such as outputting a video signal to be displayed to the HMD 5.

[0103] In the embodiment, a case where the controller 6 and the HMD 5 are directly connected will be described.

[0104] <8-2. Functional Block Diagram of Controller 6 According to the Embodiment> Fig. 35 is a functional block diagram of the controller 6 according to the embodiment. As shown in Fig. 35, the controller 6 has a distance sensor unit 61, a mouse function unit 62, a 6DoF function unit 63, a button operator switching unit 64, a button unit 65, and a communication control unit 66.

[0105] The distance sensor unit 61 measures the distance from the plane 9 on which the controller 6 is placed. If the measured distance is longer than a threshold distance, the distance sensor unit 61 outputs a switching signal to the communication control unit 66. The switching signal is a signal for switching the display mode between a two-dimensional image to which the coordinate information output from the mouse function unit 62 is applied and a three-dimensional image to which the position and orientation information output from the 6DoF function unit 63 is applied.

[0106] The mouse function unit 62 outputs coordinate information on the plane 9 on which the controller 6 is placed to the communication control unit 66. While the communication control unit 66 receives a switching signal from the distance sensor unit 61, the mouse function unit 62 is turned off and does not output coordinate information to the communication control unit 66, and when the communication control unit 66 does not receive a switching signal, the mouse function unit 62 is turned on and outputs coordinate information to the communication control unit 66.

[0107] The 6DoF function unit 63 outputs the position and orientation information of the controller 6 to the communication control unit 66. While the 6DoF function unit 63 receives a switching signal from the distance sensor unit 61, it is turned on and outputs the position and orientation information to the communication control unit 66, and when the 6DoF function unit 63 does not receive a switching signal, it is turned off and does not output the position and orientation information to the communication control unit 66.

[0108] When the button operator switching unit 64 receives a switching signal from the distance sensor unit 61, it switches the function of operation in a two-dimensional image assigned to the button unit 65 to the function of operation in a three-dimensional image. When the button operator switching unit 64 does not receive a switching signal, it assigns the function of operation in a two-dimensional image assigned to the button unit 65 to the button unit 65, and when it receives a switching signal, it assigns the function of operation in a three-dimensional image to the button unit 65.

[0109] Furthermore, the button operator switching unit 64 switches the selection operation of the button unit 65. For example, when a switching signal is not received, a 2D operation is being performed, and the button unit 65 is selected by clicking the button unit 65. When a switching signal is received, a 6DoF operation is being performed, and the button unit 65 is selected by touching the button unit 65.

[0110] The button section 65 has a plurality of buttons. The buttons include, for example, a left button 41 and a right button (not shown) of the controller 6, and a thumb button 42 provided on the side of the controller 6. When a switching signal is output from the button section 65, i.e., when a function for operating in a three-dimensional image is assigned, the button section 65 outputs a signal indicating the selected button to the communication control section 66. When a switching signal is not output from the button section 65, i.e., when a function for operating in a two-dimensional image is assigned, the button section 65 outputs a signal indicating the selected button to the communication control section 66.

[0111] The communication control unit 66 outputs the switching signal output from the distance sensor unit 61, the coordinate information output from the mouse function unit 62, the position coordinate information output from the 6DoF function unit 63, and the signal indicating the selected button output from the button unit 65 to the HMD 5, which is an information processing device.

[0112] In addition, since the HMD 5 can recognize the 2D operation or 6DoF operation of the controller 6 using the switching signal, all signals (coordinate information output from the mouse function unit 62, position coordinate information output from the 6DoF function unit 63, and a signal indicating the selected button output from the button unit 65) can be output from the controller 6, and these signals can be selected by the HMD 5.

[0113] As shown in FIG. 34, when the controller 6 is connected to the HMD via the PC 52, the communication control unit 66 outputs a switching signal and the like to the PC 52.

[0114] 36 is a functional block diagram of the HMD 5 according to the embodiment. As shown in FIG. 36, the HMD 5 has a receiving unit 71, a display mode switching unit 72, an image information processing unit 73, a flip-up detection unit 74, and a display unit 75.

[0115] The receiving unit 71 receives an output signal including a switching signal, coordinate information, position coordinate information, and a signal indicating a selected button output from the communication control unit 66 of the controller 6 .

[0116] The image information processing unit 73 stores the two-dimensional images and three-dimensional images to be displayed on the display unit 75, and outputs the two-dimensional images and / or three-dimensional images to the display mode switching unit 72 based on the output signal. The image information processing unit 73 also performs processing corresponding to the coordinate information, position coordinate information, and signal indicating the selected button of the output signal from the receiving unit 71. The image information processing unit 73 may be provided separately from the HMD 5.

[0117] The display mode switching unit 72 switches from a first display mode displaying a two-dimensional image to a second display mode displaying a three-dimensional image based on the received switching signal. If the switching signal is not received and a 2D operation is being performed, the display mode switching unit 72 switches from the second display mode displaying a three-dimensional image to the first display mode displaying a two-dimensional image.

[0118] The HMD 5 may switch from a first display mode displaying a two-dimensional image to a second display mode displaying a three-dimensional image when distance data from the plane on which the controller is placed, received from the controller 6, is longer than a threshold distance. Furthermore, the PC 52 may output a switching signal to the HMD 5 when distance data from the plane on which the controller is placed, received from the controller 6, is longer than a threshold distance.

[0119] The display mode switching unit 72 outputs the two-dimensional image and the three-dimensional image output from the image information processing unit 73 to the display unit 75 in accordance with the display mode. For example, in the first display mode, the display mode switching unit 72 outputs the two-dimensional image to the display unit 75. In the first display mode, the display mode switching unit 72 may output the two-dimensional image and the three-dimensional image to the display unit 75. In the second display mode, the display mode switching unit 72 outputs the three-dimensional image to the display unit 75. In the second display mode, the display mode switching unit 72 may output the two-dimensional image and the three-dimensional image to the display unit 75.

[0120] The flip-up detection unit 74 detects the flip-up of the HMD 5 and outputs the result to the display mode switching unit 72. If the flip-up detection unit 74 does not detect the flip-up (in the case of the flip-down), the display mode switching unit 72 outputs an image according to the display mode.

[0121] The display unit 75 displays the image output from the display mode switching unit 72 .

[0122] 37 is a flowchart for explaining the operation of the mouse function unit 62 and the 6DoF function unit 63 in response to a switching signal of the controller 6 according to the embodiment. As shown in FIG. 37, the mouse function unit 62 and the 6DoF function unit 63 determine whether or not a switching signal has been received from the distance sensor unit 61 (step S1).

[0123] If it is determined that the switching signal has not been received (No in step S1), the mouse function unit 62 is turned on and the 6DoF function unit 63 is turned off (step S2). As a result, a signal for 2D operation is output from the mouse function unit 62. Next, the button operator switching unit 64 switches the function and selection operation of the button unit 65 to the function and selection operation in the 2D image (step S3), and the process ends.

[0124] On the other hand, if it is determined in step S1 that a switching signal has been received (Yes in step S1), the mouse function unit 62 is turned off and the 6DoF function unit 63 is turned on (step S4). As a result, a signal for 3D operation is output from the 6DoF function unit 63. Next, the button operator switching unit 64 switches the function and selection operation of the button unit 65 to the function and selection operation in the 3D image (step S5), and the process ends.

[0125] <9-2. Operation of HMD 5 According to Embodiment> <9-2-1. First Operation of HMD 5 According to Embodiment> Fig. 38 is a flowchart for explaining a first example of the operation of the HMD 5 according to the embodiment in response to a switching signal. As shown in Fig. 38, the HMD 5 determines whether or not a switching signal has been received from the controller 6 (step S11).

[0126] If it is determined that a switching signal has not been received from the controller 6 (No in step S11), a 2D image (virtual screen 1) is displayed (step S12). If a switching signal has been received from the controller 6 (Yes in step S11), a 3D image (3D display object 3) is displayed (step S13).

[0127] 39 is a flowchart for explaining a second example of the operation of the HMD 5 according to the embodiment in response to a switching signal. As shown in FIG. 39, the HMD 5 determines whether or not a switching signal has been received from the controller 6 (step S21).

[0128] If it is determined that a switching signal has not been received from the controller 6 (No in step S21), a 2D image (virtual screen 1) and a 3D image are displayed (step S22).If a switching signal has been received from the controller 6 (Yes in step S21), a 3D image (3D display object 3) is displayed (step S23).

[0129] 40 is a flowchart for explaining a third example of the operation of the HMD 5 according to the embodiment in response to a switching signal. As shown in FIG. 40, the HMD 5 determines whether or not a switching signal has been received from the controller 6 (step S31).

[0130] If it is determined that a switching signal has not been received from the controller 6 (No in step S31), a 2D image (virtual screen 1) is displayed (step S32). If a switching signal has been received from the controller 6 (Yes in step S31), a 2D image (virtual screen 1) and a 3D image (3D display object 3) are displayed (step S33).

[0131] 41 is a flowchart for explaining a fourth example of the operation of the HMD 5 in accordance with the embodiment in response to a flip-down operation. As shown in FIG. 41, the HMD 5 determines whether a flip-down operation has been performed (step S41).

[0132] In step S41, if a flip-up operation is performed without a flip-down operation (No in step S41), no processing is performed. On the other hand, if a flip-down operation is performed in step S41 (Yes in step S41), a 2D image and a 3D image, a 3D image, or a 2D image is displayed on the HMD 5 (step S42).

[0133] <10. Modifications of the embodiment> <10-1. Shape of the controller 6> <10-1-1. Pen-type mouse> Fig. 42 is a diagram showing a state in which a pen-type mouse-shaped controller 306 according to a modification of the shape of the controller 6 according to the embodiment is placed on the ground. Fig. 43 is a diagram showing a state in which a pen-type mouse-shaped controller 306 according to a modification of the shape of the controller 6 according to the embodiment is lifted up. Fig. 44 is a diagram showing the configuration of a controller 306 according to a modification of the embodiment.

[0134] 42 and 43, the grip portion of the pen-type mouse-shaped controller 306 is shaped like a pen, and has a flat portion 307 at its tip. The flat portion 307 has a shape that allows it to stably contact the ground surface.

[0135] As shown in FIG. 44, the planar portion 307 has a movement detection portion 43 (not shown) that detects a two-dimensional relative movement distance with respect to the ground surface.

[0136] Furthermore, the gripping portion of the controller 306 has operation buttons 40 arranged at positions that are touched by the index finger and thumb, and the buttons are operated with the index finger and thumb. When the controller 306 is in a state where it is in contact with the ground, mouse operation is mainly performed with the index finger, thumb, and middle finger. When the controller 306 is in a state where it is lifted, button operation is performed with the index finger and thumb, and the pen-type mouse-shaped controller 306 is held by the index finger, thumb, middle finger, and the base of the index finger.

[0137] The pen-type mouse-shaped controller 306 satisfies the requirements of a mouse explained in <2. Definition of a Mouse>. The flat part 307 moves together with the grip part during 2D operation, but during 6DoF operation, it may remain on the desk like a pen holder.

[0138] <10-1-1-1. Third feature of controller 306 according to the embodiment> The controller 306 according to the modified embodiment has a structure for preventing the controller 306 from moving longitudinally within the palm of the hand due to the influence of gravity when performing 6 DoF operations with the controller 306.

[0139] As shown in FIGS. 43 and 44, the controller 306 has side portions 44 on the sides of the controller 306 that come into contact with the bases of the thumb and index finger when the controller 306 is picked up for 3D operation.

[0140] Therefore, when the controller 306 is lifted for 3D operation, the side surface portion 44 mainly comes into contact with the base of the thumb and index finger, and a force is generated to resist the forces that tend to fall and rotate, which are generated when the reaction force from the plane is lost during the lifting action, and the mouse position relative to the hand is maintained without having to strengthen the grip of the hand or change the position of the hand.

[0141] <10-1-2. Grip-type mouse> Fig. 45 is a diagram showing a state in which a grip-type mouse-shaped controller 406 according to a modified example of the shape of the controller 6 in accordance with the embodiment is placed on the ground. Fig. 46 is a diagram showing a state in which a grip-type mouse-shaped controller 406 according to a modified example of the shape of the controller 6 in accordance with the embodiment is lifted up. Fig. 47 is a diagram showing the configuration of a controller 406 according to a modified example of the embodiment.

[0142] 45 and 46, a grip-type mouse-shaped controller 406 has a flat portion 407 at its bottom, which serves as a contact surface. The flat portion 407 has a shape that allows it to stably contact the contact surface. The flat portion 407 has a movement detection unit 43 (not shown) that detects the two-dimensional relative movement distance with respect to the contact surface.

[0143] 47, the controller 406 has operation buttons 40 arranged at positions that can be touched by the index finger and thumb, and the buttons are operated with the index finger and thumb. When the controller 406 is in a state where it is in contact with the ground, mouse operation is mainly performed with the wrist and the entire arm. When the controller 406 is in a state where it is held up, the buttons are operated with the index finger and thumb, and the controller 406 is held with the entire palm of the hand.

[0144] The controller 406 has a 6DoF position detection unit 46 such as an optical marker inside. The controller 406 has a hollow portion, and this hollow portion is a portion where the user's middle finger, ring finger, and little finger are placed.

[0145] The pen-type mouse-shaped controller 406 satisfies the requirements of a mouse explained in <2. Definition of a Mouse>. The flat portion 407 moves on the plane 9 together with the grip portion during 2D operation.

[0146] <10-1-2-1. Third feature of controller 406 according to the embodiment> The controller 406 according to the modified embodiment has a structure for preventing the controller 406 from moving longitudinally within the palm of the hand due to the influence of gravity when performing 6 DoF operations with the controller 406.

[0147] As shown in FIGS. 45 to 47, the controller 406 has an inner wall portion 45 that is in contact with the middle finger, ring finger, and little finger when the controller 406 is held up for 3D operation.

[0148] Therefore, when the controller 406 is lifted for 3D operation, the inner wall portion 45 mainly comes into contact with the middle finger, ring finger, and little finger of the hand, and a force is generated to resist the forces that tend to fall and rotate, which are generated when the reaction force from the plane is lost during the lifting action, and the mouse position relative to the hand is maintained without having to strengthen the gripping force of the hand or change the position of the hand.

[0149] As a 6DoF control means, for example, right hand hand tracking using the camera of the HMD 5 may be used, and operations equivalent to button operations on the controller 406 may be performed by keyboard operations with the left hand.

[0150] 10-2. Shape of Controller 6 (Bright Spot Tracking Method) Fig. 48 is a diagram showing a mouse-shaped controller 506 using a bright spot tracking method according to a modified example of the shape of the controller 6 in accordance with the embodiment, in a state where the controller is in contact with the ground. Fig. 49 is a diagram showing a state where the controller 506 is lifted up.

[0151] 48 and 49, the controller 506 has a ring 507 at its tip. An LED (Light Emitting Diode) that serves as a bright spot is arranged inside the ring 507.

[0152] Fig. 50 is a diagram showing a state in which a bright spot tracking type mouse-shaped controller 606 according to a modified example of the shape of the controller 6 in accordance with the embodiment is placed on the ground. Fig. 51 is a diagram showing a state in which the controller 606 is covered with a hand in a different manner.

[0153] 50 and 51 , controller 606 has a protrusion 611 on its side, between the part where the thumb and index finger come into contact. A plurality of bright dots 612 (e.g., LEDs) are arranged on protrusion 611. A plurality of bright dots 612 are also arranged in the left and right button areas on the top surface of mouse-shaped controller 606.

[0154] Therefore, with controller 606, there is a risk that the LED that becomes bright spot 612 may be hidden by the hand, but with controller 606, there is no risk that bright spot 612 may be hidden by the hand, as shown in Fig. 52. Therefore, bright spots can be detected robustly.

[0155] <10-3. Examples of Switching Between 2D Operation and 6DoF Operation in Case of Other Controller Structures> In the above-described embodiment, a case has been described in which both 2D operation and 6DoF operation are realized using the controller 6 or the like.

[0156] To switch between 2D operation and 6DoF operation, a wearable controller (ring) that can coexist with mouse operation can be provided, and the ring or hand tracking can detect when the hand is raised, allowing switching between 2D operation mode and 6DoF operation mode.

[0157] Alternatively, the right hand may be freehand tracked and the left hand may be used to operate the buttons, and the lifting of the hand may be detected by hand tracking to switch between the 2D operation mode and the 6DoF operation mode.

[0158] 11. Hardware Configuration FIG. 52 is a hardware configuration diagram showing an example of a computer that realizes the information processing device of the HMD 5 according to the embodiment.

[0159] A computer that realizes the information processing device according to each of the above-described embodiments is realized, for example, by a computer 1000 configured as shown in Fig. 52. Fig. 52 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the arithmetic device. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The various components of the computer 1000 are connected by a bus 1050.

[0160] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0161] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, as well as programs that depend on the hardware of the computer 1000 .

[0162] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records an application program according to the present disclosure, which is an example of program data 1450.

[0163] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0164] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Discs), magneto-optical recording media such as MOs (Magneto-Optical Discs), tape media, magnetic recording media, and semiconductor memories.

[0165] For example, when the computer 1000 functions as the arithmetic unit of the HMD 5, the CPU 1100 of the computer 1000 executes a program loaded onto the RAM 1200 to realize the processing according to the embodiment. Note that the CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, the CPU 1100 may obtain these programs from another device via an external network 1550.

[0166] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0167] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0168] The present technology can also be configured as follows. (1) A controller having a mouse function unit that outputs coordinate information on a plane, a 6DoF function unit that outputs position and orientation information, and a switching signal output unit that outputs a switching signal to switch between a display mode of a two-dimensional image to which the coordinate information output from the mouse function unit is applied and a display mode of a three-dimensional image to which the position and orientation information output from the 6DoF function unit is applied. (2) The controller described in (1), wherein the switching signal output unit has a distance sensor that measures a distance from a plane on which the controller is placed, and outputs the switching signal when the distance measured by the distance sensor is longer than a threshold distance. (3) The controller described in (1) or (2), wherein, when the switching signal is not output, the mouse function unit outputs the coordinate information and the 6DoF function unit does not output the position and orientation information, and when the switching signal is output, the mouse function unit does not output the coordinate information and the 6DoF function unit outputs the position and orientation information. (4) The controller according to any one of (1) to (3), further comprising: a button unit to which a function of an operation in the two-dimensional image or a function of an operation in the three-dimensional image is assigned; and a button operation switching unit that switches the function of the operation in the two-dimensional image assigned to the button unit to the function of the operation in the three-dimensional image when the switching signal is received. (5) The controller according to (4), further comprising: a button operation switching unit that switches the operation of selecting the button unit. (6) The controller according to any one of (1) to (5), further comprising: a first side surface and a second side surface opposite to the first side surface; and a third side surface and a fourth side surface that is formed above the third side surface and further outward from the controller than the third side surface when the controller is viewed from the front. (7) The controller according to (6), further comprising: a fifth side surface and a sixth side surface that is above the fifth side surface and further outward from the controller than the fifth side surface when the controller is viewed from the front.(8) The controller according to any one of (1) to (7), wherein the controller has a first side surface and a second side surface opposite to the first side surface, the first side surface having a third side surface and a fourth side surface formed above the third side surface and further outward from the controller than the third side surface when viewed from the front, the second side surface having a fifth side surface and a sixth side surface formed above the fifth side surface and further outward from the controller than the fifth side surface when viewed from the front, and the center of gravity of the controller is between the fourth side surface and the sixth side surface. (9) The controller according to any one of (1) to (8), wherein the controller is dogleg-shaped. (10) The controller according to (9), wherein the center of gravity of the controller is in a central portion inside the controller. (11) An information processing device having a display mode switching unit that switches a first display mode displaying a two-dimensional image to a second display mode displaying a three-dimensional image when a switching signal for switching the display mode is received or when the distance of the controller from a plane on which the controller is placed is longer than a threshold distance. (12) The information processing device according to (11), having a display unit that displays the two-dimensional image and the three-dimensional image. (13) The information processing device according to (11) or (12), wherein the second display mode displays the two-dimensional image. (14) The information processing device according to (13), wherein the first display mode displays the three-dimensional image. (15) An information processing method, wherein the first display mode displaying the two-dimensional image is switched to the second display mode displaying the three-dimensional image when a switching signal for switching the display mode is received or when distance data from the plane on which the controller is placed, received from the controller, is longer than a threshold distance. (16) An information processing method, comprising: calculating a distance between a controller and a plane on which the controller is placed; and, when the calculated distance is longer than a threshold distance, switching from a first display mode in which the two-dimensional image is displayed to a second display mode in which the three-dimensional image is displayed.

[0169] 1 Virtual screen 2 Object to be displayed in 2D 3 Object to be displayed in 3D 5 HMD 6, 106, 206, 306, 406, 506, 606 Controller 7 Line 8 Ray 9 Plane 12 First side 13 Second side 14 Third side 15 Fourth side 16 Fifth side 17 Sixth side 21 Center of gravity 25, 26, 27 Inner part 28 Upper part 31 Distance sensor 40 Operation button 41 Left button 42 Thumb button 43 Movement detection unit 44 Side part 45 Inner wall part 46 6DoF position detection unit 51 Real monitor 52 PC 61 Distance sensor unit 62 Mouse function unit 63 6DoF function unit 64 Button operation device switching unit 65 Button unit 66 Communication control unit 107 Ring 307, 407 Flat portion 611 Protrusion 612 Bright spot

Claims

1. A controller having a mouse function unit that outputs coordinate information on a plane, a 6DoF function unit that outputs position and orientation information, and a switching signal output unit that outputs a switching signal to switch between a display mode of a two-dimensional image to which the coordinate information output from the mouse function unit is applied and a display mode of a three-dimensional image to which the position and orientation information output from the 6DoF function unit is applied.

2. A controller as described in claim 1, wherein the switching signal output unit has a distance sensor that measures the distance from a plane on which the controller is placed, and outputs the switching signal when the distance measured by the distance sensor is longer than a threshold distance.

3. A controller as described in claim 1, wherein, when the switching signal is not output, the mouse function unit outputs the coordinate information and the 6DoF function unit does not output the position and orientation information, and, when the switching signal is output, the mouse function unit does not output the coordinate information and the 6DoF function unit outputs the position and orientation information.

4. A controller as described in claim 1, having a button section to which a function of operation in the two-dimensional image or a function of operation in the three-dimensional image is assigned, and a button operation switching section which switches the function of operation in the two-dimensional image assigned to the button section to the function of operation in the three-dimensional image when the switching signal is received.

5. The controller according to claim 4, wherein the button operation switching section switches the selection operation of the button section.

6. The controller of claim 1, wherein the controller has a first side and a second side opposite the first side, the first side having a third side and a fourth side formed above the third side and further outboard of the controller than the third side when the controller is viewed from the front.

7. A controller as described in claim 6, wherein the second side has: a fifth side; and a sixth side which is an upper part of the fifth side when the controller is viewed from the front, and which is formed further outward from the controller than the fifth side.

8. The controller of claim 1, wherein the controller has a first side and a second side opposite to the first side, the first side having a third side and a fourth side formed on an upper part of the third side and further outboard of the controller than the third side when the controller is viewed from the front, the second side having a fifth side and a sixth side formed on an upper part of the fifth side and further outboard of the controller than the fifth side when the controller is viewed from the front, and the center of gravity of the controller is between the fourth side and the sixth side.

9. The controller according to claim 1, wherein the controller is doglegged in shape.

10. The controller of claim 9, wherein the center of gravity of the controller is in a central portion of the interior of the controller.

11. An information processing device having a display mode switching unit that switches from a first display mode that displays two-dimensional images to a second display mode that displays three-dimensional images when a switching signal for switching the display mode is received or when the distance of the controller from the plane on which the controller is placed is longer than a threshold distance.

12. The information processing device according to claim 11, further comprising a display unit for displaying the two-dimensional image and the three-dimensional image.

13. The information processing device according to claim 11, wherein the second display mode displays the two-dimensional image.

14. The information processing device according to claim 13, wherein the first display mode displays the three-dimensional image.

15. An information processing method comprising: receiving distance data from a controller to a plane on which the controller is placed; and, if the received distance data is longer than a threshold distance, switching from a first display mode that displays a two-dimensional image to a second display mode that displays a three-dimensional image.

16. An information processing method comprising: calculating a distance between a controller and a plane on which the controller is placed; and, if the calculated distance is longer than a threshold distance, switching from a first display mode that displays a two-dimensional image to a second display mode that displays a three-dimensional image.

Citation Information

Patent Citations

  • Computer mouse

    JP1458239S

  • computer mouse

    JP1659055S

  • Input device, method and program and storage medium

    JP2008047047A

  • System and method for multipurpose input device for two-dimensional and three-dimensional environments

    US20200192486A1

  • Rendering device and rendering method

    WO2019102825A1