Information processing system and information processing method

Abstract

The present disclosure relates to an information processing system which makes it possible to improve usability in controlling the position and posture of a virtual object using an XR technology. The present disclosure also relates to an information processing method. In the present disclosure, on the basis of an operation that is with respect to a first rotation axis for which operation is easiest and that is based on information indicating the position and posture of a controller, display is controlled such that a first operation, which is the most frequently performed, is performed on a virtual object. On the basis of an operation that is with respect to a second rotation axis for which operation is the second easiest, display is controlled such that a second operation, which is the second most frequently performed, is performed on the virtual object. It is possible to apply the present disclosure to an information processing system that manipulates the position and posture of a virtual object using an XR technology.

Description

Information processing system and information processing method

[0001] The present disclosure relates to an information processing system and an information processing method, and more particularly to an information processing system and an information processing method that improve usability related to control of the position and attitude of a virtual object in controllers of various configurations.

[0002] In a space that combines virtual and real spaces, such as XR (Cross Reality / Extended Reality), it is possible to link and control the positions and postures of real objects (objects in real space) and virtual objects (objects in virtual space). In particular, XR technology using HMDs (Head Mounted Displays) has been widely developed.

[0003] For example, a technology has been proposed in which a controller that is held in each hand and has a twisting mechanism is used to move an object in a virtual space with a sensation close to that of the real world (see Patent Document 1).

[0004] JP 2008-102789 A

[0005] However, the technology of Patent Document 1 can only be realized with a specific controller that is held in each hand and has a twisting mechanism, and when using controllers of various configurations that are held in each hand and do not have a twisting mechanism, sufficient usability cannot be obtained in controlling the position and posture of virtual objects in a linked manner.

[0006] The present disclosure has been made in light of such circumstances, and in particular aims to improve usability in controlling the position and attitude of virtual objects, even for controllers with various configurations.

[0007] An information processing system according to one aspect of the present disclosure includes a controller and a display device that displays a virtual object, and acquires information indicating the position and attitude of the controller, and controls the display of the virtual object based on the information indicating the position and attitude of the controller, the display being controlled so that a first operation is performed on the virtual object based on operation of a first rotation axis of the controller, and the display being controlled so that a second operation different from the first operation is performed on the virtual object based on operation of a second rotation axis of the controller that is different from the first rotation axis, and the first rotation axis and the first operation, and the second rotation axis and the second operation are each associated with each other based on characteristics of the controller.

[0008] An information processing method according to one aspect of the present disclosure is an information processing method for an information processing system including a controller and a display device that displays a virtual object, the information processing method including: performing an acquisition process to acquire information indicating a position and attitude of the controller; and performing a display control process to control a display of the virtual object based on the information indicating the position and attitude of the controller, wherein the display control process controls a display so that a first operation is performed on the virtual object based on an operation of a first rotation axis of the controller, and controls a display so that a second operation different from the first operation is performed on the virtual object based on an operation of a second rotation axis of the controller that is different from the first rotation axis, and the first rotation axis and the first operation, and the second rotation axis and the second operation are each associated with each other based on characteristics of the controller.

[0009] In one aspect of the present disclosure, information indicating a position and attitude of a controller is acquired, a display of the virtual object is controlled based on the information indicating the position and attitude of the controller, the display is controlled so that a first operation is performed on the virtual object based on operation of a first rotation axis of the controller, and the display is controlled so that a second operation different from the first operation is performed on the virtual object based on operation of a second rotation axis of the controller different from the first rotation axis, and the first rotation axis and the first operation, and the second rotation axis and the second operation are each associated based on characteristics of the controller.

[0010] 7 is a diagram illustrating a configuration of an information processing system for explaining an overview of the present disclosure. FIG. 7 is a diagram illustrating the ease of operation of each rotation axis in the controller of FIG. 1. FIG. 7 is a diagram illustrating the frequency of operation of each rotation axis in operating the virtual object of FIG. 1. FIG. 7 is a diagram illustrating a state when operating a virtual object using the same controller. FIG. 7 is a diagram illustrating assignment of the easiest-to-operate rotation axis in the controller to the most frequently used function in operating the virtual object. FIG. 7 is a diagram illustrating assignment of the easiest-to-operate rotation axis in the controller to the most frequently used function in operating the virtual object, and the second easiest-to-operate rotation axis in the controller to the second most frequently used function in operating the virtual object. FIG. 7 is a diagram illustrating an example configuration of an information processing system according to a first embodiment of the present disclosure. FIG. 7 is a diagram illustrating the appearance and operation of the controller of the information processing system of FIG. 7. FIG. 7 is a diagram illustrating an example configuration of the HMD of FIG. 7. FIG. 7 is a diagram illustrating an example configuration of the controller of FIG. 7. FIG. 7 is a diagram illustrating setting the position of the rotation axis of the virtual object. FIG. 7 is a diagram illustrating setting the distance (positional relationship) between the virtual object and the controller. FIG. 7 is a diagram illustrating setting the center position in zooming in and out of a virtual object. FIG. 7 is a diagram illustrating assignment of the easiest-to-operate first rotation axis in the controller to a rotation operation centered on the most frequently used rotation axis in operating a virtual object. 10 is a diagram illustrating allocation of the second rotation axis on the controller, which is the second easiest to operate, to the zoom operation, which is the second most frequently used among operations on virtual objects. FIG. 11 is a flowchart illustrating six-axis information acquisition processing. FIG. 12 is a flowchart illustrating position and orientation linkage setting processing. FIG. 13 is a flowchart illustrating virtual object operation processing. FIG. 14 is a flowchart illustrating position and orientation linkage setting transfer processing. FIG. 15 is a diagram illustrating an example in which, when the controller is rotated by a rotation amount on a first rotation axis, the virtual object is rotated by the same rotation amount. FIG. 16 is a diagram illustrating a first modified example. FIG. 17 is a diagram illustrating a second modified example. FIG. 18 is a diagram illustrating a third modified example. FIG. 19 is a diagram illustrating a fourth modified example. FIG. 20 is a diagram illustrating a fifth modified example. FIG. 21 is a diagram illustrating examples of functions assigned to the first to second rotation axes depending on the application. FIG. 22 is a diagram illustrating a sixth modified example. FIG. 23 is a diagram illustrating a seventh modified example. FIG. 24 is a diagram illustrating an eighth modified example.FIG. 37 is a diagram for explaining a ninth modified example. FIG. 38 is a diagram for explaining a tenth modified example. FIG. 39 is a diagram for explaining an eleventh modified example. FIG. 40 is a diagram for explaining an eleventh modified example. FIG. 41 is a diagram for explaining an example configuration of an information processing system according to a second embodiment of the present disclosure. FIG. 42 is a diagram for explaining adjustment of the distance of a virtual object in the ring-shaped controller of FIG. 36. FIG. 43 is a diagram for explaining adjustment of the magnification rate for scaling a virtual object in the ring-shaped controller of FIG. 36. FIG. 44 is a flowchart for explaining position and orientation linkage setting reuse processing. FIG. 45 is a diagram for explaining an example configuration of a general-purpose computer.

[0011] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0012] Hereinafter, embodiments of the present technology will be described in the following order.

[0013] 1. Overview of the Present Disclosure 2. First Embodiment 3. First Modification 4. Second Modification 5. Third Modification 6. Fourth Modification 7. Fifth Modification 8. Sixth Modification 9. Seventh Modification 10. Eighth Modification 11. Ninth Modification 12. Tenth Modification 13. Eleventh Modification 14. Second Embodiment 15. Example of Implementation by Software

[0014] <<1. Overview of the Present Disclosure>> <Information Processing System Comprising a Controller and an HMD for Controlling the Position and Attitude of a Virtual Object> The present disclosure aims to improve usability in controlling the position and attitude of a virtual object, particularly for controllers with various configurations. Therefore, first, an overview of the present disclosure will be described.

[0015] FIG. 1 shows an example configuration of an information processing system for explaining the outline of the present disclosure, which controls the position and orientation of a virtual object (an object in a virtual space) relative to a real object (an object in a real space) in a space that combines a virtual space and a real space, such as XR (Cross Reality / Extended Reality).

[0016] The information processing system in FIG. 1 is composed of an HMD (Head Mounted Display) 21 and controllers 22L and 22R.

[0017] The HMD 21 is worn on the head 11h of the user 11 and includes a pass-through, video-through, or fully immersive display (not shown) located opposite the eyes of the user 11. The HMD 21 also includes an IMU (Inertial Measurement Unit) that detects the three-dimensional position and three-axis angular velocity in accordance with the movement of the head 11h. The HMD 21 also includes a camera (not shown) for tracking the controller 22.

[0018] The HMD 21 determines the position and orientation of the controllers 22L, 22R based on the position and orientation of the controllers 22L, 22R tracked by a camera and the position and orientation information of the controller 22 detected and supplied by an IMU built into the controller 22.

[0019] The HMD 21 controls the position and attitude of the virtual object based on information about its own position and attitude and the positions and attitudes of the controllers 22L and 22R, and displays the virtual object V in accordance with the movement of the head 11h and in combination with the surrounding real space on a pass-through, video-through, or fully immersive display. Note that in Figure 1, the virtual object V is modeled after an airliner.

[0020] This allows the user 11 to view an image VP in which the virtual object V appears to be floating in real space through the display of the HMD 21 that the user is wearing.

[0021] In reality, the image VP differs depending on the type of display, and may be configured as an image as a whole, such as in a pass-through type or video-through type, in which only the virtual object V is superimposed and the remaining range is configured as an area corresponding to the real space around the HMD 21, or may be configured as a single VR image in which an image simulating real space is combined with the virtual object V, such as in a fully immersive type. However, in this specification, the image VP is treated as a single image VP in which the image of real space combined with the virtual object V within the field of view of the user 11 is combined with the virtual object in the virtual space.

[0022] The controllers 22L, 22R are used to control the position and orientation of the virtual object V in the image VP in conjunction with the user 11 changing their position and orientation while holding them. In Fig. 1, the controllers 22L, 22R are held in the left and right hands 11cL, 11cR of the user 11, respectively, and as the user 11 changes their position and orientation, the position and orientation of the virtual object V in the image VP change in conjunction with the controllers 22L, 22R.

[0023] Although FIG. 1 illustrates a situation in which two controllers 22L, 22R are held in the left and right hands 11cL, 11cR, control of the position and posture of the virtual object V in the image VP can be realized using either one of them.

[0024] Therefore, in Figure 1, the explanation will be given assuming that the position and posture of the virtual object V is controlled by the controller 22L held in the left hand 11cL, and since there is no distinction between left and right, the controller 22L will also be simply referred to as the controller 22.

[0025] Similar control is also possible with the controller 22R held in the right hand 11cR, and furthermore, only one of the controllers 22L and 22R may be used.

[0026] More specifically, the controller 22 is provided with a rod-shaped grip portion 22g, and in FIG. 1, the user 11 holds the grip portion 22g with the left hand 11cL.

[0027] The controller 22 detects the angular velocity in the R1 direction around the rotation axis AX1 along the core of the rod-shaped grip portion 22g, the position on the rotation axis AX1, the angular velocity in the R2 direction around the rotation axis AX2 perpendicular to the rotation axis AX1, the position on the rotation axis AX2, and the angular velocity in the R3 direction around the rotation axis AX3 perpendicular to both the rotation axes AX1 and AX2, and the position on the rotation axis AX3 using an IMU (not shown), and supplies the detection results to the HMD 21.

[0028] Based on its own position and orientation detected by an IMU (not shown) that the HMD 21 is equipped with, information on the position and orientation of the controller 22 tracked by a camera, and information on the position on the rotation axes AX1 to AX3 and the angular velocity in the R1 to R3 directions supplied by the controller 22, the HMD 21 controls the positions on the rotation axes AX1' to AX3' of the virtual object V and the angular velocity in the R1' to R3' directions that correspond to the positions on the rotation axes AX1 to AX3 and the angular velocity in the R1 to R3 directions, and controls and displays the position and orientation of the virtual object V in the image VP.

[0029] Here, the rotation axis AX1' of the virtual object V is the vertical direction in real space and corresponds to the Z-axis direction, which is the height direction of the virtual object V; the R1 direction corresponds to the yaw direction of the virtual object V; the rotation axis AX2' corresponds to the Y-axis direction, which is the vertical direction of the virtual object V; the R2 direction corresponds to the pitch direction of the virtual object V; the rotation axis AX3' corresponds to the X-axis direction of the virtual object V; and the R3 direction corresponds to the roll direction of the virtual object V.

[0030] Based on the HMD 21's own position and posture information, the position and posture information of the controller 22 obtained by tracking itself, and the position and posture information sequentially supplied by the controller 22, the HMD 21 controls the position and posture of the virtual object V, displays it, and allows the user 11 to view it as an image VP.

[0031] The position and attitude information of the controller 22 may be at least either determined by tracking itself or sequentially supplied by the controller 22, but by using both, position and attitude information with higher accuracy can be obtained.

[0032] This allows the user 11 to change the position and posture of the virtual object V in the image VP by rotating the controller 22 while changing its position, giving the user the feeling that he or she is manipulating the virtual object V with the controller 22.

[0033] The same controller 22 may also be handed over to another user wearing the HMD 21, allowing the other user to experience the same operating sensation with respect to the virtual object V.

[0034] <Easy and Difficult Rotation Operations Using Controller> Depending on the shape and the like, there are easy and difficult rotation operations on the controller 22.

[0035] For example, in the case of the controller 22, as shown in Figure 2, the easiest rotation operation is the R1 direction around the rotation axis AX1, which can be operated by rotating the rod-shaped core that makes up the grip portion 22g, followed by the R2 direction around the rotation axis AX2, which is the second easiest, and the R3 direction around the rotation axis AX3, which is the hardest.

[0036] <Frequently Performed Rotation Operations on Virtual Objects and Infrequently Performed Rotation Operations> On the other hand, with regard to the virtual object V, there are frequently performed rotation operations and infrequently performed rotation operations.

[0037] For example, in the case of a virtual object V simulating a passenger aircraft, as shown in FIG. 3, the frequency of operation in the R1' direction around the rotation axis AX1', which is the yaw direction, is relatively high, but the frequency of operation in the R2' and R3' directions around the rotation axes AX2' and AX3', which are the pitch and roll directions, is relatively low.

[0038] However, such a rotation operation with a high frequency of operation of the virtual object V is not necessarily assigned to an easy-to-operate rotation operation on the controller 22. For this reason, for example, if a rotation operation that is difficult to operate on the controller 22 is assigned to a rotation operation with a high frequency of operation of the virtual object V, the operational burden on the user 11 related to controlling the position and attitude of the virtual object V using the controller 22 increases, and there is a risk that the virtual object V will not be manipulated with precision.

[0039] <Operating the same virtual object by sharing the same controller with another user> Furthermore, by sharing the same controller 22 with another user wearing the HMD 21, the position and orientation of the virtual object can be operated in a coordinated manner.

[0040] When using the controller 22 to change the position and orientation of the virtual object V for viewing, the user often performs zooming in and out operations in addition to changing the position and orientation.

[0041] Here, consider a case where a user who first uses the controller 22 to view the virtual object V enlarges or reduces the virtual object V and views it, and there is a notable part of the virtual object V that the user wants other users to pay particular attention to when enlarging or reducing it.

[0042] However, even if the same controller 22 is used by a user and another user of the virtual object V, the center position serving as the reference for enlarging or reducing the virtual object V will not necessarily match unless it is specifically set. For this reason, even if the first user hands the same controller 22 to another user and instructs them to simply enlarge or reduce the controller 22 to view a portion of interest, the portion that is enlarged or reduced will be different, and the other user may not be able to properly view the portion of interest on the virtual object V.

[0043] More specifically, for example, as shown in the left part of Fig. 4, a user 11X wearing an HMD 21x wants the other user 11Y wearing an HMD 21y to "zoom in and take a look" and hands over the controller 22x to the other user 11Y, who wants the other user 11Y to "see the left engine," and the "left engine" part here is the area indicated by the area Zx surrounded by a dotted line in the virtual object Vx in Fig. 4.

[0044] In this case, as shown in the right part of Figure 4, the position of user 11Y relative to virtual object Vy may be a different position from that of user 11X, and the center position serving as the basis for zooming may be different from when user 11X performed the zooming operation.

[0045] Therefore, there is a possibility that the user 11Y may operate the handed-over controller 22x to enlarge the virtual object Vx, for example, based on the center position Cy of the zoom, and thereby enlarge and view the tail portion indicated by the area Zy.

[0046] In other words, in the case of Figure 4, when other user 11Y performs a zoom operation, the tail section shown in area Zy becomes visible, and as shown in the right part of Figure 4, the other user 11Y may think, "The tail section looks big when zoomed in, but is that what they want me to look at?" and may not be able to see the part of the "left engine" that user 11X intended to show.

[0047] Therefore, in the present disclosure, in the controller 22, the easiest rotation operation is set as the most frequently performed rotation operation on the virtual object V, and the second easiest rotation operation is set as the second most frequently performed rotation operation on the virtual object V.

[0048] More specifically, as shown in FIG. 5, the function of rotating the virtual object V in the direction R1' around the rotation axis AX', which is the yaw direction in which the virtual object V is most frequently operated, is assigned to the rotation operation in the R1 direction around the rotation axis AX1, which is the easiest to operate on the controller 22.

[0049] Also, for example, as shown in FIG. 6, the function of the zoom operation, which is the second most frequently operated operation of the virtual object V, is assigned to the rotation operation in the R2 direction of the rotation axis AX2, which is the second easiest to operate on the controller 22.

[0050] Furthermore, the position and orientation of the rotation axis in the function of rotating the virtual object V can be set by the user 11X who uses the controller 22 first.

[0051] This means that, for example, when user 11X uses controller 22, he or she can set the rotation axis for rotating virtual object V near the area of ​​interest, and even if user 11Y subsequently uses controller 22 to rotate virtual object V, the area of ​​interest will be located on the rotation axis, so that the area of ​​interest can be reliably viewed.

[0052] Furthermore, the center position C of the enlargement / reduction in the rotation operation function can be set by the user 11X who uses the controller 22 for the first time on the rotation axis AX1' in the yaw direction, which is the most frequently operated position for the virtual object V.

[0053] As a result, for example, when the user 11X uses the controller 22, the user 11X sets the zoom center position C near the area of ​​interest and on the rotation axis for rotating the virtual object V. Thereafter, even if the user 11Y uses the controller 22 to zoom in or out the virtual object V, the zoom center position C will be set on the rotation axis set near the area of ​​interest, so the user can reliably zoom in or out and view the area of ​​interest.

[0054] By doing this, in the present disclosure, the rotation operation that is easiest to operate on the controller 22 is assigned to the rotation operation that is most frequently operated on the virtual object V, and the rotation operation that is second most frequently operated on the virtual object V is assigned to the rotation operation that is second easiest to operate on the controller 22, thereby making it possible to improve the operability of the virtual object V.

[0055] In addition, the user 11X who uses the controller 22 for the first time can set the position and orientation of the rotation axis in the operation direction in which the virtual object V is most frequently operated near the area of ​​interest, and set the center position of the zoom operation on that rotation axis.

[0056] Therefore, when user 11Y uses the same controller 22 after user 11X, the part of interest will be near the rotation axis or the center position C of the zoom, even if the virtual object V is rotated or zoomed in or out. This ensures that the part of interest is displayed regardless of whether the virtual object V is rotated or zoomed in or out, so that user 11Y can reliably view the part of interest specified by user 11X.

[0057] In other words, regardless of the shape of the controller 22, the easier the operation of the controller 22 is to perform, the more frequently the operation is assigned to a function that is frequently performed on the virtual object V, thereby making the control of the position and attitude of the virtual object V more effective and efficient, reducing the effort involved and improving satisfaction.

[0058] Furthermore, since it is possible to set specific positions and orientations in conjunction with one another, such as the position and orientation of the rotation axis in a rotation operation on the virtual object V, and the center position in a zoom operation on the virtual object V, it is possible to make effective and efficient control of the position and orientation of the virtual object V when multiple users use the same controller 22, thereby reducing the effort involved and improving satisfaction.

[0059] As a result, it becomes possible to improve the effectiveness, efficiency, and satisfaction of controlling the position and posture of virtual objects using XR technology, thereby improving so-called usability.

[0060] <<2. First embodiment>> Next, an information processing system will be described that controls the position and orientation of a virtual object (an object in a virtual space) relative to a real object (an object in a real space) in a space that combines a virtual space and a real space, such as XR according to the present disclosure.

[0061] FIG. 7 shows an example configuration of an information processing system that controls the position and orientation of a virtual object (an object in a virtual space) relative to a real object (an object in a real space) in a space that combines a virtual space and a real space, such as XR.

[0062] The information processing system 101 in FIG. 7 is composed of HMDs (Head Mounted Displays) 121A and 121B and VR (Virtual Reality) controllers 122L and 122R.

[0063] The HMDs 121A, 121B and the controllers 122L, 122R have the same basic functions as the HMD 21 and the controllers 22L, 22R in FIG. 1, respectively, but further include a position and orientation linkage setting function to improve usability in controlling the position and orientation of virtual objects using XR technology.

[0064] 7, the operation of this embodiment can be achieved by using either one of the controllers 122L and 122R. For this reason, in the following description, the case where only the controller 122L is used will basically be described, and since there is no need to distinguish between the left and right, they will also be simply referred to as the controller 122.

[0065] More specifically, the HMD 121 includes a display 131, a control unit 132, an IMU (Inertial Measurement Unit) 209 (Figure 9), and a camera 210 (Figure 9), and is attached to the head 111Ahd so that the pass-through, video-through, or fully immersive display 131 is positioned facing both eyes of the user 111.

[0066] Based on the movement of the head 111Ahd detected by an IMU (Inertial Measurement Unit) 209 (Figure 9), the position and orientation of the controller 122 tracked by a camera 210 (Figure 9), and information on the position and orientation of the controller 122 detected by an IMU 152 (Figure 10) built into the controller 122 and supplied by the controller 122, the control unit 132 displays a virtual object V on the display 131 together with the surrounding real space visible through the display 131 from the user's field of view, or displays a VR image on the display 131 that combines an image simulating real space with the virtual object V, thereby presenting an image VP that combines real space and virtual space to the user 111.

[0067] This allows the user 111A to simultaneously view the real space and the virtual object V through the display 131 of the HMD 121A that he or she is wearing, thereby enabling the user 111A to view an image VP in which the virtual object V is blended into the real space.

[0068] The controller 122 is used to control the position and orientation of the virtual object V within the image VP, and by changing the position and orientation while holding it in each hand, the position and orientation of the virtual object V within the image VP are changed in tandem.

[0069] More specifically, the controller 122 is provided with a rod-shaped grip portion 122g, and in FIG. 7, the user 111A holds the controller 122 by grasping the grip portion 122g with the left hand 111Ah.

[0070] As shown in Figure 8, the controller 122 detects position and attitude information of the controller 122 as six-axis information using the built-in IMU 152, which consists of the angular velocity in the Ry direction around the rotation axis AXy formed by the core of the rod-shaped grip portion 122g and the position on the rotation axis AXy, the angular velocity in the Rp direction around the rotation axis AXp and the position on the rotation axis AXp, and the angular velocity in the Rr direction around the rotation axis AXr and the position on the rotation axis AXr, and supplies the detection results to the HMD 121.

[0071] In Figure 8, in correspondence with XYZ in a three-dimensional coordinate system, the angular velocity in the Rr direction of the rotation axis AXr, which is the X direction, is expressed as the angular velocity in the Roll direction, the angular velocity in the Rp direction of the rotation axis AXp, which is the Y direction, is expressed as the angular velocity in the Pitch direction, and the angular velocity in the Ry direction of the rotation axis AXy, which is the Z direction, is expressed as the angular velocity in the Yaw direction.

[0072] The controller 122 also includes an operation unit 151 including a plurality of buttons operated by the user 111 for various operations, and supplies operation signals according to the operation content to the HMD 121A.

[0073] When the display 131 is a pass-through type or a video-through type, the HMD 121 identifies and displays the position of the virtual object V on the display 131 from the position and orientation of the controller 122, which is a real object viewed by the user 111 wearing the HMD 121 through the display 131, based on its own position and orientation detected by an IMU 209 ( FIG. 9 ) provided in the HMD 121, position and orientation information of the controller 122 obtained from tracking results of the controller 122 captured by a camera 210 ( FIG. 9 ), and position and orientation information of the controller 122 supplied from an IMU 152 built into the controller 122. This allows the user 111 to view an image VP by identifying the position of the virtual object V on the display 131 from the position and orientation of the controller 122. When the display 131 is a fully immersive type, the HMD 121 displays a VR image that combines an image simulating real space with the virtual object V. Therefore, the HMD 121 allows the user 111 to view the image VP by displaying the controller 122 at a position within the image VP made up of a VR image, which is identified based on tracking results and detection results of the controller 122.

[0074] In addition, through a user setting process using the operation unit 151, the controller 122 generates position and orientation linkage settings consisting of linkage settings between a rotation operation that is easy to operate on the controller 122 and a function that realizes a frequently operated operation on the virtual object V, a distance setting between the controller 122, which is a real object, and the virtual object V (setting the distance between the centers of gravity of the two), and a setting of the center position for zooming operations, and supplies the set position and orientation linkage settings and six-axis information, which is information on the position and orientation of the controller 122, to the HMD 121.

[0075] More specifically, the controller 122 statistically processes the history of six-axis information, which constitutes its own operation history, and sets the order of ease of operation for each of the three rotation axes in descending order of frequency of use. Here, the order of the rotation axes is determined by assuming that frequency of use corresponds to ease of operation (ease of turning, ease of use). The order of the rotation axes changes depending on the characteristics of the controller 122, such as the shape of the controller 122, and can be thought of as representing the characteristics of the controller 122. The controller 122 also transmits the order of the rotation axes that are easy to operate to the HMD 121 as appropriate, and shares it with the HMD 121.

[0076] The controller 122 then assigns the rotation operation based on the first rotation axis, which is the easiest to operate, to the rotation operation function of the virtual object V, which is the most frequently operated operation on the virtual object V, and assigns the rotation operation based on the second rotation axis, which is the second easiest to operate, to the second most frequently operated operation of the virtual object V. In other words, as described above, the order of the rotation axes can be considered to be the characteristics of the controller 122 itself, and therefore, it can be said that the functions assigned to the rotation operations based on each rotation axis based on the order of the rotation axes are assigned based on the characteristics of the controller 122.

[0077] In this example, the rotation operations assigned to the rotation operation function and zoom operation function for the virtual object V are assigned in order of ease of rotation operation for each rotation axis of the controller 122 and frequency of operation for the virtual object V, but the rotation operations assigned to each function may be any rotation operation based on the three axes, or may be assigned arbitrarily by the user.

[0078] With this configuration, for example, after the position and orientation linkage setting is made by the user 111, even if the controller 122 is handed over to the user 111B wearing the HMD 121B and operated by the user 111B, the HMD 121B can present the image VP including the virtual object V to the user 111B based on the position and orientation linkage setting already made by the user 111A.

[0079] As a result, when user 111A, for example, wants user 111B to view a specific part of virtual object V that is of interest, he or she can set the rotation axis of the rotation operation function of virtual object V at a position near the part of interest in virtual object V, and also set the center position of zoom in and out at a position on that rotation axis that is near the part of interest in virtual object V.

[0080] When user 111B uses controller 122 to control the position and orientation of virtual object V based on the position and orientation linkage setting information in which the rotation axis in the rotation operation function of virtual object V and the center position in the zoom function of virtual object V are set, it becomes easier to display the focus area of ​​virtual object V.

[0081] As a result, user 111A can simply hand over the controller 122 for which he or she has performed position and posture linkage settings to user 111B and have user 111B control the position and posture of the virtual object V, making it easier for user 111B to view the area of ​​interest that user 111A wants user 111A to focus on.

[0082] Although the example shows that the order of ease of operation of the three rotation axes is determined by the user 111 who owns the controller 122 using the operation history, the order may be determined by other methods, for example, by using the operation history of another user. Furthermore, the axis that generates the smallest moment when the controller is rotated from the center of gravity may be determined as the easiest rotation axis based on the shape and weight of the controller 122. Furthermore, the developers of the HMD 121 and the controller 122 may set the order at the time of development based on the expected way of holding the controller 122, or similarly, the user may determine the order.

[0083] <Example of Hardware Configuration of HMD> Next, an example of the hardware configuration of the HMD 121 in FIG. 7 will be described with reference to FIG.

[0084] The HMD 121 is composed of a control unit 201, an input unit 202, an output unit 203, a memory unit 204, a communication unit 205, a drive 206, a removable storage medium 207, an IMU 209, and a camera 210, which are connected to each other via a bus 208 and can send and receive data and programs.

[0085] 7 , is composed of a processor and a memory, and controls the overall operation of the HMD 121. The control unit 201 also includes a tracking unit 230, a controller six-axis information management unit 231, a position and orientation linkage setting information management unit 232, an HMD six-axis information management unit 233, and a display control unit 234.

[0086] The tracking unit 230 tracks the controller 122 based on the image captured by the camera 210, detects the position and orientation of the controller 122, generates tracking information consisting of position and orientation information of the controller 122 in chronological order, and stores it in the tracking information storage unit 260.

[0087] The controller 6-axis information management unit 231 controls the communication unit 205 to extract, in chronological order, 6-axis information that is the position and attitude information of the controller 122 detected by the controller 122 itself from the information transmitted from the controller 122, and stores the extracted information in the controller 6-axis information storage unit 261.

[0088] The position and orientation linkage setting information management unit 232 extracts position and orientation linkage setting information from the information transmitted from the controller 122 and stores it in the position and orientation linkage setting information storage unit 262. The position and orientation linkage setting information includes, for example, the operation content of the controller 122 set for the rotation operation function of the virtual object V, the operation content of the controller 122 set for the enlargement / reduction function of the virtual object V, the position and orientation of the rotation axis used to realize the rotation operation function of the virtual object V, the distance setting between the controller 122 which is the real object and the virtual object V, and the setting of the center position related to enlargement / reduction in the enlargement / reduction operation function. Note that the operation content of the controller 122 includes information indicating which rotation axis of the controller 122 is the center of the rotation operation.

[0089] The HMD six-axis information management unit 233 acquires six-axis information, which is information about the position and orientation of the HMD 121 itself, supplied from the IMU 209 , and supplies it to the display control unit 234 .

[0090] The display control unit 234 reads out content 263 including image information of the virtual object V stored in the storage unit 204, and displays the virtual object V on the display 131 based on the position and orientation information of the controller 122, the position and orientation information of the HMD 121, and the position and orientation linkage setting information so that the user 111 can view the image VP in which the virtual object V appears to be blended into an appropriate position in real space via the display 131. Note that, in the case where the display 131 is a pass-through or video-through type, the image VP in which the virtual object V appears to be blended into an appropriate position in real space is a single image representing the entire image visually recognized when the user 111 views an image of the virtual object V that appears to be floating in an appropriate position in the transparent real space. On the other hand, in the case where the display 131 is a fully immersive type, the image VP in which the virtual object V appears to be blended into an appropriate position in real space is the image VP itself, positioned in a position in which the virtual object V appears to be floating in an appropriate position in the VR image displayed on the display 131 that simulates real space.

[0091] More specifically, the display control unit 234 displays the virtual object V based on the position and orientation information of the controller 122 stored in the tracking information storage unit 260, the six-axis information which is the position and orientation information of the controller 122 stored in the controller six-axis information storage unit 261, the six-axis information which is the position and orientation information of the HMD 122 supplied from the HMD six-axis information management unit 233, and the position and orientation linkage setting information.

[0092] In addition, since either the position and orientation information of the controller 122 stored in the tracking information storage unit 260 or the six-axis information, which is the position and orientation information of the controller 122 stored in the controller six-axis information storage unit 261, is sufficient to identify the position and orientation of the controller 122, it is possible to use at least one of them.

[0093] Furthermore, since both of these are position and orientation information of the controller 122 detected by the HMD 121 and position and orientation information detected by the controller 122 itself, both of these can be used to improve the accuracy of the position and orientation of the controller 122, and when the controller 122 moves outside the field of view of the camera 210 or enters an occlusion area and becomes unable to be tracked, only the position and orientation information detected by the controller 122 itself can be used.

[0094] The input unit 202 is composed of input devices such as a keyboard, a mouse, and a touch panel for inputting various types of information, and supplies the control unit 201 with various signals corresponding to the input information.

[0095] The output unit 203 is controlled by the control unit 201 and includes a display 131 and an audio output unit 251. As described in FIG. 7 , when the display 131 is a pass-through or video-through display, the display 131 displays the virtual object V so that the external world can be viewed as if it were transparent and can be used in a state where it appears to be blended into the external world. In the case of a fully immersive display, the display 131 displays the VR image itself, which is a composite of an image simulating the external world and the virtual object V. That is, the display 131 allows the user 111 wearing the HMD 121 to directly view the external world and indirectly view the virtual object V, or combines the external world and the virtual object V and allows the user 111 to indirectly view the external world and displays the image VP, in which the virtual object V appears to be blended into an appropriate position in the real space, via the display 131. Note that the display 131 may be any other display device as long as it allows the user 111 wearing the HMD 121 to view the external world while wearing the HMD 121 and can also display XR images.

[0096] The audio output unit 251 is made up of an audio output device such as a speaker, and outputs various types of voice, music, sound effects, and the like as audio.

[0097] The storage unit 204 is composed of a hard disk drive (HDD), a solid state drive (SSD), or semiconductor memory, and is controlled by the control unit 201 to write or read various data and programs. The storage unit 204 also stores a controller six-axis information storage unit 261, a position and orientation linkage setting information storage unit 262, and content 263, which are written or read as needed under the control of the control unit 201.

[0098] The communication unit 205 is controlled by the control unit 201 and realizes wired or wireless communication represented by LAN (Local Area Network) or Bluetooth (registered trademark), and transmits and receives various data and programs to and from other information processing devices including other users' HMDs 121 and controllers 122 via the network as necessary.

[0099] The drive 206 reads and writes data from and to a removable storage medium 207 such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor memory.

[0100] The IMU 209 is a so-called six-axis sensor that detects the position and orientation of the main body of the HMD 121 and supplies the detected position and orientation to the control unit 201 .

[0101] As described above, the camera 210 captures an image of a recognition target within the field of view of the user 111 wearing the HMD 121 and within the viewing angle visible through the display 131 , and supplies the image to the control unit 201 .

[0102] <Example of Hardware Configuration of Controller> Next, an example of the hardware configuration of the controller 122 in FIG. 7 will be described with reference to FIG.

[0103] The controller 122 is composed of a control unit 301, an input unit 302, an output unit 303, a memory unit 304, a communication unit 305, a drive 306, a removable storage medium 307, and an IMU 152, which are connected to each other via a bus 308 and can send and receive data and programs.

[0104] The control unit 301, input unit 302, output unit 303, memory unit 304, communication unit 305, drive 306, removable storage medium 307, bus 308, and IMU 152 correspond to the control unit 201, input unit 202, output unit 203, memory unit 204, communication unit 205, drive 206, removable storage medium 207, bus 208, and IMU 209 of the HMD 121, respectively, and their explanations will be omitted as appropriate.

[0105] The control unit 301 includes a six-axis information control unit 321 and a position and orientation linkage setting information management unit 322 .

[0106] The six-axis information control unit 321 acquires six-axis information, which is information on the position and orientation of the controller 122 itself, supplied from the IMU 152, and stores it in a six-axis information storage unit 331 in chronological order.

[0107] The position and orientation linkage setting information management unit 322 statistically processes the six-axis information, which is information about the position and orientation of the controller 122 stored in chronological order in the six-axis information storage unit 331, to set a ranking of rotation directions that are easiest to operate for the controller 122 based on the frequency of use of each of the three axis directions, and stores the ranking in the position and orientation linkage setting information storage unit 332.

[0108] The position and orientation linkage setting information management unit 322 then assigns the rotation operation function of the virtual object V to the rotation direction that is easiest to operate the controller 122, and assigns the zoom function of the virtual object V to the rotation direction that is second easiest to operate.

[0109] In addition, in the position and orientation linkage setting process, the position and orientation linkage setting information management unit 322 generates position and orientation linkage setting information consisting of the position and orientation setting of the rotation axis in the rotation operation function of the virtual object V, the distance setting between the controller 122, which is the real object, and the virtual object V (setting the distance between the centers of gravity of both), and the center position setting in the zoom function of the virtual object V, and supplies the set position and orientation linkage setting information and six-axis information, which is information on the position and orientation of the controller 122, to the HMD 121.

[0110] 11 , the setting of the position and orientation of the rotation axis in the rotation operation function of the virtual object V refers to the setting of the position and orientation on a plane formed by the rotation axes AXp and AXr of the rotation axis AXy, which serves as the center of rotation operation in the Ry direction around the rotation axis AXy, which serves as the yaw direction of the virtual object V. When the user 111 holds the controller 122 and reaches a desired position and orientation that they wish to set as the position and orientation of the rotation axis, and presses a predetermined button constituting the operation unit 151 (input unit 302), the position and orientation linkage setting information management unit 322 sets the position and orientation on the plane formed by the rotation axes AXp and ZXr at that time as the position and orientation of the rotation axis of the virtual object V.

[0111] In this example, the explanation is given assuming that the rotation axis of the virtual object V is fixed in the vertical direction, and therefore the orientation of the rotation axis of the virtual object V is fixed in the vertical direction, and the explanation is given assuming that the processing involves setting only the position of the rotation axis of the virtual object V. However, the orientation can also be set.

[0112] Furthermore, the distance setting between the controller 122, which is a real object, and the virtual object V (setting of the distance between the centers of gravity of the controller 122 and the virtual object V) is, for example, setting of the distance between the centers of gravity of the controller 122 and the virtual object V, as shown in FIG. 12 . When the user 111 holds the controller 122 and changes the distance VP from the virtual object V, and presses a predetermined button constituting the operation unit 151 (input unit 302) at a position where the desired distance VP is reached, the position and orientation linkage setting information management unit 322 sets the distance VP at that time as the distance between the virtual object V and the controller 122. Note that, although an example is shown here in which only the distance between the controller 122 and the virtual object V (the distance between the centers of gravity of the controller 122 and the virtual object V) is set, a three-dimensional positional relationship between the centers of gravity of the controller 122 and the virtual object V may also be set. Therefore, hereinafter, the distance setting between the controller 122 and the virtual object V is assumed to also set the positional relationship including the distance, but either one of these may be used. When the positional relationship between the controller 122 and the virtual object V is set, it can be said that the display position of the virtual object V is set substantially based on the position of the controller 122 .

[0113] 13, the setting of the center position in the zoom function of the virtual object V refers to the setting of the center position of the zoom operation on the rotation axis of the virtual object V. When the user 111 holds the controller 122 and changes the position SCP on the rotation axis of the virtual object V (on the arrow indicated by the dotted line) and presses a predetermined button constituting the operation unit 151 (input unit 302) at a desired position SCP, the position and orientation linkage setting information management unit 322 sets the position SCP at that time as the center position in the zoom operation of the virtual object V.

[0114] In this way, by setting the position and orientation linkage in the controller 122, the rotation axis in the operation direction that is easiest to operate is set as the rotation axis for the rotation operation function of the virtual object V, the rotation axis in the operation direction that is second easiest to operate is set as the operation direction for the zoom function, and further, the position and orientation of the rotation axis of the virtual object V, the distance (positional relationship) between the controller 122, which is the real object, and the virtual object V, and the center position for the zoom operation of the virtual object V are set.

[0115] With this configuration, the controller 122 can supply the HMD 121 with position and orientation linked installation information in addition to six-axis information that is information about its own position and orientation.

[0116] As a result, the HMD 121 is supplied with position and attitude linkage settings and six-axis information from the controller 122, making it possible to rotate the virtual object V in the yaw direction using the easiest rotation operation, which is the most frequently operated operation, and also making it possible to enlarge or reduce the virtual object V using the second easiest rotation operation.

[0117] In addition, the position and orientation linkage setting information, which is configured by the first user, and consists of the position (orientation) of the rotation axis in the rotation operation function of the virtual object V, the distance (positional relationship) between the controller 122 and the virtual object V, and the center position of zooming, is stored in the controller 122 and supplied to the HMD 121 together with the six-axis information.

[0118] As a result, even if a second user is handed the same controller 122, he or she will be able to control the position and orientation of the virtual object V with a similar feel.

[0119] That is, for example, the second user 111B uses the controller 122 for which position and posture linkage has been set by the first user 111A to operate the controller 122 in the rotation direction Ry11 centered on the rotation axis AXy11, which is the first rotation axis that is easiest to operate, as shown in FIG. 14, thereby realizing a rotation operation function in the rotation direction Ry11' centered on the rotation axis AXy11', which is the most frequently used axis for operating the virtual object V.

[0120] At this time, the rotation axis AXy11' can be set to a focus position set by the first user 111A that the first user 111A wants to focus on with respect to the virtual object V, making it easier for the second user 111B to see the focus position and preventing the second user 111B from overlooking the focus position. Also, at this time, the distance from the controller 122 to the virtual object V can be appropriately set, making it possible to prevent the focus position from being too far from the controller 122 to be seen or too close to be difficult to recognize.

[0121] 15 , the second user 111B can use the controller 122 for which position and orientation linkage has been set by the first user 111A to operate the controller 122 in the rotation direction Rp12 about the rotation axis AXp12, which is the second easiest to operate, to realize the zoom function, which is the second most frequently used function for operating the virtual object V. This makes it possible to zoom in and out of the virtual object V at a magnification factor SC according to the rotation angle in the rotation direction Ry12 about the rotation axis AXp12.

[0122] In this case, the center position C of zooming in and out is on the rotation axis of the rotation operation function, which is set near the focus position that the first user 111A wants to draw attention to, and can be set near the focus position. Therefore, even if the second user 111B is handed the controller 122 and zooms in and out, the focus position is easily displayed, making it possible to zoom in and out appropriately without losing sight of the focus position.

[0123] <Six-Axis Information Acquisition Processing> Next, six-axis information acquisition processing of the controller 122 will be described with reference to the flowchart of FIG.

[0124] In steps S31 and S51, the control unit 201 in the HMD 121 and the control unit 301 in the controller 122 control the communication units 205 and 305, respectively, to exchange IDs with each other and start communication according to a predetermined protocol.

[0125] In step S32, the six-axis information control unit 321 determines whether or not a predetermined time has elapsed, and if the predetermined time has elapsed, the process proceeds to step S33.

[0126] In step S33, the six-axis information control unit 321 controls the IMU 152 to acquire six-axis information.

[0127] In step S34, the six-axis information control unit 321 stores the acquired six-axis information in the six-axis information storage unit 331 of the storage unit 304 in chronological order.

[0128] In step S35, the six-axis information control unit 321 accesses the six-axis information storage unit 331 in the storage unit 304, reads out the six-axis information stored in chronological order, and calculates the ranking of the ease of operation (ease of use) of the three rotation axes in descending order of the frequency of occurrence of rotation operations on the three axes through predetermined statistical processing or the like, and updates and stores the ranking in the position and orientation linkage setting information storage unit 332.

[0129] In step S36 , the six-axis information control unit 321 transmits the six-axis information to the HMD 121 .

[0130] In step S52, the controller six-axis information management unit 231 of the control unit 201 in the HMD 121 determines whether six-axis information has been transmitted from the controller 122.

[0131] In step S52, if six-axis information is transmitted from the controller 122, the process proceeds to step S53.

[0132] In step S53 , the controller six-axis information management unit 231 acquires the six-axis information transmitted from the controller 122 .

[0133] In step S54, the controller six-axis information management unit 231 accesses the controller six-axis information storage unit 261 of the storage unit 204 in chronological order, and stores the acquired six-axis information from the controller 122 in chronological order.

[0134] In step S55, the tracking section 230 determines whether or not a predetermined time has elapsed, and if the predetermined time has elapsed, the process proceeds to step S56.

[0135] In step S56 , the tracking unit 230 controls the camera 210 to track the position and orientation of the controller 122 .

[0136] In step S57, the tracking unit 230 stores the position and orientation of the controller 122, which are the tracking results, in chronological order in the tracking information storage unit 260 of the storage unit 204.

[0137] In steps S37 and S58, it is determined whether or not an instruction to end the process has been given. If an instruction to end the process has not been given, the process returns to steps S32 and S52, and the subsequent processes are repeated.

[0138] If an instruction to end the process is given in steps S37 and S58, the process ends.

[0139] If it is determined in step S32 that the predetermined time has not elapsed, the processes in steps S33 to S36 are skipped.

[0140] Also, if it is determined in step S52 that six-axis information has not been transmitted, the processes of steps S53 and S54 are skipped.

[0141] Furthermore, if it is determined in step S55 that the predetermined time has not elapsed, the processes of steps S56 and S57 are skipped.

[0142] Through the above processing, the controller 122 acquires the six-axis information generated by the IMU 209 at predetermined time intervals, stores it in chronological order, and sets the order of frequency of occurrence of rotation operations on the three rotation axes (in order of ease of use).

[0143] The ranking of the frequency of use of the three rotation axes (ranking of ease of use) based on the time-series six-axis information of this controller 122 changes depending on the characteristics of the controller 122, such as the shape and structure, and therefore can also be considered characteristic information of the controller 122.

[0144] The HMD 121 also acquires six-axis information, which is position and orientation information of the controller 122, supplied from the controller 122 at predetermined time intervals, and stores the information in chronological order in a controller six-axis information storage unit 261. The HMD 121 also stores the position and orientation information of the controller 122 tracked by the tracking unit 230 in chronological order in a tracking information storage unit 260. As a result, in subsequent processing, the position and orientation information measured by the controller 122 itself and the position and orientation of the controller 122 measured by the HMD 121 are stored, making it possible to grasp changes in the position and orientation of the controller 122 in real time.

[0145] In the above, an example has been described in which six-axis information is detected based on the IMU 152 of the controller 122, and the order of the three rotation axes that are easy to use for the controller 122 is determined based on the detection results.

[0146] However, the HMD 121 may determine the order of the three rotation axes of the controller 122 that are easy to use based on the tracking results.

[0147] In this case, the controller 122 only needs to be able to detect its position and orientation by tracking based on the image captured by the camera 210 , so for example, the palm of the hand can also be used as the controller 122 .

[0148] <Position and Orientation Linked Setting Process> Next, the position and orientation linked setting process will be described with reference to the flowchart in FIG.

[0149] In steps S71 and S91, the control unit 201 in the HMD 121 and the control unit 301 in the controller 122 exchange IDs and start communication using a predetermined protocol by controlling the communication units 205 and 305, respectively. Note that if communication has been started in advance by the processing of the flowchart in Fig. 16, the processing of steps S71 and S91 may be omitted.

[0150] In step S92, the position and orientation linkage setting information management unit 322 of the control unit 301 in the controller 122 determines whether the user has operated the input unit 302 to request that the function assignment for each rotation axis (functions assigned to each rotation axis) be changed from the initial setting.

[0151] In addition, in the initial setting, the functions assigned to each rotation axis of the controller 122 are as follows: the first rotation axis, which is assumed to be the most frequently used and easiest to use, is assigned the function that is assumed to be the most frequently used related to operating the virtual object V; and the second rotation axis, which is assumed to be the second most frequently used and easiest to use, is assigned the function that is assumed to be the second most frequently used related to operating the virtual object V.

[0152] As the six-axis information of the controller 122 is repeatedly acquired, the initially expected order of frequency of use of the three rotation axes may change, so the initial setting can be changed as necessary.

[0153] In step S92, if a request is made to change the functions assigned to each rotation axis from the initial settings, the process proceeds to step S93.

[0154] In step S93, the position and orientation linkage setting information management unit 322 assigns the function of rotating the virtual object V to the first rotation axis, which is the most frequently used and easiest to use, based on the ranking of the three rotation axes stored in the position and orientation linkage setting information, and assigns the function of enlarging and reducing the virtual object V to the second most frequently used and second easiest to use rotation axis.

[0155] In step S92, if it is not required to change the function allocation for each rotation axis from the initial setting, the process of step S93 is skipped.

[0156] In step S94, the position and orientation linkage setting information management unit 322 supplies the HMD 121 with information on function allocation for each rotation axis, and requests the HMD 121 to present (display) a virtual object V for position and orientation linkage setting.

[0157] In step S72, the position and orientation linkage setting information management unit 232 of the control unit 201 in the HMD 121 acquires information on function allocation for each rotation axis sent from the controller 122, and controls the display control unit 234 based on a request to present (display) the virtual object V for position and orientation linkage setting to display the virtual object V for position and orientation linkage setting on the display 131.

[0158] This process enables the user 111 to view the virtual object V for position and orientation linkage setting displayed on the display 131. The virtual object V for position and orientation linkage setting cannot be rotated or enlarged or reduced by the operation of the controller 122, and can be viewed while fixed at a predetermined position.

[0159] Therefore, the user 111 can perform various position and orientation linkage settings while checking the positional relationship of his / her own controller 122 and viewing the virtual object V for position and orientation linkage settings displayed on the display 131 .

[0160] In step S73, the position and orientation linkage setting information management unit 232 controls the display control unit 234 to display on the display 131 text or the like that prompts the user to set the position (orientation) of the rotation axis of the virtual object, which is one of the position and orientation linkage settings.

[0161] With this display, the user 111 wearing the HMD 121 can move the controller 122 to the position of the rotation axis while looking at the displayed virtual object V, and determine the position (posture) of the rotation axis by pressing a specific button that constitutes the input unit 302 (151) at the desired position (posture) of the rotation axis.

[0162] In step S95, the position and orientation linkage setting information management unit 322 of the controller 122 determines whether the user has operated the input unit 302 (151) to set the position (orientation) of the rotation axis of the virtual object V, and the same processing is repeated until it is set.

[0163] Then, in step S95, if the user operates the buttons of the input unit 302 (151) to set the position and orientation of the rotation axis of the virtual object V, the process proceeds to step S95.

[0164] In step S96, the position and orientation linkage setting information management unit 322 acquires the six-axis information of the IMU 152 at the time when the user operates the button on the input unit 302 (151), and transmits it to the HMD 121 as position (orientation) information set as the position (orientation) of the rotation axis of the virtual object V.

[0165] When position information set as the position and orientation of the rotation axis of the virtual object V is transmitted from the controller 122, in step S74, the position and orientation linkage setting information management unit 232 of the HMD 121 acquires the position (orientation) information set as the position (orientation) of the rotation axis of the virtual object V from the controller 122.

[0166] In addition, the position and orientation linkage setting information management unit 232 tracks the controller 122 from the information captured by the camera 210, identifies the position of the controller 122, and generates position (orientation) information that is independently set as the position (orientation) of the rotation axis of the virtual object V on the HMD 121 side.

[0167] Then, the position and orientation linkage setting information management unit 232 stores the position and orientation information of the rotation axis of the virtual object V transmitted from the controller 122 and the position (orientation) information of the rotation axis of the virtual object V identified by tracking in the HMD 121 in association with the ID of the controller 122 in the position and orientation linkage setting information storage unit 262.

[0168] In step S75, the position and orientation linkage setting information management unit 232 controls the display control unit 234 to display on the display 131 text or the like that prompts the user to set the distance (positional relationship) between the virtual object V and the controller 122 as part of the position and orientation linkage settings.

[0169] With this display, the user 111 wearing the HMD 121 looks at the displayed virtual object V and moves the controller 122 so that the distance (positional relationship) between the virtual object V and the controller 122 is the same, and when the desired distance (positional relationship) between the virtual object V and the controller 122 is achieved, the user 111 presses a predetermined button that constitutes the input unit 302 (151), thereby setting the distance (positional relationship) between the virtual object V and the controller 122.

[0170] In step S97, the position and orientation linkage setting information management unit 322 of the controller 122 determines whether the user has operated the input unit 302 (151) to set the distance (positional relationship) between the virtual object V and the controller 122, and the same processing is repeated until the distance (positional relationship) is set.

[0171] Then, in step S97, if the user operates the buttons of the input unit 302 (151) to set the distance (positional relationship) between the virtual object V and the controller 122, the process proceeds to step S98.

[0172] In step S98, the position and posture linkage setting information management unit 322 acquires the six-axis information of the IMU 152 at the time when the user operates the button on the input unit 302 (151), and transmits it to the HMD 121 as position information set as the distance (positional relationship) between the virtual object V and the controller 122.

[0173] When the controller 122 transmits position information set as the distance (positional relationship) between the virtual object V and the controller 122, in step S76, the position and posture linkage setting information management unit 232 of the HMD 121 acquires the position information set as the distance (positional relationship) between the virtual object V and the controller 122 from the controller 122.

[0174] In addition, the position and orientation linkage setting information management unit 232 controls the tracking unit 230 to track the controller 122 from the information captured by the camera 210, identifies the position of the controller 122, and generates position information on the HMD 121 side that is set independently as the distance (positional relationship) between the virtual object V and the controller 122.

[0175] Then, the position and orientation linkage setting information management unit 232 stores information on the distance (positional relationship) between the virtual object V and the controller 122, which is identified from the position information that sets the distance (positional relationship) between the virtual object V and the controller 122 and which is transmitted from the controller 122, and the distance (positional relationship) between the virtual object V and the controller 122, which is identified from the position information that sets the distance (positional relationship) between the virtual object V and the controller 122, which is identified by tracking in the HMD 121, in association with the ID of the controller 122 in the position and orientation linkage setting information storage unit 262.

[0176] In step S77, the position and orientation linkage setting information management unit 232 controls the display control unit 234 to display on the display 131 text or the like that prompts the user to set the center position of the enlargement and reduction of the virtual object V, which is one of the position and orientation linkage settings.

[0177] With this display, the user 111 wearing the HMD 121 moves the controller 122 while looking at the displayed virtual object V so that it becomes the center position for scaling the virtual object V, and presses a specified button that constitutes the input unit 302 (151) at the desired center position for scaling the virtual object V, thereby setting the center position for scaling the virtual object V.

[0178] In step S99, the position and orientation linkage setting information management unit 322 of the controller 122 determines whether the user has operated the input unit 302 (151) to set the center position for scaling the virtual object V, and the same processing is repeated until it is set.

[0179] Then, in step S99, if the user operates the buttons of the input unit 302 (151) to set the center position of the enlargement / reduction of the virtual object V, the process proceeds to step S100.

[0180] In step S100, the position and orientation linkage setting information management unit 322 acquires the six-axis information of the IMU 152 at the time when the user operates the button on the input unit 302 (151), and transmits this information to the HMD 121 as position information set as the center position of the enlargement and reduction of the virtual object V.

[0181] When the controller 122 transmits position information set as the center position for scaling the virtual object V, in step S78, the position and orientation linkage setting information management unit 232 of the HMD 121 acquires the position information set as the center position for scaling the virtual object V from the controller 122.

[0182] In addition, the position and orientation linkage setting information management unit 232 controls the tracking unit 230 to track the controller 122 from the information captured by the camera 210, identifies the position of the controller 122, and generates position information on the HMD 121 side that is independently set as the center position for enlarging or reducing the virtual object V.

[0183] Then, the position and orientation linkage setting information management unit 232 stores the position information that sets the center position of the scaling of the virtual object V transmitted from the controller 122 and the center position of the scaling of the virtual object V identified by tracking in the HMD 121 in association with the ID of the controller 122 in the position and orientation linkage setting information storage unit 262.

[0184] In steps S79 and S101, the position and orientation linkage setting information management unit 322 of the HMD 121 and the position and orientation linkage setting information management unit 232 of the controller 122 share the position and orientation linkage setting information that they mutually possess, namely, information on function assignment for each rotation axis, the position (orientation) of the rotation axis, the distance (positional relationship) between the virtual object V and the controller 122, and the center position of zooming, and store these in their respective position and orientation linkage setting information storage units 262 and 332.

[0185] The above processing enables the position and orientation linkage setting information to be shared and held by the HMD 121 and the controller 122. Therefore, if the HMD 121 can acquire time-series six-axis information from the controller 122, it can rotate and enlarge / reduce the virtual object V using the information including the position and orientation linkage setting information and display it on the display 131.

[0186] In this example, the orientation of the rotation axis of the virtual object is fixed in the vertical direction, so the orientation of the rotation axis of the virtual object is not set; however, the orientation of the rotation axis may be set at the same time as setting the position of the rotation axis of the virtual object.

[0187] Furthermore, in the above processing, an example has been described in which the position (posture) of the rotation axis, the distance (positional relationship) between the virtual object V and the controller 122, and the center position of zooming are obtained by both values ​​based on information obtained by tracking the image from the camera 210 by the tracking unit 230 on the HMD 121 side and values ​​from information obtained by the IMU 152 of the controller 122.

[0188] However, to rotate or scale the virtual object V, it is sufficient to use at least one of values ​​based on information obtained by tracking the image from the camera 210 on the HMD 121 side and values ​​from information obtained by the IMU 152 of the controller 122.

[0189] As described above, the controller 122 may be, for example, a palm, whose position and orientation can be detected by tracking based on an image captured by the camera 210 .

[0190] In other words, when the controller 122 is held in the palm of the hand, the HMD 121 determines and stores the order of the three axes of rotation based on tracking, and therefore the HMD 121 also allocates functions to each axis of rotation.

[0191] Furthermore, the position of the rotation axis, the distance between the virtual object V and the palm that serves as the controller 122, and the center position of the zooming may be set, for example, by placing one palm that will be used as the controller 122 at the desired position and operating the input unit 202 provided on the HMD 121 with the other hand.

[0192] Therefore, when the controller 122 is held in the palm of the hand, the position and orientation link setting process described with reference to the flowchart in FIG. 17 is substantially realized by only steps S72 to S79 ​​in the flowchart in FIG.

[0193] Furthermore, in steps S79 and S101, the process of mutually sharing the position and orientation linkage setting information between the HMD 121 and the controller 122 is not essential and may be omitted.

[0194] However, this processing causes the position and orientation linkage setting information to be shared between the HMD 121 and the controller 122, and when either one of them operates with another controller 122 or another HMD 121, the position and orientation linkage setting information held therein can be transferred to the other controller 122 or another HMD 121, making it possible to make the focus areas of the virtual object V that the user 111 who set the position and orientation linkage setting information wants to focus on easier to operate and view.

[0195] In other words, by mutually sharing the position and orientation linkage setting information between the HMD 121 and the controller 122, the intentions and intentions of the user who set the position and orientation linkage setting information regarding viewing the virtual object V can be conveyed to other users.

[0196] <Virtual Object Manipulation Processing> Next, the virtual object manipulation processing will be described with reference to the flowchart in Fig. 18. Note that this processing is premised on the fact that the position and orientation linkage setting information of the corresponding controller 122 has been set for at least the HMD 121 by the position and orientation linkage setting processing described with reference to Fig. 17.

[0197] In steps S111 and S131, the control unit 201 in the HMD 121 and the control unit 301 in the controller 122 exchange IDs and start communication using a predetermined protocol by controlling the communication units 205 and 305, respectively. Note that if communication has been started in advance by the processing of the flowcharts in Figures 16 and 17, the processing of steps S111 and S131 may be omitted.

[0198] In step S112, the six-axis information control unit 321 of the control unit 301 in the controller 122 determines, based on the six-axis information stored in chronological order in the six-axis information storage unit 331, whether or not the controller 122 is rotating by a rotation operation around the rotation axis set as the first rotation axis, i.e., the rotation axis that controls the rotation of the virtual object V.

[0199] If it is determined in step S112 that rotation is occurring, the process proceeds to step S113.

[0200] In step S113 , the six-axis information control unit 321 transmits the amount of rotation of the first rotation axis to the HMD 121 .

[0201] In step S132, the display control unit 234 determines whether or not there is rotation about the first rotation axis of the controller 122 based on at least one of the movement of the controller 122 tracked by the tracking unit 230 from the image captured by the camera 210 and the notification of the amount of rotation from the controller 122.

[0202] If it is determined in step S132 that the first rotation axis of the controller 122 is rotating, the process proceeds to step S133.

[0203] In step S133, the display control unit 234 generates an image of rotating the virtual object V based on the movement of the controller 122 tracked from the image captured by the camera 210, the amount of rotation of the first rotation axis based on at least one of notifications from the controller 122, position and orientation linkage setting information, and six-axis information (its own position and orientation) acquired from the IMU 209, and displays the image on the display 131.

[0204] In step S114, the six-axis information control unit 321 of the control unit 301 in the controller 122 determines, based on the six-axis information stored in chronological order in the six-axis information storage unit 331, whether or not the controller 122 is being rotated by a rotation operation around the rotation axis set as the second rotation axis, i.e., the rotation axis that controls the enlargement and reduction of the virtual object V.

[0205] If it is determined in step S114 that rotation is occurring, the process proceeds to step S115.

[0206] In step S115 , the six-axis information control unit 321 transmits the amount of rotation about the second rotation axis to the HMD 121 .

[0207] In step S134, the display control unit 234 determines whether or not there is rotation about the second rotation axis of the controller 122 based on at least one of the movement of the controller 122 tracked from the image captured by the camera 210 and the notification of the amount of rotation from the controller 122.

[0208] If it is determined in step S134 that the second rotation axis of the controller 122 is rotating, the process proceeds to step S135.

[0209] In step S135, the display control unit 234 generates an image that enlarges or reduces the virtual object V based on the movement of the controller 122 tracked from the image captured by the camera 210, the amount of rotation of the second rotation axis based on at least one of notifications from the controller 122, position and orientation linkage setting information, and six-axis information (its own position and orientation) acquired from the IMU 209, and displays the image on the display 131.

[0210] In steps S116 and S136, it is determined whether or not an instruction to end the process has been given. If an instruction to end the process has not been given, the process returns to steps S112 and S132, and the subsequent processes are repeated.

[0211] If an instruction to end the process is given in steps S116 and S136, the process ends.

[0212] If no rotation is detected in steps S112 and S114, the processes in steps S113 and S115 are skipped.

[0213] If no rotation is detected in steps S132 and S134, the processes in steps S133 and S135 are skipped.

[0214] Through the above processing, when operating the virtual object V, the rotation operation of the rotation axis that is frequently operated and easy to operate is assigned to the operation that is frequently operated on the virtual object V, thereby improving the usability of the virtual object V.

[0215] In addition, the rotation operations of the rotation axes that are frequently operated and easy to operate are statistically determined based on the characteristics of the controller 122, such as the shape and structure, and the ranking of the ease of use of the three rotation axes is determined based on the characteristics of the controller 122.

[0216] Therefore, based on the ranking of ease of use of the three rotation axes, the operation functions with high frequency of operation of the virtual object V are assigned to the rotation operations of the rotation axes that are operated frequently and easily.In other words, in accordance with the characteristics of the controller 122, functions related to the operation of the virtual object V are appropriately assigned to the rotation operations of each rotation axis of the controller.

[0217] As a result, easier operations of the controller 122 are assigned to more frequently operated virtual objects, thereby reducing the burden associated with operating the virtual objects and enabling comfortable and satisfying operations.

[0218] In the above, among the position and orientation linkage setting information, the information on the position and orientation of the rotation axis of the virtual object V, the distance between the virtual object V and the controller 122, and the center position for zooming in and out includes both information obtained by tracking from the image captured by the camera 210 on the HMD 121 side and information obtained by the IMU 152 of the controller 122, so the display control unit 234 may use both of these to generate an image that rotates the virtual object V or an image that zooms in and out with high precision.

[0219] Furthermore, the display control unit 234 may use at least one of information obtained by tracking an image captured by the camera 210 on the HMD 121 side using the tracking unit 230 and information obtained by the IMU 152 of the controller 122 to generate an image that rotates the virtual object V or an image that enlarges or reduces the virtual object V, for information regarding the position and orientation of the rotation axis of the virtual object V, the distance between the virtual object V and the controller 122, and the center position for enlarging or reducing the virtual object V.

[0220] Furthermore, in the above, we have explained an example in which the controller 122 itself detects whether or not it is rotating and notifies the HMD 121, but the HMD 121 can track the controller 122 based on images captured by the camera 210, so it is sufficient if the controller 122 is held by the user 111.

[0221] That is, the processes of steps S111 to S116 by the controller 122 are not essential to the process of the flowchart of Fig. 18. Therefore, the process of the flowchart of Fig. 18 may be only the processes of steps S132 to S136.

[0222] That is, when the controller 122 is held in the palm of the hand, the virtual object operation process is substantially realized by only the processes of steps S132 to S136 in the flowchart of FIG.

[0223] Furthermore, in the above virtual object operation process, the explanation is given on the assumption that all of the set information is used when using the position and orientation linkage setting information, but it is not necessary to use all of the set information. For example, in the above, an example has been explained in which only the position of the rotation axis of the virtual object V is set and the orientation is fixed in the vertical direction, but even if the position and orientation are set separately, it is also possible to use only the position information and use the orientation in a state where it is fixed in the vertical direction.

[0224] <Position and orientation linked setting information transfer process> The above has described an example in which a first user sets position and orientation linked setting information in the controller 122, and uses the set position and orientation linked setting information to view and operate a virtual object V using his or her own HMD 121.

[0225] However, the controller 122 holding the position and orientation linkage setting information set by the first user may be handed over to the second user, and the information may be displayed on the HMD 121 held by the second user.

[0226] In this case, the controller 122 that holds the position and orientation linkage setting information set by the first user needs to transfer the position and orientation linkage setting information that it holds to the HMD 121 held by the second user.

[0227] Therefore, referring to the flowchart in Figure 19, we will explain the position and orientation linked setting information transfer process in which the controller 122, which holds the position and orientation linked setting information set by the first user, transfers the position and orientation linked setting information to the HMD 121 of the second user.

[0228] In steps S151 and S171, the control unit 201 in the HMD 121 and the control unit 301 in the controller 122 control the communication units 205 and 305, respectively, to exchange IDs with each other and start communication according to a predetermined protocol.

[0229] In step S152, the position and orientation linkage setting information management unit 322 of the control unit 301 in the controller 122 reads out the position and orientation linkage setting information set by the first user that is stored in the position and orientation linkage setting information storage unit 332 of the storage unit 304, and transfers it to the HMD 121 of the second user.

[0230] In step S172, the position and orientation linkage setting information management unit 232 of the control unit 201 (132) in the second user's HMD 121 acquires the position and orientation linkage setting information set by the first user transferred from the controller 122, and stores it in the position and orientation linkage setting information storage unit 262 of the storage unit 204.

[0231] Through the above processing, a second user who has inherited the controller 122 in which the position and orientation linkage setting information has been set by the first user can operate the virtual object V using the position and orientation linkage setting information set by the first user in his or her own HMD 121.

[0232] As a result, as explained with reference to Figure 7, the position and posture linkage setting information is set so that the first user can easily view the part of the virtual object V that the first user wants to focus on, making it easier for the part of the virtual object V that the first user wants to focus on to be displayed, thereby making it possible to prevent the second user from overlooking the part of interest.

[0233] As described above, when the controller 122 is implemented in the palm of the hand, the controller 122 itself does not exist, and therefore the position and orientation linkage setting information is simply stored in the HMD 121. In this case, the same operation can be achieved by having the HMD 121 of the first user directly transfer the position and orientation linkage setting information set by the first user to the HMD 121 held by the second user. The position and orientation linkage setting information transfer process in this case is basically the same as the process described with reference to the flowchart of FIG. 19 , and therefore a description thereof will be omitted.

[0234] Furthermore, although the above description has mainly focused on an example in which a common controller 122 is used by multiple users 111, multiple users 111 may each hold and use their own individual controller 122.

[0235] In this case, after the first user 111A sets the position of the rotation axis, etc. and registers the position and orientation linkage setting information, the second user 111B can use his / her own controller 122B and HMD 121B to execute the position and orientation linkage setting process described with reference to the flowchart in Figure 17, for example, to set the position and orientation of the rotation axis of the virtual object V to a position that is easy for the second user to see, move the center position of zooming in and out, or duplicate and display the virtual object.

[0236] In addition, the position and orientation of the rotation axis may be set to a display position that is easy to see, and the center position of the zoom may be moved using information such as the position of the second user's controller 122, the orientation of the second user's HMD 121, and height and distance according to information such as the display virtual image distance and angle of view.

[0237] Similarly, the position of the rotation axis and the center position of the zooming in and out may be moved to a display position that is easy to see by using information on the height and arm length of the user 111 that has been acquired in advance.

[0238] That is, for example, for a user 111 with long arms, the distance between the virtual object V and the controller 122 may be set to be relatively far, so that the virtual object V is displayed at a position that is far away from the user.

[0239] Also, multiple users may use controllers with different characteristics.

[0240] That is, for example, if the direction of the rotation axis that is easy to rotate, which is a characteristic of the controller 122 used by each user, is different, even if the direction of the rotation axis of the virtual object V is common, the direction of the rotation axis that is easy to rotate can be set for each controller 122, so that each user can comfortably rotate the virtual object V with their respective controller 122.

[0241] If the controller 122 used by some of the multiple users 111 is not capable of tracking in six-axis directions, a mouse that can be controlled in six-axis directions may be used as the controller 122.

[0242] Even if the controller 122 is capable of tracking in six axial directions, if rotational operation is relatively difficult, for example, if a joystick is installed, the joystick may be used as the controller 122.

[0243] For example, even if only the first user has a desired portion of the virtual object V to be corrected, the position of the rotation axis or the center position of the enlargement / reduction may be set around the portion to be corrected, and the position and orientation may be registered as position and orientation linked setting information. Then, based on the registered position and orientation linked setting information, the portion to be corrected may be checked at a different time, and the portion to be corrected may be corrected at a different time.

[0244] As described above, according to the present disclosure, the easier the operation of the controller 122 is, depending on the characteristics of the controller 122 due to its shape, weight, etc., the more frequently the function that is operated on the virtual object V is assigned to it. This makes it possible to make effective and efficient control of the position and attitude of the virtual object V, reduce the effort involved, and improve satisfaction.

[0245] Furthermore, by setting specific positions and orientations in a linked manner, such as the position and orientation of the rotation axis in a rotation operation on the virtual object V, and the center position in a zoom operation on the virtual object V, it is possible to unify the operational feel when multiple users use the same controller 122, thereby making it possible to effectively and efficiently control the position and orientation when multiple users view a common area of ​​interest, reducing the effort involved and improving satisfaction.

[0246] As a result, it becomes possible to improve the effectiveness, efficiency, and satisfaction of controlling the position and posture of virtual objects using XR technology, thereby improving so-called usability.

[0247] <<3. First Modified Example>> In the above, as shown in FIG. 20 , when the user 111 rotates the controller 122 that the user 111 is holding in the Ry1 direction around the rotation axis AXy, which is the yaw direction, an image VP1 is displayed on the HMD 121, basically showing the virtual object V rotated in the Ry1′ direction around the rotation axis AXy′ by the same amount of rotation as the amount of rotation of the controller 122 (the same rotation angle as the rotation angle of the controller 122).

[0248] However, when the user 111 holds the controller 122 as shown in FIG. 20, the range of motion of a human hand is actually limited to about 180 degrees, and it is not possible to rotate the virtual object V by any greater amount (rotation angle), so the rotation of the virtual object V is also limited to about 180 degrees.

[0249] Therefore, when the controller 122 is maintained in a state where it has rotated in a predetermined direction beyond a predetermined rotation amount, the virtual object V may continue to rotate in the predetermined direction at a predetermined rotational speed (angular velocity).

[0250] That is, as shown in FIG. 21, for example, when the user 111 continues to hold the controller 122 and rotate it in the Rp31 direction around the rotation axis AXp, which is the pitch direction, by, for example, 30 degrees or more, the virtual object V may continue to rotate as shown in images VP11 to VP13 depending on the duration.

[0251] Image VP11 shows that as a result of user 111 continuing to rotate controller 122 by 30 degrees or more in the Rp31 direction, virtual object V begins to rotate in the Ry31' direction around rotation axis AXy', and one second later, rotates a predetermined amount (rotation angle) (e.g., 30 degrees) and changes into virtual object V31.

[0252] Image VP12 shows that as the user 111 continues to rotate the controller 122 by 30 degrees or more in the Rp31 direction, the virtual object V starts to rotate in the Ry31'' direction, and two seconds later, the virtual object V rotates by an amount of rotation (rotation angle) (e.g., 60 degrees > 30 degrees) and changes into virtual object V32.

[0253] Image VP13 shows that as a result of the user 111 continuing to rotate the controller 122 in the Rp31 direction by more than 30 degrees, virtual object V starts to rotate in the Ry31''' direction, and three seconds later rotates by an amount of rotation (rotation angle) (e.g., 90 degrees > 60 degrees) and changes into virtual object V33.

[0254] In this way, the user 111 can rotate the virtual object V in accordance with the rotation angle by rotating the controller 122 in the Ry direction centered on the rotation axis AXy, which is the pitch direction, and can also rotate the virtual object V in the Ry direction centered on the rotation axis AXy in accordance with the duration by simply continuing to rotate the controller 122 in the Rp direction centered on the rotation axis AXp, which is the pitch direction, by a predetermined angle or more.

[0255] In other words, the controller 122 can be rotated by an amount corresponding to the desired angle in the yaw direction around the first rotation axis, which is the easiest to operate, making it possible to fine-tune the rotation angle of the virtual object V.

[0256] Furthermore, by simply continuing to rotate the controller 122 in the pitch direction about the second rotation axis, which is the second easiest to operate, by a predetermined angle or more, the virtual object V can be rotated according to the duration, making it possible to rotate the virtual object V to a large angle with a relatively rough operation.

[0257] <<4. Second Modification>> In the above, an example has been described in which the rotation operation of the virtual object V is switched by switching between a rotation operation about the first rotation axis, which is the easiest to operate, and a rotation operation about the second rotation axis, which is the second easiest to operate, but the operation performed on the first rotation axis may also be switched on the second rotation axis. In this way, the following operations can be realized.

[0258] First, as shown in state (a) of Figure 22, when the user 111 rotates the controller 122 that he or she is holding in the direction Ry51 around the rotation axis AXy, which is the yaw direction, an image VP31 is displayed on the HMD 121, showing the virtual object V rotated in the direction Ry51' around the rotation axis AXy' by the same amount of rotation as the controller 122 (the same rotation angle as the rotation angle of the controller 122).

[0259] Next, as shown in state (b) of FIG. 22, the user 111 rotates the controller 122 in his / her hand in the direction of Rp61 by a predetermined angle around the rotation axis AXp, which is the pitch direction.

[0260] Then, as shown in state (c) of Figure 22, when the user 111 rotates the controller 122 that he or she is holding in the Ry51 direction around the rotation axis AXy, which is the yaw direction, an image VP32 is displayed on the HMD 121 in which the virtual object V has rotated in the Rp71' direction around the rotation axis AXp' by the same amount of rotation as the controller 122 (the same rotation angle as the rotation angle of the controller 122).

[0261] That is, when the controller 122 is rotated in the pitch direction by a predetermined angle, the rotation of the virtual object V switches from rotation in the Ry51' direction around the rotation axis AXy' to rotation in the Rp71' direction around the rotation axis AXp'.

[0262] By doing this, the user 111 can rotate the controller 122 in the Ry direction around the rotation axis AXy, which is the yaw direction, and rotate the virtual object V in both the yaw direction and the pitch direction depending on the rotation angle.

[0263] In other words, by rotating the controller 122 in the yaw direction around the first rotation axis, which is the easiest to operate, the virtual object V can be rotated in both the yaw direction and the pitch direction by an amount of rotation corresponding to the desired angle of rotation, making it possible to fine-tune the rotation angle of the virtual object V in both the yaw direction and the pitch direction.

[0264] In this case, the rotation axis of the virtual object V can be switched by simply rotating the controller 122 by a predetermined angle or more in the first direction in the pitch direction centered on the second rotation axis, which is the second easiest to operate, so that the rotation axis of the virtual object V can be switched with a relatively rough operation.

[0265] Although not shown, when the operation of rotating the controller 122 in the pitch direction by a predetermined angle or more is performed again, the rotation axis of the virtual object V returns from the pitch direction to the yaw direction.

[0266] Furthermore, when the controller 122 is rotated by a predetermined angle or more in a second direction opposite to the first direction in the pitch direction about a second rotation axis that is the second easiest to operate, the rotation axis of the virtual object V may be switched to the roll direction. In this way, the user 111 can adjust the rotation angle in the roll direction of the virtual object V by adjusting the amount of rotation in the yaw direction about the first rotation axis that is the easiest to operate the controller 122.

[0267] Furthermore, when a first operation is performed to rotate the controller 122 in a first direction in the pitch direction centered on the second rotation axis, which is the second easiest to operate, by a predetermined angle or more, the rotation axis of the virtual object V switches to the pitch direction, and then when a second operation is performed, it switches to the roll direction, and then when a third operation is performed, it returns to the yaw direction, and thereafter, when similar operations are repeated, the rotation axis of the virtual object V switches in the order of the yaw direction, pitch direction, and roll direction.

[0268] In this case, each time an operation is performed to rotate the controller 122 by a predetermined angle or more in a second direction opposite to the first direction in the pitch direction around the second rotation axis that is the second easiest to operate, the rotation axis of the virtual object V may be switched in the order of the roll direction, pitch direction, and yaw direction, in the reverse order of the yaw direction, pitch direction, and roll direction.

[0269] <<5. Third Modification>> In the above, an example has been described in which the rotation axis of the virtual object V performed on the first rotation axis is switched by an operation using the second rotation axis. However, although the rotation operations performed on the virtual object V on the first rotation axis and the second rotation axis are the same, it is also possible to switch the magnification of the amount of rotation of the virtual object V relative to the amount of rotation of the controller 122.

[0270] That is, as shown in state (a) of Figure 23, when the user 111 rotates the controller 122 that he or she is holding in the Ry91 direction around the rotation axis AXy, which is the yaw direction, an image VP51 is displayed on the HMD 121 in which the virtual object V has rotated in the Ry91' direction around the rotation axis AXy' by the same amount of rotation as the amount of rotation of the controller 122 (for example, 90 degrees) (the same rotation angle as the rotation angle of the controller 122) (for example, 90 degrees).

[0271] On the other hand, as shown in state (b) of Figure 23, when the user 111 rotates the controller 122 that he or she is holding in the Rp92 direction around the rotation axis AXp, which is the pitch direction, an image VP52 is displayed on the HMD 121, showing the virtual object V rotated in the Ry92' direction around the rotation axis AXy' by an amount of rotation that is twice the amount of rotation of the controller 122 (e.g., 90 degrees) (the same rotation angle as the rotation angle of the controller 122) (e.g., 180 degrees).

[0272] By doing this, the user 111 can rotate the controller 122 in the Ry direction around the rotation axis AXy, which is the yaw direction, to rotate the virtual object V in the yaw direction by the same rotation angle as the rotation angle.

[0273] In addition, the user 111 can rotate the controller 122 in the Rp direction around the rotation axis AXp, which is the pitch direction, to rotate the virtual object V in the yaw direction by a rotation angle that is a predetermined magnification of the rotation angle.

[0274] In other words, by rotating the controller 122 in the yaw direction around the first rotation axis, which is the easiest to operate, the controller 122 can be rotated by an amount corresponding to the desired angle of rotation, making it possible to fine-tune the rotation angle of the virtual object V in the yaw direction.

[0275] On the other hand, by rotating the controller 122 in the pitch direction around the second rotation axis, which is the second easiest to operate, it is possible to rotate the virtual object V by an amount of rotation that is several times (or a fraction) of the angle of rotation, so that it is possible to roughly rotate the yaw direction of the virtual object V by a large (or small) angle.

[0276] The operation of rotating the virtual object V may be configured to rotate discretely rather than completely synchronized with the rotation of the controller 122. For example, in the case of the controller 122 for which fine adjustment of rotation is difficult, the virtual object V may be configured to rotate by a predetermined angle in the yaw direction when the controller 122 is rotated by a predetermined angle or more.

[0277] <<6. Fourth Modification>> Various operations that can be applied to the virtual object V using the controller 122 are envisioned, and an operation is envisioned in which, when a straight virtual line called a ray extending from the controller 122 passes through the virtual object V, a point at which the ray intersects with an end of the virtual object V can be selected.

[0278] In this operation, when the ray passes through the virtual object V, there may be multiple points where the ray intersects with the end of the virtual object V, and it may be possible to select these multiple intersection points using the second rotation axis of the controller 122.

[0279] 24, it is assumed that a ray Lx extends from near the center of gravity of the controller 122 to a virtual object V111. In this case, the ray Lx and the virtual object V111 intersect at three points, intersection points PS1 to PS3, and it is assumed that the intersection point PS1 is selected by default.

[0280] At this time, when the user 111 rotates the controller 122 by a predetermined angle (e.g., 15 degrees) or more in the Rp111 direction centered on the rotation axis AXp, which is the pitch direction, the state in which intersection point PS1 is selected changes to the state in which intersection point PS2 is selected, and when the controller is rotated again by a predetermined angle (e.g., 15 degrees) or more, the state in which intersection point PS2 is selected changes to the state in which intersection point PS3 is selected, and when the controller is rotated again by a predetermined angle (e.g., 15 degrees) or more, the state in which intersection point PS3 is selected changes back to the state in which intersection point PS1 is selected, and thereafter the selected intersection point may be switched in the same manner.

[0281] <<7. Fifth Modification>> In the above, an example has been described in which, among operations applied to the virtual object V using the controller 122, only one axis of rotation (yaw direction) is set for the virtual object V. However, any two axes of the yaw direction, pitch direction, and roll direction may be set, or all three axes may be set.

[0282] That is, for example, as shown in FIG. 25, an image VP81 showing a virtual object V121 rotated by an angle corresponding to the amount of rotation applied by the user 111 to the controller 122 held in each of the three axial directions may be displayed on the HMD 121.

[0283] 25 , when the controller 122 is rotated in the Ry direction around the rotation axis AXy, which is the yaw direction, the virtual object V rotates in the Ry' direction around the rotation axis AXy' in accordance with the amount of rotation of the controller 122. Furthermore, when the controller 122 is rotated in the Rp direction around the rotation axis AXp, which is the pitch direction, the virtual object V rotates in the Rp' direction around the rotation axis AXp' in accordance with the amount of rotation of the controller 122. Furthermore, when the controller 122 is rotated in the Rr direction around the rotation axis AXr, which is the roll direction, the virtual object V rotates in the Rr' direction around the rotation axis AXr' in accordance with the amount of rotation of the controller 122.

[0284] In addition, the controller 122 may be able to be switched between a state in which it has the function of rotating the virtual object V in all three axial directions, which are the operation directions, and a state in which it has the function in only one axial direction.

[0285] That is, for example, under normal circumstances, rotation in all three axial directions is possible, but under specific conditions, for example, when it is necessary to focus on a specific virtual object, only one axis of rotation may be used, and the specific virtual object may be rotated using the axis of rotation that is easiest for the user 111 to operate among the three axial rotation operations performed by operating the controller 122.

[0286] In addition to the first to fifth variants described above, there are many variations in the functions that can be assigned to the first rotation axis, second rotation axis, and third rotation axis that can be set as the operation direction of the controller 122, depending on the application of the information processing system 101.

[0287] Figure 26 shows examples of functions that can be assigned to the first rotation axis, the second rotation axis, and the third rotation axis depending on the use of the information processing system 101, and from left to right, shows examples of use, functions that can be assigned to the first rotation axis, functions that can be assigned to the second rotation axis, and functions that can be assigned to the third rotation axis.

[0288] As shown in the top row, if the application is rough 3D CAD assembly, the first rotation axis is assigned to rotate the virtual object V, the second rotation axis is assigned to scale the virtual object V, and the third rotation axis is unassigned (N / A).

[0289] As shown in the second row, if the application is looking around a detailed 3D model, the first rotation axis is assigned to rotate the virtual object V, the second rotation axis is assigned to move the virtual object V, and the third rotation axis is unassigned (N / A).

[0290] As shown in the third row, when the application is to look around a rough 3D model, the first rotation axis is assigned to rotate the virtual object V by angle input, the second rotation axis is assigned to rotate the virtual object V by angular velocity input, and the third rotation axis is unassigned (N / A).

[0291] As shown in the fourth row, when the application is design checking, the first rotation axis is assigned to rotate the virtual object V, the second rotation axis is assigned to scale the virtual object V, and the third rotation axis is assigned to move the cross-sectional position.

[0292] As shown in the fifth row, when the application is exhibition layout consideration, the first rotation axis is assigned to the rotation of the virtual object V (normal adjustment), the second rotation axis is assigned to the rotation of the virtual object V (fine adjustment), and the third rotation axis is unassigned (N / A).

[0293] As shown in the sixth row, when the application is a target specification operation on a virtual object, the first rotation axis is assigned to scaling the virtual object V, the second rotation axis is assigned to specifying a point or face to be selected with a ray on the virtual object V, and the third rotation axis is unassigned (N / A).

[0294] The functions for each rotation axis as shown in FIG. 26 can be switched depending on the application by being recorded in the position and orientation linkage setting information.

[0295] <<8. Sixth Modification>> In the above, an example has been described in which the rotation axis of the virtual object V is determined according to the attitude of the controller 122, but the rotation axis of the virtual object V may also be determined according to the shape of the controller 122.

[0296] For example, consider the case where a virtual object V171 as shown in the left part of FIG. 27 is operated with a controller 122 as shown in the second from the left of FIG.

[0297] The controller 122 in Figure 27 is provided with a characteristic structure portion 122W that has a characteristic structure for facilitating recognition by images such as a ring structure, which makes it easier to capture posture changes of the controller 122 when tracking using images captured by the camera 210 of the HMD 121 and improves tracking accuracy.

[0298] For example, as shown from the left in Figure 27, if the characteristic structure portion 122W has a ring-shaped structure, it is possible to assume that a disk-shaped turntable FL exists on this characteristic structure portion 122W, and to display the virtual object V171 so that its position and posture change in tandem so that it appears as if it is attached to the turntable FL, as shown in the right part of Figure 27.

[0299] That is, as shown in the second from the right in FIG. 27, when the controller 122 is held so that the turntable FL on the controller 122 is perpendicular to the vertical direction, the virtual object V171 is displayed as if it were placed perpendicular to the vertical direction, just like the turntable FL.

[0300] In contrast, as shown in the right part of Figure 27, when the rotation axis AXy of the controller 122 is tilted toward the rotation axis AXy', the turntable FL is tilted toward the turntable FL' in accordance with the tilt, and the virtual object V171 is also displayed as a tilted virtual object V171'.

[0301] In this way, the virtual object V171 is displayed in accordance with the movement of the controller 122, so that the virtual object V171 can be operated using the controller 122 more intuitively.

[0302] In addition, at this time, the size at which the virtual object V171 is displayed may be set according to the size of the characteristic structural portion 122W, and by doing so, it becomes possible to make the operation of the virtual object V171 using the controller 122 even more intuitive.

[0303] <<9. Seventh Modification>> In the above, an example has been described in which the first user sets the position and attitude (direction) of the rotation axis for rotating the virtual object V. However, conversely, the first user may not specify the position and attitude (direction) of the rotation axis, and the second user may dynamically adjust them when looking around.

[0304] For example, the second user may be allowed to rotate the virtual object V only in the same direction as the rotation axis of the controller 122 that is easy to rotate while pressing a predetermined button on the operation unit 151 of the controller 122.

[0305] The direction of this rotation axis may change dynamically while a button on the controller 122 is being pressed, or may be fixed in the direction it is in at the moment a predetermined button on the operation unit 151 is pressed.

[0306] In this case, the position of the rotation axis of the virtual object V191 may be aligned with the rotation axis of the controller 122, as shown in the left part of Fig. 28. Alternatively, the rotation axis of the controller 122 may be the central axis of the virtual object V191, as shown in the right part of Fig. 28.

[0307] That is, in the left part of Figure 28, when the user 111 holds the controller 122 and rotates it in the Ry191 direction around a predetermined rotation axis AXy191, the virtual object V191 also rotates around the rotation axis AXy191 by an angle corresponding to the amount of rotation of the controller 122.

[0308] Also, in the right part of Figure 28, when the user 111 holds the controller 122 and rotates it in the Ry191 direction around a predetermined rotation axis AXy191, the virtual object V191 rotates around the central axis AXy191' of the virtual object V191 by an angle corresponding to the amount of rotation of the controller 122.

[0309] <<10. Eighth Modification>> When displaying information about a virtual object, the information to be displayed may be switched depending on the operation history of the first user and the operation method of the second user.

[0310] For example, numerical values ​​on CAD data may be displayed to other users only when the posture of the virtual object when displayed is similar to (or the same as) the posture of the virtual object at the time the first user sets the numerical value.

[0311] For example, when sharing numerical values ​​such as the size of a specific part or clearance in CAD data with others, displaying all the numerical values ​​as shown in FIG. 29 may be distracting to other users.

[0312] That is, in Figure 29, a first user 111A holds a controller 122 and views a virtual object V211 and an image VP91 using an HMD 121 that he or she is wearing, and after inputting and setting all the numerical values ​​indicating the sizes of parts, gaps, etc., he or she hands over the controller 122 to a second user 111B.

[0313] Thereafter, the second user 111B uses the controller 122 to view the image VP91 using his / her HMD 121B, and tries to find the numerical value indicating the size of the engine, which is his / her objective.

[0314] However, since multiple values ​​other than those for the engine are displayed, it can be difficult to find the value that indicates the desired engine size.

[0315] Therefore, as shown in image VP101 on the left side of FIG. 30, the display posture of virtual object V211 when first user 111A inputs the engine value (3 m) desired by second user 111B is maintained.

[0316] As shown in image VP101' on the right side of Figure 30, the numerical value of the target engine (3m) may be displayed only when the second user views virtual object V211 in a position similar to the display position when the second user inputs the retained engine numerical value.

[0317] This allows second user 111B to view the target numerical value only when first user 111A assumes a display posture close to the display posture in which first user 111A inputs the numerical value for each target.

[0318] In this example, when a first user 111A is viewing in a display position similar to the display position when the first user 111A inputs the engine numerical value, only the engine numerical value is displayed when the second user 111B is viewing, but the same applies to other configurations.

[0319] For example, when second user 111B watches in a display position close to the display position when first user 111A inputs the numerical value of the tail, only the numerical value of the tail is displayed.

[0320] That is, when second user 111B watches in a display position close to the display position in which first user 111A inputs the numerical values ​​for all body parts, only the numerical values ​​for each body part are displayed.

[0321] As a result, the numerical values ​​are displayed only in the display posture when the numerical values ​​are input, and the numerical values ​​of only the specific part are displayed only when the display posture required for viewing the specific part is taken, so that it becomes possible to properly check only the numerical values ​​of the target part by simply finding and viewing the target part.

[0322] For example, information associating the display attitude with an input numerical value may be registered together with the position and attitude linkage setting information, and the display control unit 234 may switch the displayed numerical value according to the display attitude. Specifically, the display attitude here refers to an operation history added to a virtual object that indicates which position of the virtual object was viewed, and is an operation history that indicates, for example, the angle at which the virtual object was rotated for viewing, or the size at which the virtual object was enlarged or reduced for viewing.

[0323] As a result, it becomes possible to properly read the numerical values ​​that one wants to check, thereby improving usability.

[0324] <<11. Ninth Modification>> When a first user uses the controller 122 to display a virtual object, the first user may set the cross-sectional position when the cross-section is displayed, the movable direction of the cross-sectional position, and the movable range as position and orientation linkage setting information, and share this information with a second user who uses the same controller 122.

[0325] For example, if you want to check the area around the screw hole H of a part such as virtual object V251 in Figure 31, it is expected that the cross section you want to check for virtual object V251 will be set to cross section Svz passing through the center Sv of the screw hole H, as shown in virtual object V251'''.

[0326] In this case, it is expected that the cross section will attempt to confirm information about a cross section parallel to the direction in which the screw is inserted into the screw hole H, but it is expected that there will be no need to confirm the cross section Sv away from the screw hole H, as shown by the virtual object V251' in Figure 31, or the cross section Sh in a direction perpendicular to the screw hole H, as shown by the virtual object V251'' in Figure 31.

[0327] Therefore, as shown by virtual object V251''', the first user can use controller 122 to set position and posture linkage setting information that limits the direction and range in which the cross section is moved to the radius of screw hole H from the center Sv of screw hole H as shown by arrow Zs, and then hand over controller 122 to the second user, thereby sharing the restrictions on the direction and range in which the cross section is moved with the second user.

[0328] This prevents unnecessary cross sections from being displayed, making it impossible to identify which part the cross section refers to, and also makes it possible to quickly find the cross section you want to check, thereby improving usability.

[0329] In addition to restricting the direction and range of the cross section, the size of the virtual object may also be restricted.

[0330] That is, the first user may set the upper and lower limits of the display size of the virtual object V that the first user expects, and then hand over the controller 122 to the second user.

[0331] This makes it possible to prevent a situation in which the second user displays a virtual object at an extremely large or small size that the first user did not expect, leaving the first user unsure of where to focus and check the virtual object.

[0332] Furthermore, in addition to restricting the direction and range of the cross section or restricting the size, restrictions may also be placed on the distance (positional relationship) between the virtual object V and the controller 122 depending on the characteristics of the controller 122, such as its shape.

[0333] <<12. Tenth Modification>> After registering the position and orientation linkage setting information between the controller 122 and the virtual object V, the display form of the virtual object V may be changed by an additional operation. For example, when examining products with the same handle but different sizes, the display may be switched to match the specifications by operating the operation unit 151, which is made up of buttons and the like that constitute the input unit 302.

[0334] 32 , when a racket-shaped virtual object is displayed, a small racket-shaped virtual object V271 may be displayed by default, and when the operation unit 151 is operated once, the display of the virtual object V271 may be switched to a larger racket-shaped virtual object V271′. At this time, when the operation unit 151 is operated again, the display of the large virtual object V271 may be switched to a smaller racket-shaped virtual object V271′.

[0335] Similarly, a movable weight may be placed inside the controller 122 to change the weight balance, allowing a user to check the feel of the virtual object beforehand.

[0336] <<13. Eleventh Modification>> In the above, an example has been described in which the controller 122 includes the grip portion 122g, and the user 111 uses the controller by changing the position or direction while holding the grip portion 122g, but the controller may have any other shape, such as a finger ring shape.

[0337] FIG. 33 shows an example of the external configuration of a ring-shaped controller 122X.

[0338] The controller 122X has a hole 122Xh the size of a typical human finger, and the user 111 puts their finger through the hole 122Xh to wear the controller 122X.

[0339] The controller 122X includes an operation unit 151X that is a touchpad that can detect unidirectional movement when a finger is placed over it, for example, on the same surface as the back of the hand. The controller 122X also includes an IMU 152X built in, and can detect its own position and orientation.

[0340] With this configuration, when the user 111A wears the controller 122X, for example, as shown in the image VP151 viewed on the HMD 121A in FIG. 34, the center of the arm can be rotated in the Ry direction around the rotation axis AXy' of the virtual object V311 as the first rotation axis, and the rotation can be performed in the Ry' direction at an angle corresponding to the amount of rotation.

[0341] In addition, the touchpad of the operation unit 151X can be given the function of a second rotation axis, and for example, by touching it with a finger and moving it back and forth in the direction of the arrow FB, the position of the rotation axis AXy' of the virtual object can be moved back and forth in the direction of the arrow FB'.

[0342] Furthermore, as shown in image VP171 viewed on HMD 121A in Figure 35, the position of the rotation axis AXy' of the virtual object can be moved left and right in the direction of arrow LR' by rotating the palm of the hand in the Rp direction around the rotation axis AXp in the horizontal direction and touching the touchpad of operation unit 151X with a finger and moving it back and forth in the direction of arrow FB.

[0343] That is, in the case of Figure 35, the function of the third rotation axis that rotates the palm horizontally is assigned to move the virtual object V311 left and right from the position of the rotation axis AXy' in conjunction with the operation of the touchpad of the operation unit 151X.

[0344] This allows the virtual object V to be rotated while moving the position of the rotation axis of the virtual object V according to the position to be focused on.

[0345] Furthermore, the function of the second rotation axis may be something other than the function of moving the position of the rotation axis of the virtual object V. For example, the function of enlarging or reducing the virtual object V may be assigned by touching the touchpad of the operation unit 151X with a finger and moving it back and forth in the direction of the arrow FB.

[0346] The hardware configuration of the controller 122X is basically the same as that of the controller 122 described with reference to FIG. 10, except for the size, and therefore a description thereof will be omitted.

[0347] Furthermore, the six-axis information acquisition process, the position and orientation linked setting process, the virtual object operation process, and the position and orientation linked setting information transfer process are the same as those described with reference to the flowcharts in Figures 16 to 19, and therefore their description will be omitted.

[0348] That is, even in the ring-shaped controller 122X, functions for performing processing on frequently used virtual objects V are assigned in order of ease of operation for operations such as the rotation axis and touchpad, so that a controller that is easy to operate and has a low operational load can be realized for the user 111.

[0349] <<14. Second Embodiment>> In the above, an example has been described in which the controller 122 used by the first user 111A together with the HMD 121A is passed to the second user 111B and used together with the HMD 121B owned by the second user 111B based on the position and orientation linkage setting information set by the first user 111A, thereby making it easier for the first user 111A to view a desired point of interest in the virtual object V.

[0350] That is, in the above example, it is assumed that the same controller 122 is used by both the first user 111A and the second user 111B.

[0351] However, the controller 122 used by the first user 111A and the controller 122 used by the second user 111B may be different, and furthermore, the characteristics of the controllers 122 used by the two users may be different.

[0352] FIG. 36 shows an example configuration of an information processing system 101′ in which the controller 122 used by the first user 111A and the controller 122 used by the second user 111B are different, and further, the characteristics of the controllers 122 used by the two users are different.

[0353] That is, in the information processing system 101′ of Fig. 36, the first user 111A uses the controller 122 described with reference to Fig. 8, and the second user 111B uses the controller 122X described with reference to Fig. 33. Note that the first user 111A and the second user 111B both use the same HMDs 121A and 121B.

[0354] In the information processing system 101' of FIG. 36, first, a first user 111A uses the controller 122 to register position and orientation linkage setting information.

[0355] Thereafter, first user 111A hands controller 122 to second user 111B, who then uses controller 122 once according to the procedure described in the first embodiment.

[0356] At this time, as described above, the position and orientation linkage setting information registered in the controller 122 is registered in the HMD 121B used by the second user 111B.

[0357] Next, the second user 111B replaces the controller 122 with a controller 122X and uses it together with the HMD 121B.

[0358] At this time, the HMD 121B reuses the position and orientation linkage setting information set by the first user 111A and acquired via the controller 122 to generate new position and orientation linkage setting information and transfers it to the ring-shaped controller 122X.

[0359] More specifically, the HMD 121B acquires information on the ranking of ease of operation of the rotation axes involved in the operation from the ring-shaped controller 122X, reuses the position and posture interlocking setting information set by the first user 111A, and reconfigures only the functions assigned to the rotation axes based on the acquired ranking of the rotation axes, generates new position and posture interlocking setting information, and transfers it to the ring-shaped controller 122X.

[0360] The new position and attitude linkage setting information is assigned in accordance with the characteristics of the ring-shaped controller 122X for functions assigned to each rotation axis, but other information, such as the position (orientation) of the rotation axis of the virtual object, the distance (positional relationship) between the controller 122X and the virtual object V, and the center position of zooming, that has been set by the first user 111A is used.

[0361] As a result, the second user 111B can use the HMD 121B and the ring-shaped controller 122X to appropriately assign functions to each rotation axis according to the characteristics of the ring-shaped controller 122X, and can use other position and posture linkage setting information in the state set by the first user 111A.

[0362] However, since the controller 122 and the controller 122X have different shapes and characteristics, adjustments are required as necessary.

[0363] For example, as shown in the left part of Figure 37, a controller 122 equipped with a grip portion 122g is provided with a characteristic structural portion 122W having a ring structure, and therefore, a distance D311 from the center of gravity position close to the operation portion 151 to the virtual object V331 is set so as not to interfere with the position of the characteristic structural portion 122W.

[0364] However, as shown in the right part of Figure 37, the controller 122X does not have a structure corresponding to the characteristic structural portion 122W on its external surface, so the distance D311 from the center of gravity position close to the operation unit 151X to the virtual object V331' is too far.

[0365] In such a case, it is necessary to adjust the position and orientation linkage setting information so that the distance (positional relationship) between the controller 122X and the virtual object V331′ becomes an appropriate distance (positional relationship).

[0366] Specifically, only the process of setting an appropriate distance (positional relationship) between the controller 122X and the virtual object V331′ may be executed in the flowchart of Fig. 17. That is, for example, only the processes of steps S71, S75, S76, S79, S91, S97, S98, and S101 may be executed in the flowchart of Fig. 17.

[0367] Furthermore, since the operation unit 151X of the ring-shaped controller 122X is composed of a touchpad with only one direction, the function of the second rotation axis is set to enlarge or reduce the virtual object V331' at a magnification factor SC' according to the distance TB traced by tracing the operation unit 151X consisting of the touchpad with a finger, as shown in the right part of Figure 38.

[0368] In this case, the controller 122 is assigned a function to enlarge or reduce the virtual object V331 at a magnification factor SC according to the amount of rotation in the Rp direction around the rotation axis AXp, as shown in the left part of FIG.

[0369] Here, the amount of rotation in the Rp direction around the rotation axis AXp of the controller 122 can take a relatively large value, but the touchpad that serves as the operation unit 151X of the ring-shaped controller 122X has a small tracing distance TB, and there is a possibility that a sufficient magnification SC cannot be secured.

[0370] In such a case, the enlargement factor SC′ in accordance with the tracing distance TB may be reset to a value greater than the enlargement factor SC in the controller 122 .

[0371] As a specific method for setting the magnification ratio SC', for example, the magnification ratio SC' may be reset by tracing the distance of a finger across the operation unit 151X made up of a touchpad.

[0372] <Position and Posture Linked Setting Information Utilization Processing> Next, with reference to the flowchart in FIG. 39, a position and posture linked setting information utilization processing using the HMD 121B of the second user 111B and the ring-shaped controller 122X will be described.

[0373] Note that this processing is premised on the premise that the first user 111A has used the controller 122 to register position and orientation linkage setting information, the controller 122 has been handed over to the second user 111B, and the second user 111B has once used the controller 122 together with the HMD 121B according to the procedure described in the first embodiment. In other words, it is premised on the premise that the position and orientation linkage setting information set by the first user 111A has been registered in the HMD 121B used by the second user 111B.

[0374] In steps S191 and S201, the control unit 201 in the HMD 121B and the control unit 301 in the controller 122X control the communication units 205 and 305, respectively, to exchange IDs with each other and start communication according to a predetermined protocol.

[0375] In step S192, the position and orientation linkage setting information management unit 232 of the control unit 201 in the HMD 121B presents a list of available position and orientation linkage setting information registered in the position and orientation linkage setting information storage unit 262 to the second controller, controller 122X, on the display 131, and requests the controller 122X to select the position and orientation linkage setting information that the controller wishes to use.

[0376] In step S202, when the user 111B of the HMD 121B operates the controller 122X to perform an operation input to select, for example, the position and orientation linked setting information set by the first user 111A from a list of available position and orientation linked setting information, the position and orientation linked setting information management unit 322 of the controller 122X supplies information specifying the selected position and orientation linked setting information together with its own rotation three-axis order information to the HMD 121B.

[0377] In step S193, the position and orientation linked setting information management unit 232 reads out the position and orientation linked setting information set by the selected first user 111A from the position and orientation linked setting information storage unit 262 based on information from the controller 122X.

[0378] Furthermore, the position and orientation linkage setting information management unit 232 reallocates the function of rotating a virtual object from the read position and orientation linkage setting information to the function of the rotation axis with the highest priority that is easiest to operate on the controller 122X. Similarly, the position and orientation linkage setting information management unit 232 reallocates the function of enlarging or reducing a virtual object to the function of the rotation axis with the second highest priority that is second easiest to operate on the controller 122X. Then, the position and orientation linkage setting information management unit 232 combines these information, leaving the other information as they are, to generate new position and orientation linkage setting information and registers it in the position and orientation linkage setting information storage unit 262.

[0379] In steps S194 and S203, the position and orientation linkage setting information management unit 322 of the HMD 121B and the position and orientation linkage setting information management unit 232 of the controller 122X share the position and orientation linkage setting information that they mutually possess, namely, information on function assignment for each rotation axis, the position (orientation) of the rotation axis, the distance (positional relationship) between the virtual object V and the controller 122, and the center position of zooming, and store these in their respective position and orientation linkage setting information storage units 262 and 332.

[0380] Through the above processing, new position and orientation linkage setting information is generated in which a function is assigned to each rotation axis according to the characteristics of the controller 122X, and other information remains as set by the user 111A. The generated new position and orientation linkage setting information is then shared and held by the HMD 121B and the controller 122X. As a result, if the HMD 121B can acquire information on the position and orientation of the controller 122X, it can realize a rotation operation function and a zoom function for the virtual object V using the information to which the new position and orientation linkage setting information has been added, and display the information on the display 131.

[0381] As a result, even if the controller 122 used by the first user 111A and the controller 122X used by the second user 111B have different characteristics such as shape, the rotation axis for the easiest operation is set in correspondence with the operation function for the virtual object V that is operated most frequently in the controller 122X, and the rotation axis for the second easiest operation is set in correspondence with the operation function for the virtual object V that is operated second most frequently in the controller 122X.

[0382] As a result, even if the controller 122 used by the first user 111A and the controller 122X used by the second user 111B have different characteristics such as shape, it is possible for them to comfortably operate the virtual object V.

[0383] The above has been described as an example in which the controller 122 used by the first user 111A and the controller 122X used by the second user 111B have different characteristics such as shape, but similar processing can be used even in cases in which both the first user 111A and the second user 111B use different controllers 122 that have the same shape and characteristics. In this case, even though the controllers 122 are different, they have the same characteristics, so the position and orientation linkage setting information is transferred without any changes.

[0384] Furthermore, as described above, the position and orientation linkage setting information reuse process alone, which has been described with reference to the flowchart in FIG. 39 , includes position and orientation linkage setting information that is not necessarily set appropriately depending on the shape of the controller 122X, such as the distance (positional relationship) between the virtual object and the controller 122X and the magnification rate when enlarging or reducing the virtual object, and therefore it may be necessary to actually execute the virtual object operation process, check for any defects, and make adjustments.

[0385] <<15. Example of Execution by Software>> The above-described series of processes can be executed by hardware, but can also be executed by software. When the series of processes is executed by software, the program constituting the software is installed from a recording medium into a computer incorporated in dedicated hardware, or into, for example, a general-purpose computer that can execute various functions by installing various programs.

[0386] 40 shows an example of the configuration of a general-purpose computer. This computer has a built-in processing circuit 1001. An input / output interface 1005 is connected to the processing circuit 1001 via a bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

[0387] The input / output interface 1005 is connected to an input unit 1006 including input devices such as a keyboard and a mouse through which a user inputs operation commands, an output unit 1007 that outputs a processing operation screen and images of processing results to a display device, a storage unit 1008 including a hard disk drive or the like that stores programs and various data, and a communication unit 1009 including a LAN (Local Area Network) adapter or the like that executes communication processing via a network typified by the Internet. Also connected is a drive 1010 that reads and writes data from / to a removable storage medium 1011 such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor memory.

[0388] The processing circuit 1001 executes various processes in accordance with a program stored in a ROM 1002 or a program read from a removable storage medium 1011 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, installed in a storage unit 1008, and loaded from the storage unit 1008 into a RAM 1003. The RAM 1003 also stores data necessary for the processing circuit 1001 to execute various processes, as appropriate.

[0389] In a computer configured as described above, the processing circuit 1001 performs the above-described series of processes by, for example, loading a program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0390] The program executed by the computer (processing circuit 1001) can be provided by being recorded on a removable storage medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0391] In the computer, the program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting the removable storage medium 1011 into the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Alternatively, the program can be pre-installed in the ROM 1002 or the storage unit 1008. The communication unit 1009 corresponds to the communication unit 205 in FIG. 9 and the communication unit 305 in FIG. 10.

[0392] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0393] Furthermore, when the computer in FIG. 40 functions as the information processing system 101, 101′ according to an embodiment of the present disclosure, the processing circuit 1001 of the computer functions as the control unit 201, 301 by executing a program loaded onto the RAM 1003. Furthermore, the storage unit 1008 and removable storage medium 1011 constituting the secondary storage device store the information processing program according to the present disclosure and various data stored in the storage units 204, 304. The processing circuit 1001 reads and executes program data from the storage unit 1008 or removable storage medium 1011. Alternatively, the processing circuit 1001 may obtain these programs from another device via the communication unit 1009. In other words, the secondary storage device is not limited to being located inside the computer of FIG. 40 , like the storage unit 1008 or removable storage medium 1011, but may also be located outside the computer of FIG. 40 . The processing circuit 1001 is an example of an integrated circuit, and a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphical Processing Unit), an APU (Accelerated Processing Unit), an ASIC (Application Specific Integrated Circuit), and an FPGA (Field Programmable Gate Array) can all be considered to be integrated circuits.

[0394] The processing circuit 1001 realizes the functions of the control unit 201 in FIG. 9 and the control unit 301 in FIG.

[0395] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0396] Furthermore, the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure.

[0397] For example, the present disclosure can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0398] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0399] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0400] Furthermore, among the processes described in the above-described embodiments of the present disclosure, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0401] Furthermore, the components of each device shown in the figure are functional concepts and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0402] Furthermore, the above-described embodiments of the present disclosure can be combined as appropriate within the scope of the processing content without causing inconsistencies. Furthermore, the order of the steps shown in the sequence diagrams or flowcharts of the present embodiments can be changed as appropriate. For example, the steps may be processed in chronological order, repeatedly, or partially in parallel.

[0403] The present disclosure may also be configured as follows. <1> An information processing system including a controller and a display device that displays a virtual object, wherein: information indicating the position and orientation of the controller is acquired; and display of the virtual object is controlled based on the information indicating the position and orientation of the controller; the display is controlled so that a first operation is performed on the virtual object based on operation of a first rotation axis of the controller; and the display is controlled so that a second operation different from the first operation is performed on the virtual object based on operation of a second rotation axis of the controller different from the first rotation axis; and the first rotation axis and the first operation, and the second rotation axis and the second operation are associated with each other based on characteristics of the controller. <2> The information processing system described in <1>, wherein the characteristics of the controller include the shape and weight of the controller, and the ease of turning the rotation axis of the controller. <3> The information processing system described in <2>, wherein the ease of turning the rotation axis of the controller is based on an operation frequency determined from an operation history of the controller. <4> The information processing system described in <3>, wherein the ease of turning the rotation axis of the controller increases as the operation frequency increases. <5> The information processing system described in <3>, wherein the first rotation axis and the first operation, and the second rotation axis and the second operation, are associated with each other based on the ease of turning the rotation axis, which is based on the operation frequency of the controller, and the frequency of operations performed on the virtual object. <6> The information processing system described in <5>, wherein the first rotation axis is the rotation axis with the highest operation frequency and the easiest to turn, and the first operation associated with the first rotation axis is the operation most frequently performed on the virtual object, and the second rotation axis is the rotation axis with the second highest operation frequency and the second easiest to turn, and the second operation associated with the second rotation axis is the operation second most frequently performed on the virtual object. <7> The information processing system described in <1>, wherein the first operation is an operation to rotate the virtual object. <8> The information processing system according to <1>, wherein the first operation is an operation for enlarging or reducing the virtual object.<9> The information processing system described in <1> further stores at least one of a setting value and an operation history set by a first user using the controller as position and orientation linkage setting information, and the position and orientation linkage setting information is transferred to another controller different from the controller used by a second user different from the first user. <10> The information processing system described in <9>, wherein the setting value set by the first user is information about the position and orientation of a rotation axis in an operation to rotate the virtual object. <11> The information processing system described in <9>, wherein the setting value set by the first user is information about a center position in an operation to enlarge or reduce the virtual object. <12> The information processing system described in <9>, wherein the setting value set by the first user is information indicating a positional relationship between the controller and the virtual object. <13> The information processing system described in <9>, wherein the setting value set by the first user is a range of display sizes of the virtual object. <14> The information processing system described in <9>, wherein the setting value set by the first user is a display position of the virtual object. <15> The information processing system according to <9>, wherein the characteristics of the controller and the other controller are different. <16> The position and orientation linkage setting information includes, in addition to at least one of the setting value and the operation history, information indicating the first rotation axis and the first operation, and the second rotation axis and the second operation, which are associated based on the characteristics, and when the position and orientation linkage setting information is transferred from the controller to the other controller, the information indicating the first rotation axis and the first operation, and the second rotation axis and the second operation is changed so as to be associated based on the characteristics of the other controller.<17> An information processing method for an information processing system including a controller and a display device that displays a virtual object, comprising: performing an acquisition process to acquire information indicating the position and orientation of the controller; and performing a display control process to control the display of the virtual object based on the information indicating the position and orientation of the controller, wherein the display control process controls the display so that a first operation is performed on the virtual object based on an operation of a first rotation axis of the controller, and controls the display so that a second operation different from the first operation is performed on the virtual object based on an operation of a second rotation axis of the controller different from the first rotation axis, wherein the first rotation axis and the first operation, and the second rotation axis and the second operation are associated with each other based on characteristics of the controller. <18> The information processing method described in <17>, wherein the characteristics of the controller include the shape and weight of the controller, and the ease of turning the rotation axis of the controller. <19> The information processing method described in <17>, wherein the first operation is an operation to rotate the virtual object. <20> The information processing method according to <17>, wherein the first operation is an operation for enlarging or reducing the virtual object.

[0404] 101, 101' Information processing system, 111, 111A, 111B User, 121, 121A, 121B HMD, 122, 122A, 122B, 122X Controller, 131 Display, 151 Operation unit, 152 IMU, 209 IMU, 231 Controller 6-axis information management unit, 232 Position and orientation interlocking setting information management unit, 233 HMD 6-axis information management unit, 234 Display control unit, 261 Controller 6-axis information storage unit, 262 Position and orientation interlocking setting information storage unit, 322 Position and orientation interlocking setting information management unit, 331 6-axis information storage unit, 332 Position and orientation interlocking setting information storage unit

Claims

1. An information processing system comprising a controller and a display device that displays a virtual object, the information processing system acquiring information indicating the position and attitude of the controller, controlling the display of the virtual object based on the information indicating the position and attitude of the controller, controlling the display so that a first operation is performed on the virtual object based on operation of a first rotation axis of the controller, and controlling the display so that a second operation different from the first operation is performed on the virtual object based on operation of a second rotation axis of the controller that is different from the first rotation axis, and the first rotation axis and the first operation, and the second rotation axis and the second operation are each associated based on characteristics of the controller.

2. The information processing system according to claim 1, wherein the characteristics of the controller include the shape, weight, and ease of rotation of the rotation axis of the controller.

3. An information processing system according to claim 2, wherein the ease of rotation of the rotation shaft of the controller is based on the frequency of operation determined from the operation history of the controller.

4. The information processing system according to claim 3, wherein the rotational axis of the controller is easier to turn as the frequency of operation increases.

5. An information processing system as described in claim 3, wherein the first rotation axis and the first operation, and the second rotation axis and the second operation are associated with each other based on the ease of turning the rotation axis, which is based on the frequency of operation of the controller, and the frequency of operation performed on the virtual object.

6. The information processing system of claim 5, wherein the first rotation axis is the rotation axis that is most frequently operated and easiest to turn, the first operation associated with the first rotation axis is the operation performed most frequently on the virtual object, the second rotation axis is the rotation axis that is second most frequently operated and second easiest to turn, and the second operation associated with the second rotation axis is the operation performed second most frequently on the virtual object.

7. The information processing system according to claim 1, wherein the first operation is an operation of rotating the virtual object.

8. The information processing system according to claim 1, wherein the first operation is an operation for enlarging or reducing the virtual object.

9. An information processing system as described in claim 1, further storing at least one of the setting values ​​and operation history set by a first user using the controller as position and orientation linked setting information, and the position and orientation linked setting information being transferred to another controller different from the controller used by a second user different from the first user.

10. An information processing system according to claim 9, wherein the setting value set by the first user is information about the position and orientation of a rotation axis in an operation to rotate the virtual object.

11. An information processing system according to claim 9, wherein the setting value set by the first user is information about a center position in an operation to enlarge or reduce the virtual object.

12. An information processing system according to claim 9, wherein the setting value set by the first user is information indicating the positional relationship between the controller and the virtual object.

13. An information processing system according to claim 9, wherein the setting value set by the first user is a range of the size of the display of the virtual object.

14. An information processing system according to claim 9, wherein the setting value set by the first user is the display position of the virtual object.

15. The information processing system according to claim 9, wherein the controller and the other controller have different characteristics.

16. The information processing system of claim 15, wherein the position and attitude interlocking setting information includes, in addition to at least one of the setting value and the operation history, information indicating the first rotation axis and the first operation, and the second rotation axis and the second operation, which are associated based on the characteristics, and when the position and attitude interlocking setting information is transferred from the controller to the other controller, the information indicating the first rotation axis and the first operation, and the second rotation axis and the second operation is changed so as to be associated based on the characteristics of the other controller.

17. An information processing method for an information processing system comprising a controller and a display device that displays a virtual object, the information processing method comprising: performing an acquisition process to acquire information indicating the position and attitude of the controller; and performing a display control process to control the display of the virtual object based on the information indicating the position and attitude of the controller, wherein the display control process controls the display so that a first operation is performed on the virtual object based on operation of a first rotation axis of the controller; and controls the display so that a second operation different from the first operation is performed on the virtual object based on operation of a second rotation axis of the controller that is different from the first rotation axis, and the first rotation axis and the first operation, and the second rotation axis and the second operation are respectively associated based on characteristics of the controller.

18. The information processing method according to claim 17, wherein the characteristics of the controller include the shape and weight of the controller, and the ease of turning the rotation axis of the controller.

19. The information processing method according to claim 17, wherein the first operation is an operation of rotating the virtual object.

20. The information processing method according to claim 17, wherein the first operation is an operation for enlarging or reducing the virtual object.

Images