Information processing apparatus and information processing method

Abstract

The present disclosure pertains to an information processing device and an information processing method that make it possible to improve operability with respect to a 3D object. An axis posture determination unit determines a posture of an adjustment axis serving as a reference for performing adjustment processing on a 3D object disposed in a three-dimensional space of a computer graph to be superimposed on a real space, based on a posture of a user's finger, and modifies and fixes the posture of the adjustment axis. The adjustment processing unit performs adjustment processing on the 3D object in response to an operation by a user in accordance with an axial direction of the adjustment shaft whose posture has been fixed. The present technology can be applied, for example, to an AR system for displaying a 3D object superimposed on a real space.

Description

Technical Field

[0001] The present disclosure relates to an information processing device and an information processing method, and more particularly, to an information processing device and an information processing method capable of improving the operability of a 3D object. Background Art

[0002] In the related art, in order to move or rotate a 3D object displayed through AR (Augmented Reality) or VR (Virtual Reality), a user's hand performs an operation of "grasping" the 3D object to perform the operation of moving or rotating the 3D object according to the movement of the user's hand.

[0003] For example, Patent Document 1 discloses a technique for correcting the positions of virtual objects displayed by AR by a plurality of AR devices among the AR devices.

[0004] Reference List

[0005] Patent Literature

[0006] Patent document 1: WO 2021 / 177132 Summary of the invention

[0007] Technical issues

[0008] When a 3D object is operated as discussed above, the axis as a reference for the operation is not fixed. Therefore, the 3D object may be operated in a direction that the user does not intend, and it is desired to improve operability.

[0009] The present disclosure has been made in view of such circumstances, and can improve the operability of 3D objects.

[0010] Solution to the problem

[0011] One aspect of the present disclosure provides an information processing device, including: an axis posture determination unit, which determines the posture of an adjustment axis based on the posture of a user's finger, and changes and fixes the posture of the adjustment axis, the adjustment axis being used as a reference for performing adjustment processing on a 3D object set in a three-dimensional space on computer graphics to be superimposed on a real space; and an adjustment processing unit, which performs adjustment processing on the 3D object according to the axial direction of the adjustment axis in the fixed posture and according to an operation performed by the user.

[0012] One aspect of the present disclosure provides an information processing method that enables an information processing device to perform processing, the processing including: determining a posture of an adjustment axis based on a posture of a user's finger, and changing and fixing the posture of the adjustment axis, the adjustment axis being used as a reference for performing adjustment processing on a 3D object set in a three-dimensional space on computer graphics to be superimposed on a real space; and performing adjustment processing on the 3D object according to an operation performed by the user based on the axial direction of the adjustment axis in the fixed posture.

[0013] In one aspect of the present disclosure, a posture of an adjustment axis is determined based on a posture of a user's finger, and the posture of the adjustment axis is changed and fixed, the adjustment axis being used as a reference for performing adjustment processing on a 3D object set in a three-dimensional space on computer graphics to be superimposed on a real space; and adjustment processing on the 3D object is performed according to an operation performed by the user based on the axial direction of the adjustment axis in the fixed posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] [ Figure 1 ] Figure 1 An example of the configuration of an embodiment of an AR system to which the present technology is applied is shown.

[0015] [ Figure 2 ] Figure 2 is a perspective view showing an example of a schematic configuration of a ring-type device.

[0016] [ Figure 3 ] Figure 3 is a block diagram showing an example of a hardware configuration of an AR system.

[0017] [ Figure 4 ] Figure 4 is a block diagram showing an example of a functional configuration of an AR system.

[0018] [ Figure 5 ] Figure 5 The adjustment mode determination range is shown.

[0019] [ Figure 6 ] Figure 6 The adjustment axis is shown displayed in adjustment mode.

[0020] [ Figure 7 ] Figure 7 An operation of changing the posture of the adjustment axis is shown.

[0021] [ Figure 8 ] Figure 8 The operation of fixing the posture of the adjustment axis is shown.

[0022] [ Fig. 9 ] Fig. 9 An operation of adjusting the adjustment axis to a specified angle is shown.

[0023] [ Fig.10 ] Fig.10 The adjustment process of the position of the mobile 3D object is shown.

[0024] [ Fig.11 ] Fig.11 The adjustment process of the posture of rotating a 3D object is shown.

[0025] [ Fig.12 ] Fig.12 An adjustment process of enlarging / reducing the size of a 3D object is shown.

[0026] [ Fig.13 ] Fig.13 is a flow chart illustrating a 3D object adjustment process.

[0027] [ Fig.14 ] Fig.14 Another processing example of the adjustment process of the position of the mobile 3D object is shown.

[0028] [ Fig.15 ] Fig.15 The magnification in the adjustment process for the 3D object is shown.

[0029] [ Fig.16 ] Fig.16 Examples of adjustment axes indicated with bold lines are shown.

[0030] [ Fig.17 ] Fig.17 Assistance in detecting a touch location on a touch pad is shown.

[0031] [ Fig.18 ] Fig.18 The process of changing the posture of the adjustment shaft at every predetermined angle is shown.

[0032] [ Fig.19 ] Fig.19 A process for initially displaying the adjustment axis according to a default posture is shown.

[0033] [ Fig. 20 ] Fig. 20 An example of initially displaying a 3D object according to a default pose is shown.

[0034] [ Fig.21 ] Fig.21 Selection of a 3D object that is the target of an adjustment process is shown.

[0035] [ Fig. 22 ] Fig. 22 A pen-type device is shown.

[0036] [ Fig.23 ] Fig.23Shows a mouse-type device.

[0037] Fig.24 Fig.24 Is a block diagram showing an example of the configuration of an embodiment of a computer to which the present technology is applied. Detailed Embodiment

[0038] Hereinafter, the detailed embodiments of the application of the present technology will be described in detail with reference to the accompanying drawings.

[0039] <Configuration Example of AR System>

[0040] Figure 1 Shows an example of the configuration of an embodiment of an AR system to which the present technology is applied.

[0041] Figure 1 The AR system 11 shown is configured to include a glasses-type wearable terminal 12 and a ring-type device 13.

[0042] The glasses-type wearable terminal 12 is configured to have a shape wearable by a user like ordinary glasses, and is configured to be provided with a transmissive display unit 14L and a transmissive display unit 14R in the lens portion of the glasses. The glasses-type wearable terminal 12 is also configured to be provided with a camera device 15 at a position where the camera device 15 can capture the field of view of the user wearing the glasses-type wearable terminal 12.

[0043] The transmissive display unit 14L and the transmissive display unit 14R display a 3D object 21 represented by computer graphics, and make the light in the area other than the area where the 3D object 21 is displayed transmit. Therefore, the user can visually recognize the 3D object 21 displayed by the transmissive display unit 14L and the transmissive display unit 14R as if the 3D object 21 is set to be superimposed on the real space.

[0044] For example, the 3D object 21 displayed by the transmissive display unit 14L and the transmissive display unit 14R is displayed according to a world coordinate system 22 that is a coordinate system defining the entire three-dimensional space on the computer graphics. In addition, for example, the 3D object 21 is provided with a local coordinate system 23, and the local coordinate system 23 defines the front-back direction, left-right direction, and up-down direction of the 3D object 21 with the center of the 3D object 21 as the origin.

[0045] For example, the camera device 15 can capture the movement of the user's hand, fingers, etc.

[0046] The ring-type device 13 is an input device for inputting operations performed by the user on the 3D object 21, and is used, for example, in a state of being worn on the user's index finger as shown.

[0047] ​​In the AR system 11, for example, when the user touches the touch panel 33 ( Figure 2 ) is touch-on (from non-contact to contact), an operation of “grasping” the 3D object 21 touched or entered by the fingertip of the index finger is performed. Then, while performing the operation of “grasping” the 3D object 21, the 3D object 21 is fixed to the user's fingertip, so that the 3D object 21 can move or rotate according to the movement of the user's fingertip. At this time, the AR system 11 can estimate the position of the user's fingertip using the ring device 13, or recognize the position of the user's fingertip based on the video captured by the camera device 15.

[0048] Afterwards, when the user touches the touch panel 33 ( Figure 2 ) performs touch-off (changes from contact to non-contact), the operation of "grasping" the 3D object 21 is canceled. Therefore, the state in which the 3D object 21 is fixed to the user's fingertips is canceled. Therefore, the 3D object 21 is fixed at the position and posture in the three-dimensional space at the moment when the user performs touch-off, for example, so that the 3D object 21 can be kept in the air.

[0049] Furthermore, the AR system 11 may perform adjustment processing on the 3D object 21 using the ring-type device 13 as will be discussed later.

[0050] Figure 2 is a perspective view showing an example of a schematic configuration of the ring-type device 13 .

[0051] like Figure 2 As shown, the ring-type device 13 is configured to include a main body portion 31 , a belt portion 32 , a touch pad 33 , and a tactile switch 34 .

[0052] The main body 31 is configured to include a battery 41, a vibration actuator 42, an audio module 43, an IMU (Inertial Measurement Unit) 44, a wireless communication substrate 45, etc. integrated therein. For example, in the main body 31, the power provided by the battery 41 is used to drive each part, so that the vibration actuator 42 vibrates, the audio module 43 outputs sound, the IMU 44 measures the three-dimensional angular velocity and acceleration of the ring device 13, and the wireless communication substrate 45 communicates with the glasses-type wearable terminal 12.

[0053] The band portion 32 is used to enable the ring-type device 13 to be worn on the user's finger. For example, the user can fix the ring-type device 13 to his / her index finger by inserting his / her index finger into the band portion 32 along the fixed line indicated in the figure and then adjusting the fit of the band portion 32.

[0054] The touch panel 33 is provided on the side surface of the band portion 32, and detects a touch operation performed by the user on the surface of the touch panel 33. For example, the user may wear the ring-type device 13 on his / her index finger so that the touch panel 33 faces the finger as shown in FIG. Figure 1 The user is shown on the thumb side to be able to perform a touch operation with his / her thumb on the touch pad 33. As indicated by the X-axis and Y-axis in the figure, an operation axis is set for the touch pad 33, wherein the direction along the axis of the belt portion 32 on the surface of the touch pad 33 is defined as the X-axis direction, and the direction orthogonal to the X-axis direction is defined as the Y-axis direction.

[0055] For example, the tactile switch 34 is a switch that is turned on only while being pressed by a user's finger, and is turned off when the user releases his / her finger. Figure 2 A configuration example is shown in which the tactile switch 34 is provided below the touch panel 33. However, a configuration may be used in which the tactile switch 34 is provided on the back side of the touch panel 33 so that the tactile switch 34 is turned on when the user presses the touch panel 33.

[0056] Figure 3 is a block diagram showing an example of the hardware configuration of the AR system 11 .

[0057] like Figure 3 As shown, the glasses-type wearable terminal 12 is configured such that a display 51 , a speaker 52 , and a sensor 53 are connected to a device body 54 .

[0058] For example, the display 51 is composed of Figure 1 The transmissive display unit 14L and the transmissive display unit 14R are composed of the transmissive display unit 14L and the transmissive display unit 14R, and display the 3D object 21, the adjustment axis 25 ( Figure 6 )wait.

[0059] The speaker 52 outputs sound based on the sound data supplied from the device body 54 .

[0060] For example, the sensor 53 is composed of Figure 1 The device body 54 is composed of the camera 15 and provides the visual data captured by the camera 15 to the device body 54.

[0061] The device body 54 is configured to include a CPU 55, a memory 56, a communication device 57, and an input / output interface 58 integrated therein. The CPU 55 controls the various parts of the glasses-type wearable terminal 12 by executing a program. The memory 56 stores data required for various processes performed in the glasses-type wearable terminal 12. The communication device 57 communicates with the ring device 13. The input / output interface 58 performs data input and output with the display 51, the speaker 52, and the sensor 53.

[0062] The ring-type device 13 is configured such that a vibrator 61 and a sensor 62 are connected to a device body 63 .

[0063] For example, the vibrator 61 is composed of Figure 2 The device is composed of a vibration actuator 42 and vibrates according to tactile data provided from the device body 54.

[0064] For example, the sensor 62 is Figure 2 The IMU 44 in the device body 63 is used to construct the device body 63, and the angular velocity data and acceleration data about the ring-shaped device 13 measured by the IMU 44 are provided to the device body 63.

[0065] The device body 63 is configured to include a CPU 64, a memory 65, a communication device 66, and an input / output interface 67 integrated therein. The CPU 64 controls the various parts of the ring-type device 13 by executing a program. The memory 65 stores data required for various processes performed in the ring-type device 13. The communication device 66 communicates with the glasses-type wearable terminal 12. The input / output interface 67 performs data input and output with the vibrator 61 and the sensor 62.

[0066] The glasses-type wearable terminal 12 may be various head-mounted displays used in optical see-through AR, video see-through AR, VR, and the like. A portion of the hardware configuration of the glasses-type wearable terminal 12 may be implemented by a personal computer (not shown). Alternatively, the glasses-type wearable terminal 12 may be replaced by a projector or a large or small display device. In the AR system 11, the ring-type device 13 may not be used, and the functions of the ring-type device 13 may be integrated in the glasses-type wearable terminal 12. For example, in the AR system 11, an operation terminal for operating a robot, a drone, and the like may be used.

[0067] Figure 4 is a block diagram showing an example of the functional configuration of the AR system 11 .

[0068] like Figure 4 As shown, the AR system 11 is configured to include a sensor control unit 71 , an information processing unit 72 , a visual display unit 73 , a tactile presentation unit 74 , and a sound presentation unit 75 .

[0069] For example, the sensor control unit 71 controls Figure 1 The camera device 15, Figure 2 The sensor control unit 71 then provides the information processing unit 72 with the visual data captured by the camera 15, the angular velocity data and acceleration data about the ring-type device 13 measured by the IMU 44, the touch position data detected by the touch panel 33, the on / off data detected by the tactile switch 34, and the like.

[0070] In addition, the sensor control unit 71 can control various sensors such as an RGB camera and a depth camera (not shown). For example, such a sensor can be set in the environment where the AR system 11 is used, or attached to a part of the user's body other than the face or hand, rather than being set in the glasses-type wearable terminal 12 or the ring-type device 13. A single sensor or multiple sensors can be set.

[0071] The information processing unit 72 is configured to include a sensor information acquisition unit 81 , an adjustment mode determination unit 82 , a finger posture determination unit 83 , an axis posture determination unit 84 , a finger operation determination unit 85 , an adjustment processing unit 86 , and an output control unit 87 .

[0072] The sensor information acquisition unit 81 acquires visual data supplied from the sensor control unit 71 , angular velocity data and acceleration data on the ring device 13 , touch position data of the touch panel 33 , on / off data of the tactile switch 34 , and the like as sensor information.

[0073] For example, the adjustment mode determination unit 82 determines whether to transition to an adjustment mode for performing adjustment processing on the 3D object 21 and whether to end the adjustment mode based on sensor information acquired by the sensor information acquisition unit 81, such as visual data or angular velocity data or acceleration data about the ring-shaped device 13.

[0074] The finger posture determination unit 83 acquires bone position posture information indicating the position and posture of each finger by analyzing the skeletal frame of the user's fingers captured in the visual data acquired by the sensor control unit 71. The finger posture determination unit 83 also acquires device position posture information indicating the position and posture of the ring-shaped device 13 based on the angular velocity data and acceleration data about the ring-shaped device 13 acquired by the sensor control unit 71. Then, the finger posture determination unit 83 can determine the posture of the user's fingers based on the bone position posture information and the device position posture information. In addition, the finger posture determination unit 83 can also use the depth information acquired by the depth sensor (not shown) to determine the posture of the user's fingers.

[0075] The axis posture determination unit 84 determines the adjustment axis 25 ( 24 ) which is a reference for the adjustment process for the 3D object 21 based on the posture of the user's finger determined by the finger posture determination unit 83. Figure 6 ) and changes and fixes the posture of the adjustment shaft 25. For example, the shaft posture determination unit 84 determines the posture of the adjustment shaft 25 to be aligned with the adjustment shaft ( Figure 2The posture of the adjustment axis 25 is consistent with the posture of the X-axis and Y-axis in the ring-shaped device 13, and the adjustment axis 25 is displayed. In addition, when the user does not touch the touch panel 33, the axis posture determination unit 84 changes the posture of the adjustment axis 25 according to the change of the posture of the adjustment axis of the touch panel 33 of the ring-shaped device 13. On the other hand, when the user touches the touch panel 33, the axis posture determination unit 84 fixes the posture of the adjustment axis 25.

[0076] The finger operation determination unit 85 determines an operation performed by the user's finger based on the touch position data of the touch panel 33 and the on / off data of the tactile switch 34 acquired by the sensor control unit 71. For example, the finger operation determination unit 85 can determine whether the touch panel 33 is touched, the touch position and the movement amount of the touch on the touch panel 33, and whether the tactile switch 34 is on or off.

[0077] When the user touches the touch panel 33, the adjustment processing unit 86 performs adjustment processing of moving the position of the 3D object 21, rotating the posture of the 3D object 21, and enlarging / reducing the size of the 3D object 21 based on the touch position and movement amount of the touch on the touch panel 33 determined by the finger operation determination unit 85.

[0078] The output control unit 87 controls output of visual data to the visual display unit 73 , output of tactile data to the tactile presentation unit 74 , and output of sound data to the sound presentation unit 75 .

[0079] For example, the visual display unit 73 is composed of Figure 1 The transmissive display unit 14L and the transmissive display unit 14R are configured, and a video is displayed based on visual data supplied from the output control unit 87.

[0080] For example, the tactile presentation unit 74 is composed of Figure 2 The vibration actuator 42 in the embodiment is configured, and based on the tactile data provided from the output control unit 87, the tactile sensation is presented through vibration.

[0081] The sound presentation unit 75 is composed of Figure 2 The audio module 43 in the output control unit 87 is configured to present sound based on the sound data provided from the output control unit 87.

[0082] <Adjustment mode determination range>

[0083] Will refer to Figure 5 Description Adjustment mode determination range 24.

[0084] In the AR system 11 , for example, an adjustment mode determination range 24 for determining transition and end of an adjustment mode for performing an adjustment process on the 3D object 21 is set around the 3D object 21 .

[0085] For example, Figure 5As shown in A of FIG. 1 , the adjustment mode determination range 24 is set in a certain range around the 3D object 21. When the user moves his / her finger wearing the ring-type device 13 close to the 3D object 21 and the fingertip is located at a position within the adjustment mode determination range 24 and the fingertip is not in contact with the 3D object 21, the adjustment mode is changed. Thereafter, when the user moves his / her finger wearing the ring-type device 13 away from the 3D object 21 and the fingertip is located at a position outside the adjustment mode determination range 24, the adjustment mode is ended.

[0086] Here, when the position of the 3D object 21 is adjusted in the adjustment mode, the position of the adjustment mode determination range 24 may be adjusted together with the 3D object 21 so that the positional relationship of the adjustment mode determination range 24 relative to the 3D object 21 does not change. Alternatively, when the position of the 3D object 21 is adjusted in the adjustment mode, the position of the adjustment mode determination range 24 may be kept fixed so that the positional relationship of the adjustment mode determination range 24 relative to the 3D object 21 is as shown in FIG. Figure 5 The changes shown in B.

[0087] In the AR system 11 , transition and end of the adjustment mode may be determined without using the adjustment mode determination range 24 .

[0088] For example, when the user moves his / her finger wearing the ring-shaped device 13 close to the 3D object 21 and performs an operation of clicking, double-clicking, or long pressing the tactile switch 34, it is possible to transition to the adjustment mode. Thereafter, when the user performs an operation of clicking, double-clicking, or long pressing the tactile switch 34, the adjustment mode can be ended.

[0089] Similarly, when the user moves his / her finger wearing the ring-shaped device 13 close to the 3D object 21 and performs a single tap, a double tap, or a long press operation on the touch pad 33, it is possible to transition to the adjustment mode. Thereafter, when the user performs a single tap, a double tap, or a long press operation on the touch pad 33, the adjustment mode can be ended.

[0090] Alternatively, when the user moves his / her finger wearing the ring-type device 13 close to the 3D object 21 and performs an operation of tapping, double-tapping, or long-pressing the 3D object 21 with the fingertip, it may be transitioned to the adjustment mode. Thereafter, when the user taps, double-taps, or long-presses the 3D object 21 with the fingertip, the adjustment mode may be terminated. In addition, a specific gesture such as snapping fingers may be used. Such movement of the fingertip may be detected by the IMU 44, the camera 15, or the like.

[0091] <Adjustment axis>

[0092] Will refer to Figures 6 to 9 Description of the adjustment axis 25 displayed in the adjustment mode.

[0093] Figure 6 The display of the adjustment axis 25 is shown.

[0094] In the AR system 11, for example, according to the posture of the touch panel 33 of the ring-type device 13 at the timing of transition to the adjustment mode, the adjustment axis 25 is displayed according to the origin of the local coordinate system 23 of the 3D object 21. As discussed above, the operation axis ( Figure 2 The X-axis and Y-axis in FIG. 2 are shown, and the display adjustment axis 25 is aligned with the direction of the operation axis. Figure 6 As shown, when the touch panel 33 is shifted to the adjustment mode with the X axis pointing horizontally and the Y axis pointing vertically, the adjustment axis 25 is displayed for the 3D object 21 in a posture with the X axis pointing horizontally and the Y axis pointing vertically.

[0095] In the AR system 11, the adjustment axis 25 is displayed from the timing of transition to the adjustment mode until the timing of ending the adjustment mode, so that the user can recognize whether the adjustment mode is activated based on the adjustment axis 25. In this case, feedback may be given by sound, vibration, etc., or may be given visually by means other than the adjustment axis 25.

[0096] Figure 7 An operation of changing the posture of the adjustment shaft 25 is shown.

[0097] In the AR system 11, for example, the user can change the posture of the adjustment axis 25 displayed in the adjustment mode by tilting the ring-shaped device 13. That is, when the posture of the operation axis of the touch pad 33 of the ring-shaped device 13 changes, the adjustment axis 25 whose posture is changed to match the posture of the operation axis is displayed.

[0098] like Figure 7 As shown in FIG. 1A , when the user tilts the ring-type device 13 to rotate leftward about an axis orthogonal to the surface of the touch panel 33 , the adjustment shaft 25 is shown to rotate according to the rotation angle of the ring-type device 13 .

[0099] like Figure 7 As shown in FIG. 1B , when the user tilts the ring-type device 13 to rotate leftward about the Y-axis of the surface of the touch panel 33 , the adjustment shaft 25 is shown to rotate according to the rotation angle of the ring-type device 13 .

[0100] Figure 8 An operation of fixing the posture of the adjustment shaft 25 is shown.

[0101] In the AR system 11, for example, when in the adjustment mode, while the user is performing an operation of turning on the tactile switch 34, the display of the adjustment axis 25 may be fixed to the posture at the timing of turning on the tactile switch 34. That is, even if the user tilts the ring-shaped device 13 with the tactile switch 34 kept turned on, the posture of the adjustment axis 25 may not change.

[0102] Even if the user adjusts the axis to be displayed as Figure 8 The operation of turning on the tactile switch 34 is performed in the case shown on the left side of Figure 8 As shown on the right side of the figure, the annular device 13 can be tilted, and the posture of the adjustment shaft 25 can also be kept fixed.

[0103] Thereafter, when the user performs an operation to turn off the tactile switch 34, the relative relationship between the adjustment axis 25 and the operation axis of the touch panel 33 at the timing of performing the operation is maintained. Figure 8 As shown on the right side of FIG. 1 , when the ring-shaped device 13 is tilted relative to the adjustment shaft 25 , a deviation occurs, and the posture of the adjustment shaft 25 can be changed by tilting the ring-shaped device 13 while maintaining the deviation.

[0104] When the user performs a specific cancel operation such as double-tapping the touch panel 33 or shaking the ring-type device 13 , the offset may be canceled, and the adjustment axis 25 may be displayed to coincide with the posture of the operation axis of the touch panel 33 .

[0105] Fig. 9 An operation of adjusting the adjustment shaft 25 to a specified angle is shown.

[0106] In the AR system 11, for example, the designated angle of the adjustment axis 25 may be set so that when the 3D object 21 is viewed from the user's position, the X axis of the adjustment axis 25 is in the horizontal direction and the Y axis of the adjustment axis 25 is in the vertical direction. Then, when the user changes the posture of the adjustment axis 25 by tilting the ring-shaped device 13, when the current posture of the adjustment axis 25 differs from the designated angle by a certain angle or less (e.g., ±10° or less), a user interface indicating the designated angle may be displayed. This enables the user to match the posture of the adjustment axis 25 with the designated angle.

[0107] Fig. 9 An example of displaying the user interface 26 indicating the designated angle is shown. When the current posture of the adjustment shaft 25 differs from the designated angle by a certain angle or less, by displaying the user interface 26 indicating the designated angle in this manner, the user can easily match the posture of the adjustment shaft 25 with the designated angle by tilting the posture of the ring-shaped device 13 to coincide with the user interface 26.

[0108] <3D object adjustment processing>

[0109] Will refer to Figures 10 to 12 The adjustment process of the 3D object 21 is described.

[0110] In the AR system 11, for example, it is possible to perform adjustment processing of moving the position of the 3D object 21, adjustment processing of rotating the posture of the 3D object 21, and adjustment processing of enlarging / reducing the size of the 3D object 21. By single-tapping, double-tapping, or long-pressing the touch pad 33, clicking, double-clicking, or long-pressing the tactile switch 34, etc., it is possible to switch between the adjustment processing.

[0111] Fig.10 The adjustment process of the position of the mobile 3D object 21 is shown.

[0112] For example, Fig.10 As shown on the left side of FIG. 1 , when the adjustment mode is switched to, the adjustment axis 25 is displayed for the 3D object 21. Then, when the user Fig.10 As shown in the center of the touch panel 33, the left side is touched and while maintaining the touch, the following operations are performed: Fig.10 When the user moves the touch to the right side of the touch panel 33 as shown in the right side of the touch panel 33, the position of the 3D object 21 moves toward the right side of the X-axis of the adjustment shaft 25 according to the amount of movement of the touch. At this time, even if the ring-shaped device 13 is tilted while the user performs the operation of moving the touch, the adjustment shaft 25 is continuously in a fixed state while the touch is detected.

[0113] When the user performs an operation of moving the touch from the right side toward the left side of the touch panel 33, the position of the 3D object 21 moves toward the left side of the X axis of the adjustment axis 25. At the same time, when the user performs an operation of moving the touch from the upper side toward the lower side of the touch panel 33, the position of the 3D object 21 moves toward the lower side of the Y axis of the adjustment axis 25. Similarly, when the user performs an operation of moving the touch from the lower side toward the upper side of the touch panel 33, the position of the 3D object 21 moves toward the upper side of the Y axis of the adjustment axis 25.

[0114] In this way, the adjustment process of moving the position of the 3D object 21 is performed according to the movement direction and the movement amount of the touch operation performed by the user on the touch panel 33 .

[0115] Fig.11 The adjustment process of the posture of the rotated 3D object 21 is shown.

[0116] For example, Fig.11 As shown on the left side of FIG. 1 , when the adjustment mode is switched to, the adjustment axis 25 is displayed for the 3D object 21. Then, when the user Fig.11 As shown in the center of the touch panel 33, the upper side is touched and while maintaining the touch, the following operations are performed: Fig.11As shown in the right side of the figure, when the user moves the touch to the lower side of the touch panel 33, the posture of the 3D object 21 rotates toward the front side around the X-axis of the adjustment shaft 25 according to the amount of movement of the touch. At this time, even if the ring-shaped device 13 is tilted while the user is performing the operation of moving the touch, the adjustment shaft 25 is continuously in a fixed state while the touch is detected.

[0117] When the user performs an operation of moving the touch from the lower side toward the upper side of the touch panel 33, the posture of the 3D object 21 rotates toward the rear side around the X-axis of the adjustment shaft 25. At the same time, when the user performs an operation of moving the touch from the left side toward the right side of the touch panel 33, the posture of the 3D object 21 rotates toward the right side around the Y-axis of the adjustment shaft 25. Similarly, when the user performs an operation of moving the touch from the right side toward the left side of the touch panel 33, the posture of the 3D object 21 rotates toward the left side around the Y-axis of the adjustment shaft 25.

[0118] In this way, the adjustment process of the posture of rotating the 3D object 21 is performed according to the movement direction and the movement amount of the touch operation performed by the user on the touch panel 33 .

[0119] Fig.12 An adjustment process of enlarging / reducing the size of the 3D object 21 is shown.

[0120] For example, Fig.12 As shown on the left side of FIG. 1 , when the adjustment mode is switched to, the adjustment axis 25 is displayed for the 3D object 21. Then, when the user Fig.12 As shown in the center of the touch panel 33, the left side is touched and while maintaining the touch, the following operations are performed: Fig.12 When the user moves the touch to the right side of the touch panel 33 as shown in the right side of the figure, the size of the 3D object 21 is enlarged in the direction of the X-axis along the adjustment axis 25 according to the amount of movement of the touch. At this time, even if the ring-shaped device 13 is tilted while the user is performing the operation of moving the touch, the adjustment axis 25 is continuously in a fixed state while the touch is detected.

[0121] When the user performs an operation of moving the touch from the right side toward the left side of the touch panel 33, the size of the 3D object 21 is reduced in the direction along the X axis of the adjustment axis 25. At the same time, when the user performs an operation of moving the touch from the upper side toward the lower side of the touch panel 33, the size of the 3D object 21 is reduced in the direction along the Y axis of the adjustment axis 25. Similarly, when the user performs an operation of moving the touch from the lower side toward the upper side of the touch panel 33, the size of the 3D object 21 is enlarged in the direction along the Y axis of the adjustment axis 25.

[0122] In this way, the adjustment process of enlarging / reducing the size of the 3D object 21 is performed according to the moving direction and the moving amount of the touch operation performed by the user on the touch panel 33 .

[0123] <Processing Example of 3D Object Adjustment Processing>

[0124] Will refer to Fig.13 The illustrated flowchart describes the 3D object adjustment process performed in the AR system 11 .

[0125] In step S11, the adjustment mode determination unit 82 determines whether to transition to the adjustment mode for performing the adjustment process for the 3D object 21 based on the sensor information provided from the sensor information acquisition unit 81. For example, when it is detected that the user moves his / her finger wearing the ring-shaped device 13 close to the 3D object 21 and the fingertip is located at a position within the adjustment mode determination range 24 and the fingertip is not in contact with the 3D object 21, the adjustment mode determination unit 82 determines to transition to the adjustment mode.

[0126] The process waits until the adjustment mode determination unit 82 determines the transition to the adjustment mode in step S11. When the adjustment mode determination unit 82 determines the transition to the adjustment mode, the process proceeds to step S12.

[0127] In step S12, the axis posture determination unit 84 determines the posture of the adjustment axis 25 at the timing when the transition to the adjustment mode is made based on the device position posture information about the ring-shaped device 13 provided from the finger posture determination unit 83. For example, the axis posture determination unit 84 determines the posture of the adjustment axis 25 to be consistent with the posture of the operating axis of the surface of the touch panel 33 at the timing when the transition to the adjustment mode is made. Then, the axis posture determination unit 84 instructs the output control unit 87 to display the adjustment axis 25 according to the determined posture, and the output control unit 87 provides the generated visual data about the adjustment axis 25 to be displayed in the posture indicated by the axis posture determination unit 84 to the visual display unit 73. Therefore, as discussed above with reference to Figure 6 As described above, the adjustment axis 25 in a posture consistent with the posture of the operation axis of the surface of the touch panel 33 at the timing of transition to the adjustment mode is displayed at the origin of the local coordinate system 23 of the 3D object 21 .

[0128] In step S13 , the finger operation determination unit 85 determines whether a touch of the touch panel 33 of the ring-type device 13 by the user is detected based on the sensor information supplied from the sensor information acquisition unit 81 .

[0129] When the finger operation determination unit 85 determines in step S13 that the user's touch on the touch panel 33 of the ring-type device 13 is not detected, the process proceeds to step S14 .

[0130] In step S14, the axis posture determination unit 84 changes the posture of the adjustment axis 25 according to the change (if any such change) in the posture of the operating axis of the surface of the touch panel 33 based on the device position posture information about the ring-type device 13 provided from the finger posture determination unit 83. For example, the axis posture determination unit 84 instructs the output control unit 87 to display the adjustment axis 25 in the posture based on the device position posture information, and the output control unit 87 provides the generated visual data about the adjustment axis 25 to be displayed in the posture instructed by the axis posture determination unit 84 to the visual display unit 73. Therefore, as discussed above with reference to Figure 7 As described above, the adjustment shaft 25 is shown, wherein the posture of the adjustment shaft 25 changes according to the change of the posture of the ring-shaped device 13.

[0131] On the other hand, when the finger operation determination unit 85 determines in step S13 that the user's touch on the touch panel 33 of the ring-type device 13 is detected, the process proceeds to step S15 .

[0132] In step S15, in a case where the posture of the adjustment axis 25 is fixed, the adjustment processing unit 86 performs adjustment processing on the 3D object 21 based on the touch position and movement amount of the touch on the touch panel 33 provided from the finger operation determination unit 85. For example, in a case where the adjustment processing of moving the position of the 3D object 21 has been set, the adjustment processing unit 86 instructs the output control unit 87 to display the 3D object 21 at a position after it has been moved by a certain movement amount based on the movement amount of the touch. Therefore, the output control unit 87 provides the visual display unit 73 with the generated visual data about the 3D object 21 to be displayed at the position indicated by the adjustment processing unit 86. Therefore, as discussed above with reference to FIG. Fig.10 As described above, the moved 3D object 21 is displayed.

[0133] After the processing in step S14 or S15, the processing proceeds to step S16, and the adjustment mode determination unit 82 determines whether to end the adjustment mode for performing the adjustment processing on the 3D object 21 based on the sensor information supplied from the sensor information acquisition unit 81. For example, when it is detected that the user moves his / her finger wearing the ring-type device 13 away from the 3D object 21 and the fingertip is located at a position outside the adjustment mode determination range 24, the adjustment mode determination unit 82 determines to end the adjustment mode.

[0134] When the adjustment mode determination unit 82 determines in step S16 that the adjustment mode is not to be ended, the process returns to step S13, and similar processes are then repeatedly performed. On the other hand, when the adjustment mode determination unit 82 determines in step S16 that the adjustment mode is to be ended, the process returns to step S11, and similar processes are then repeatedly performed.

[0135] In the AR system 11, as described above, the adjustment process of the 3D object 21 can be performed in the 3D object adjustment process, wherein the posture of the adjustment axis 25 is changed when the touch panel 33 is not touched, and the posture of the adjustment axis 25 is fixed when the touch panel 33 is touched. Therefore, the adjustment process can be performed on the 3D object 21 as intended by the user, thereby improving the operability of the 3D object 21.

[0136] <Various Processing Examples>

[0137] Will refer to Figures 14 to 21 Examples of various processes that can be performed by the AR system 11 are described.

[0138] Fig.14 Another processing example of the adjustment processing of the position of the mobile 3D object 21 is shown.

[0139] In the AR system 11, for example, in addition to performing the adjustment process of moving the position of the 3D object 21 using the touch panel 33 as discussed above, the adjustment process of moving the position of the 3D object 21 may be performed by tilting the ring-type device 13. That is, the adjustment processing unit 86 may move the position of the 3D object 21 according to the tilt of the ring-type device 13 based on the angular velocity data and acceleration data about the ring-type device 13 acquired by the sensor information acquisition unit 81.

[0140] like Fig.14 As shown on the left side of FIG. , when the adjustment mode is switched to, the adjustment axis 25 is displayed for the 3D object 21. Then, as shown in FIG. Fig.14 As shown on the right side of , when the user tilts the ring-shaped device 13 to the right, the position of the 3D object 21 moves toward the right side of the X-axis of the adjustment axis 25 according to the tilt of the ring-shaped device 13.

[0141] The adjustment process performed using a touch on the touch panel 33 and the adjustment process performed by tilting the ring-type device 13 may be combined with each other.

[0142] In addition, when the ring-type device 13 includes an operating tool such as a pointing stick or a scroll wheel, the adjustment process can be performed using such an operating tool. Alternatively, even when the ring-type device 13 is not worn, for example, the adjustment process can be performed by detecting the movement of one finger in contact with another finger using a skin strain sensor, an electromyographic sensor, etc., or by detecting the movement in the depth direction using a pressure sensor, etc.

[0143] Fig.15 The magnification in the adjustment process for the 3D object 21 is shown.

[0144] In the AR system 11 , for example, when the adjustment process is performed by touching the touch panel 33 , the magnification of the movement amount of the position of the 3D object 21 can be changed.

[0145] like Fig.15 As shown in A, when the user performs an operation of touching near the upper side of the touch panel 33 and moving the touch in the X-axis direction while maintaining the touch, the position of the 3D object 21 can be significantly moved in the X-axis direction by multiplying a reference movement amount that matches the movement amount of the touch by 1 or more.

[0146] like Fig.15 As shown in B, when the user performs an operation of touching near the lower side of the touch panel 33 and moving the touch in the X-axis direction while maintaining the touch, the position of the 3D object 21 can be slightly moved in the X-axis direction by multiplying a reference movement amount that matches the movement amount of the touch by a factor of 1 or less.

[0147] Similarly, when the user performs an operation of touching near the right side of the touch panel 33 and moving the touch in the Y-axis direction while maintaining the touch, the position of the 3D object 21 may move significantly in the Y-axis direction. Meanwhile, when the user performs an operation of touching near the left side of the touch panel 33 and moving the touch in the Y-axis direction while maintaining the touch, the position of the 3D object 21 may move slightly in the Y-axis direction.

[0148] As with the magnification for moving the position of the 3D object 21 , the magnification may or may not be used for the rotation amount for turning the posture of the 3D object 21 or the enlargement or reduction amount for enlarging / reducing the size of the 3D object 21 .

[0149] The magnification may vary according to the size of the 3D object 21. For example, when the size of the 3D object 21 is large, the position of the 3D object 21 may be significantly moved, and when the size of the 3D object 21 is small, the position of the 3D object 21 may be slightly moved.

[0150] When the magnification in the adjustment process of the 3D object 21 is changed in this way, the display on the user interface may be changed to allow the user to recognize such a change. Fig.16 As shown, the adjustment axis 25 may be indicated by a thick line.

[0151] Before performing the adjustment process of the 3D object 21, the magnification may be changed according to the time from when the finger touches the touch panel 33 until the finger starts to move. For example, when the finger starts to move immediately after touching the touch panel 33, the magnification may be low, and when the finger starts to move several seconds after touching the touch panel 33, the magnification may be high.

[0152] For each user who uses the AR system 11, the magnification may be stored in advance in the ring-type device 13. In this case, by identifying the user wearing the ring-type device 13, the adjustment process is performed using the magnification stored in the ring-type device 13.

[0153] Fig.17 Auxiliary detection of a touch position on the touch panel 33 is shown.

[0154] Occasionally, for example, when the user touches the upper side of the touch panel 33 and moves the touched position to the lower side of the touch panel 33 while maintaining the touch, the touched position cannot be accurately detected.

[0155] like Fig.17 As shown in A of FIG. 1 , even if the user intends to touch the upper side of the touch panel 33, when the tip of the thumb contacts the touch panel 33 when the user starts touching, the center of the touch panel 33 is detected as the touched position. Therefore, in this case, Fig.17 As shown in FIG. 1B , even if the user moves the touch position downward, only the movement from the center of the touch panel 33 to the downward side is detected. That is, the touch operation performed by the user cannot be accurately detected.

[0156] Therefore, in the AR system 11, the detection of the touch position on the touch panel 33 can be assisted by detecting and tracking the position of the user's thumb based on the video of the user's fingers captured by the camera 15. In the AR system 11, by using video-based tracking in this way, the touch position can be detected more accurately.

[0157] Fig.18 The process of changing the posture of the adjustment shaft 25 at every predetermined angle is shown.

[0158] In the AR system 11, as discussed above, the posture of the adjustment axis 25 can be freely changed to any angle according to the posture of the ring-type device 13. However, it is assumed that it is occasionally difficult to change the posture of the adjustment axis 25 to a desired angle. Therefore, in the AR system 11, for example, when the ring-type device 13 is tilted to a certain angle or more, the posture of the adjustment axis 25 can be changed at each predetermined angle.

[0159] For example, Fig.18 FIG. 2 shows a processing example of a process of changing the posture of the adjustment shaft 25 by every 90° when the ring-shaped device 13 is tilted to 45° or more. Fig.18 As shown on the left side of FIG. , when the mode is switched to the adjustment mode, the adjustment axis 25 is displayed for the 3D object 21. Fig.18 As shown in the center of FIG. 1 , when the ring-shaped device 13 is tilted to an angle less than 45°, even if the user tilts the ring-shaped device 13, the posture of the adjustment shaft 25 does not change. Fig.18As shown on the right side of FIG. 1 , when the user further tilts the ring-shaped device 13 to 45° or more, the posture of the adjustment shaft 25 is changed by 90°.

[0160] That is, when the ring-type device 13 is tilted to the right or left by an angle less than 45°, the adjustment shaft 25 remains fixed even if the ring-type device 13 is tilted. When the ring-type device 13 is tilted to the right or left by 45° or more, the posture of the adjustment shaft 25 can be rotated 90° to the right or left. Therefore, the posture of the adjustment shaft 25 can be accurately adjusted to a predetermined angle on either side.

[0161] Of course, the change in the posture of the adjustment shaft 25 is not limited to every 90°, but may be arranged so that the posture of the adjustment shaft 25 is changed every any predetermined angle such as 45° or 60°.

[0162] Fig.19 A process of initially displaying the adjustment axis 25 according to a default posture is shown.

[0163] In the AR system 11, as discussed above, the adjustment axis 25 is displayed according to the posture of the touch pad 33 of the ring-type device 13 at the timing of transition to the adjustment mode. However, when the posture of the adjustment axis 25 initially displayed at the timing of transition to the adjustment mode is different according to the posture of the touch pad 33 of the ring-type device 13, it is assumed that it is occasionally difficult to perform adjustment processing on the 3D object 21.

[0164] Therefore, in the AR system 11, regardless of the posture of the touch panel 33 of the ring-type device 13 at the timing of transition to the adjustment mode, the adjustment axis 25 can always be initially displayed according to the default posture. For example, a posture in which the X axis of the adjustment axis 25 is in the horizontal direction and the Y axis of the adjustment axis 25 is in the vertical direction in the case where the 3D object 21 is viewed from the user's position can be set as the default posture for initially displaying the adjustment axis 25. For example, the default posture for initially displaying the adjustment axis 25 can be determined to match the result of estimating the psychological position and posture of the user's hand.

[0165] For example, when the ring device 13 is Fig.19 When the tilting state shown in A is changed to the adjustment mode, and when the ring device 13 is as shown in Fig.19 When the adjustment mode is changed to the tilted state shown in B, the adjustment axis 25 can be displayed in the same posture in either case by initially displaying the adjustment axis 25 according to the default posture. This enables the user to easily perform the adjustment process by naturally moving his / her hand close to the 3D object 21.

[0166] Here, as Fig.19As shown in FIG. 1B , when a deviation occurs between the posture of the adjustment shaft 25 and the posture of the operation shaft of the touch pad 33 of the ring-shaped device 13, when the posture of the adjustment shaft 25 is changed thereafter, the posture of the adjustment shaft 25 is changed while maintaining the deviation. Fig.19 The posture of the adjustment shaft 25 is changed while the relative relationship shown in B is changed.

[0167] In the AR system 11 , for example, the direction of the Z axis (not shown) may be fixed so that the Z axis is always in the direction of the user.

[0168] In the AR system 11, furthermore, the 3D object 21 may be initially displayed according to a default posture. For example, a default posture for initially displaying the 3D object 21 may be set so that when the 3D object 21 is displayed with reference to the ring-shaped device 13, the Z-axis direction of the local coordinate system 23 of the 3D object 21 coincides with the Z-axis direction of the world coordinate system 22. Therefore, when the 3D object 21 is initially displayed, the 3D object 21 is initially displayed so that the posture of the 3D object 21 in the up-down direction is always the same.

[0169] For example, when displaying the 3D object 21 with reference to the ring-type device 13, it is not desirable to Fig. 20 As shown in A, the 3D object 21 is initially displayed according to the posture of the ring device 13 (that is, the up and down direction of the 3D object 21 is not consistent with the Z-axis direction of the world coordinate system 22). Fig. 20 As shown in FIG. 1B , regardless of the posture of the ring-shaped device 13, the 3D object 21 is preferably initially displayed according to the default posture so that the Z-axis direction of the local coordinate system 23 of the 3D object 21 is consistent with the Z-axis direction of the world coordinate system 22. Therefore, the 3D object 21 is initially displayed so that when the user sees the 3D object 21, the upper side in the Z-axis direction of the local coordinate system 23 of the 3D object 21 always points upward.

[0170] Will refer to Fig.21 The selection of the 3D object 21 as a target of the adjustment process is described.

[0171] In the AR system 11 , for example, selection of the 3D object 21 as a target of the adjustment process may be changed according to a touch position on the touch panel 33 of the ring-type device 13 .

[0172] Fig.21 FIG. 2 shows a processing example in which the 3D object 21 is a vehicle and the 3D object 21a is a door. Fig.21 As shown in A of FIG. 1 , when the touch position on the touch panel 33 is on the lower side, the 3D object 21 is selected as the target of the adjustment process. Fig.21 As shown in B, when the touch position on the touch panel 33 is on the upper side, the 3D object 21a is selected as the target of the adjustment process.

[0173] For example, when the user touches the lower side of the touch panel 33, a larger 3D object 21 may be selected, and when the user touches the upper side of the touch panel 33, a smaller 3D object 21a may be selected. This enables the user to select the 3D object 21 quickly and easily.

[0174] Of course, in addition to when the 3D object 21 is selected as the target of the adjustment process, when the 3D object 21 is selected as the target of the operation of "grasping" the 3D object 21 as in the related art, the target to be selected can also be changed according to the touch position on the touch panel 33. In addition, the same can also be applied to the operation of switching the appearance of the object, intensity simulation, or display of layers such as temperature.

[0175] Here, while the ring-type device 13 is used by way of example in the present embodiment to adjust the 3D object 21, a device other than the ring-type device 13, such as a glove-type device or a device whose shape covers the entirety of a single finger, may also be used. The touch pad 33 is also not limited to the touch pad capable of dual-axis detection discussed above, and a touch pad capable of single-axis detection may also be used.

[0176] For example, Fig. 22 As shown, a pen-type device 91 may also be used. The pen-type device 91 is provided with a touch pad 92 capable of uniaxial detection at a position near the index finger when held by the user. Although not shown, the pen-type device 91 is also provided with a tactile switch 34.

[0177] To use the pen-type device 91, Fig. 22 B and Fig. 22 As shown in FIG. 9C, a single-axis adjustment axis 27 is displayed for the 3D object 21 according to the tilt of the pen-type device 91. Along the single-axis adjustment axis 27 displayed in this manner, adjustment processing for the 3D object 21 can be performed.

[0178] like Fig.23 As shown, a mouse type device 93 may also be used. For example, Fig.23 As shown in A of FIG. 1 , an operation axis centered at the center of the top end of the mouse type device 93 is set for the mouse type device 93. Among the operation axes represented by the three axes of the X axis, the Y axis, and the Z axis indicated in the figure, an operation axis represented by two axes of the X axis and the Z axis or the X axis and the Y axis, or an operation axis represented by a single axis, etc. are set.

[0179] For example, the user can perform an operation of performing adjustment processing on the 3D object 21 while holding the mouse-type device 93 in the air. Fig.23As shown in FIG. 1B , according to the posture of the mouse type device 93 at the timing of transition to the adjustment mode, the adjustment axis 25 is displayed according to the origin of the local coordinate system 23 of the 3D object 21. Then, after the adjustment axis 25 is fixed, the adjustment process of the 3D object 21 can be performed according to the operation direction and the operation amount of the operation on the operation unit such as the capacitive touch panel 94 and the scroll wheel 95 provided in the mouse type device 93.

[0180] <Computer Configuration Example>

[0181] The above-described series of processing (information processing method) can be executed by hardware or software. In the case where the series of processing is executed by software, a program configuring the software is installed in a general-purpose computer or the like.

[0182] Fig.24 : is a block diagram showing a schematic configuration of an embodiment of a computer on which a program that executes the above-described series of processing steps is installed.

[0183] The program may be recorded in advance in the hard disk 105 or the ROM 103 which is a recording medium built in the computer.

[0184] Alternatively, the program may be stored (recorded) in a removable recording medium 111 driven by the drive 109. Such a removable recording medium 111 may be provided as so-called packaged software. Here, examples of the removable recording medium 111 include a floppy disk, a compact disc read-only memory (CD-ROM, Compact Disc Read Only Memory), a magneto-optical (MO, Magneto Optical) disk, a digital versatile disk (DVD, Digital Versatile Disc), a magnetic disk, and a semiconductor memory.

[0185] The program may be installed in the computer from the removable recording medium 111 as described above, or may be downloaded to the computer via a communication network or a broadcasting network and installed in the built-in hard disk 105. That is, for example, the program is transmitted to the computer wirelessly from a download site via an artificial satellite for digital satellite broadcasting, or is transmitted to the computer by wire via a network such as a LAN (Local Area Network) or the Internet.

[0186] The computer includes a central processing unit (CPU) 102 . An input / output interface 110 is connected to the CPU 102 via a bus 101 .

[0187] When the user inputs an instruction through the input / output interface 110 by manipulating the input unit 107, the CPU 102 executes a program stored in the read only memory (ROM) 103 according to the instruction. Alternatively, the CPU 102 loads a program stored in the hard disk 105 into the random access memory (RAM) 104 and executes the program.

[0188] Therefore, the CPU 102 executes the processing according to the above flowchart or the processing executed by the components of the above block diagram. Then, the CPU 102 causes the output unit 106 to output the processing result, causes the communication unit 108 to send the processing result, and causes the hard disk 105 to record the processing result, for example, via the input / output interface 110 as needed.

[0189] The input unit 107 is constituted by a keyboard, a mouse, a microphone, etc. Furthermore, the output unit 106 is configured by a liquid crystal display (LCD), a speaker, and the like.

[0190] The processing performed by the computer according to the program described in this article may not necessarily be performed in chronological order in the order described as the flowchart. In other words, the processing performed by the computer according to the program also includes processing performed in parallel or individually (for example, parallel processing or object processing).

[0191] The program may be a program that is processed by one computer (processor) or may be distributed and processed by a plurality of computers. In addition, the program may be a program that is sent to a remote computer for execution.

[0192] In addition, in this specification, a system means a collection of multiple constituent elements (including devices and modules (components)), regardless of whether all constituent elements are contained in the same housing. Therefore, multiple devices contained in separate housings and connected via a network and a device in which multiple modules are contained in one housing are both systems.

[0193] For example, a configuration described as one device (or processing unit) may be divided and configured as a plurality of devices (or processing units). On the other hand, the above configuration as a plurality of devices (or processing units) may be configured together as one device (or processing unit). In addition, of course, configurations other than the above may be added to the configuration of each device (or each processing unit). In addition, as long as the configuration or operation of the system as a whole is substantially the same, a part of the configuration of a device (or processing unit) may be included in the configuration of another device (or another processing unit).

[0194] Furthermore, for example, the present technology may have a cloud computing configuration in which one function is shared and processed by a plurality of devices via a network.

[0195] In addition, for example, the above-mentioned program can be executed in any device. In this case, the device only needs to have necessary functions (functional blocks, etc.) and be able to obtain necessary information.

[0196] In addition, for example, each step described in the above flowchart can be performed by one device, or by multiple devices in a shared manner. In addition, in the case where multiple processing steps are included in one step, the multiple processing steps included in one step can be performed by one device or by multiple devices in a shared manner. In other words, it is also possible to perform multiple processing included in one step as processing of multiple steps. On the other hand, it is also possible to perform the processing described as multiple steps together as one step.

[0197] For example, for a program executed by a computer, the processing of the steps describing the program may be performed in chronological order in the order described in this specification, or may be performed in parallel or individually at necessary timings such as when called. That is, the processing of the individual steps may be performed in an order different from the above order as long as there is no contradiction. In addition, the processing of the steps describing the program may be performed in parallel with the processing of another program, or may be performed in combination with the processing of another program.

[0198] Note that, as long as no contradiction occurs, the present technology described in this specification as various modes can be implemented separately and independently. Of course, any multiple technologies can be implemented together. For example, some or all of the present technology described in several embodiments can be implemented in combination with some or all of the present technology described in other embodiments. Any part or all of the above-mentioned present technology can also be implemented together with another technology not described above.

[0199] <Configuration combination example>

[0200] The present disclosure may also be configured as follows. (1)

[0202] An information processing device, comprising:

[0203] an axis posture determination unit that determines a posture of an adjustment axis based on a posture of a user's finger, and changes and fixes the posture of the adjustment axis, the adjustment axis serving as a reference for performing adjustment processing on a 3D object set in a three-dimensional space on computer graphics to be superimposed on a real space; and

[0204] An adjustment processing unit performs adjustment processing on the 3D object according to an operation performed by a user, according to an axial direction of the adjustment axis in the fixed posture. (2)

[0206] The information processing device according to (1), further comprising:

[0207] A sensor information acquisition unit that acquires sensor information provided by a wearable device that is worn on a user's finger and has a touch panel for inputting a touch operation, wherein:

[0208] an axis posture determination unit that determines a posture of the adjustment axis by estimating a posture of a finger of the user based on sensor information indicating acceleration data and angular velocity data detected by the wearable device; and

[0209] The adjustment processing unit performs adjustment processing on the 3D object based on sensor information indicating a touch operation performed on the touch panel by a user. (3)

[0211] The information processing device according to (2), wherein:

[0212] a shaft posture determination unit that changes the posture of the adjustment shaft when a touch to the touch panel is not detected, and fixes the posture of the adjustment shaft at a timing when a touch to the touch panel is detected; and

[0213] When a touch to the touch panel is detected, the adjustment processing unit performs adjustment processing on the 3D object. (4)

[0215] The information processing device according to (3), wherein:

[0216] The adjustment processing unit performs adjustment processing of moving the position of the 3D object, adjustment processing of rotating the posture of the 3D object, or adjustment processing of enlarging / reducing the size of the 3D object. (5)

[0218] The information processing device according to any one of (1) to (4), wherein:

[0219] An adjustment mode determination range for determining an adjustment mode in which an adjustment process for a 3D object is performed is set in a certain range around the 3D object. (6)

[0221] The information processing device according to (3), wherein:

[0222] The axis posture determination unit displays the adjustment axis according to the posture of the wearable device at the timing when transitioning to the adjustment mode in which the adjustment process for the 3D object is performed. (7)

[0224] The information processing device according to (6), wherein:

[0225] When in the adjustment mode, when the posture of the wearing device changes, the axis posture determination unit changes the posture of the adjustment axis according to the posture of the wearing device. (8)

[0227] The information processing device according to (6) or (7), wherein:

[0228] When in the adjustment mode, during a period in which a user performs a predetermined operation, the axis posture determination unit fixes the adjustment axis at a posture at the timing of performing the predetermined operation. (9)

[0230] The information processing device according to any one of (6) to (8), wherein:

[0231] When the current posture of the adjustment axis differs from a preset designated angle by a certain angle or less, the axis posture determination unit displays a user interface indicating the designated angle. (10)

[0233] The information processing device according to (3), wherein:

[0234] The adjustment processing unit moves the position of the 3D object according to the movement direction and movement amount of the touch operation performed by the user on the touch panel based on sensor information indicating the touch operation performed by the user. (11)

[0236] The information processing device according to (3), wherein:

[0237] The adjustment processing unit rotates the posture of the 3D object according to the moving direction and the moving amount of the touch operation performed by the user on the touch panel based on sensor information indicating the touch operation performed on the touch panel by the user. (12)

[0239] The information processing device according to (3), wherein:

[0240] The adjustment processing unit enlarges or reduces the size of the 3D object according to a moving direction and a moving amount of the touch operation performed by the user on the touch panel based on sensor information indicating the touch operation performed by the user. (13)

[0242] The information processing device according to (3), wherein:

[0243] The adjustment processing unit moves the position of the 3D object according to the tilt of the wearable device based on sensor information indicating acceleration data and angular velocity data detected by the wearable device. (14)

[0245] The information processing device according to (3), wherein:

[0246] The adjustment processing unit changes a magnification of a movement amount of the position of the 3D object according to a touch position of the user on the touch panel. (15)

[0248] The information processing device according to (14), wherein:

[0249] When the magnification is changed, the display of the adjustment axis changes. (16)

[0251] The information processing device according to (3), wherein:

[0252] Tracking based on capturing a video of the user's fingers is used to assist in detecting the location of a user's touches on the touchpad. (17)

[0254] The information processing device according to (3), wherein:

[0255] When the wearable device is tilted at a certain angle or more, the axis posture determination unit changes the posture of the adjustment axis at each predetermined angle. (18)

[0257] The information processing device according to (3), wherein:

[0258] When transitioning to the adjustment mode in which the adjustment process for the 3D object is performed, the axis posture determination unit displays the adjustment axis according to a predetermined default posture when the adjustment axis is initially displayed. (19)

[0260] The information processing device according to (3), wherein:

[0261] The selection of a 3D object as a target to be processed is changed according to a touch position of the touch panel by the user. (20)

[0263] An information processing method causes an information processing device to perform a process, the process comprising:

[0264] determining a posture of an adjustment axis used as a reference for performing adjustment processing on a 3D object set in a three-dimensional space on computer graphics to be superimposed on a real space based on a posture of a user's finger, and changing and fixing the posture of the adjustment axis; and

[0265] According to the axial direction of the adjustment axis in the fixed posture, the adjustment process is performed on the 3D object according to the operation performed by the user.

[0266] Note that the embodiments of the present disclosure are not limited to the above-mentioned embodiments, and may be modified in various ways without departing from the scope and spirit of the present disclosure. The advantageous effects described in this specification are merely exemplary and not restrictive, and other advantageous effects may be achieved.

[0267] Reference numerals list

[0268] 11AR system, 12 glasses-type wearable terminal, 13 ring-type device, 14 transmissive display unit, 15 camera device, 213D object, 22 world coordinate system, 23 local coordinate system, 24 adjustment mode determination range, 25 adjustment axis, 31 main body, 32 belt part, 33 touch panel, 34 tactile switch, 41 battery, 42 vibration actuator, 43 audio module, 44IMU, 45 wireless communication substrate, 51 display, 52 speaker, 53 sensor, 54 device body, 55CPU, 56 memory, 57 communication device, 58 input / output interface, 61 vibrator, 62 sensor, 63 device body, 64 CPU, 65 memory, 66 communication device, 67 input / output interface, 71 sensor control unit, 72 information processing unit, 73 visual display unit, 74 tactile presentation unit, 75 sound presentation unit, 81 sensor information acquisition unit, 82 adjustment mode determination unit, 83 finger posture determination unit, 84 axis posture determination unit, 85 finger operation determination unit, 86 adjustment processing unit, 87 output control unit, 91 pen type device, 92 touch pad, 93 mouse type device, 94 capacitive touch pad, 95 scroll wheel

Claims

1. An information processing device, comprising: an axis posture determination unit that determines a posture of an adjustment axis used as a reference for performing adjustment processing on a 3D object set in a three-dimensional space on computer graphics to be superimposed on a real space based on the posture of a user's finger, and changes and fixes the posture of the adjustment axis; as well as An adjustment processing unit performs the adjustment processing on the 3D object according to the operation performed by the user according to the axial direction of the adjustment axis in the fixed posture.

2. The information processing device according to claim 1, further comprising: a sensor information acquisition unit that acquires sensor information provided from a wearable device that is worn on a finger of the user and has a touch panel for inputting a touch operation, wherein: the axis posture determination unit determines the posture of the adjustment axis by estimating the posture of the user's finger based on sensor information indicating acceleration data and angular velocity data detected by the wearable device; and The adjustment processing unit performs the adjustment processing on the 3D object based on sensor information indicating a touch operation performed on the touch panel by the user.

3. The information processing device according to claim 2, wherein: the axis posture determination unit changes the posture of the adjustment axis when a touch to the touch panel is not detected, and fixes the posture of the adjustment axis at a timing when a touch to the touch panel is detected; as well as When a touch on the touch panel is detected, the adjustment processing unit performs the adjustment processing on the 3D object.

4. The information processing device according to claim 3, wherein: The adjustment processing unit performs an adjustment process of moving a position of the 3D object, an adjustment process of rotating a posture of the 3D object, or an adjustment process of enlarging / reducing a size of the 3D object.

5. The information processing device according to claim 1, wherein: An adjustment mode determination range for determining an adjustment mode for performing an adjustment process on the 3D object is set in a certain range around the 3D object.

6. The information processing device according to claim 3, wherein: The axis posture determination unit displays the adjustment axis according to the posture of the wearable device at the timing of transition to an adjustment mode in which an adjustment process for the 3D object is performed.

7. The information processing device according to claim 6, wherein: When in the adjustment mode, when the posture of the wearable device changes, the axis posture determination unit changes the posture of the adjustment axis according to the posture of the wearable device.

8. The information processing device according to claim 6, wherein: When in the adjustment mode, during a period in which the user performs a predetermined operation, the axis posture determination unit fixes the adjustment axis at a posture at a timing when the predetermined operation is performed.

9. The information processing device according to claim 6, wherein: When the current posture of the adjustment axis differs from a preset designated angle by a certain angle or less, the axis posture determination unit displays a user interface indicating the designated angle.

10. The information processing device according to claim 3, wherein: The adjustment processing unit moves the position of the 3D object according to a moving direction and a moving amount of the touch operation performed by the user on the touch panel based on sensor information indicating the touch operation performed on the touch panel by the user.

11. The information processing device according to claim 3, wherein: The adjustment processing unit rotates the posture of the 3D object according to a moving direction and a moving amount of the touch operation performed by the user on the touch panel based on sensor information indicating the touch operation performed on the touch panel by the user.

12. The information processing device according to claim 3, wherein: The adjustment processing unit enlarges or reduces the size of the 3D object according to a moving direction and a moving amount of the touch operation performed by the user on the touch panel based on sensor information indicating the touch operation performed on the touch panel by the user.

13. The information processing device according to claim 3, wherein: The adjustment processing unit moves the position of the 3D object according to the tilt of the wearable device based on sensor information indicating acceleration data and angular velocity data detected by the wearable device.

14. The information processing device according to claim 3, wherein: The adjustment processing unit changes a magnification of a movement amount when moving a position of the 3D object according to a touch position of the user on the touch panel.

15. The information processing device according to claim 14, wherein: When the magnification is changed, the display of the adjustment axis is changed.

16. The information processing device according to claim 3, wherein: Tracking based on capturing video of the user's fingers is used to assist in detecting the location of the user's touches on the touchpad.

17. The information processing device according to claim 3, wherein: When the wearable device is tilted at a certain angle or more, the axis posture determination unit changes the posture of the adjustment axis at each predetermined angle.

18. The information processing device according to claim 3, wherein: When transitioning to an adjustment mode in which an adjustment process for the 3D object is performed, the axis posture determination unit displays the adjustment axis according to a predetermined default posture when the adjustment axis is initially displayed.

19. The information processing device according to claim 3, wherein: Selection of the 3D object as a target to be processed is changed according to a touch position of the touch panel by the user.

20. An information processing method, the information processing method causing an information processing device to perform a process, the process comprising: determining a posture of an adjustment axis based on a posture of a user's finger, and changing and fixing the posture of the adjustment axis, the adjustment axis serving as a reference for performing adjustment processing on a 3D object set in a three-dimensional space on computer graphics to be superimposed on a real space; as well as The adjustment process is performed on the 3D object according to an operation performed by the user, according to the axial direction of the adjustment axis in the fixed posture.

Images