Input device, control apparatus, control method, information processing apparatus, and information processing method

Abstract

The present technology relates to an input device, a control device, a control method, an information processing device, and an information processing method, which are capable of improving operability of an input device for operations such as augmented reality (XR). An input device includes: a ring portion into which a finger is inserted; an operation part which can be operated by a finger inserted into the ring part in the first direction; and a gripping part which is gripped by a palm when the operation part is operated by a finger. The present technology can be applied to, for example, an XR system.

Description

Technical Field

[0001] The present technology relates to input devices, control apparatuses, control methods, information processing apparatuses and information processing methods, and more specifically, to input devices, control apparatuses, control methods, information processing apparatuses and information processing methods that improve the operability of input devices used for operations such as extended reality (XR) space. Background Art

[0002] Generally, a handheld input device is used for operation of a virtual space (for example, see Patent Document 1). In addition, generally, a pen-type input device is used for indicating a position in a virtual space (for example, see Patent Document 2).

[0003] Citation List

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2020-181320

[0006] Patent Document 2: WO 2019 / 220803 A Summary of the invention

[0007] Problems to be solved by the present invention

[0008] However, if a strap or the like is not provided, the handheld input device may fall when the hand is released. In addition, for example, when a button is provided in a pen-type input device, there is a possibility that the button operation causes the pen tip to shake and the position indicated by the input device to shift.

[0009] The present technology has been made in view of such circumstances, and an object is to improve the operability of an input device used for operations of an XR space or the like.

[0010] Solution to the problem

[0011] An input device according to a first aspect of the present technology includes: a ring portion into which a finger is inserted; an operation portion operable by the finger inserted into the ring portion in a first direction; and a grip portion gripped by a palm when the operation portion is operated by the finger.

[0012] According to the second aspect of the present technology, a control device includes an identification unit and an operation control unit, wherein the identification unit identifies the position and orientation of an input device, wherein the input device includes a ring portion into which fingers of a user's hand are inserted, an operation portion that can be operated by the fingers inserted into the ring portion along a first direction, and a holding portion that is held by the palm of the hand when the operation portion is operated by the fingers; the operation control unit controls the operation of the input device based on the position and orientation of the input device.

[0013] According to the second aspect of the present technology, the control method includes: identifying the position and orientation of an input device, and controlling the operation of the input device based on the position and orientation of the input device; wherein the input device includes a ring portion into which the fingers of a user's hand are inserted, an operating portion that can be operated by the fingers inserted into the ring portion along a first direction, and a holding portion that is held by the palm of the hand when the operating portion is operated by the fingers.

[0014] In an information processing device according to the third aspect of the present technology, an operation input signal output from an input device is input, and display control information for controlling the display of an XR space based on the operation input signal is output; wherein the input device includes a ring portion into which a finger is inserted, an operating portion that can be operated by the finger inserted into the ring portion, and a holding portion that is held by the palm of the hand when the operating portion is operated by the finger.

[0015] According to the fourth aspect of the present technology, an information processing device includes: a control unit that receives input information, executes an application based on the input information, and outputs output information for controlling the display of a virtual object including computer-aided design (CAD) information in an XR space; wherein the input information indicates at least one of the following: a state of a terminal device that presents the XR space to a user, a state around the terminal device, a state of the user, a behavior of the user, or an operation input to an input device for operation input of the XR space.

[0016] According to the fourth aspect of the present technology, the information processing method includes: receiving input information through an information processing device, executing an application based on the input information, and outputting output information for controlling the display of a virtual object including CAD information in an XR space; wherein the input information indicates at least one of the following: the state of a terminal device that presents the XR space to a user, the state around the terminal device, the state of the user, the user's behavior, or operation input to an input device for operation input of the XR space.

[0017] According to the fifth aspect of the present technology, an information processing device includes: a control unit, which executes an application for presenting an XR space to a user, and outputs output information indicating a change or abnormality in the state of the application to a terminal device that presents the XR space to the user or an input device for inputting operations of the XR space.

[0018] According to the fifth aspect of the present technology, the information processing method includes: executing an application for presenting an XR space to a user through an information processing device, and outputting output information indicating a change or abnormality in the state of the application to a terminal device for presenting the XR space to the user or an input device for inputting operations of the XR space.

[0019] In the first aspect of the present technology, the operation portion is operated by the finger inserted into the ring portion, and the grip portion is gripped by the palm while the operation portion is operated by the finger.

[0020] In a second aspect of the present technology, the position and orientation of an input device are identified, and the operation of the input device is controlled based on the position and orientation of the input device; wherein the input device includes a ring portion into which fingers of a user's hand are inserted, an operating portion that can be operated by the fingers inserted into the ring portion along a first direction, and a holding portion that is held by the palm of the hand when the operating portion is operated by the fingers.

[0021] In a third aspect of the present technology, an operation input signal output from an input device is input, and display control information for controlling the display of an XR space based on the operation input signal is output; wherein the input device includes a ring portion into which a finger is inserted, an operating portion that can be operated by the finger inserted into the ring portion, and a holding portion that is held by a palm when the operating portion is operated by the finger.

[0022] In a fourth aspect of the present technology, input information is received, an application is executed based on the input information, and output information for controlling the display of a virtual object including computer-aided design (CAD) information in an XR space is output; wherein the input information indicates at least one of: a state of a terminal device that presents the XR space to a user, a state around the terminal device, a state of the user, a behavior of the user, or an operation input to an input device for operation input of the XR space.

[0023] In a fifth aspect of the present technology, an application for presenting an XR space to a user is executed, and output information indicating a change or abnormality in the state of the application is output to a terminal device that presents the XR space to the user or an input device for inputting operations of the XR space. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a diagram showing an embodiment of an XR system to which the present technology is applied.

[0025] Figure 2 is a diagram showing a display example of an XR system.

[0026] Figure 3 is a diagram showing a display example of an XR system.

[0027] Figure 4 is a block diagram showing a configuration example of an information processing device and a terminal device.

[0028] Figure 5 is an external view showing a configuration example of a controller device.

[0029] Figure 6is a diagram illustrating a method of grasping a controller device.

[0030] Figure 7 is a diagram illustrating a method of grasping a controller device.

[0031] Figure 8 is a diagram illustrating a method of grasping a controller device.

[0032] Fig. 9 is a diagram showing an example of arrangement of operating members of a controller device.

[0033] Fig.10 is a diagram showing an arrangement example of symbols of a controller device.

[0034] Fig.11 is a diagram showing an example of how markers appear on a controller device.

[0035] Fig.12 is a diagram for explaining a method of identifying the position and orientation of a controller device.

[0036] Fig.13 is a diagram showing an example of the internal configuration of a controller device.

[0037] Fig.14 is a diagram showing an arrangement example of a haptic device of a controller device.

[0038] Fig.15 Detailed description of the flowchart of the operation member control process performed by the XR system.

[0039] Fig.16 It is a diagram for explaining the operation member control processing performed by the XR system.

[0040] Fig.17 is a diagram showing an example of a method of grasping a controller device.

[0041] Fig.18 is a diagram showing an example of a method of grasping a controller device.

[0042] Fig.19 is a diagram showing an example of a method of grasping a controller device.

[0043] Fig. 20 is a flowchart for illustrating a haptic feedback control process performed by the XR system.

[0044] Fig.21 is a diagram for explaining an example of tactile feedback.

[0045] Fig. 22 is a diagram for explaining an example of tactile feedback.

[0046] Fig.23This is a diagram for explaining an example of haptic feedback.

[0047] Fig.24 This is a block diagram showing an example configuration of a computer. Detailed implementation

[0048] Hereinafter, implementations for executing the present technology will be described. The description will be given in the following order.

[0049] 1. Implementation

[0050] 2. Modification example

[0051] 3. Others

[0052] <<1. Implementation>>

[0053] Reference will be made to Figures 1 to 23 to describe an implementation of the present technology.

[0054] <Example configuration of XR system 101>

[0055] Figure 1 Shows an example configuration of an extended reality (XR) system 101, which is an implementation of an information processing system to which the present technology is applied.

[0056] XR system 101 is a system that realizes XR, which is a technology that fuses the real world and the virtual world, such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and substitute reality (SR). XR system 101 is a system that presents to a user a space (hereinafter referred to as an XR space) obtained by fusing a real space and a virtual space. For example, XR system 101 can present to a user a non-real virtual object (hereinafter referred to as a virtual object or virtual entity), such as a model created by computer-aided design (CAD) (hereinafter referred to as a CAD model), as if the virtual object exists at that location.

[0057] XR system 101 includes an information processing device 111, a terminal device 112, and a controller device 113.

[0058] The information processing device 111 and the terminal device 112 can communicate with each other wirelessly or via a wire and send and receive data from each other. The terminal device 112 and the controller device 113 can communicate with each other wirelessly or via a wire and send and receive data from each other. The information processing device 111 and the controller device 113 communicate with each other via the terminal device 112 and send and receive data from each other.

[0059] For example, the information processing device 111 can independently receive a user's operation and present various types of information such as visual information and auditory information to the user.

[0060] In addition, for example, the information processing device 111 executes a predetermined application (hereinafter referred to as an XR application) to control the terminal device 112, and controls the terminal device 112 to present an XR space to the user. For example, the information processing device 111 executes the XR application to control the output of various types of information such as visual information and auditory information in the terminal device 112, and constructs an XR space presented by the terminal device 112.

[0061] Figure 1 The following example is shown: In this example, the information processing device 111 includes a personal computer (PC) including an operation input unit including a mouse and a keyboard. For example, the information processing device 111 may include another information processing device such as a smart phone or a tablet terminal. For example, the information processing device 111 may include a plurality of information processing devices. For example, the information processing device 111 may include a system constructed by cloud computing via a network.

[0062] The terminal device 112 is a device that presents an XR space to a user.

[0063] Figure 1 The following example is shown: In this example, the terminal device 112 is a head-mounted display device that can be mounted on the user's head and includes a head-mounted display (HMD), which is a device that presents an XR space to the user. More specifically, an example is shown in which the terminal device 112 is a non-transmissive HMD that covers the user's field of view.

[0064] For example, the terminal device 112 includes a video see-through HMD, which has an imaging function capable of imaging the real space based on the user's viewpoint, and can present a synthetic image to the user, which is obtained by synthesizing the real image obtained by imaging the real space with an image of a virtual space such as computer graphics (CG) (hereinafter referred to as a virtual image).

[0065] For example, the terminal device 112 includes left and right imaging units corresponding to the left and right eyes of the user, respectively, and left and right display units corresponding to the left and right eyes of the user, respectively.

[0066] For example, the left imaging unit and the right imaging unit constitute a stereo camera device, and capture images in the direction of the user's line of sight (hereinafter referred to as field of view images) from multiple viewpoints corresponding to the user's left eye and right eye. That is, each of the left imaging unit and the right imaging unit captures an image of an object in the real space (hereinafter referred to as a real object or a real entity) viewed from the user's viewpoint.

[0067] The left display unit and the right display unit can display different images for the left eye and the right eye, respectively, and can present a three-dimensional virtual object by displaying images with parallax of the left eye and the right eye. For example, the left display unit and the right display unit display a left field of view image and a right field of view image captured by the left imaging unit and the right imaging unit, respectively.

[0068] Note that the terminal device 112 may include, for example, another XR terminal device such as AR glasses or a smartphone set with goggles. In addition, for example, a display device such as a spatial reproduction display may be used instead of the terminal device 112.

[0069] The controller device 113 is used to perform operations and inputs (hereinafter referred to as operation inputs) with respect to the XR space presented to the user by the terminal apparatus 112. For example, the user can perform various operations on a virtual object displayed by the terminal apparatus 112 by using the controller device 113.

[0070] For example, the controller device 113 detects at least one of the user's operation input or the user's behavior (e.g., gesture) through at least one of the operation members such as buttons or sensors. The controller device 113 sends a signal (hereinafter referred to as a controller signal) to the information processing device 111 via the terminal device 112, the signal including at least one of the operation input signal indicating the user's operation input or the behavior signal indicating the user's behavior.

[0071] Furthermore, for example, the controller device 113 a includes a tactile device that presents tactile stimulation such as vibration, and presents the tactile stimulation to the user under the control of the information processing apparatus 111 or the terminal apparatus 112 .

[0072] The controller device 113 includes, for example, one or more types of input devices among a controller, a ring-type input device, a pointing device, and a six-degree-of-freedom (6DoF) input apparatus.

[0073] The controller is an input device grasped by, for example, a user's hand. The controller may include, for example, an operation member operable by the user, such as a button. For example, the user may perform a selection operation, a determination operation, a scroll operation, etc. on a virtual object displayed on the terminal device 112 by pressing a button of the controller. In addition, the controller may include, for example, a touch sensor and a motion sensor.

[0074] Note that the controller is not limited to being grasped by the user's hand, and may, for example, be worn on a part of the user's body (eg, elbow, arm, knee, ankle, or thigh).

[0075] The ring-shaped device is a ring-shaped input device worn on the user's finger. The ring-shaped device may include, for example, operation members operable by the user, such as buttons. For example, the user can change the position and orientation of a virtual object (such as a three-dimensional model) in the XR space by operating the ring-shaped device with six degrees of freedom (6DoF).

[0076] The pointing device is an input device capable of indicating any position in the XR space. For example, the information processing device 111 identifies the 6DOF position and orientation of the pointing device via the terminal device 112 by a tracking method such as a bright spot tracking method, a magnetic tracking method, or an ultrasonic tracking method.

[0077] The 6DoF input device is, for example, an input device capable of operating with 6DoF.

[0078] For example, while viewing various objects (display objects) displayed on the information processing device 111 or the terminal device 112, the user can perform operation inputs using the controller device 113.

[0079] Note that the type and number of the controller devices 113 are not particularly limited. For example, as the controller device 113, an input device other than the above types can be used, or an input device obtained by combining multiple types of input devices can be used.

[0080] For example, the XR system 101 can be applied to various fields such as the manufacturing field and the medical field.

[0081] For example, the XR system 101 can perform product design support and assembly support in the manufacturing field. For example, in the product design stage, by using the XR system 101, the user can freely edit a three-dimensional object as a virtual object, or can preview the design results and designs before prototype production by comparing with the real world.

[0082] For example, the XR system 101 can support surgeries and education in the medical field. For example, by using the XR system 101, the user can display the internal state of the body on the surface of the patient to preview the surgical site or perform training.

[0083] Note that, for example, when the XR space is shared by multiple users, such as in the XR system 101, the terminal device 112 and the controller device 113 are provided for each user.

[0084] <Display example of the XR system 101>

[0085] Here, reference will be made to Figure 2 and Figure 3 to describe the display example of the display object in the XR system 101.

[0086] Figure 2 and Figure 3 An example of display of a display object in the XR system 101 in the case where a CAD model is created is shown.

[0087] For example, Figure 2 As shown in FIG. 1A , a two-dimensional CAD model is displayed by the information processing device 111 , and a user can edit the two-dimensional CAD model.

[0088] For example, Figure 2 As shown in FIG. 1B , the three-dimensional CAD model is displayed by the terminal device 112 , and the user can edit the three-dimensional CAD model.

[0089] For example, Figure 2 As shown in C, a two-dimensional object such as a design drawing or a specification is displayed by the terminal device 112, and the user can confirm the design drawing, specification, etc.

[0090] Figure 3 A display example of the XR space of the terminal device 112 is shown.

[0091] The display 151, keyboard 152, mouse 153, and table 154 of the information processing device 111 are displayed as a video perspective of a real image obtained by imaging the real space. On the other hand, the two-dimensional image of the terminal device 112 is superimposed on the display 151 as a virtual monitor. For example, a two-dimensional CAD model to be designed is displayed on the virtual monitor. For example, from the viewpoint of high accuracy of position detection and ease of position maintenance, it is preferred to operate the two-dimensional CAD model displayed by the virtual monitor through the keyboard 152 and the mouse 153.

[0092] Furthermore, in this example, a three-dimensional CAD model 155 to be designed is displayed in front of the display 151 by the terminal device 112 .

[0093] For example, the CAD model 155 is operated by a controller device 113a grasped by the user's dominant hand (the right hand in this example) and a controller device 113b (ring-type device) worn on the index finger of the user's non-dominant hand (the left hand in this example).

[0094] For example, the information processing device 111 recognizes the position, orientation, and behavior of the hand grasping the controller device 113a and the hand of the user wearing the controller device 113b by performing hand tracking based on an image captured by an imaging unit included in the terminal device 112. In addition, for example, the information processing device 111 receives controller signals from the controller device 113a and the controller device 113b via the terminal device 112, and recognizes the operation of the controller device 113a and the controller device 113b on the CAD model 155 based on the controller signals.

[0095] For example, a user may use controller device 113a or controller device 113b to grab, release, or move and rotate CAD model 155 in 6 DoF.

[0096] Note that, for example, in the case where the hand wearing the controller device 113a or the controller device 113b moves without picking up the CAD model 155, the CAD model 155 may not move, or the CAD model 155 may move along with the movement of the virtual point.

[0097] For example, the user can point to any point, line, surface, etc. of the CAD model 155 by using a light ray (virtual beam) etc. by using the controller device 113 a. For example, the user can perform line drawing using the controller device 113 a to draw a line on the CAD model 155 .

[0098] For example, the user may perform editing (eg, modeling, wiring, disassembly, etc.) on the CAD model 155 by using the controller device 113 a or the controller device 113 b .

[0099] <Configuration Example of Information Processing Device 111 and Terminal Device 112>

[0100] Figure 4 1 is a block diagram showing a configuration example of functions of the information processing device 111 and the terminal device 112 of the XR system 101 .

[0101] The information processing apparatus 111 includes an operation input unit 201 , a control unit 202 , a display unit 203 , a storage unit 204 , and a communication unit 205 .

[0102] The operation input unit 201 includes, for example, an input device such as a keyboard and a mouse. The operation input unit 201 receives an operation input from a user, and supplies an operation input signal indicating the content of the operation input from the user to the control unit 202.

[0103] The control unit 202 includes, for example, electronic circuits such as a CPU and a microprocessor. Furthermore, the control unit 202 may include a ROM storing programs to be used, calculation parameters, and the like, and a RAM temporarily storing parameters and the like that are changed appropriately.

[0104] For example, the control unit 202 functions as a calculation processing device and a control device, and controls the overall operation of the information processing device 111 or performs various types of processing according to various programs.

[0105] For example, the control unit 202 implements the information processing unit 211 by executing the information processing device 111 and the XR application, and the XR application enables the user to experience and edit virtual objects in the XR space. The information processing unit 211 includes an identification unit 221, an operation control unit 222, a space control unit 223, an audio control unit 224, a tactile presentation control unit 225, and a learning unit 226. That is, the identification unit 221, the operation control unit 222, the space control unit 223, the audio control unit 224, the tactile presentation control unit 225, and the learning unit 226 are implemented by the control unit 202 executing the XR application. In addition, the input and output of each unit of the information processing unit 211 (i.e., the identification unit 221, the operation control unit 222, the space control unit 223, the audio control unit 224, the tactile presentation control unit 225, and the learning unit 226) are all performed via the XR application.

[0106] The identification unit 221 identifies the state of the information processing device 111, the state of the terminal device 112, the state around the terminal device 112, the state of the controller device 113, the state of the user, the user operation, the state of the XR space, etc. based on at least one of the operation input signal from the operation input unit 201, the information from the control unit 202, the information from the display unit 203, the information from the communication unit 205, the sensing data sent from the terminal device 112, the controller signal sent from the controller device 113, the information from the operation control unit 222, or the information from the space control unit 223.

[0107] The state of the information processing device 111 to be identified includes, for example, the state of each unit of the information processing device 111, the state of each application such as an XR application, the communication state between the information processing device 111 and another device, or at least one of various types of setting information (such as setting values ​​of various setting items, etc.). The state of each unit of the information processing device 111 includes, for example, at least one of the operating state of each unit, the presence or absence of an abnormality, or the content of the abnormality. The state of each application includes, for example, at least one of the start and end of each application, the operating state, the presence or absence of an abnormality, or the content of the abnormality. The communication state between the information processing device 111 and another device includes, for example, the communication state with the terminal device 112, and the communication state with the controller device 113 via the terminal device 112.

[0108] The state of the terminal device 112 to be identified includes, for example, at least one of the position, orientation, or behavior of the terminal device 112, or various types of setting information (e.g., setting values ​​of various setting items, etc.). Note that, for example, in the case where the terminal device 112 is worn by a user, the position, orientation, and behavior of the terminal device 112 indirectly indicate the position, orientation, and behavior of the part of the user wearing the terminal device 112.

[0109] The state around the terminal device 112 to be recognized includes, for example, at least one of the type, position, orientation, behavior, size, shape, appearance, or feature amount of a real object around the terminal device 112 (user).

[0110] The state of the controller device 113 to be recognized includes, for example, at least one of the position, orientation, or behavior of the controller device 113 , or various types of setting information (eg, setting values ​​of various setting items, etc.).

[0111] The state of the user to be identified includes, for example, at least one of the user's position, orientation, overall behavior, behavior of a body part, or sight direction.

[0112] The user operation to be recognized includes, for example, at least one of an operation input of the operation input unit 201 , an operation input of the controller device 113 , an operation input of a user's gesture, or an operation input of a virtual tool in an XR space.

[0113] The state of the XR space to be recognized includes, for example, at least one of the type, position, orientation, behavior, size, shape, appearance, or feature amount of the virtual object in the XR space.

[0114] The recognition unit 221 supplies information on the recognition result to each unit of the information processing device 111 .

[0115] In addition, the recognition unit 221 sends information about the recognition result to the terminal device 112 via the communication unit 205, or sends the information to the controller device 113 via the communication unit 205 and the terminal device 112. For example, in the case where a change in state or an abnormality of the terminal device 112 or the input device 113 is detected, the recognition unit 221 sends information indicating the detected content to the terminal device 112 via the communication unit 205, or sends the information to the controller device 113 via the communication unit 205 and the terminal device 112. For example, in the case where a change in state (e.g., start, stop, etc.) or an abnormality of an application such as an XR application is detected, the recognition unit 221 sends information indicating the detected content to the terminal device 112 via the communication unit 205, or sends the information to the controller device 113 via the communication unit 205 and the terminal device 112.

[0116] Note that, for example, any method such as image recognition or object recognition may be used for recognition processing of various recognition targets by the recognition unit 221 .

[0117] In addition, for example, in the case where the XR space is shared by multiple users, the recognition unit 221 performs recognition processing for each user, for example. For example, the recognition unit 221 recognizes the state of the terminal device 112 of each user, the state around the terminal device 112 of each user, the state of the controller device 113 of each user, the state of each user, and the user operation of each user. The result of the recognition processing for each user can be shared between users, for example, by sending it to the terminal device 112 or the controller device 113 of each user.

[0118] The operation control unit 222 controls the operation process of the controller device 113 based on at least one of the recognition result of the recognition unit 221 or the controller signal transmitted from the controller device 113 .

[0119] For example, the operation control unit 222 controls the operation process of the controller device 113 based on at least one of the position or orientation of the controller device 113 or the controller signal. For example, the operation control unit 222 controls the enabling or disabling of each operating member included in the controller device 113, the function assigned to each operating member, the operation method of the function assigned to each operating member, etc., based on the installation method, gripping method, use method, etc. of the controller device 113.

[0120] The operation control unit 222 provides information related to control of operation processing by the controller device 113 to each unit of the information processing apparatus 111 .

[0121] The space control unit 223 controls the presentation of the two-dimensional space or the three-dimensional space by the display unit 203 and the presentation of the XR space by the terminal device 112 based on at least a part of the recognition result of the recognition unit 221 .

[0122] For example, the space control unit 223 generates a display object to be displayed in a two-dimensional space or a three-dimensional space based on at least a part of the recognition result of the recognition unit 221, and performs various calculations required for the construction, display, etc. of the two-dimensional space or the three-dimensional space (e.g., the behavior of the display object). The space control unit 223 generates display control information for controlling the display of the two-dimensional space or the three-dimensional space based on the calculation result, and provides the display control information to the display unit 203, thereby controlling the display of the two-dimensional space or the three-dimensional space by the display unit 203. Note that the display control information may include, for example, information on using the two-dimensional space or the three-dimensional space (e.g., operation menus, guides, messages, etc.), and information notifying the status of the information processing device 111 (e.g., setting information, remaining battery power, error display, etc.).

[0123] For example, the space control unit 223 generates a virtual object to be displayed in the XR space based on at least a part of the recognition result of the recognition unit 221, and performs various calculations required for the construction, display, etc. of the XR space (such as the behavior of the virtual object). The recognition result of the recognition unit 221 includes, for example, the operation content for the controller device 113a recognized by the recognition unit 221 based on the controller signal including the operation input signal from the controller device 113a, etc. The space control unit 223 generates display control information for controlling the display of the XR space based on the calculation result, and sends the display control information to the terminal device 112 via the communication unit 205, thereby controlling the display of the XR space by the terminal device 112. Note that the display control information may include, for example, information on using the XR space (e.g., operation menus, guides, messages, etc.), and information notifying the status of the XR system 101 (e.g., setting information, remaining battery power, error display, etc.).

[0124] The space control unit 223 supplies information about the two-dimensional space, the three-dimensional space, and the XR space to each unit of the information processing device 111 .

[0125] The audio control unit 224 controls the audio output of the terminal device 112 based on at least one of the recognition result of the recognition unit 221 or the information from the space control unit 223. For example, the space control unit 223 generates audio control information for outputting audio in the terminal device 112. The audio control information includes, for example, information related to at least one of the type, content, frequency, amplitude, or waveform of the audio to be output. The audio control unit 224 controls the audio output of the terminal device 112 by sending the audio control information to the terminal device 112 via the communication unit 205.

[0126] The tactile presentation control unit 225 controls the presentation of tactile stimulation to the user based on at least one of the recognition result of the recognition unit 221 or the information from the space control unit 223. For example, the tactile presentation control unit 225 generates tactile control information for presenting tactile stimulation in the controller device 113. The tactile control information includes, for example, information related to at least one of the type, mode, intensity, or length of the tactile sensation to be presented. The tactile presentation control unit 225 sends the tactile control information to the controller device 113 via the communication unit 205 and the terminal device 112, thereby controlling the presentation of tactile stimulation by the controller device 113.

[0127] The learning unit 226 performs learning processing related to the processing of the XR system 101 based on at least one of the recognition result of the recognition unit 221 or the learning data given from the outside. For example, the learning unit 226 learns the user's preferences, action patterns, etc., and adjusts various types of processing and parameters of the XR system 101 based on the learning results to appropriately correspond to the user's preferences, action patterns, etc. For example, the learning unit 226 learns the difference between the XR space and the real space, and adjusts the design data based on the learning results to make the characteristics, behaviors, etc. of the virtual objects in the XR space closer to the characteristics, behaviors, etc. of the real objects.

[0128] For example, the learning unit 226 causes the storage unit 204 to store information indicating the learning result (eg, a learning model, etc.).

[0129] Note that the control unit 202 can execute not only the XR application but also other applications.

[0130] The storage unit 204 includes, for example, a read only memory (ROM) that stores programs to be used for processing of the control unit 202 , calculation parameters, and the like, and a random access memory (RAM) that temporarily stores parameters and the like that are changed appropriately.

[0131] The communication unit 205 communicates with an external device to send and receive data. For example, the communication unit 205 communicates with the terminal device 112 to send and receive data. For example, the communication unit 205 sends display control information, audio control information, and tactile control information to the terminal device 112. For example, the communication unit 205 receives sensing data and controller signals from the terminal device 112.

[0132] The communication method of the communication unit 205 may be wired or wireless, for example, using a wired LAN, a wireless LAN, Wi-Fi, Bluetooth, etc. Furthermore, the communication unit 205 may support two or more types of communication methods.

[0133] The terminal device 112 includes an operation input unit 251 , a sensing unit 252 , a control unit 253 , a display unit 254 , an audio output unit 255 , and a learning unit 226 .

[0134] The operation input unit 251 includes, for example, an operation input device such as a button. The operation input unit 251 receives the user's operation input and provides an operation input signal indicating the content of the user's operation input to the control unit 253. For example, the operation input unit 251 receives user operation inputs such as turning on or off the power of the terminal device 112 and adjusting the brightness of the display unit 254.

[0135] The sensing unit 252 includes various sensors for sensing the terminal device 112, the periphery of the terminal device 112, and the state of the user. For example, the sensing unit 252 includes a camera or a depth sensor for imaging the periphery of the terminal device 112. For example, the sensing unit 252 includes a camera or a depth sensor for imaging both eyes of the user. For example, the sensing unit 252 includes an inertial measurement unit (IMU) for detecting the acceleration, angular velocity, etc. of the terminal device 112. For example, the sensing unit 252 includes a global navigation satellite system (GNSS) receiver for detecting the current position of the terminal device 112 (user). The sensing unit 252 provides sensing data indicating the detection results of at least one or more sensors to the control unit 253.

[0136] The control unit 253 includes, for example, electronic circuits such as a CPU and a microprocessor. Furthermore, the control unit 253 may include a ROM storing programs to be used, calculation parameters, and the like, and a RAM temporarily storing parameters and the like that are changed appropriately.

[0137] For example, the control unit 253 functions as a calculation processing device and a control device, and controls the overall operation of the terminal device 112, and performs various types of processing according to various programs based on the operation input signal from the operation input unit 251, the sensing data from the sensing unit 252, the display control information and the audio control information from the information processing device 111, the controller signal from the controller device 113, etc. For example, the control unit 253 controls the display unit 254 to display the XR space, etc. based on the display control information. For example, the control unit 253 controls the audio output unit 255 to output audio based on the audio control information.

[0138] The display unit 254 includes various display devices. For example, in the case where the terminal device 112 is an HMD, the display unit 254 includes a display fixed to the left eye and the right eye of the user, and displays the left eye image and the right eye image. The display includes, for example, a display panel such as a liquid crystal display or an organic electroluminescent (EL) display, or a laser scanning display such as a retinal direct projection display. In addition, the display unit 254 may include, for example, an imaging optical system that enlarges and projects a display screen to form an enlarged virtual image with a predetermined viewing angle on the user's pupil. For example, the display unit 254 displays an XR space including a virtual object under the control of the control unit 253.

[0139] The audio output unit 255 includes, for example, an audio output device such as a headphone, an earphone, or a speaker. The audio output unit 255 outputs audio under the control of the control unit 253 .

[0140] The communication unit 256 communicates with an external device to send and receive data. For example, the communication unit 256 communicates with the terminal device 112 and the controller device 113 to send and receive data. For example, the communication unit 256 sends the sensing data and the controller signal to the information processing device 111. For example, the communication unit 256 receives the display control information, the audio control information, and the tactile control information from the information processing device 111. For example, the communication unit 256 sends the tactile control information to the controller device 113. For example, the communication unit 256 receives the controller signal from the controller device 113.

[0141] The communication method of the communication unit 256 may be wired or wireless, for example, using a wired LAN, a wireless LAN, Wi-Fi, Bluetooth, etc. In addition, the communication unit 256 may support two or more types of communication methods. In addition, the communication unit 256 may perform communication between the information processing apparatus 111 and the controller device 113 by using different communication methods.

[0142] The following is an example of processing by the information processing device 111 using the XR application.

[0143] For example, the communication unit 205 receives input information indicating at least one of the state of the terminal device 112, the state around the terminal device 112, the state of the user, the behavior of the user, or the operation input to the input device 113 from the terminal device 112, or from the controller device 113 via the terminal device 112, and provides the input information to the control unit 221. The control unit 221 executes the XR application based on the input information, generates output information for controlling the display of the virtual object (including CAD information related to CAD) in the XR space, and outputs the output information to the terminal device 112. The communication unit 205 sends the output information to the terminal device 112.

[0144] Furthermore, for example, the control unit 221 executes the XR application and outputs output information indicating a change or abnormality in the state of the XR application to the terminal device 112 or the controller device 113. The communication unit 205 transmits the output information to the terminal device 122 or to the controller device 113 via the terminal device 112.

[0145] On the other hand, for example, the terminal device 112 notifies the change or abnormality of the state of the XR application through an image, message, audio, vibration, etc. based on the output information. For example, the controller device 113 notifies the change or abnormality of the state of the XR application through vibration, etc. based on the output information.

[0146] Note that in the following, in the case where each unit of the information processing device 111 communicates with the outside via the communication unit 205, the description of the communication unit 205 may be omitted. For example, in the case where the space control unit 223 of the information processing device 111 communicates with the terminal device 112 via the communication unit 205, it can be simply described as the space control unit 223 of the information processing device 111 communicating with the terminal device 112.

[0147] Hereinafter, in the case where each unit of the terminal device 112 communicates with the outside via the communication unit 256, the description of the communication unit 256 may be omitted. For example, in the case where the control unit 253 of the terminal device 112 communicates with the information processing device 111 via the communication unit 256, it may be simply described as the control unit 253 of the terminal device 112 communicating with the information processing device 111.

[0148] For example, in the XR system 101, the space control unit 223 of the information processing device 111 generates display control information and transmits the display control information to the terminal device 112 via the communication unit 205, and the control unit 253 of the terminal device 112 receives the display control information via the communication unit 256 and controls the display unit 254 based on the display control information. Hereinafter, the description of a series of processes may be simplified, and for example, it may be described as follows: the space control unit 223 of the information processing device 111 controls the display unit 254 of the terminal device 112.

[0149] For example, in the XR system 101, the audio control unit 224 of the information processing device 111 generates audio control information and transmits the audio control information to the terminal device 112 via the communication unit 205, and the control unit 253 of the terminal device 112 receives the audio control information via the communication unit 256 and controls the audio output unit 255 based on the audio control information. In the following, the description of a series of processes is simplified, and for example, it can be described as follows: the audio control unit 224 of the information processing device 111 controls the audio output unit 255 of the terminal device 112.

[0150] For example, in the XR system 101, the tactile presentation control unit 225 of the information processing device 111 generates tactile control information, and transmits the tactile control information to the controller device 113 via the communication unit 205 and the terminal device 112, and the controller device 113 presents tactile stimulation based on the tactile control information. In the following, the description of a series of processes is simplified, and for example, it can be described as follows: the tactile presentation control unit 225 of the information processing device 111 controls the controller device 113 via the terminal device 112.

[0151] <Configuration Example of Controller Device 113a>

[0152] Next, we will refer to Figures 5 to 14 describe Figure 3 An example of the configuration of the controller device 113a.

[0153] Figure 5 A configuration example of the appearance of the controller device 113 a is shown. Figure 5 A is a left side view of the controller device 113a. Figure 5 B is a front view of the controller device 113a. Figure 5 C is a bottom view of the controller device 113a. Figure 5 D is a stereoscopic view of the controller device 113a as viewed from the right side of the front oblique direction.

[0154] Note that in the following, Figure 5 The upward direction in A is defined as the upward direction of the controller device 113a, Figure 5 The downward direction in A is defined as the downward direction of the controller device 113a. Figure 5 The right direction in A is defined as the front direction of the controller device 113a, Figure 5 The left direction in A is defined as the rear direction of the controller device 113a.

[0155] When the controller device 113a is viewed from any of the front, rear, left, right, top, and bottom directions, it has a symmetrical shape. In addition, in the controller device 113a, the shape of the front surface viewed from the front is similar to the shape of the rear surface viewed from the rear, and the shape of the right side surface viewed from the right direction is similar to the shape of the left side surface viewed from the left direction.

[0156] The controller device 113a is roughly divided into three parts: a ring portion 301, an operating portion 302a, and a grip portion 302b.

[0157] like Figure 5 As shown in A, the ring portion 301 extends upward from near the center of gravity of the left side surface 314b. When viewed from the direction of the side surface of the ring portion 301 (for example, the left side surface 314b of the controller device 113a), the operating portion 302a and the holding portion 302b have a shape symmetrical about the ring portion 301. The operating portion 302a extends forward and obliquely downward from near the center of gravity of the left side surface 314b (near the lower end of the ring portion 301). The holding portion 302b extends backward and obliquely downward from near the center of gravity of the left side surface 314b (near the lower end of the ring portion 301), symmetrical with the operating portion 302a. When the end of the ring portion 301, the end of the operating portion 302a, and the end of the holding portion 302b are connected, an isosceles triangle with the end of the ring portion 301 as a vertex is formed. The angle between the ring portion 301 and the operating portion 302a, the angle between the ring portion 301 and the grip portion 302b, and the angle between the operating portion 302a and the grip portion 302b are each approximately 120 degrees, and the isosceles triangle is a substantially equilateral triangle.

[0158] The end of the side surface of the ring portion 301 extends linearly, while the root extends in a curved shape. The end of the side surface of the operating portion 302a extends linearly, while the root extends in a curved shape. The end of the side surface of the grip portion 302b extends linearly, while the root extends in a curved shape. The boundary between the ring portion 301 and the operating portion 302a, the boundary between the ring portion 301 and the grip portion 302b, and the boundary between the operating portion 302a and the grip portion 302b are all curved.

[0159] like Figure 5 As shown in FIG. 1B , a hole 301A penetrating in the front-rear direction is formed in the ring portion 301. The outer periphery of the ring portion 301 expands gently toward the end, and the front end is curved. Similarly, the hole 301A expands gently toward the end, and the front end and the tail end are curved.

[0160] like Figure 5As shown in FIG. 1B , the operation portion 302a tapers toward the end, and the front end is curved. The upper surface 312a of the operation portion 302a is inclined forward and downward. On the upper surface 312a of the operation portion 302a, a shallow groove is formed that is curved in the lateral direction and extends in the front-to-back direction. The end of the upper surface 312a of the operation portion 302a is slightly recessed relative to the end of the operation portion 302a. Therefore, the upper surface 312a of the operation portion 302a has the following shape: wherein, in the case where the user's finger is inserted into the hole 301A of the ring portion 301 from the rear in the forward direction, the inserted finger can be easily placed.

[0161] The grip portion 302b has a shape similar to that of the operation portion 302a, and an upper surface 312b (not shown) having a shape similar to that of the upper surface 312a is formed.

[0162] like Figure 5 As shown in FIG. 3C, the bottom surface 313 curved in the front-rear direction is formed by the lower surface of the operation portion 302a and the lower surface of the grip portion 302b. The bottom surface 313 is formed with a shallow groove curved in the lateral direction and extending in the front-rear direction.

[0163] For example, a rubber-like material such as silicon or an elastomer is used for the inner peripheral surface 311, the upper surface 312a, the upper surface 312b, and the bottom surface 313 of the controller device 113a. For the other parts of the controller device 113a, for example, an IR transmissive resin is used.

[0164] Figures 6 to 8 An example of a method of grasping the controller device 113a is shown.

[0165] For example, Figure 6 As shown in A, the index finger of the right hand is inserted into the ring portion 301 from the rear in the forward direction, the tip of the index finger is placed near the tip of the upper surface 312a of the operating portion 302a, and the operating portion 302a can be operated by the index finger. Since the size of the hole 301A of the ring portion 301 has a margin relative to the thickness of the index finger, the index finger is easily inserted. The tip of the thumb of the right hand is gently placed near the tip of the side surface of the operating portion 302a, and the grip portion 302b is gently grasped and held by the palm of the right hand.

[0166] For example, Figure 6 As indicated by the arrow in A, when the index finger is pressed downward near the end of the operating portion 302a, as shown in FIG. Figure 6 As shown in FIG. 1B , the end of the grip portion 302b abuts against the palm, thereby preventing the controller device 113a from rotating in the pressing direction. Therefore, the vicinity of the end of the operating portion 302a is prevented from shaking in space, and the user can reliably press the vicinity of the end of the operating portion 302a in a state where the end direction of the operating portion 302a is stable.

[0167] Furthermore, as described above, in the controller device 113a, the shape viewed from the front is similar to the shape viewed from the rear, and the shape viewed from the right direction is similar to the shape viewed from the left direction. Therefore, the user can hold the controller device 113a without worrying about the front and rear. That is, Figure 7 As shown in FIG. 1A , the user can hold the controller device 113 a so that the operation portion 302 a faces the fingertip direction and the right side surface 314 a faces the thumb direction. Figure 7 As shown in FIG. 1B , the user may hold the controller device 113 b such that the grip portion 302 b faces the fingertips and the left side surface 314 b faces the thumb.

[0168] Note that in the following, Figure 7 As shown in FIG. 1A , grasping the controller device 113a so that the operating portion 302a faces the fingertip direction is referred to as grasping the controller device 113a in the forward direction. Figure 7 As shown in FIG. 8B , grasping the controller device 113 a so that the grip portion 302 b faces the fingertip direction is referred to as grasping the controller device 113 a in the rearward direction.

[0169] When the controller device 113a is gripped in the backward direction, the roles of the operating portion 302a and the grip portion 302b are exchanged. That is, the grip portion 302b serves as an operating portion that can be operated by the index finger of the right hand, while the operating portion 302a serves as a grip portion that is grasped by the palm of the right hand.

[0170] In addition, if Figure 8 As shown in , even if the user releases his hand from the controller device 113a, the ring portion 301 is grasped by the index finger, and the controller device 113a does not fall. Therefore, without setting a belt or the like, the user is prevented from accidentally dropping the controller device 113a.

[0171] Fig. 9 An arrangement example of operating members of the controller device 113 a is shown. Fig. 9 A is a stereoscopic view of the controller device 113a viewed from the upper right side. Fig. 9 B is a stereoscopic view of the controller device 113a viewed from the upper left. Fig. 9 C is a stereoscopic view of the controller device 113a viewed from the rear and downward direction.

[0172] The operation members are arranged symmetrically about the ring portion 301 in the front-rear direction and the left-right direction of the controller device 113 a .

[0173] For example, the operating member 331 is arranged at the lower end of the inner peripheral surface 311 (the hole 301A) of the ring portion 301. For example, the user bends the index finger and operates the operating member 331 with the fingertip of the index finger.

[0174] An operating member 332a is arranged near the end of the upper surface 312a of the operating portion 302a. An operating member 332b is arranged near the end of the upper surface 312b of the holding portion 302b. For example, the user operates the operating member 332a or the operating member 332b with the tip of the index finger.

[0175] The operating member 333a and the operating member 333b are respectively arranged near the front and rear ends of the bottom surface 313. For example, the user operates the operating member 333a or the operating member 333b with the fingertip of the ring finger or the little finger.

[0176] The operation member 334 is arranged in the front-rear direction at the center of the bottom surface 313. For example, the user operates the operation member 334 with the tip of the thumb, ring finger, or little finger.

[0177] For example, any type of operating member such as a button, a touch pad, or a joystick may be used as the operating member 331, the operating member 332a, the operating member 332b, the operating member 333a, the operating member 333b, and the operating member 334. However, the same type of operating member is used for the operating member 332a and the operating member 332b arranged at positions symmetrical about the ring portion 301. Similarly, the same type of operating member is used for the operating member 333a and the operating member 333b arranged at positions symmetrical about the ring portion 301.

[0178] For example, any function may be assigned to the operating member 331, the operating member 332a, the operating member 332b, the operating member 333a, the operating member 333b, and the operating member 334. However, similar functions are assigned to the operating member 332a and the operating member 332b arranged at positions symmetrical about the ring portion 301. Similarly, similar functions are assigned to the operating member 333a and the operating member 333b arranged at positions symmetrical about the ring portion 301.

[0179] Specifically, for example, the function of calling up the main menu screen is assigned to the operating member 331. For example, the function of selecting a virtual object is assigned to the operating member 332a and the operating member 332b. For example, functions other than the selection function of the operating member 332a and the operating member 332b are assigned to the operating member 333a and the operating member 333b. For example, the function of calling up the submenu screen is assigned to the operating member 334.

[0180] Note that, for example, different functions may be assigned to the operating member 332a and the operating member 332b, and the functions of the two may be switched depending on the direction in which the controller device 113a is held. Similarly, for example, different functions may be assigned to the operating member 333a and the operating member 333b, and the functions of the two may be switched depending on the direction in which the controller device 113a is held.

[0181] As described above, whether the user grasps the controller device 113a in the forward direction or in the backward direction, the user can perform similar operations.

[0182] Note that although it is assumed that the index finger is inserted into the ring portion 301 , for example, the middle finger or the ring finger may be inserted for use.

[0183] Note that, hereinafter, when there is no need to distinguish the operating member 332a and the operating member 332b from each other, they are simply referred to as the operating member 332. hereinafter, when there is no need to distinguish the operating member 333a and the operating member 333b from each other, they are simply referred to as the operating member 333.

[0184] <Example of layout of mark>

[0185] For example, a marker such as an IR light emitting element may be provided in the controller device 113a. Then, the recognition unit 221 of the information processing device 111 may detect the marker of the controller device 113a based on an image sensed by the sensing unit 252 of the terminal device 112, and recognize the relative position and orientation between the terminal device 112 and the controller device 113 based on the position of the detected marker.

[0186] Fig.10 An arrangement example of the marks 351 of the controller device 113a is shown. Each mark 351 is indicated by a black circle.

[0187] For example, Fig.10 As shown in FIG. 1A , the marks 351 are arranged in the vertical direction on the right side surface 314a and the left side surface 314b to surround the outer circumference of the ring portion 301. For example, the marks 351 are arranged near the ends of both side surfaces of the operation portion 302a and near the ends of both side surfaces of the grip portion 302b. For example, the marks 351 are arranged near the front end and the rear end of the bottom surface 313.

[0188] Therefore, if Fig.11 As shown in A to D, in any orientation of the controller device 113a, at least a portion of the mark 351 becomes visible without being covered by the user's hand.

[0189] On the other hand, for example, Fig.12As shown in FIG. 1 , the terminal device 112 includes a plurality of camera devices 401. Each camera device 401 constitutes a sensing unit 252 ( Figure 4 ). Each camera 401 captures an image of the controller device 113a. The terminal device 112 transmits sensing data including captured image data obtained by imaging to the information processing device 111.

[0190] On the other hand, the control unit 202 of the information processing device 111 receives the sensing data. The identification unit 221 of the control unit 202 identifies the position and orientation of the controller device 113a relative to the terminal device 112 based on the light emission pattern of the marker 351 of the controller device 113a.

[0191] Note that, for example, Fig.10 As shown in FIG. 1B , the marks 351 may be arranged in two rows in the transverse direction to surround the outer circumference of the ring portion 301. In addition, for example, Fig.10 As shown in FIG. 3C , the marks 351 may be arranged in three rows along the lateral direction to surround the outer circumference of the ring portion 301 .

[0192] In this way, by arranging the marks 351 on the outer circumference of the ring portion 301, the size of the controller device 113a can be reduced.

[0193] <Example of Internal Structure of Controller Device 113a>

[0194] Next, we will refer to Fig.13 An example of the internal structure of the controller device 113a will be described.

[0195] The controller device 113 a includes a haptic device 371 , a haptic device 372 a , a haptic device 372 b , a substrate 373 , and a battery 374 .

[0196] Each of the haptic devices 371 , 372 a , and 372 b includes, for example, a device that presents (transmits) tactile stimulation, such as vibration of a linear resonant actuator (LRA), an eccentric rotating mass (ERM), a piezoelectric element, or the like.

[0197] The tactile device 371 is arranged near the lower end of the inner peripheral surface 311 of the ring portion 301 (the operation member 331 ( Fig. 9 ) and presents tactile stimulation near the lower end of the inner circumferential surface 311.

[0198] The haptic device 372a is arranged near the end (operating member 332a ( Fig. 9 ) and transmits tactile stimulation to the vicinity of the end of the operating portion 302a.

[0199] The haptic device 372b is disposed near the end in the holding part 302b (near the operation member 332b( Fig. 9 )) and transmits a haptic stimulus near the end of the holding part 302b.

[0200] The substrate 373 is a substrate for controlling the controller device 113a and is substantially disposed at the center in the controller device 113a and below the haptic device 371.

[0201] The battery 374 is disposed below the substrate 373 in the controller device 113a and supplies power to each unit of the controller device 113a.

[0202] For example, as Fig.14 shown, when the user grasps the controller device 113a with the right hand in the forward direction, the haptic device 371 presents a haptic stimulus near the base joint of the thumb. The haptic device 372a presents haptic stimuli near the tip of the thumb and near the tip of the index finger. The haptic device 372b presents haptic stimuli near the root of the thumb and near the palm.

[0203] Note that the haptic device 371, the haptic device 372a, and the haptic device 372b are disposed at symmetric positions with respect to the ring part 301 in the front-rear direction of the controller device 113a. Therefore, even if the user holds the controller device 113a in the forward direction or the backward direction, similar haptic stimuli are presented to the user's hand.

[0204] <Processing of the XR system 101>

[0205] Next, the processing of the XR system 101 will be described with reference to Figures 15 to 23 .

[0206] <Operation member control processing>

[0207] First, the operation member control processing executed by the XR system 101 will be described with reference to the flowchart of Fig.15 .

[0208] For example, this processing is executed when the user holds or switches the controller device 113a.

[0209] In step S1, the information processing device 111 performs hand recognition by hand tracking.

[0210] For example, the control unit 253 of the terminal device 112 sends sensing data including captured image data indicating images captured by each imaging device 401 to the information processing device 111.

[0211] On the other hand, the control unit 202 of the information processing device 111 receives the sensing data. The recognition unit 221 of the control unit 202 performs hand recognition by hand tracking based on the captured image data included in the sensing data. Therefore, for example, the recognition unit 221 tracks the hand of the user grasping the controller device 113a based on the marker 351 set in the controller device 113a.

[0212] In step S2, the recognition unit 221 determines whether the hand holding the controller device 113a has been recognized based on the processing result in step S1. In the case where it is determined that the hand holding the controller device 113a has not been recognized, the process returns to step S1.

[0213] Thereafter, the processes of step S1 and step S2 are repeatedly performed until it is determined in step S2 that the hand grasping the controller device 113a is recognized.

[0214] On the other hand, in the event that determination is made in step S2 that the hand gripping the controller device 113a is recognized, the process proceeds to step S3.

[0215] In step S3, the recognition unit 221 recognizes the light emission pattern of the controller device 113a based on the captured image data. That is, the recognition unit 221 recognizes the light emission pattern of the mark 351 in the controller device 113a that is not blocked by the user's hand.

[0216] In step S4, the recognition unit 221 determines whether the gripping direction of the controller device 113a has been recognized. Specifically, the recognition unit 221 attempts to recognize the gripping direction of the controller device 113a based on the recognition result of the hand of the user gripping the controller device 113a and the recognition result of the light emission pattern of the controller device 113a. Then, in the case where it is determined that the gripping direction of the controller device 113a has not been recognized, the process returns to step S3.

[0217] Thereafter, the processes of step S3 and step S4 are repeatedly performed until it is determined in step S4 that the gripping direction of the controller device 113a is recognized.

[0218] On the other hand, in a case where it is determined in step S4 that the gripping direction of the controller device 113 a is recognized, the process proceeds to step S5 .

[0219] In step S5, the operation control unit 222 disables the operation member on the palm side. Fig.16 As shown in , in the case where the controller device 113a is gripped in the forward direction, the operation member 332b on the palm side is disabled. Thereafter, for example, the recognition unit 221 and the operation control unit 222 ignore the operation input signal of the operation member 332b.

[0220] On the other hand, for example, in the case where the controller device 113a is gripped in the rearward direction, the palm-side operation member 332a is disabled.

[0221] Thereafter, the operating member control process ends.

[0222] Therefore, the operation member 332 is prevented from being erroneously operated by the palm of the user.

[0223] As described above, the hand grasping the controller device 113a and the grasping direction are recognized, and the operability of the controller device 113a does not change regardless of the grasping direction of the controller device 113a.

[0224] So, for example, Fig.17 As shown in A and B of FIG. 1 , even if no special setting is made on the terminal apparatus 112 side, the user can grasp and use the controller device 113 a with the dominant hand regardless of which hand the user has.

[0225] Note that, for example, Fig.18 As shown in A and B of , the user can wear another controller device 113b (e.g., a ring-type device) on the non-dominant hand side for use.

[0226] In addition, for example, Fig.19 As shown in , the user can wear the controller device 113a on both hands for use.

[0227] <Haptic feedback control processing>

[0228] Next, we will refer to Fig. 20 1 is used to describe the haptic feedback control process performed by XR system 101.

[0229] This process starts, for example, when the power of the information processing device 111 is turned on, and ends when the power is turned off.

[0230] In step S51 , the information processing device 111 recognizes the state of the terminal device 112 , the surrounding state, and the like.

[0231] Specifically, the sensing unit 252 of the terminal device 112 senses the state of the terminal device 112 and the state around the terminal device 112, and provides sensing data indicating the sensing result to the control unit 253. The control unit 253 transmits the sensing data to the information processing device 111.

[0232] On the other hand, the control unit 202 of the information processing device 111 receives the sensing data.

[0233] The controller device 113 a transmits a controller signal including an operation input signal indicating the operation content of each operation member to the information processing apparatus 111 via the terminal device 112 .

[0234] On the other hand, the control unit 202 of the information processing device 111 receives the controller signal.

[0235] The recognition unit 221 of the control unit 202 recognizes the state of the terminal device 112, the state around the terminal device 112, the state of the controller device 113, the state of the user, the user operation, etc. based on the sensing data and the controller signal. For example, the recognition unit 221 recognizes the position and orientation of the terminal device 112. For example, the recognition unit 221 recognizes the direction of the user's line of sight. For example, the recognition unit 221 recognizes the position and orientation of the controller device 113a relative to the terminal device 112. For example, the recognition unit 221 recognizes the operation content of the controller device 113a.

[0236] In step S52, the space control unit 223 of the information processing device 111 controls the XR space. Specifically, the space control unit 223 generates a virtual object to be displayed in the XR space based on at least a part of the recognition result of the recognition unit 221, and performs various calculations required for the construction, display, etc. of the XR space (such as the behavior of the virtual object). The space control unit 223 generates display control information for controlling the display of the XR space based on the calculation result, and sends the display control information to the terminal device 112 via the communication unit 205, thereby controlling the display of the XR space by the terminal device 112.

[0237] The recognition unit 221 recognizes the type, position, orientation, and the like of virtual objects around the terminal device 112 (user) based on information from the space control unit 223 and the like.

[0238] In step S53, the tactile presentation control unit 225 determines whether it is time to present tactile feedback based on at least one of the recognition result of the recognition unit 221 or the information from the space control unit 223. If it is determined not to be time to present tactile feedback, the process returns to step S51.

[0239] Thereafter, the processes of step S51 to step S53 are repeatedly performed until it is determined in step S53 that it is time to present tactile feedback.

[0240] On the other hand, in a case where it is determined in step S53 that it is the timing to present haptic feedback, the process proceeds to step S54 .

[0241] In step S54 , the information processing device 111 controls the presentation of tactile feedback. Specifically, the tactile presentation control unit 225 generates tactile control information for causing the controller device 113a to present tactile stimulation. The tactile presentation control unit 225 sends the tactile control signal to the controller device 113a via the terminal device 112 .

[0242] On the other hand, the controller device 113a receives the haptic control information. Each haptic device of the controller device 113a presents haptic stimulation based on the haptic control information.

[0243] Thereafter, the process returns to step S51 , and the processes in step S51 and subsequent steps are performed.

[0244] As described above, tactile stimulation is appropriately presented to the user by the controller device 113a.

[0245] Here, we will refer to Figure 21 to Figure 23 An example of a method of presenting tactile feedback of the controller device 113a is described below.

[0246] For example, an operating member 332a ( Fig. 9 ) includes a touch panel, and the tip of the index finger slides on the operating member 332a in the front-rear direction (such as Fig.21 In the case of FIG. 1A ), by arranging the haptic device 372a ( Fig.13 ) presents tactile stimulation to the tip of the index finger.

[0247] For example, Fig.21 As shown in FIG. 1B, when the user touches the button 431 in the XR space using the end of the operation unit 302a of the controller device 113a, the haptic device 372a ( Fig.13 ) presents tactile stimulation to the tip of the index finger.

[0248] For example, in a case where the controller device 113a or a hand grasping the controller device 113a collides with a virtual object in the XR space, an impact due to the collision is expressed using each haptic device of the controller device 113a.

[0249] For example, Fig. 22 A shows an example of a case where the tip of the operating section 302a of the controller device 113a collides with the virtual object 441 in the XR space from above. In this case, for example, the haptic device 372a ( Fig.13 ) presents an upward vibration near the end of the operating portion 302a, and is provided by the haptic device 372b ( Fig.13) presents a downward vibration near the end of the grip portion 302b. Therefore, the user can feel a rotational force (torque) in an upward direction relative to the controller device 113a.

[0250] For example, Fig. 22 B shows an example of a case where the tip of the operating section 302a of the controller device 113a collides with the virtual object 441 in the XR space from below. In this case, for example, the haptic device 372a ( Fig.13 ) presents a downward vibration near the end of the operating portion 302a, and is provided by the haptic device 372b ( Fig.13 ) presents an upward vibration near the end of the grip portion 302b. Therefore, the user can feel a rotational force (torque) in a downward direction relative to the controller device 113a.

[0251] For example, Fig.23 113a collides with the virtual object 441 in the XR space from the front. In this case, for example, by making the haptic device 371 ( Fig.13 ) vibrates to vibrate the entire controller device 113a. Therefore, the user can feel the reaction force from the virtual object 441 to the controller device 113a.

[0252] As described above, the operability of the controller device 113a can be improved. Therefore, the operability for the XR space is improved.

[0253] <<2. Modifications>>

[0254] Hereinafter, modified examples of the embodiments of the present technology described above will be described.

[0255] <Regarding Modification Example of Controller Device 113a>

[0256] In the above description, an example has been described in which the controller device 113a can be grasped in the forward direction or the backward direction. However, for example, the controller device may be configured to be grasped only in the forward direction.

[0257] At this time, the operating portion 302a and the grip portion 302b do not necessarily have symmetrical shapes about the ring portion 301, and for example, the operating portion 302a and the grip portion 302b may have different shapes. In addition, the operating member 332b and the operating member 333b of the grip portion 302b may be removed.

[0258] For example, the controller device 113a may use a material other than resin (eg, metal).

[0259] <About Modification Example of Sharing Processing>

[0260] For example, part of the processing of the information processing device 111 may be performed by the terminal device 112 .

[0261] For example, the terminal device 112 may perform all or part of the processing of the information processing unit 211 of the information processing device 111. For example, the terminal device 112 may independently present the XR space without being controlled by the information processing device 111. For example, the information processing device 111 and the terminal device 112 may independently share and perform processing such as constructing the XR space.

[0262] <Other Modifications>

[0263] For example, the controller device 113a may be used not only for operations in the XR space but also for operations in two-dimensional space and three-dimensional space (eg, games).

[0264] <<3. Others>>

[0265] <Computer Configuration Example>

[0266] The above series of processing can be performed by hardware, and can also be performed by software. In the case of performing a series of processing by software, the program constituting the software is installed in the computer. Here, examples of the computer include a computer included in dedicated hardware, and, for example, a general-purpose personal computer that can perform various functions by installing various programs.

[0267] Fig.24 : is a block diagram showing a configuration example of hardware of a computer that executes the above-described series of processes by a program.

[0268] In a computer 2000 , a central processing unit (CPU) 2001 , a read only memory (ROM) 2002 , and a random access memory (RAM) 2003 are connected to one another via a bus 2004 .

[0269] An input / output interface 2005 is also connected to the bus 2004. An input unit 2006, an output unit 2007, a storage unit 2008, a communication unit 2009, and a drive 2010 are connected to the input / output interface 2005.

[0270] The input unit 2006 includes an input switch, a button, a microphone, an image sensor, etc. The output unit 2007 includes a display, a speaker, etc. The storage unit 2008 includes a hard disk, a nonvolatile memory, etc. The communication unit 2009 includes a network interface, etc. The drive 2010 drives a removable medium 2011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0271] In the computer 2000 configured as described above, the above-described series of processes is performed, for example, by the CPU 2001 loading a program stored in the storage unit 2008 into the RAM 2003 via the input / output interface 2005 and the bus 2004 and executing the program.

[0272] The program executed by the computer 2000 (CPU 2001) can be provided by being recorded on the removable medium 2011 as, for example, a package medium, etc. Furthermore, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0273] In the computer 2000, by mounting the removable medium 2011 on the drive 2010, the program can be installed in the storage unit 2008 via the input / output interface 2005. In addition, the program can be received by the communication unit 2009 via a wired or wireless transmission medium and installed in the storage unit 2008. In addition to the above, the program can also be pre-installed in the ROM 2002 or the storage unit 2008.

[0274] Note that the program executed by the computer may be a program that executes processing in time series in the order described in this specification, or may be a program that executes processing in parallel or at necessary timing (for example, when a call is made).

[0275] In addition, in this specification, a system means an assembly of multiple components (devices, modules (parts), etc.), and it is not important whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected to each other via a network and a device in which multiple modules are housed in one housing are both systems.

[0276] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications may be made without departing from the scope of the present technology.

[0277] For example, the present technology may be embodied in cloud computing in which one function is shared and performed by a plurality of devices via a network.

[0278] Furthermore, each step described in the above flowchart may be performed by one device, or may be shared and performed by a plurality of devices.

[0279] Furthermore, in the case where a plurality of processes are included in one step, the plurality of processes included in one step may be executed by one device, or may be executed by a plurality of devices in a shared manner.

[0280] <Configuration combination example>

[0281] This technology can also be configured as follows: (1)

[0283] An input device, comprising:

[0284] a ring portion into which a finger is inserted;

[0285] an operating portion operable by a finger inserted into the ring portion in a first direction; and

[0286] The gripping portion is gripped by a palm when the operating portion is operated by the finger. (2)

[0288] The input device according to (1), wherein:

[0289] When viewed from a side surface of the ring portion, the operation portion and the grip portion have symmetrical shapes with respect to the ring portion. (3)

[0291] The input device according to (2), wherein:

[0292] The grip portion is operable by a finger inserted into the loop portion in a second direction opposite to the first direction, and

[0293] When the grip portion is operated by the finger, the operating portion is grasped by the palm of the hand. (4)

[0295] The input device according to (3), further comprising:

[0296] a first operating member disposed near a distal end of the operating portion; and

[0297] A second operating member is disposed near a distal end of the grip portion. (5)

[0299] The input device according to (4), further comprising:

[0300] a first tactile device disposed adjacent to the first operating member; and

[0301] A second haptic device is disposed adjacent to the second operating member. (6)

[0303] The input device according to (4) or (5), further comprising:

[0304] An operating member is arranged symmetrically about the ring portion on a surface formed by a lower surface of the operating portion and a lower surface of the grip portion. (7)

[0306] The input device according to any one of (4) to (6), wherein:

[0307] The operating portion and the gripping portion present tactile stimulation. (8)

[0309] The input device according to any one of (4) to (7), further comprising:

[0310] Multiple tags, among which

[0311] Some of the markings are arranged along the periphery of the ring portion, and

[0312] Some of the marks are arranged on both side surfaces of the operation portion and both side surfaces of the grip portion. (9)

[0314] The input device according to (8), wherein

[0315] Some of the marks are arranged on a surface formed by a lower surface of the operation portion and a lower surface of the grip portion. (10)

[0317] The input device according to any one of (2) to (9), wherein:

[0318] With the distal end of the ring portion oriented upward, the operating portion extends from near the lower end of the ring portion along the first direction and obliquely downward, and the grip portion extends from near the lower end of the ring portion along a second direction opposite to the first direction and obliquely downward. (11)

[0320] The input device according to any one of (1) to (10), further comprising:

[0321] An operating member is disposed near a lower end of the hole of the ring portion. (12)

[0323] The input device according to (11), further comprising:

[0324] A haptic device is disposed adjacent to the operating member. (13)

[0326] The input device according to any one of (1) to (12), wherein:

[0327] The operating portion includes an inclined surface which is inclined from the vicinity of the lower end of the ring portion in the first direction and in an oblique downward direction with the distal end of the ring portion oriented upward, the input device further comprising:

[0328] An operating member is arranged on the inclined surface. (14)

[0330] The input device according to (13), further comprising:

[0331] A haptic device is disposed adjacent to the operating member. (15)

[0333] A control device, comprising:

[0334] an identification unit that identifies a position and orientation of an input device, wherein the input device includes: a ring portion into which a finger of a user's hand is inserted; an operation portion that can be operated by the finger inserted into the ring portion in a first direction; and a grip portion that is gripped by a palm of the hand when the operation portion is operated by the finger; and

[0335] An operation control unit controls the operation of the input device based on the position and orientation of the input device. (16)

[0337] The control device according to (15), wherein:

[0338] When viewed from the side surface of the ring portion, the operating portion and the grip portion have symmetrical shapes with respect to the ring portion,

[0339] The grip portion is operable by a finger inserted into the ring portion in a second direction opposite to the first direction,

[0340] When the gripping portion is operated by the finger, the operating portion is held by the palm of the hand.

[0341] The recognition unit recognizes the direction in which the input device is grasped, and

[0342] The operation control unit controls operations for the operation portion and the grip portion based on a direction in which the input device is gripped. (17)

[0344] The control device according to (15) or (16), wherein:

[0345] The recognition unit recognizes operation content of the input device based on at least one of: an operation input signal output from the input device, or a position and orientation of the input device. (18)

[0347] The control device according to (17), further comprising:

[0348] A tactile presentation control unit controls presentation of tactile stimulation by the input device based on the operation content. (19)

[0350] A control method, comprising:

[0351] Identifying the position and orientation of an input device, wherein the input device comprises: a ring portion into which a finger of a user's hand is inserted; an operating portion operable by the finger inserted into the ring portion in a first direction; and a grip portion, which is gripped by a palm of the hand when the operating portion is operated by the finger; and

[0352] Operation of the input device is controlled based on the position and orientation of the input device. (20)

[0354] An information processing device, wherein:

[0355] An operation input signal is input, and the operation input signal is output from an input device, the input device including: a ring portion into which a finger is inserted; an operation portion that can be operated by the finger inserted into the ring portion; and a grip portion that is gripped by a palm when the operation portion is operated by the finger; and

[0356] Display control information is output, the display control information being used to control the display of the XR space based on the operation input signal. (twenty one)

[0358] An information processing device, comprising:

[0359] A control unit that receives input information, executes an application based on the input information, and outputs output information for controlling display of a virtual object including computer-aided design (CAD) information in an XR space, wherein the input information indicates at least one of: a state of a terminal device that presents the XR space to a user, a state around the terminal device, a state of the user, a behavior of the user, or an operation input to an input device for operation input of the XR space. (twenty two)

[0361] An information processing method, comprising:

[0362] By the information processing device,

[0363] receiving input information indicating at least one of: a state of a terminal device presenting an XR space to a user, a state around the terminal device, a state of the user, a behavior of the user, or an operation input to an input device for operation input of the XR space;

[0364] executing an application based on the input information; and

[0365] Output information for controlling display of a virtual object including CAD information in the XR space is output. (twenty three)

[0367] An information processing device, comprising:

[0368] A control unit that executes an application for presenting an XR space to a user, and outputs output information indicating a change or abnormality in the state of the application to a terminal device that presents the XR space to the user or an input device for inputting operations of the XR space. (twenty four)

[0370] An information processing method, comprising:

[0371] By the information processing device,

[0372] executing an application for presenting the XR space to a user; and

[0373] Output information indicating a change in the state or an abnormality of the application is output to a terminal device that presents the XR space to the user or an input device for operation input of the XR space.

[0374] Note that the effects described in this specification are merely examples and are not limiting, and other effects can be achieved.

[0375] Reference numerals list

[0376] 101 XR System

[0377] 111 Information processing device

[0378] 112 Terminal Device

[0379] 113, 113a, 113b controller device

[0380] 202 Control unit

[0381] 203 Display unit

[0382] 211 Information Processing Unit

[0383] 221 Identification Unit

[0384] 222 Operation control unit

[0385] 223 Space Control Unit

[0386] 224 Audio Control Unit

[0387] 225 Tactile presentation control unit

[0388] 226 study units

[0389] 252 sensor units

[0390] 253 Control Unit

[0391] 254 Display units

[0392] 255 audio output unit

[0393] 301 Ring

[0394] 301A hole

[0395] 302a Operation unit

[0396] 302b Grip

[0397] 312a, 312b upper surface

[0398] 313 bottom surface

[0399] 331 to 334 operating components

[0400] 351 Mark

[0401] 371 to 372b Haptic devices

[0402] 401 Camera Device

Claims

1. An input device, include: a ring portion into which a finger is inserted; an operating portion operable by a finger inserted into the ring portion along a first direction; as well as The gripping portion is gripped by a palm when the operating portion is operated by the finger.

2. The input device according to claim 1, in, When viewed from a side surface of the ring portion, the operation portion and the grip portion have symmetrical shapes with respect to the ring portion.

3. The input device according to claim 2, in, The grip portion is operable by a finger inserted into the loop portion in a second direction opposite to the first direction, and When the grip portion is operated by the finger, the operating portion is grasped by the palm of the hand.

4. The input device according to claim 3, further comprising: include: a first operating member disposed near a distal end of the operating portion; as well as A second operating member is disposed near a distal end of the grip portion.

5. The input device according to claim 4, further comprising: include: a first tactile device disposed proximate to the first operating member; as well as A second haptic device is disposed adjacent to the second operating member.

6. The input device according to claim 4, further comprising: include: The operating members are arranged symmetrically with respect to the ring portion on a surface formed by a lower surface of the operating portion and a lower surface of the grip portion.

7. The input device according to claim 4, in, The operating portion and the gripping portion present tactile stimulation.

8. The input device according to claim 4, further comprising: include: Multiple tags, among which Some of the markings are arranged along the periphery of the ring portion, and Some of the marks are arranged on both side surfaces of the operation portion and both side surfaces of the grip portion.

9. The input device according to claim 8, in, Some of the marks are arranged on a surface formed by a lower surface of the operation portion and a lower surface of the grip portion.

10. The input device according to claim 2, in, With the distal end of the ring portion oriented upward, the operating portion extends from near the lower end of the ring portion along the first direction and obliquely downward, and the grip portion extends from near the lower end of the ring portion along a second direction opposite to the first direction and obliquely downward.

11. The input device according to claim 1, further comprising: include: An operating member is disposed near a lower end of the hole of the ring portion.

12. The input device according to claim 11, further comprising: include: A haptic device is disposed adjacent to the operating member.

13. The input device according to claim 1, in, The operating portion includes an inclined surface which is inclined from the vicinity of the lower end of the ring portion in the first direction and in an oblique downward direction with the distal end of the ring portion oriented upward, The input device also includes: An operating member is arranged on the inclined surface.

14. The input device according to claim 13, further comprising: include: A haptic device is disposed adjacent to the operating member.

15. A control device, include: an identification unit that identifies a position and orientation of an input device, wherein the input device includes: a ring portion into which a finger of a user's hand is inserted; an operation portion that can be operated by the finger inserted into the ring portion in a first direction; and a grip portion that is gripped by a palm of the hand when the operation portion is operated by the finger; and An operation control unit controls the operation of the input device based on the position and orientation of the input device.

16. The control device according to claim 15, in, When viewed from the side surface of the ring portion, the operating portion and the grip portion have symmetrical shapes with respect to the ring portion, The grip portion is operable by a finger inserted into the ring portion in a second direction opposite to the first direction, When the gripping portion is operated by the finger, the operating portion is held by the palm of the hand. The recognition unit recognizes the direction in which the input device is grasped, and The operation control unit controls operations for the operation portion and the grip portion based on a direction in which the input device is gripped.

17. The control device according to claim 15, in, The recognition unit recognizes operation content of the input device based on at least one of: an operation input signal output from the input device, or a position and orientation of the input device.

18. The control device according to claim 17, further comprising: include: A tactile presentation control unit controls presentation of tactile stimulation by the input device based on the operation content.

19. A control method, include: Identifying the position and orientation of an input device, wherein the input device comprises: a ring portion into which a finger of a user's hand is inserted; an operating portion operable by the finger inserted into the ring portion in a first direction; and a grip portion, which is gripped by a palm of the hand when the operating portion is operated by the finger; and Operation of the input device is controlled based on the position and orientation of the input device.

20. An information processing device, in, An operation input signal is input, and the operation input signal is output from an input device, the input device including: a ring portion into which a finger is inserted; an operation portion that can be operated by the finger inserted into the ring portion; and a grip portion that is gripped by a palm when the operation portion is operated by the finger; and Display control information is output, the display control information being used to control the display of the XR space based on the operation input signal.

21. An information processing device, include: A control unit that receives input information, executes an application based on the input information, and outputs output information for controlling the display of a virtual object including computer-aided design (CAD) information in an XR space, wherein the input information indicates at least one of: a state of a terminal device that presents the XR space to a user, a state around the terminal device, a state of the user, a behavior of the user, or an operation input to an input device, wherein the input device is used for operation input to the XR space.

22. An information processing method, include: By the information processing device, receiving input information, the input information indicating at least one of the following: a state of a terminal device presenting an XR space to a user, a state around the terminal device, a state of the user, a behavior of the user, or an operation input to an input device, wherein the input device is used for operation input of the XR space; executing an application based on the input information; and Output information for controlling display of a virtual object including CAD information in the XR space is output.

23. An information processing device, include: A control unit that executes an application for presenting an XR space to a user, and outputs output information indicating a change or abnormality in the state of the application to a terminal device that presents the XR space to the user or an input device for inputting operations of the XR space.

24. A method of information processing, include: By the information processing device, executing an application for presenting an XR space to a user; as well as Output information indicating a change in the state or an abnormality of the application is output to a terminal device that presents the XR space to the user or an input device for operation input of the XR space.

Images