System, control device, control method, information processing device, and information processing method

Abstract

The present technology pertains to an input device, a control apparatus, a control method, an information processing apparatus, and an information processing method, which can improve operability of the input device used for operation of cross reality (XR) and the like. The input device comprises: a ring part into which a finger is to be inserted; an operation part which can be operated by the finger inserted into the ring part in a first direction; and a held part which is held by a palm when the operation part is operated by the finger. The present technology can be applied to an XR system, for example.

Description

Input device, control device, control method, information processing device, and information processing method

[0001] The present technology relates to an input device, a control device, a control method, an information processing device, and an information processing method, and in particular to an input device, a control device, a control method, an information processing device, and an information processing method that improve the operability of an input device used for operating an XR (cross reality) space or the like.

[0002] Conventionally, hand-held input devices have been used to operate virtual spaces (see, for example, Patent Document 1), and pen-type input devices have been used to specify positions within virtual spaces (see, for example, Patent Document 2).

[0003] JP 2020-181320 A International Publication No. 2019 / 220803

[0004] However, if an input device that is held in the hand is not provided with a strap or the like, it will fall when released. Also, for example, if a pen-type input device is provided with a button, the pen tip may shake when the button is operated, which may cause the position indicated by the input device to shift.

[0005] The present technology has been made in view of such circumstances, and aims to improve the operability of an input device used for operating an XR space or the like.

[0006] An input device according to a first aspect of the present technology includes a ring portion into which a finger is inserted, an operation portion operable by the finger inserted into the ring portion in a first direction, and a holding portion to be held by the palm of a hand when operating the operation portion with the finger.

[0007] A control device according to a second aspect of the present technology includes a recognition unit that recognizes the position and orientation of an input device that includes a ring portion into which a finger of a user's hand is inserted, an operation unit that can be operated by the finger inserted into the ring portion in a first direction, and a holding unit that is held by the palm of the hand when the operation unit is operated by the finger, and an operation control unit that controls operation by the input device based on the position and orientation of the input device.

[0008] A control method according to a second aspect of the present technology recognizes the position and orientation of an input device that includes a ring portion into which a finger of a user's hand is inserted, an operation portion that can be operated by the finger inserted into the ring portion in a first direction, and a holding portion that is held by the palm of the hand when the operation portion is operated by the finger, and controls operation by the input device based on the position and orientation of the input device.

[0009] An information processing device according to a third aspect of the present technology receives an operation input signal output from an input device including a ring portion into which a finger is inserted, an operation portion operable by the finger inserted into the ring portion, and a holding portion to be held by the palm of the hand when the operation portion is operated by the finger, and outputs display control information that controls the display of an XR space based on the operation input signal.

[0010] An information processing device according to a fourth aspect of the present technology includes a control unit that receives input information indicating at least one of a state of a terminal device that presents an XR space to a user, a state around the terminal device, a state of the user, behavior of the user, and an operation input to an input device used for operation input in the XR space, and executes an application based on the input information, and outputs output information that controls display of a virtual object including CAD (Computer Aided Design) information in the XR space.

[0011] An information processing method according to a fourth aspect of the present technology includes an information processing device that receives input information indicating at least one of a state of a terminal device that presents an XR space to a user, a state around the terminal device, a state of the user, behavior of the user, and an operation input to an input device used for operation input in the XR space, and executes an application based on the input information and outputs output information that controls display of a virtual object including CAD information in the XR space.

[0012] An information processing device according to a fifth aspect of the present technology includes a control unit that executes an application for presenting an XR space to a user, and outputs output information indicating a change in or an abnormality in the status of the application to a terminal device that presents the XR space to the user, or an input device that is used for inputting operations in the XR space.

[0013] In an information processing method according to a fifth aspect of the present technology, an information processing device executes an application for presenting an XR space to a user, and outputs output information indicating a change in or an abnormality in the state of the application to a terminal device that presents the XR space to the user or an input device used for inputting operations in the XR space.

[0014] According to the first aspect of the present technology, the operation portion is operated by a finger inserted into the ring portion, and the holding portion is held by the palm when the operation portion is operated by the finger.

[0015] In a second aspect of the present technology, the position and orientation of an input device having a ring portion into which a finger of a user's hand is inserted, an operation portion that can be operated by the finger inserted into the ring portion in a first direction, and a holding portion that is held by the palm of the hand when the operation portion is operated by the finger is recognized, and operation by the input device is controlled based on the position and orientation of the input device.

[0016] In a third aspect of the present technology, an operation input signal is input from an input device that includes a ring portion into which a finger is inserted, an operation portion that can be operated by the finger inserted into the ring portion, and a holding portion that is held by the palm of a hand when the operation portion is operated by the finger, and display control information that controls the display of an XR space based on the operation input signal is output.

[0017] In a fourth aspect of the present technology, input information indicating at least one of a state of a terminal device that presents an XR space to a user, a state around the terminal device, a state of the user, a behavior of the user, and an operation input to an input device used for operation input in the XR space is received, an application is executed based on the input information, and output information that controls the display of a virtual object including CAD (Computer Aided Design) information in the XR space is output.

[0018] In a fifth aspect of the present technology, an application for presenting an XR space to a user is executed, and output information indicating a change in or an abnormality in the state of the application is output to a terminal device that presents the XR space to the user or an input device used for inputting operations in the XR space.

[0019] 1 is a diagram illustrating an embodiment of an XR system to which the present technology is applied. FIG. 1 is a diagram illustrating an example display of the XR system. FIG. 2 is a diagram illustrating an example display of the XR system. FIG. 3 is a block diagram illustrating an example configuration of an information processing device and a terminal device. FIG. 4 is an external view illustrating an example configuration of a controller device. FIG. 5 is a diagram illustrating a method of holding the controller device. FIG. 6 is a diagram illustrating a method of holding the controller device. FIG. 7 is a diagram illustrating a method of holding the controller device. FIG. 8 is a diagram illustrating an example arrangement of operation members of the controller device. FIG. 9 is a diagram illustrating an example arrangement of markers on the controller device. FIG. 10 is a diagram illustrating an example of how markers on the controller device appear. FIG. 11 is a diagram illustrating a method of recognizing the position and orientation of the controller device. FIG. 12 is a diagram illustrating an example internal configuration of the controller device. FIG. 13 is a diagram illustrating an example arrangement of haptic devices on the controller device. FIG. 14 is a flowchart illustrating operation member control processing executed by the XR system. FIG. 15 is a diagram illustrating operation member control processing executed by the XR system. FIG. 16 is a diagram illustrating an example method of holding the controller device. FIG. 17 is a diagram illustrating an example method of holding the controller device. FIG. 18 is a diagram illustrating a haptic feedback control processing executed by the XR system. FIG. 19 is a diagram illustrating an example of haptic feedback. FIG. 19 is a block diagram illustrating an example configuration of a computer.

[0020] Hereinafter, embodiments of the present technology will be described. The description will be made in the following order: 1. Embodiment 2. Modification 3. Other

[0021] <<1. Embodiment>> An embodiment of the present technology will be described with reference to FIGS. 1 to 23 .

[0022] <Configuration Example of XR System 101> FIG. 1 shows a configuration example of an XR (cross reality) system 101, which is an embodiment of an information processing system to which the present technology is applied.

[0023] The XR system 101 is a system that realizes XR, which is a technology that combines the real world and the virtual world, such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and alternate reality (SR). The XR system 101 is a system that presents a space (hereinafter referred to as XR space) that combines real space and virtual space to a user. The XR system 101 can present a virtual object (hereinafter referred to as a virtual object or virtual object) that does not exist in reality, such as a model created by computer-aided design (CAD) (hereinafter referred to as a CAD model), to the user as if it were actually present in the actual location.

[0024] The XR system 101 includes an information processing device 111 , a terminal device 112 , and a controller device 113 .

[0025] The information processing device 111 and the terminal device 112 can communicate wirelessly or via a cable, and transmit and receive data to and from each other. The terminal device 112 and the controller device 113 can communicate wirelessly or via a cable, and transmit and receive data to and from each other. The information processing device 111 and the controller device 113 communicate via the terminal device 112, and transmit and receive data to and from each other.

[0026] The information processing device 111 can, for example, independently accept operations by a user and present various types of information such as visual information and auditory information to the user.

[0027] Furthermore, the information processing device 111 controls the terminal device 112 by executing a predetermined application (hereinafter referred to as an XR app), for example, and controls the presentation of an XR space to a user by the terminal device 112. For example, by executing the XR app, the information processing device 111 controls the output of various types of information, such as visual information and auditory information, in the terminal device 112, and constructs the XR space presented by the terminal device 112.

[0028] 1 illustrates an example in which the information processing device 111 is configured as a personal computer (PC) equipped with an operation input unit including a mouse and a keyboard. For example, the information processing device 111 may be configured as another information processing device such as a smartphone or a tablet terminal. For example, the information processing device 111 may be configured as a plurality of information processing devices. For example, the information processing device 111 may be configured as a system established by cloud computing via a network.

[0029] The terminal device 112 is a device that presents the XR space to the user.

[0030] 1 shows an example in which the terminal device 112 is configured as a head-mounted display (HMD) that is a head-mounted display device that can be worn on the user's head and that presents an XR space to the user. More specifically, the terminal device 112 is shown as an example in which the terminal device 112 is a non-transparent HMD that covers the user's field of vision.

[0031] For example, the terminal device 112 is configured as a video see-through HMD that has an imaging function that can capture images of real space based on the user's viewpoint and can present to the user a composite image that combines a real image captured of real space with an image of a virtual space such as computer graphics (CG) (hereinafter referred to as a virtual image).

[0032] For example, the terminal device 112 includes left and right imaging units corresponding to the left and right eyes of the user, respectively, and left and right display units corresponding to the left and right eyes of the user, respectively.

[0033] For example, the left and right imaging units configure a stereo camera and capture images in the user's line of sight (hereinafter referred to as "viewing image") from multiple viewpoints corresponding to the user's left and right eyes. That is, the left and right imaging units capture images of objects in real space (hereinafter referred to as "real objects" or "real objects") that are visible from the user's viewpoints.

[0034] The left and right display units can display different images to the left and right eyes, respectively, and by displaying images with parallax to the left and right eyes, it is possible to present a three-dimensional virtual object. For example, the left and right display units display left and right field images captured by the left and right imaging units, respectively.

[0035] The terminal device 112 may be configured as another XR terminal device, such as a smartphone used in conjunction with AR glasses or goggles. Alternatively, a display device, such as a spatial reproduction display, may be used instead of the terminal device 112.

[0036] The controller device 113 is used for operations and inputs (hereinafter referred to as operation inputs) to the XR space presented to the user by the terminal device 112. For example, the user can use the controller device 113 to perform various operations on virtual objects displayed by the terminal device 112.

[0037] For example, the controller device 113 detects at least one of a user's operation input and a user's behavior (e.g., a gesture) using at least one of an operation member such as a button and a sensor. The controller device 113 transmits a signal (hereinafter referred to as a controller signal) including at least one of an operation input signal indicating the user's operation input and a behavior signal indicating the user's behavior to the information processing device 111 via the terminal device 112.

[0038] Furthermore, for example, the controller device 113a includes a haptic device that presents haptic stimuli such as vibrations, and presents the haptic stimuli to the user under the control of the information processing device 111 or the terminal device 112.

[0039] The controller device 113 includes, for example, one or more types of input devices selected from the group consisting of a controller, a ring-type input device, a pointing device, and a 6DoF (six degrees of freedom) input device.

[0040] The controller is, for example, an input device held in the user's hand. The controller may include, for example, operation members such as buttons that can be operated by the user. For example, by pressing a button on the controller, the user can perform a selection operation, a confirmation operation, a scroll operation, or the like on a virtual object displayed on the terminal device 112. The controller may also include, for example, a touch sensor and a motion sensor.

[0041] The controller is not limited to being held in the user's hand, but may be attached to a part of the user's body, such as the elbow, arm, knee, ankle, or thigh.

[0042] The ring-shaped device is a ring-shaped input device worn on a user's finger. The ring-shaped device may include an operating member such as a button that can be operated by the user. For example, by operating the ring-shaped device, the user can change the position and orientation of a virtual object (e.g., a three-dimensional model) in an XR space with six degrees of freedom (DoF).

[0043] The pointing device is an input device capable of pointing to any position in the XR space. For example, the information processing device 111 recognizes the 6DoF position and orientation of the pointing device via the terminal device 112 using a tracking method such as a bright spot tracking method, a magnetic tracking method, or an ultrasonic tracking method.

[0044] The 6DoF input device is, for example, an input device that can be operated in 6DoF.

[0045] For example, the user can perform operation input using the controller device 113 while viewing various objects (display objects) displayed on the information processing device 111 or the terminal device 112 .

[0046] There is no particular limitation on the type and number of the controller device 113. For example, the controller device 113 may be an input device other than the types described above, or may be an input device that combines multiple types of input devices.

[0047] For example, the XR system 101 can be applied to various fields such as the manufacturing field and the medical field.

[0048] For example, in the manufacturing field, the XR system 101 can provide support for product design and assembly. For example, during the product design stage, a user can use the XR system 101 to freely edit a three-dimensional object, which is a virtual object, and compare it with the real world to understand the design results and design in advance before prototyping.

[0049] For example, the XR system 101 can support surgery and education in the medical field. For example, a user can use the XR system 101 to display the state of the inside of a patient's body on the surface of the body, thereby understanding the surgical site in advance or conducting training.

[0050] For example, when an XR space is shared by multiple users, a terminal device 112 and a controller device 113 are provided for each user in the XR system 101.

[0051] <Display Example of XR System 101> Here, a display example of a display object in the XR system 101 will be described with reference to FIGS. 2 and 3. FIG.

[0052] 2 and 3 show examples of display objects in the XR system 101 when creating a CAD model.

[0053] For example, as shown in FIG. 2A, the information processing device 111 displays a two-dimensional CAD model, and the user can edit the two-dimensional CAD model.

[0054] For example, as shown in FIG. 2B, a three-dimensional CAD model is displayed on the terminal device 112, and the user can edit the three-dimensional CAD model.

[0055] For example, as shown in FIG. 2C, a two-dimensional object such as a design drawing or specification sheet is displayed on the terminal device 112, and the user can check the design drawing or specification sheet.

[0056] FIG. 3 shows an example of the XR space displayed by the terminal device 112.

[0057] The display 151, keyboard 152, mouse 153, and desk 154 of the information processing device 111 are displayed as a video see-through using a real image captured from real space. Meanwhile, a two-dimensional image from the terminal device 112 is superimposed on the display 151 as a virtual monitor. For example, a two-dimensional CAD model that is the design object is displayed on the virtual monitor. The two-dimensional CAD model displayed on the virtual monitor is preferably operated using the keyboard 152 and mouse 153, for example, because of the high accuracy of position detection and the ease of position retention.

[0058] In this example, a three-dimensional CAD model 155 that is the design object is displayed in front of the display 151 by the terminal device 112 .

[0059] The CAD model 155 is operated, for example, by a controller device 113a held in the user's dominant hand (in this example, the right hand) and a controller device 113b, which is a ring-shaped device worn on the index finger of the user's non-dominant hand (in this example, the left hand).

[0060] For example, the information processing device 111 recognizes the position, posture, and behavior of the hand holding the controller device 113a and the hand of the user wearing the controller device 113b by performing hand tracking based on an image captured by an imaging unit included in the terminal device 112. Furthermore, for example, the information processing device 111 receives controller signals from the controller devices 113a and 113b via the terminal device 112, and recognizes operations performed on the CAD model 155 by the controller devices 113a and 113b based on the controller signals.

[0061] For example, a user can use controller device 113a or controller device 113b to grab, release, move and rotate CAD model 155 with 6 DoF.

[0062] For example, when the hand wearing the controller device 113a or the controller device 113b is moved without gripping the CAD model 155, the CAD model 155 may not move, or the CAD model 155 may move so as to move a virtual point.

[0063] For example, the user can use the controller device 113a to point to any point, line, surface, etc. on the CAD model 155 using a ray (a virtual ray) or the like. For example, the user can use the controller device 113a to perform line drawing, which is to draw a line on the CAD model 155.

[0064] For example, a user can use the controller device 113a or the controller device 113b to edit (for example, model, wire, disassemble, etc.) the CAD model 155.

[0065] <Configuration Example of Information Processing Device 111 and Terminal Device 112> FIG. 4 is a block diagram showing a configuration example of the functions of the information processing device 111 and the terminal device 112 of the XR system 101.

[0066] The information processing device 111 includes an operation input unit 201 , a control unit 202 , a display unit 203 , a storage unit 204 , and a communication unit 205 .

[0067] The operation input unit 201 includes input devices such as a keyboard, a mouse, etc. The operation input unit 201 receives an operation input from the user and supplies an operation input signal indicating the content of the operation input from the user to the control unit 202.

[0068] The control unit 202 includes electronic circuits such as a CPU, a microprocessor, etc. The control unit 202 may also include a ROM for storing programs to be used, calculation parameters, etc., and a RAM for temporarily storing parameters that change as needed.

[0069] For example, the control unit 202 functions as an arithmetic processing unit and a control unit, and controls the overall operation of the information processing device 111 and executes various processes in accordance with various programs.

[0070] For example, the control unit 202 realizes the information processing unit 211 by executing an XR app that enables a user experience in the information processing device 111 and the XR space and editing of virtual objects. The information processing unit 211 includes a recognition unit 221, an operation control unit 222, a space control unit 223, an audio control unit 224, a tactile presentation control unit 225, and a learning unit 226. That is, the recognition unit 221, the operation control unit 222, the space control unit 223, the audio control unit 224, the tactile presentation control unit 225, and the learning unit 226 are realized by the control unit 202 executing the XR app. Furthermore, input and output of each unit of the information processing unit 211, that is, the recognition unit 221, the operation control unit 222, the space control unit 223, the audio control unit 224, the tactile presentation control unit 225, and the learning unit 226, is performed via the XR app.

[0071] The recognition unit 221 recognizes the state of the information processing device 111, the state of the terminal device 112, the state of the surroundings of the terminal device 112, the state of the controller device 113, the state of the user, the user operation, the state of the XR space, etc., based on at least one of the operation input signal from the operation input unit 201, information from the control unit 202, information from the display unit 203, information from the communication unit 205, sensing data transmitted from the terminal device 112, the controller signal transmitted from the controller device 113, information from the operation control unit 222, and information from the space control unit 223.

[0072] The state of the information processing device 111 to be recognized includes, for example, at least one of the state of each part of the information processing device 111, the state of each application such as an XR app, the communication state between the information processing device 111 and other devices, and various setting information (for example, setting values ​​of various setting items). The state of each part of the information processing device 111 includes, for example, at least one of the operation state of each part, the presence or absence of an abnormality, and the content of the abnormality. The state of each application includes, for example, at least one of the start, termination, operation state, presence or absence of an abnormality, and the content of the abnormality of each application. The communication state between the information processing device 111 and other devices includes, for example, the communication state with the terminal device 112 and the communication state with the controller device 113 via the terminal device 112.

[0073] The state of the terminal device 112 to be recognized includes, for example, at least one of the position, posture, behavior, and various setting information (for example, setting values ​​of various setting items) of the terminal device 112. Note that, for example, when the terminal device 112 is worn by a user, the position, posture, and behavior of the terminal device 112 indirectly represent the position, posture, and behavior of the part of the user on which the terminal device 112 is worn.

[0074] The state of the surroundings of the terminal device 112 to be recognized includes, for example, at least one of the type, position, posture, behavior, size, shape, appearance, and feature amount of real objects around the terminal device 112 (user).

[0075] The state of the controller device 113 to be recognized includes, for example, at least one of the position, posture, behavior, and various setting information of the controller device 113 (for example, setting values ​​of various setting items).

[0076] The state of the user to be recognized includes, for example, at least one of the user's position, posture, overall behavior, behavior of body parts, and gaze direction.

[0077] The user operations to be recognized include, for example, at least one of operation input by the operation input unit 201, operation input by the controller device 113, operation input by a user's gesture, and operation input by a virtual tool or the like in the XR space.

[0078] The state of the XR space to be recognized includes, for example, at least one of the type, position, posture, behavior, size, shape, appearance, and feature amount of a virtual object in the XR space.

[0079] The recognition unit 221 supplies information about the recognition result to each unit of the information processing device 111 .

[0080] Furthermore, the recognition unit 221 transmits information regarding the recognition result to the terminal device 112 via the communication unit 205, or transmits it to the controller device 113 via the communication unit 205 and the terminal device 112. For example, when the recognition unit 221 detects a change in or abnormality in the state of the terminal device 112 or the input device 113, the recognition unit 221 transmits information indicating the detected content to the terminal device 112 via the communication unit 205, or transmits it to the controller device 113 via the communication unit 205 and the terminal device 112. For example, when the recognition unit 221 detects a change in or abnormality in the state of an application such as an XR app (e.g., startup, stop, etc.), the recognition unit 221 transmits information indicating the detected content to the terminal device 112 via the communication unit 205, or transmits it to the controller device 113 via the communication unit 205 and the terminal device 112.

[0081] The recognition process of the recognition unit 221 for various recognition targets can use any method such as image recognition or object recognition.

[0082] Furthermore, for example, when an XR space is shared by multiple users, the recognition unit 221 performs recognition processing for each user. For example, the recognition unit 221 recognizes the state of each user's terminal device 112, the state of the surroundings of each user's terminal device 112, the state of each user's controller device 113, the state of each user, and the user operations of each user. The results of the recognition processing for each user may be shared among users by, for example, being transmitted to each user's terminal device 112 or controller device 113.

[0083] The operation control unit 222 controls the operation processing by the controller device 113 based on at least one of the recognition result by the recognition unit 221 and the controller signal transmitted from the controller device 113 .

[0084] For example, the operation control unit 222 controls operation processing by the controller device 113 based on at least one of the position and orientation of the controller device 113 and the controller signal. For example, the operation control unit 222 controls the enabling or disabling of each operation member provided in the controller device 113, the functions assigned to each operation member, the operation method of the functions assigned to each operation member, etc., based on the method of wearing, holding, using, etc., of the controller device 113.

[0085] The operation control unit 222 supplies information relating to the control of operation processing by the controller device 113 to each unit of the information processing device 111 .

[0086] The space control unit 223 controls the presentation of a two-dimensional space or a three-dimensional space by the display unit 203 and the presentation of an XR space by the terminal device 112 based on at least a part of the recognition result by the recognition unit 221 .

[0087] For example, the space control unit 223 generates display objects to be displayed in a two-dimensional space or a three-dimensional space based on at least a part of the recognition results by the recognition unit 221, and performs various calculations necessary for constructing and displaying the two-dimensional space or the three-dimensional space, such as the behavior of the display objects. The space control unit 223 generates display control information for controlling the display of the two-dimensional space or the three-dimensional space based on the calculation results, and supplies the information to the display unit 203, thereby controlling the display of the two-dimensional space or the three-dimensional space by the display unit 203. Note that the display control information may include, for example, information for using the two-dimensional space or the three-dimensional space (e.g., operation menus, guidance, messages, etc.) and information for notifying the status of the information processing device 111 (e.g., setting information, remaining battery power, error display, etc.).

[0088] For example, the space control unit 223 generates virtual objects to be displayed in the XR space based on at least a portion of the recognition results by the recognition unit 221, and performs various calculations necessary for constructing and displaying the XR space, such as the behavior of the virtual objects. The recognition results by the recognition unit 221 include, for example, operation details for the controller device 113a recognized by the recognition unit 221 based on a controller signal including an operation input signal from the controller device 113a. The space control unit 223 generates display control information for controlling the display of the XR space based on the calculation results, and transmits the display control information to the terminal device 112 via the communication unit 205, thereby controlling the display of the XR space by the terminal device 112. Note that the display control information may include, for example, information for using the XR space (e.g., operation menus, guidance, messages, etc.) and information for notifying the status of the XR system 101 (e.g., setting information, remaining battery power, error display, etc.).

[0089] The space control unit 223 supplies information relating to the two-dimensional space, the three-dimensional space, and the XR space to each unit of the information processing device 111 .

[0090] The audio control unit 224 controls the output of audio by the terminal device 112 based on at least one of the recognition result by the recognition unit 221 and information from the spatial control unit 223. For example, the spatial control unit 223 generates audio control information for outputting audio in the terminal device 112. The audio control information includes, for example, information on at least one of the type, content, frequency, amplitude, and waveform of the sound to be output. The audio control unit 224 controls the output of audio by the terminal device 112 by transmitting the audio control information to the terminal device 112 via the communication unit 205.

[0091] The tactile presentation control unit 225 controls the presentation of tactile stimuli to the user based on at least one of the recognition result by the recognition unit 221 and information from the spatial control unit 223. For example, the tactile presentation control unit 225 generates tactile control information for presenting tactile stimuli in the controller device 113. The tactile control information includes, for example, information on at least one of the type, pattern, strength, and length of the tactile sensation to be presented. The tactile presentation control unit 225 controls the presentation of tactile stimuli by the controller device 113 by transmitting the tactile control information to the controller device 113 via the communication unit 205 and the terminal device 112.

[0092] The learning unit 226 executes learning processing related to the processing of the XR system 101 based on at least one of the recognition result by the recognition unit 221 and externally provided learning data. For example, the learning unit 226 learns the preferences, behavioral patterns, etc. of the user, and adjusts various processes and parameters of the XR system 101 based on the learning results so as to appropriately respond to the preferences, behavioral patterns, etc. of the user. For example, the learning unit 226 learns the differences between the XR space and the real space, and based on the learning results, adjusts design data, etc. so as to bring the characteristics, behavior, etc. of virtual objects in the XR space closer to those of real objects.

[0093] The learning unit 226 stores, for example, information indicating the learning results (for example, a learning model) in the storage unit 204 .

[0094] The control unit 202 may execute other applications in addition to the XR app.

[0095] The storage unit 204 includes, for example, a ROM (Read Only Memory) that stores programs and calculation parameters used in the processing of the control unit 202, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[0096] The communication unit 205 communicates with an external device to transmit and receive data. For example, the communication unit 205 communicates with the terminal device 112 to transmit and receive data. For example, the communication unit 205 transmits display control information, audio control information, and haptic control information to the terminal device 112. For example, the communication unit 205 receives sensing data and a controller signal from the terminal device 112.

[0097] The communication method of the communication unit 205 may be either wired or wireless, such as a wired LAN, a wireless LAN, Wi-Fi, Bluetooth, etc. The communication unit 205 may also be compatible with two or more types of communication methods.

[0098] The terminal device 112 includes an operation input unit 251 , a sensing unit 252 , a control unit 253 , a display unit 254 , an audio output unit 255 , and a learning unit 226 .

[0099] The operation input unit 251 includes an operation input device such as a button. The operation input unit 201 accepts an operation input from the user and supplies an operation input signal indicating the content of the operation input from the user to the control unit 253. For example, the operation input unit 251 accepts operation inputs such as turning on or off the power of the terminal device 112 and adjusting the brightness of the display unit 254 by the user.

[0100] The sensing unit 252 includes various sensors for sensing the terminal device 112, the surroundings of the terminal device 112, and the state of the user. For example, the sensing unit 252 includes a camera or a depth sensor for capturing images of the surroundings of the terminal device 112. For example, the sensing unit 252 includes a camera or a depth sensor for capturing images of the user's eyes. For example, the sensing unit 252 includes an IMU (Inertial Measurement Unit) for detecting the acceleration, angular velocity, etc. of the terminal device 112. For example, the sensing unit 252 includes a GNSS (Global Navigation Satellite System) receiver for detecting the current position of the terminal device 112 (user). The sensing unit 252 supplies sensing data indicating the detection results of at least one of the sensors to the control unit 253.

[0101] The control unit 253 includes electronic circuits such as a CPU, a microprocessor, etc. The control unit 253 may also include a ROM for storing programs to be used, calculation parameters, etc., and a RAM for temporarily storing parameters that change as appropriate.

[0102] For example, the control unit 253 functions as an arithmetic processing unit and a control unit, and controls the overall operation of the terminal device 112 and executes various processes in accordance with various programs, based on operation input signals from the operation input unit 251, sensing data from the sensing unit 252, display control information and audio control information from the information processing device 111, and controller signals from the controller device 113. For example, the control unit 253 controls the display of an XR space or the like by the display unit 254 based on the display control information. For example, the control unit 253 controls the output of audio by the audio output unit 255 based on audio control information.

[0103] The display unit 254 includes various display devices. For example, if the terminal device 112 is an HMD, the display unit 254 includes displays fixed to the user's left and right eyes, respectively, and displays an image for the left eye and an image for the right eye. The displays may be, for example, display panels such as a liquid crystal display or an organic electroluminescence (EL) display, or laser scanning displays such as a retinal imaging display. The display unit 254 may also include, for example, an imaging optical system that enlarges and projects the display screen and forms an enlarged virtual image with a predetermined angle of view on the user's pupil. For example, the display unit 254 displays an XR space including virtual objects under the control of the control unit 253.

[0104] The audio output unit 255 includes an audio output device such as headphones, earphones, a speaker, etc. The audio output unit 255 outputs audio under the control of the control unit 253.

[0105] The communication unit 256 communicates with external devices and transmits and receives data. For example, the communication unit 256 communicates with the terminal device 112 and the controller device 113 and transmits and receives data. For example, the communication unit 256 transmits sensing data and a controller signal to the information processing device 111. For example, the communication unit 256 receives display control information, audio control information, and haptic control information from the information processing device 111. For example, the communication unit 256 transmits haptic control information to the controller device 113. For example, the communication unit 256 receives a controller signal from the controller device 113.

[0106] The communication method of the communication unit 256 may be either wired or wireless, and for example, a wired LAN, a wireless LAN, Wi-Fi, Bluetooth, etc. may be used. The communication unit 256 may also be compatible with two or more types of communication methods. Furthermore, the communication unit 256 may perform communication between the information processing device 111 and the controller device 113 using different communication methods.

[0107] The following is an example of processing performed by the information processing device 111 using an XR app.

[0108] For example, the communication unit 205 receives input information indicating at least one of the state of the terminal device 112, the state of the surroundings of the terminal device 112, the state of the user, the user's behavior, and an operation input to the input device 113 from the terminal device 112 or from the controller device 113 via the terminal device 112, and supplies the input information to the control unit 221. The control unit 221 executes an XR app based on this input information, generates output information for controlling the display of a virtual object including CAD information related to CAD in the XR space, and outputs the output information to the terminal device 112. The communication unit 205 transmits this output information to the terminal device 112.

[0109] Furthermore, for example, the control unit 221 executes the XR app and outputs output information indicating a change in or an abnormality in the state of the XR app to the terminal device 112 or the controller device 113. The communication unit 205 transmits this output information to the terminal device 122, or to the controller device 113 via the terminal device 112.

[0110] In response to this, for example, the terminal device 112 notifies the user of a change in or an abnormality in the state of the XR app based on the output information by using an image, a message, a sound, vibration, etc. For example, the controller device 113 notifies the user of a change in or an abnormality in the state of the XR app based on the output information by using vibration, etc.

[0111] In the following, when each unit of the information processing device 111 communicates with the outside via the communication unit 205, the description of the communication unit 205 may be omitted. For example, when the spatial control unit 223 of the information processing device 111 communicates with the terminal device 112 via the communication unit 205, it may be simply described as "the spatial control unit 223 of the information processing device 111 communicates with the terminal device 112."

[0112] Hereinafter, when each unit of the terminal device 112 communicates with the outside via the communication unit 256, the description of the communication unit 256 may be omitted. For example, when the control unit 253 of the terminal device 112 communicates with the information processing device 111 via the communication unit 256, it may be simply stated that the control unit 253 of the terminal device 112 communicates with the information processing device 111.

[0113] For example, in the XR system 101, the spatial control unit 223 of the information processing device 111 generates display control information and transmits it to the terminal device 112 via the communication unit 205, and the control unit 253 of the terminal device 112 receives the display control information via the communication unit 256 and controls the display unit 254 based on the display control information. Hereinafter, the description of this series of processes will be simplified, and may be described as, for example, the spatial control unit 223 of the information processing device 111 controls the display unit 254 of the terminal device 112.

[0114] For example, in the XR system 101, the audio control unit 224 of the information processing device 111 generates audio control information and transmits it to the terminal device 112 via the communication unit 205, and the control unit 253 of the terminal device 112 receives the audio control information via the communication unit 256 and controls the audio output unit 255 based on the audio control information. Hereinafter, the description of this series of processes will be simplified, and may be described as, for example, the audio control unit 224 of the information processing device 111 controls the audio output unit 255 of the terminal device 112.

[0115] For example, in the XR system 101, the haptic presentation control unit 225 of the information processing device 111 generates haptic control information and transmits it to the controller device 113 via the communication unit 205 and the terminal device 112, and the controller device 113 presents a haptic stimulus based on the haptic control information. Hereinafter, the description of this series of processes will be simplified, and may be described as, for example, the haptic presentation control unit 225 of the information processing device 111 controls the controller device 113 via the terminal device 112.

[0116] <Configuration Example of Controller Device 113a> Next, a configuration example of the controller device 113a in FIG. 3 will be described with reference to FIGS. 5 to 14. FIG.

[0117] Fig. 5 shows an example of the external configuration of the controller device 113a. Fig. 5A is a left side view of the controller device 113a. Fig. 5B is a front view of the controller device 113a. Fig. 5C is a bottom view of the controller device 113a. Fig. 5D is a perspective view of the controller device 113a seen from diagonally forward right.

[0118] In the following description, the upward direction of A in Fig. 5 will be referred to as the upward direction of the controller device 113a, the downward direction of A in Fig. 5 will be referred to as the downward direction of the controller device 113a, the right direction of A in Fig. 5 will be referred to as the forward direction of the controller device 113a, and the left direction of A in Fig. 5 will be referred to as the backward direction of the controller device 113a.

[0119] The controller device 113a has a symmetrical shape when viewed from any direction, including the front, back, left, right, top, and bottom. The shape of the front of the controller device 113a when viewed from the front is the same as the shape of the back when viewed from the back, and the shape of the right side when viewed from the right is the same as the shape of the left side when viewed from the left.

[0120] The controller device 113a is roughly divided into three parts: a ring part 301, an operation part 302a, and a holding part 302b.

[0121] As shown in A of FIG. 5, the ring portion 301 extends upward from near the center of gravity of the left side surface 314b. When viewed from the side of the ring portion 301 (for example, the left side surface 314b of the controller device 113a), the operation portion 302a and the holding portion 302b have shapes that are symmetrical about the ring portion 301. The operation portion 302a extends forward and diagonally downward from near the center of gravity of the left side surface 314b (near the bottom end of the ring portion 301). The holding portion 302b extends backward and diagonally downward from near the center of gravity of the left side surface 314b (near the bottom end of the ring portion 301), symmetrically with respect to the operation portion 302a. When the tip of the ring portion 301, the tip of the operation portion 302a, and the tip of the holding portion 302b are connected, an isosceles triangle is formed with the tip of the ring portion 301 as the vertex. The angle between the ring portion 301 and the operating portion 302a, the angle between the ring portion 301 and the holding portion 302b, and the angle between the operating portion 302a and the holding portion 302b are each approximately 120 degrees, and the above-mentioned isosceles triangle becomes an approximately equilateral triangle.

[0122] The tip of the side surface of ring portion 301 extends linearly and widens in a curved shape at its base. The tip of the side surface of operating portion 302a extends linearly and widens in a curved shape at its base. The tip of the side surface of holding portion 302b extends linearly and widens in a curved shape at its base. The boundary between ring portion 301 and operating portion 302a, the boundary between ring portion 301 and holding portion 302b, and the boundary between operating portion 302a and holding portion 302b are all curved.

[0123] As shown in Fig. 5B, a hole 301A is formed in the ring portion 301, penetrating it in the front-rear direction. The outer periphery of the ring portion 301 gradually widens toward the tip, and the tip is curved. Similarly, the hole 301A gradually widens toward the tip, and the tip and the end are curved.

[0124] As shown in FIG. 5B , the operating portion 302a gradually tapers toward the tip, and the tip is curved. The upper surface 312a of the operating portion 302a is inclined diagonally downward and forward. A shallow groove that curves laterally and extends in the front-to-rear direction is formed on the upper surface 312a of the operating portion 302a. The tip of the upper surface 312a of the operating portion 302a is slightly recessed relative to the tip of the operating portion 302a. As a result, the upper surface 312a of the operating portion 302a has a shape that makes it easy to place the inserted finger when the user inserts the finger into the hole 301A of the ring portion 301 from back to front.

[0125] The holding portion 302b has the same shape as the operating portion 302a, and is formed with an upper surface 312b (not shown) that has the same shape as the upper surface 312a.

[0126] 5C, the lower surfaces of the operating portion 302a and the holding portion 302b form a bottom surface 313 that curves in the front-to-rear direction. A shallow groove that curves laterally and extends in the front-to-rear direction is formed in the bottom surface 313.

[0127] A rubber-like material such as silicone or elastomer is used for the inner circumferential surface 311, the top surface 312a, the top surface 312b, and the bottom surface 313 of the controller device 113a. For example, an IR-transmitting resin is used for the other parts of the controller device 113a.

[0128] 6 to 8 show examples of how the controller device 113a can be held.

[0129] For example, as shown in Fig. 6A, the index finger of the right hand is inserted from back to front into ring portion 301, and the tip of the index finger is placed near the tip of upper surface 312a of operating portion 302a, making it possible to operate operating portion 302a with the index finger. The size of hole 301A in ring portion 301 is large enough to accommodate the thickness of the index finger, so the index finger can be inserted easily. The tip of the thumb of the right hand is lightly placed near the tip of the side surface of operating portion 302a, and holding portion 302b is lightly grasped and held in the palm of the right hand.

[0130] For example, when the vicinity of the tip of operation unit 302a is pressed downward with the index finger as shown by the arrow in A of Fig. 6, the tip of holding unit 302b comes into contact with the palm of the hand, preventing controller device 113a from rotating in the pressing direction, as shown in B of Fig. 6. This prevents the vicinity of the tip of operation unit 302a from shaking in space, allowing the user to reliably press the vicinity of the tip of operation unit 302a with the orientation of the tip of operation unit 302a stable.

[0131] As described above, the controller device 113a has the same shape when viewed from the front and the back, and also the same shape when viewed from the right and the left. Therefore, the user can hold the controller device 113a without worrying about the front or back. That is, the user can hold the controller device 113a with the operation unit 302a facing the fingertips and the right side surface 314a facing the thumb, as shown in FIG. 7A. The user can also hold the controller device 113b with the holding unit 302b facing the fingertips and the left side surface 314b facing the thumb, as shown in FIG. 7B.

[0132] Hereinafter, holding the controller device 113a so that the operation unit 302a faces the fingertips as shown in Fig. 7A will be referred to as holding the controller device 113a forward. Hereinafter, holding the controller device 113a so that the holding unit 302b faces the fingertips as shown in Fig. 7B will be referred to as holding the controller device 113a backward.

[0133] When the controller device 113a is held backward, the roles of the operation unit 302a and the holding unit 302b are reversed. That is, the holding unit 302b functions as an operation unit that can be operated by the index finger of the right hand, and the operation unit 302a functions as a holding unit that is held by the palm of the right hand.

[0134] 8, even if the user lets go of the controller device 113a, the ring portion 301 catches on the index finger, preventing the controller device 113a from falling. This prevents the user from accidentally dropping the controller device 113a, even without providing a strap or the like.

[0135] Fig. 9 shows an example of the arrangement of operation members of the controller device 113a. Fig. 9A is a perspective view of the controller device 113a seen from diagonally above right. Fig. 9B is a perspective view of the controller device 113a seen from diagonally above left. Fig. 9C is a perspective view of the controller device 113a seen from diagonally below rear.

[0136] The operation members are arranged symmetrically about the ring portion 301 in the front-rear and left-right directions of the controller device 113a.

[0137] For example, the operating member 331 is disposed at the lower end of the inner circumferential surface 311 (hole 301A) of the ring portion 301. For example, the user bends his / her index finger and operates the operating member 331 with the tip of the index finger.

[0138] An operating member 332a is disposed near the tip of an upper surface 312a of the operating portion 302a. An operating member 332b is disposed near the tip of an upper surface 312b of the holding portion 302b. For example, a user operates the operating member 332a or the operating member 332b with the tip of the index finger.

[0139] An operation member 333a and an operation member 333b are respectively arranged near the front end and the rear end of the bottom surface 313. For example, the user operates the operation member 333a or the operation member 333b with the tip of the ring finger or the little finger.

[0140] An operating member 334 is disposed in the center in the front-rear direction of the bottom surface 313. For example, the user operates the operating member 334 with the tip of the thumb, ring finger, or little finger.

[0141] Any type of operation member, such as a button, a touchpad, or a joystick, can be used for operation member 331, operation member 332a, operation member 332b, operation member 333a, operation member 333b, and operation member 334. However, the same type of operation member is used for operation member 332a and operation member 332b, which are arranged symmetrically about ring portion 301. Similarly, the same type of operation member is used for operation member 333a and operation member 333b, which are arranged symmetrically about ring portion 301.

[0142] For example, any function can be assigned to the operating members 331, 332a, 332b, 333a, 333b, and 334. However, the same function is assigned to the operating members 332a and 332b, which are arranged symmetrically about the ring portion 301. Similarly, the same function is assigned to the operating members 333a and 333b, which are arranged symmetrically about the ring portion 301.

[0143] Specifically, for example, the operation member 331 is assigned a function to call a main menu screen. For example, the operation members 332a and 332b are assigned a function to select a virtual object. For example, the operation members 333a and 333b are assigned a function other than the selection function of the operation members 332a and 332b. For example, the operation member 334 is assigned a function to call a sub-menu screen.

[0144] For example, different functions may be assigned to the operation members 332a and 332b, and the functions of the two may be switched depending on the direction in which the controller device 113a is held. Similarly, for example, different functions may be assigned to the operation members 333a and 333b, and the functions of the two may be switched depending on the direction in which the controller device 113a is held.

[0145] In this way, the user can perform the same operations whether holding the controller device 113a facing forward or backward.

[0146] Although it is assumed that the index finger is inserted into the ring portion 301, it is also possible to insert, for example, the middle finger or ring finger.

[0147] Hereinafter, when there is no need to distinguish between the operating members 332a and 332b, they will simply be referred to as operating members 332. Hereinafter, when there is no need to distinguish between the operating members 333a and 333b, they will simply be referred to as operating members 333.

[0148] <Example of Marker Arrangement> For example, a marker such as an IR light-emitting element may be provided on the controller device 113a. Then, the recognition unit 221 of the information processing device 111 may detect the marker on the controller device 113a based on an image sensed by the sensing unit 252 of the terminal device 112, and recognize the relative position and orientation between the terminal device 112 and the controller device 113 based on the position of the detected marker.

[0149] 10 shows an example of the arrangement of the markers 351 of the controller device 113a. Each marker 351 is indicated by a black circle.

[0150] 10A, the markers 351 are arranged in the vertical direction on the right side surface 314a and the left side surface 314b so as to surround the outer periphery of the ring portion 301. For example, the markers 351 are arranged near the tips of both sides of the operating portion 302a and the holding portion 302b. For example, the markers 351 are arranged near the front end and the rear end of the bottom surface 313.

[0151] As a result, as shown in A to D of Figures 11, in all positions of the controller device 113a, at least a portion of the marker 351 is visible without being covered by the user's hand.

[0152] 12, the terminal device 112 is assumed to be equipped with a plurality of cameras 401. Each camera 401 constitutes the sensing unit 252 (FIG. 4) of the terminal device 112. Each camera 401 captures an image of the controller device 113a. The terminal device 112 transmits sensing data including captured image data obtained by capturing the image to the information processing device 111.

[0153] In response, the control unit 202 of the information processing device 111 receives the sensing data. The recognition unit 221 of the control unit 202 recognizes the position and orientation of the controller device 113a relative to the terminal device 112 based on the light emission pattern of the marker 351 of the controller device 113a.

[0154] 10B, the markers 351 may be arranged in two rows in the horizontal direction so as to surround the outer periphery of the ring portion 301. Alternatively, as shown in FIG. 10C, the markers 351 may be arranged in three rows in the horizontal direction so as to surround the outer periphery of the ring portion 301.

[0155] In this way, by arranging the markers 351 on the outer periphery of the ring portion 301, the controller device 113a can be made smaller.

[0156] <Example of Internal Structure of Controller Device 113a> Next, an example of the internal structure of the controller device 113a will be described with reference to FIG.

[0157] A haptic device 371, a haptic device 372a, a haptic device 372b, a circuit board 373, and a battery 374 are built into the controller device 113a.

[0158] The haptic device 371, the haptic device 372a, and the haptic device 372b are configured by devices that present (transmit) tactile stimuli such as vibrations, for example, a linear resonant actuator (LRA), an eccentric rotating mass (ERM), or a piezoelectric element.

[0159] The tactile device 371 is disposed near the lower end of the inner circumferential surface 311 of the ring portion 301 (near the operating member 331 ( FIG. 9 )), and presents a tactile stimulus near the lower end of the inner circumferential surface 311 .

[0160] The tactile device 372a is disposed near the tip of the operation unit 302a (near the operation member 332a (FIG. 9)) and transmits tactile stimulation to the tip of the operation unit 302a.

[0161] The tactile device 372b is disposed near the tip of the holding portion 302b (near the operating member 332b (FIG. 9)) and transmits a tactile stimulus to the tip of the holding portion 302b.

[0162] The board 373 is a board for controlling the controller device 113 a, and is disposed below the haptic device 371, approximately in the center of the controller device 113 a.

[0163] The battery 374 is disposed below the substrate 373 within the controller device 113a, and supplies power to each part of the controller device 113a.

[0164] 14 , when the controller device 113a is held in the user's right hand facing forward, tactile devices 371 and 372a present tactile stimuli near the base of the thumb and the tip of the index finger, respectively. Tactile devices 372b present tactile stimuli near the base of the thumb and the palm.

[0165] The tactile devices 371, 372a, and 372b are arranged symmetrically in the front-to-back direction of the controller device 113a around the ring portion 301. Therefore, whether the user holds the controller device 113a in the front or back direction, the same tactile stimulation is presented to the user's hand.

[0166] <Processing of XR System 101> Next, processing of the XR system 101 will be described with reference to FIGS.

[0167] <Operation Member Control Processing> First, the operation member control processing executed by the XR system 101 will be described with reference to the flowchart of FIG.

[0168] This process is executed, for example, when the user picks up or changes the controller device 113a.

[0169] In step S1, the information processing device 111 performs hand recognition by hand tracking.

[0170] For example, the control unit 253 of the terminal device 112 transmits sensing data including captured image data indicating images captured by each camera 401 to the information processing device 111 .

[0171] In response, the control unit 202 of the information processing device 111 receives the sensing data. The recognition unit 221 of the control unit 202 performs hand recognition by hand tracking based on the captured image data included in the sensing data. As a result, for example, the recognition unit 221 tracks the hand of the user holding the controller device 113a based on the marker 351 provided on the controller device 113a.

[0172] In step S2, the recognition unit 221 determines whether or not the hand holding the controller device 113a has been recognized based on the result of the processing in step S1. If it is determined that the hand holding the controller device 113a has not been recognized, the processing returns to step S1.

[0173] Thereafter, the processes of steps S1 and S2 are repeatedly executed until it is determined in step S2 that the hand holding the controller device 113a has been recognized.

[0174] On the other hand, if it is determined in step S2 that the hand holding the controller device 113a has been recognized, the processing proceeds to step S3.

[0175] In step S3, the recognition unit 221 recognizes the light emission pattern of the controller device 113a based on the captured image data. That is, the recognition unit 221 recognizes the light emission pattern of the marker 351 that is not hidden by the user's hand on the controller device 113a.

[0176] In step S4, the recognition unit 221 determines whether or not the holding direction of the controller device 113a has been recognized. Specifically, the recognition unit 221 attempts to recognize the holding direction of the controller device 113a based on the recognition result of the hand with which the user is holding the controller device 113a and the recognition result of the light emission pattern of the controller device 113a. If it is determined that the holding direction of the controller device 113a has not been recognized, the processing returns to step S3.

[0177] Thereafter, the processes of steps S3 and S4 are repeatedly executed until it is determined in step S4 that the holding direction of the controller device 113a has been recognized.

[0178] On the other hand, if it is determined in step S4 that the holding direction of the controller device 113a has been recognized, the processing proceeds to step S5.

[0179] In step S5, the operation control unit 222 disables the operation member on the palm side. For example, as shown in Fig. 16, when the controller device 113a is held facing forward, the operation member 332b on the palm side is disabled. After that, for example, the recognition unit 221 and the operation control unit 222 ignore the operation input signal of the operation member 332b.

[0180] On the other hand, for example, when the controller device 113a is held backwards, the operation member 332a on the palm side is disabled.

[0181] Thereafter, the operation member control process ends.

[0182] This prevents the operating member 332 from being accidentally operated by the palm of the user's hand.

[0183] As described above, the hand holding the controller device 113a and the holding direction are recognized, and the operability of the controller device 113a does not change regardless of the holding direction of the controller device 113a.

[0184] Therefore, for example, as shown in A and B of Figure 17, without making any special settings on the terminal device 112 side, the user can hold and use the controller device 113a with their dominant hand, regardless of their dominant hand.

[0185] For example, as shown in A and B of FIG. 18, the user can also wear and use another controller device 113b, such as a ring-shaped device, on the non-dominant hand.

[0186] Also, for example, as shown in FIG. 19, the user can wear controller devices 113a on both hands.

[0187] <Haptic Feedback Control Processing> Next, the haptic feedback control processing executed by the XR system 101 will be described with reference to the flowchart of FIG.

[0188] This process starts, for example, when the information processing device 111 is powered on and ends when it is powered off.

[0189] In step S51, the information processing device 111 recognizes the state of the terminal device 112 and the surrounding area.

[0190] Specifically, the sensing unit 252 of the terminal device 112 senses the state of the terminal device 112 and the state of the surroundings of the terminal device 112, and supplies sensing data indicating the sensing results to the control unit 253. The control unit 253 transmits the sensing data to the information processing device 111.

[0191] In response to this, the control unit 202 of the information processing device 111 receives the sensing data.

[0192] The controller device 113 a transmits a controller signal including an operation input signal indicating the operation content for each operation member to the information processing device 111 via the terminal device 112 .

[0193] In response to this, the control unit 202 of the information processing device 111 receives the controller signal.

[0194] The recognition unit 221 of the control unit 202 recognizes the state of the terminal device 112, the state of the surroundings of the terminal device 112, the state of the controller device 113, the state of the user, user operations, etc. based on the sensing data and the controller signal. For example, the recognition unit 221 recognizes the position and orientation of the terminal device 112. For example, the recognition unit 221 recognizes the direction of the user's line of sight. For example, the recognition unit 221 recognizes the position and orientation of the controller device 113a relative to the terminal device 112. For example, the recognition unit 221 recognizes the content of operations performed on the controller device 113a.

[0195] In step S52, the space control unit 223 of the information processing device 111 controls the XR space. Specifically, the space control unit 223 generates virtual objects to be displayed in the XR space based on at least a portion of the recognition results by the recognition unit 221, and performs various calculations necessary for constructing and displaying the XR space, such as the behavior of the virtual objects. The space control unit 223 generates display control information for controlling the display of the XR space based on the calculation results, and transmits this information to the terminal device 112 via the communication unit 205, thereby controlling the display of the XR space by the terminal device 112.

[0196] The recognition unit 221 recognizes the type, position, orientation, etc. of virtual objects around the terminal device 112 (user) based on information from the space control unit 223, etc.

[0197] In step S53, the tactile presentation control unit 225 determines whether or not it is time to present tactile feedback based on at least one of the recognition result by the recognition unit 221 and information from the spatial control unit 223. If it is determined that it is not time to present tactile feedback, the process returns to step S51.

[0198] Thereafter, the processes of steps S51 to S53 are repeatedly executed until it is determined in step S53 that it is time to present haptic feedback.

[0199] On the other hand, if it is determined in step S53 that it is time to present haptic feedback, the process proceeds to step S54.

[0200] In step S54, the information processing device 111 controls the presentation of haptic feedback. Specifically, the haptic presentation control unit 225 generates haptic control information for causing the controller device 113a to present a haptic stimulus. The haptic presentation control unit 225 transmits a haptic control signal to the controller device 113a via the terminal device 112.

[0201] In response, the controller device 113a receives the haptic control information. Each haptic device of the controller device 113a presents a haptic stimulus based on the haptic control information.

[0202] Thereafter, the process returns to step S51, and the processes from step S51 onwards are executed.

[0203] In this way, the controller device 113a appropriately presents a tactile stimulus to the user.

[0204] Here, examples of methods for presenting haptic feedback by the controller device 113a will be described with reference to FIGS.

[0205] For example, when the operating member 332a (FIG. 9) near the tip of the operating unit 302a of the controller device 113a is configured as a touchpad, and the operating member 332a is slid back and forth with the tip of the index finger as shown in A of FIG. 21, a tactile stimulus is presented to the tip of the index finger by the tactile device 372a (FIG. 13) arranged near the operating member 332a.

[0206] For example, as shown in B of FIG. 21, when the user touches a button 431 in the XR space with the tip of the operation unit 302a of the controller device 113a, a tactile stimulus is presented to the tip of the index finger by the haptic device 372a (FIG. 13).

[0207] For example, when the controller device 113a or the hand holding the controller device 113a collides with a virtual object in the XR space, the impact of the collision is expressed using each haptic device of the controller device 113a.

[0208] For example, A in Fig. 22 shows an example where the tip of the operation unit 302a of the controller device 113a collides with a virtual object 441 in XR space from above. In this case, for example, an upward vibration is presented by the haptic device 372a (Fig. 13) near the tip of the operation unit 302a, and a downward vibration is presented by the haptic device 372b (Fig. 13) near the tip of the holding unit 302b. This makes it possible for the user to feel an upward rotational force (moment) on the controller device 113a.

[0209] For example, B in Fig. 22 shows an example where the tip of the operation unit 302a of the controller device 113a collides with a virtual object 441 in the XR space from below. In this case, for example, a downward vibration is presented by the haptic device 372a (Fig. 13) near the tip of the operation unit 302a, and an upward vibration is presented by the haptic device 372b (Fig. 13) near the tip of the holding unit 302b. This makes it possible for the user to feel a downward rotational force (moment) on the controller device 113a.

[0210] For example, Figure 23 shows an example where the tip of the operation unit 302a of the controller device 113a collides head-on with a virtual object 441 in XR space. In this case, for example, the haptic device 371 (Figure 13) near the center of the controller device 113a is vibrated, thereby vibrating the entire controller device 113a. This makes it possible for the user to feel the reaction force from the virtual object 441 acting on the controller device 113a.

[0211] In this way, the operability of the controller device 113a can be improved, resulting in improved operability in the XR space.

[0212] <<2. Modifications>> Modifications of the above-described embodiments of the present technology will now be described.

[0213] <Modifications of Controller Device 113a> In the above description, an example has been shown in which the controller device 113a can be held in either the forward or backward direction, but it is also possible to make it possible to hold it only facing forward, for example.

[0214] In this case, the operating portion 302a and the holding portion 302b do not necessarily have to be symmetrical about the ring portion 301. For example, the operating portion 302a and the holding portion 302b may have different shapes. Also, it is possible to eliminate the operating members 332b and 333b of the holding portion 302b.

[0215] For example, materials other than resin, such as metal, can be used for the controller device 113a.

[0216] <Modifications Regarding Sharing of Processing> For example, some of the processing of the information processing device 111 may be executed by the terminal device 112 .

[0217] For example, the terminal device 112 may execute all or part of the processing of the information processing unit 211 of the information processing device 111. For example, the terminal device 112 may independently present an XR space without being controlled by the information processing device 111. For example, the information processing device 111 and the terminal device 112 may independently share and execute processing such as construction of an XR space.

[0218] <Other Modifications> For example, the controller device 113a can be used not only for XR space, but also for operations in two-dimensional and three-dimensional spaces such as games.

[0219] <<3. Others>> <Example of Computer Configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs that make up the software are installed on a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, for example, that can execute various functions by installing various programs.

[0220] FIG. 24 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0221] In the computer 2000 , a CPU (Central Processing Unit) 2001 , a ROM (Read Only Memory) 2002 , and a RAM (Random Access Memory) 2003 are interconnected by a bus 2004 .

[0222] An input / output interface 2005 is further connected to the bus 2004. To the input / output interface 2005, an input unit 2006, an output unit 2007, a storage unit 2008, a communication unit 2009, and a drive 2010 are connected.

[0223] The input unit 2006 includes input switches, buttons, a microphone, an image sensor, etc. The output unit 2007 includes a display, a speaker, etc. The storage unit 2008 includes a hard disk, a nonvolatile memory, etc. The communication unit 2009 includes a network interface, etc. The drive 2010 drives removable media 2011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0224] In the computer 2000 configured as described above, the CPU 2001 performs the above-described series of processes by, for example, loading a program recorded in the memory unit 2008 into the RAM 2003 via the input / output interface 2005 and the bus 2004 and executing it.

[0225] The program executed by the computer 2000 (CPU 2001) can be provided by being recorded on a removable medium 2011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0226] In the computer 2000, the program can be installed in the storage unit 2008 via the input / output interface 2005 by inserting the removable medium 2011 into the drive 2010. The program can also be received by the communication unit 2009 via a wired or wireless transmission medium and installed in the storage unit 2008. Alternatively, the program can be installed in advance in the ROM 2002 or the storage unit 2008.

[0227] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0228] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0229] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0230] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0231] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0232] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0233] <Examples of Combinations of Configurations> The present technology can also have the following configurations.

[0234] (1) An input device comprising: a ring portion into which a finger is inserted; an operation portion operable by the finger inserted into the ring portion in a first direction; and a holding portion held by the palm of a hand when the operation portion is operated by the finger. (2) The input device described in (1), wherein the operation portion and the holding portion have shapes symmetrical about the ring portion when viewed from a side of the ring portion. (3) The input device described in (2), wherein the holding portion is operable by the finger inserted into the ring portion in a second direction opposite to the first direction, and the operation portion is held by the palm of the hand when the holding portion is operated by the finger. (4) The input device described in (3), further comprising: a first operation member arranged near a tip of the operation portion; and a second operation member arranged near a tip of the holding portion. (5) The input device described in (4), further comprising: a first tactile device arranged in the vicinity of the first operation member; and a second tactile device arranged in the vicinity of the second operation member. (6) The input device according to (4) or (5), further comprising: an operating member arranged symmetrically about the ring portion on a plane formed by a lower surface of the operating unit and a lower surface of the holding unit. (7) The input device according to any of (4) to (6), wherein the operating unit and the holding unit present a tactile stimulus. (8) The input device according to any of (4) to (7), further comprising: a plurality of markers, some of which are arranged along the outer periphery of the ring unit, and some of which are arranged on both side surfaces of the operating unit and both side surfaces of the holding unit. (9) The input device according to (8), wherein some of the markers are arranged on a plane formed by the lower surface of the operating unit and the lower surface of the holding unit. (10) The input device according to any of (2) to (9), wherein, when a tip of the ring unit is directed upward, the operating unit extends from a vicinity of a lower end of the ring unit in the first direction and diagonally downward, and the holding unit extends from a vicinity of the lower end of the ring unit in a second direction opposite to the first direction and diagonally downward.(11) The input device according to any one of (1) to (10), further comprising an operation member arranged near a lower end of a hole in the ring portion. (12) The input device according to (11), further comprising a tactile device arranged near the operation member. (13) The input device according to any one of (1) to (12), wherein the operation member has an inclined surface that slopes diagonally downward in the first direction from near a lower end of the ring portion when the tip of the ring portion is facing upward, and further comprises an operation member arranged on the inclined surface. (14) The input device according to (13), further comprising a tactile device arranged near the operation member. (15) A control device comprising: a recognition unit that recognizes a position and orientation of an input device that includes a ring portion into which a finger of a user's hand is inserted, an operation unit operable by the finger inserted into the ring portion in a first direction, and a holding unit that is held by the palm of the hand when the operation unit is operated by the finger; and an operation control unit that controls an operation by the input device based on the position and orientation of the input device. (16) The control device according to (15), wherein the operation unit and the holding unit have shapes symmetrical about the ring unit when viewed from a side of the ring unit, the holding unit can be operated by the finger inserted into the ring unit in a second direction opposite to the first direction, the operation unit is held by the palm of the hand when the holding unit is operated by the finger, the recognition unit recognizes a direction in which the input device is held, and the operation control unit controls an operation on the operation unit and the holding unit based on the direction in which the input device is held. (17) The control device according to (15) or (16), wherein the recognition unit recognizes the content of an operation performed by the input device based on an operation input signal output from the input device and at least one of a position and an attitude of the input device. (18) The control device according to (17), further comprising a tactile presentation control unit that controls presentation of a tactile stimulus by the input device based on the content of the operation.(19) A control method that recognizes a position and orientation of an input device that includes a ring portion into which a finger of a user is inserted, an operation portion operable by the finger inserted into the ring portion in a first direction, and a holding portion to be held by the palm of the hand when the operation portion is operated by the finger, and controls operation by the input device based on the position and orientation of the input device. (20) An information processing device that receives an operation input signal output from an input device that includes a ring portion into which a finger is inserted, an operation portion operable by the finger inserted into the ring portion, and a holding portion to be held by the palm of the hand when the operation portion is operated by the finger, and outputs display control information that controls display of an XR space based on the operation input signal. (21) An information processing device comprising: a control unit that receives input information indicating at least one of a state of a terminal device that presents an XR space to a user, a state around the terminal device, a state of the user, behavior of the user, and an operation input to an input device used for operation input in the XR space, executes an application based on the input information, and outputs output information that controls display of a virtual object that includes CAD (Computer Aided Design) information in the XR space. (22) An information processing method in which an information processing device receives input information indicating at least one of a state of a terminal device that presents an XR space to a user, a state around the terminal device, a state of the user, behavior of the user, and an operation input to an input device used for operation input in the XR space, executes an application based on the input information, and outputs output information that controls display of a virtual object that includes CAD information in the XR space. (23) An information processing device including a control unit that executes an application for presenting an XR space to a user and outputs output information indicating a change in or an abnormality in the state of the application to a terminal device that presents the XR space to the user or an input device used for inputting operations in the XR space.(24) An information processing method, in which an information processing device executes an application for presenting an XR space to a user, and outputs output information indicating a change in or an abnormality in the state of the application to a terminal device that presents the XR space to the user or an input device used for inputting operations in the XR space.

[0235] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0236] 101 XR system, 111 information processing device, 112 terminal device, 113, 113a, 113b controller device, 202 control unit, 203 display unit, 211 information processing unit, 221 recognition unit, 222 operation control unit, 223 space control unit, 224 audio control unit, 225 tactile presentation control unit, 226 learning unit, 252 sensing unit, 253 control unit, 254 display unit, 255 audio output unit, 301 ring unit, 301A hole, 302a operation unit, 302b holding unit, 312a, 312b top surface, 313 bottom surface, 331 to 334 operation member, 351 marker, 371 to 372b tactile device, 401 camera

Claims

1. A ring portion, a first portion operable by a finger inserted into the ring portion in a first direction; a second part that is held by the palm of the hand to which the finger inserted in the first direction belongs; and an input device comprising: Including system.

2. the second portion is operable by the finger inserted into the ring portion in a second direction opposite to the first direction, The first portion is held by the palm of the hand to which the finger inserted in the second direction belongs. The system of claim 1 .

3. The input device is a first operating member disposed near a tip of the first portion; a second operating member disposed near a tip of the second portion; and Further equipped The system of claim 2 .

4. The first portion and the second portion present a tactile stimulus. The system of claim 3.

5. The input device is a first tactile device disposed near the first operation member; a second tactile device disposed near the second operation member; Further equipped The system of claim 4.

6. The input device is a third operating member disposed on a lower surface of the first portion; a fourth operating member disposed on a lower surface of the second portion; Further equipped The system of claim 3.

7. The input device includes a plurality of markers; Some of the markers are arranged along the outer periphery of the ring portion, A part of the marker is disposed on both side surfaces of the first part and both side surfaces of the second part. The system of claim 2 .

8. An information processing device that controls display based on an operation input signal output from the input device. The system of claim 2 further comprising:

9. The information processing device determines whether to execute control corresponding to operations on the first part and the second part based on which of the first part and the second part is held in the palm of a hand. The system of claim 8.

10. The information processing device determining not to execute a control corresponding to an operation on the first portion when the first portion is held in a palm of a hand; determining not to execute a control corresponding to an operation on the second portion when the second portion is held in a palm of a hand; The system of claim 9.

11. When the tip of the ring portion is in the upward direction and the first direction is in the forward direction, at least one of the two-dimensional images obtained by orthogonal projection from the top, bottom, left, right, front, and back of the input device has line symmetry. The system of claim 1 .

12. The input device is plane-symmetrical with respect to a plane passing through the center of the hole in the ring portion in the thickness direction. The system of claim 11.

13. The input device extends in the vertical direction on a plane passing through the center of the hole in the ring portion in the thickness direction, and is rotationally symmetrical by 180 degrees with respect to an axis of symmetry that bisects the hole. The system of claim 11.

14. The first portion extends from near the lower end of the ring portion in the first direction and diagonally downward, and the second portion extends from near the lower end of the ring portion in a second direction opposite to the first direction and diagonally downward. The system of claim 11.

15. The input device further includes an operating member disposed near a lower end of the hole in the ring portion. The system of claim 1 .

16. The input device further includes a tactile device disposed near the operating member. The system of claim 15.

17. the first portion has an inclined surface that is inclined obliquely downward in the first direction from a vicinity of a lower end of the ring portion when the tip of the ring portion is in an upward direction, The input device further includes an operating member disposed on the inclined surface. The system of claim 1 .

18. The input device further comprises a tactile device disposed near the operating member.

20. The system of claim 17.

19. A recognition unit that recognizes the position and orientation of an input device that includes a ring portion, a first portion that can be operated by a finger inserted into the ring portion in a first direction, and a second portion that is held by the palm of the hand to which the finger inserted into the ring portion in the first direction belongs; an operation control unit that controls an operation by the input device based on the position and orientation of the input device; A control device comprising:

20. The second part can be operated by a finger inserted into the ring portion in a second direction opposite to the first direction, the first portion is held by the palm of a hand to which a finger inserted in the ring portion in the first direction belongs, the recognition unit recognizes which of the first part and the second part is held in the palm of the hand; The operation control unit determines whether or not to execute control corresponding to operations on the first part and the second part, based on which of the first part and the second part is held in the palm of the hand.

20. The control device of claim 19.

21. The recognition unit recognizes the operation content of the input device based on an operation input signal output from the input device and at least one of a position and an orientation of the input device.

20. The control device of claim 19.

22. a tactile presentation control unit that controls presentation of a tactile stimulus by the input device based on the operation content; The control device of claim 21 further comprising:

23. A method for recognizing a position and an attitude of an input device having a ring portion, a first portion operable by a finger inserted in the ring portion, and a second portion held by the palm of a hand to which the finger inserted in the ring portion belongs, Controlling an operation by the input device based on the position and orientation of the input device Control method.

24. A display of an XR space is controlled based on an operation input signal output from an input device having a ring portion, a first portion operable by a finger inserted into the ring portion, and a second portion held by the palm of the hand to which the finger inserted into the ring portion belongs. Information processing device.

25. a control unit that receives input information indicating at least one of the state of a terminal device that presents an XR space to a user, the state of the surroundings of the terminal device, the state of the user, the behavior of the user, and an operation input to an input device used for operation input in the XR space, executes an application based on the input information, and outputs output information that controls the display of a virtual object including CAD (Computer Aided Design) information in the XR space; An information processing device.

26. The information processing device receiving input information indicating at least one of a state of a terminal device presenting an XR space to a user, a state around the terminal device, a state of the user, a behavior of the user, and an operation input to an input device used for operation input in the XR space; Execute an application based on the input information; outputting output information for controlling the display of a virtual object including CAD information in the XR space; Information processing methods.

27. A control unit that executes an application for presenting an XR space to a user and outputs output information indicating a change in or an abnormality in the state of the application to a terminal device that presents the XR space to the user or an input device used for inputting operations in the XR space. An information processing device.

28. The information processing device Execute an application for presenting an XR space to a user; Output information indicating a change in or an abnormality in the state of the application is output to a terminal device that presents the XR space to the user or to an input device used for inputting operations in the XR space. Information processing methods.