Information processing apparatus and information processing method

Abstract

The present technology relates to an information processing device and an information processing method for improving operability of an operation related to indication of a position in an XR (cross reality) space. The information processing apparatus includes: an identification unit that identifies a distance to a virtual object in an XR space; and a space control unit that controls a method for indicating a position in the XR space through the input device based on the distance from the virtual object. This technology is applicable to, for example, an XR system.

Description

Technical Field

[0001] The present technology relates to an information processing device and an information processing method, and more particularly, to an information processing device and an information processing method suitable for indicating a position in a cross reality (XR) space. Background Art

[0002] A pen input device or the like is conventionally used to indicate a position in the XR space (eg, see Patent Document 1).

[0003] Citation List

[0004] Patent Literature

[0005] Patent document 1: WO2019 / 220803A Summary of the invention

[0006] Problems to be solved by the present invention

[0007] Meanwhile, with the popularization of XR technology in recent years, there is an increasing demand for technology that makes it easier to indicate a position in the XR space.

[0008] The present technology has been developed in view of such circumstances, and an object of the present technology is to improve the operability of operations related to indicating a position in an XR space.

[0009] Solution to the problem

[0010] An information processing apparatus according to one aspect of the present technology includes: a recognition unit that recognizes a distance to a virtual object in an XR space; and a space control section that controls a method for indicating a position in the XR space using an input device based on the distance to the virtual object.

[0011] An information processing method according to one aspect of the present technology is an information processing method performed by an information processing apparatus, the information processing method including: identifying a distance to a virtual object in an XR space; and controlling a method for indicating a position in the XR space using an input device based on the distance to the virtual object.

[0012] In aspects of the present technology, a distance to a virtual object in XR space is identified, and based on the distance to the virtual object, a method for specifying a position in the XR space using an input device is controlled. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a diagram showing an embodiment of an XR system to which the present technology is applied.

[0014] Figure 2 is a diagram showing a display example of an XR system.

[0015] Figure 3 is a diagram showing another display example of the XR system.

[0016] Figure 4 is a block diagram showing a configuration example of an information processing device and a terminal device.

[0017] Figure 5 is an external view showing a configuration example of a controller device.

[0018] Figure 6 is a diagram illustrating a method for maintaining a controller device.

[0019] Figure 7 is a diagram showing another method for maintaining a controller device.

[0020] Figure 8 is a diagram showing another method for maintaining a controller device.

[0021] Fig. 9 is a diagram showing an example of arrangement of operating members of a controller device.

[0022] Fig.10 is a diagram showing an arrangement example of symbols of a controller device.

[0023] Fig.11 is a diagram showing an example of how the indicia of a controller device may look.

[0024] Fig.12 A diagram for explaining a method of recognizing the position and posture of a controller device.

[0025] Fig.13 is a diagram showing an example of the internal configuration of a controller device.

[0026] Fig.14 is a diagram showing an arrangement example of a haptic device of a controller device.

[0027] Fig.15 is a flowchart for explaining a process performed by the XR system for controlling an operating member.

[0028] Fig.16 is a diagram for explaining a process for controlling an operating member executed by the XR system.

[0029] Fig.17 is a diagram illustrating an example of a method for maintaining a controller device.

[0030] Fig.18 is a diagram showing other examples of a method for maintaining a controller device.

[0031] Fig.19 is a diagram illustrating another example of a method for maintaining a controller device.

[0032] Fig. 20 is a flow chart illustrating a process performed by an XR system for controlling haptic feedback.

[0033] Fig.21 is a diagram for explaining an example of tactile feedback.

[0034] Fig. 22 is a diagram for explaining another example of tactile feedback.

[0035] Fig.23 is a diagram for explaining another example of tactile feedback.

[0036] Fig.24 is a diagram showing an example of assembly.

[0037] Fig.25 is a flow chart for illustrating a first embodiment of a process performed by the XR system for controlling activation.

[0038] Fig.26 is a diagram showing a display example of a designated point.

[0039] Fig. 27 is a diagram showing a display example of rays.

[0040] Fig.28 is a diagram showing a display example of a designated point and a ray.

[0041] Fig.29 is a diagram for explaining a method for activating all components.

[0042] Fig.30 is a diagram showing a display example of activated components and inactivated components.

[0043] Fig.31 A diagram for explaining a method of activating a component using a gesture.

[0044] Fig.32 is a flow chart for illustrating a second embodiment of a process performed by the XR system to control activation.

[0045] Fig.33 It is a diagram for explaining the second embodiment of the process for controlling activation.

[0046] Fig.34 is a block diagram showing a configuration example of a computer. DETAILED DESCRIPTION

[0047] The following will describe a mode for implementing the present technology. The description will be made in the following order.

[0048] 1. Implementation Method

[0049] 2. Modified Example

[0050] 3. Others

[0051] <<1. Embodiment>>

[0052] The embodiments of the present technology will be described with reference to Figures 1 to 33 the following.

[0053] <Configuration Example of XR System 101>

[0054] Figure 1 A configuration example of a cross-reality (XR) system 101 is shown. The XR system 101 is an embodiment of an information processing system to which the present technology is applied.

[0055] The XR system 101 is a system that implements XR. XR is a technology that fuses the real world and the virtual world and includes virtual reality (VR), augmented reality (AR), mixed reality (MR), and substitute reality (SR). The XR system 101 is a system that presents to a user a space (hereinafter referred to as an XR space) obtained by fusing the real space and the virtual space. For example, the XR system 101 can present to a user a non-real virtual object (hereinafter referred to as a virtual object), such as a model created by computer-aided design (CAD) (hereinafter referred to as a CAD model), as if the virtual object were present on-site.

[0056] The XR system 101 includes an information processing device 111, a terminal device 112, and a controller device 113.

[0057] The information processing device 111 and the terminal device 112 can communicate with each other wirelessly or wiredly and send and receive data to and from each other. The terminal device 112 and the controller device 113 can communicate with each other wirelessly or wiredly and send and receive data to and from each other. The information processing device 111 and the controller device 113 communicate with each other via the terminal device 112 and send and receive data to and from each other.

[0058] For example, the information processing device 111 can independently receive a user's operation and present various types of information such as visual information and auditory information to the user.

[0059] In addition, for example, the information processing device 111 controls the terminal device 112 by executing a predetermined application (hereinafter referred to as an XR application) and controls the terminal device 112 to present the XR space to the user. For example, the information processing device 111 executes the XR application to control the output of various types of information such as visual information and auditory information in the terminal device 112 and constructs the XR space presented by the terminal device 112.

[0060] Figure 1The following example is shown, in which the information processing device 111 includes a personal computer (PC) including an operation input unit including a mouse and a keyboard. For example, the information processing device 111 may include another information processing device such as a smart phone or a tablet terminal. For example, the information processing device 111 may include a plurality of information processing devices. For example, the information processing device 111 may include a system constructed by cloud computing on a network.

[0061] The terminal device 112 is a device that presents an XR space to a user.

[0062] Figure 1 The following example is shown, in which the terminal device 112 is a head-mounted display device that can be mounted on the user's head and includes a head-mounted display (HMD) as a device for presenting an XR space to the user. More specifically, Figure 1 An example is shown in which the terminal device 112 is a non-transmissive HMD that covers the user's field of view.

[0063] For example, the terminal device 112 includes a video see-through HMD, which has an imaging function of capturing an image of a real space based on the user's viewpoint, and is capable of presenting to the user a composite image obtained by combining a real image obtained by capturing the image of the real space with an image of a virtual space such as computer graphics (CG) (hereinafter referred to as a virtual image).

[0064] For example, the terminal device 112 includes: a left imaging unit and a right imaging unit corresponding to the left eye and the right eye of the user, respectively; and a left display unit and a right display unit corresponding to the left eye and the right eye of the user, respectively.

[0065] For example, the left imaging unit and the right imaging unit constitute a stereo camera device, and capture images in the direction of the user's line of sight (hereinafter referred to as field of view images) from multiple viewpoints corresponding to the user's left eye and right eye. That is, the left imaging unit and the right imaging unit capture images of objects in the real space (hereinafter referred to as real objects) viewed from the user's viewpoint.

[0066] The left display unit and the right display unit can display different images for the left eye and the right eye, respectively, and can present a three-dimensional virtual object by displaying images with parallax for the left eye and the right eye. For example, the left display unit and the right display unit display the left field image and the right field image captured by the left imaging unit and the right imaging unit, respectively.

[0067] Note that the terminal device 112 may include, for example, another terminal device for XR, such as a smartphone set in, for example, AR glasses or goggles for use, etc. In addition, for example, a display device such as a spatial reproduction display may be used instead of the terminal device 112 .

[0068] The controller device 113 is used to operate and input (hereinafter referred to as operation input) the XR space presented to the user by the terminal device 112. For example, the user can use the controller device 113 to perform various operations on the virtual object displayed on the terminal device 112.

[0069] For example, the controller device 113 uses at least one of the operating members (such as buttons or sensors) to detect at least one of the operation input by the user or the user's behavior (such as gestures). The controller device 113 sends a signal (hereinafter referred to as a controller signal) to the information processing device 111 via the terminal device 112, and the signal includes at least one of the operation input signal indicating the operation input by the user or the behavior signal indicating the user's behavior.

[0070] Furthermore, for example, the controller device 113 a includes a haptic device that presents tactile stimulation such as vibration, and presents the tactile stimulation to the user under the control of the information processing apparatus 111 or the terminal apparatus 112 .

[0071] The controller device 113 includes, for example, one or more types of input devices such as a controller, a ring-shaped input device, a pointing device, and a six-degree-of-freedom (6DoF) input device.

[0072] For example, the controller is an input device held by a user's hand. For example, the controller may include an operating member (such as a button) that the user can operate. For example, the user can perform a selection operation, a determination operation, a scroll operation, etc. on a virtual object displayed on the terminal device 112 by pressing a button of the controller. In addition, the controller may include, for example, a touch sensor and a motion sensor.

[0073] Note that the controller is not limited to being held by the user's hand, and may be worn on one of the user's body parts (including the elbow, arm, knee, ankle, and thigh), for example.

[0074] The ring device is a ring-shaped input device worn on the user's finger. For example, the ring device may include an operating member such as a button that the user can operate. For example, the user can change the position and posture of a virtual object (e.g., a three-dimensional model) in an XR space with six degrees of freedom (6DoF) by operating the ring device.

[0075] A pointing device is an input device capable of indicating any position in the XR space. For example, the information processing device 111 identifies the 6DoF position and orientation of the pointing device via the terminal device 112 using a tracking method such as the bright spot tracking method, the porcelain tracking method, or the ultrasonic tracking method.

[0076] A 6DoF input device is, for example, an input device capable of performing 6DoF operations.

[0077] For example, when viewing various objects (display objects) displayed on the information processing device 111 or the terminal device 112, the user can perform operation inputs using the controller device 113.

[0078] Note that the type and number of the controller device 113 are not particularly limited. For example, as the controller device 113, an input device other than the above types can be used, or an input device obtained by combining multiple types of input devices can be used instead.

[0079] For example, the XR system 101 can be applied to various fields such as the manufacturing field and the medical field.

[0080] For example, the XR system 101 can perform product design support and assembly support in the manufacturing field. For example, in the product design stage, the user can freely edit a three-dimensional object as a virtual object using the XR system 101, or can compare the virtual object with the real world to grasp the design result and design in advance before trial production.

[0081] For example, the XR system 101 can support surgeries and education in the medical field. For example, using the XR system 101, the user can display the state inside the patient's body on the body surface to grasp the surgical site in advance or perform training.

[0082] Note that, for example, when the XR space is shared by multiple users, for example, in the XR system 101, the terminal device 112 and the controller device 113 are provided for each user.

[0083] <Display Examples in the XR System 101>

[0084] Here, reference will be made to Figure 2 and Figure 3 to describe the display examples of the display objects in the XR system 101.

[0085] Figure 2 and Figure 3 show the display examples of the display objects in the XR system 101 in the case of creating a CAD model.

[0086] For example, as Figure 2As shown in A, the information processing device 111 displays a two-dimensional CAD model, and the user can edit the two-dimensional CAD model.

[0087] For example, Figure 2 As shown in B, the terminal device 112 displays a three-dimensional CAD model, and the user can edit the three-dimensional CAD model.

[0088] For example, Figure 2 As shown in C, the terminal device 112 displays a two-dimensional object, such as a design drawing or an instruction manual, and the user can check the design drawing, instruction manual, etc.

[0089] Figure 3 A display example of the XR space of the terminal device 112 is shown.

[0090] The display 151, keyboard 152, mouse 153 and table 154 of the information processing device 111 are displayed as a video perspective based on a real image of the real space. On the other hand, the two-dimensional image of the terminal device 112 is superimposed on the display 151 as a virtual monitor. For example, a two-dimensional CAD model to be designed is displayed on the virtual monitor. For example, from the perspective of high accuracy of position detection and easy position maintenance, the two-dimensional CAD model displayed on the virtual monitor is preferably operated using the keyboard 152 and the mouse 153.

[0091] Furthermore, in this example, the terminal device 112 displays a three-dimensional CAD model 155 to be designed on the display 151 .

[0092] The CAD model 155 is operated by, for example, controller device 113a held by the user's dominant hand (the right hand in this example) and controller device 113b which is a ring-shaped device worn on the index finger of the user's non-dominant hand (the left hand in this example).

[0093] For example, the information processing device 111 recognizes the position, posture, and behavior of the user's hand holding the controller device 113a and the user's hand wearing the controller device 113b by performing hand tracking based on the image captured by the imaging unit included in the terminal device 112. In addition, for example, the information processing device 111 receives controller signals from the controller device 113a and the controller device 113b via the terminal device 112, and recognizes operations performed on the CAD model 155 by the controller device 113a and the controller device 113b based on the controller signals.

[0094] For example, the user may use the controller device 113a or the controller device 113b to grasp, release, or move and rotate the CAD model 155 in 6 DoF.

[0095] Note that, for example, in the case where the user moves his or her hand wearing the controller device 113a or the controller device 113b without grasping the CAD model 155, the CAD model 155 may remain stationary, or the CAD model 155 may move in a manner that moves a virtual point.

[0096] For example, the user can use the controller device 113a to point to any point, line, surface, etc. of the CAD model 155 with a ray (virtual ray), etc. For example, the user can use the controller device 113a to perform line drawing on the CAD model 155, wherein the line is drawn.

[0097] For example, a user may edit (eg, model, route, disassemble, etc.) the CAD model 155 using the controller device 113 a or the controller device 113 b .

[0098] <Configuration Example of Information Processing Device 111 and Terminal Device 112>

[0099] Figure 4 1 is a block diagram showing a configuration example of functions of the information processing device 111 and the terminal device 112 of the XR system 101 .

[0100] The information processing apparatus 111 includes an operation input unit 201 , a control unit 202 , a display unit 203 , a storage unit 204 , and a communication unit 205 .

[0101] The operation input unit 201 includes, for example, an input device such as a keyboard and a mouse. The operation input unit 201 receives an operation input by a user, and supplies an operation input signal indicating the content of the operation input by the user to the control unit 202.

[0102] For example, the control unit 202 includes an electronic circuit such as a CPU or a microprocessor. In addition, the control unit 202 may include a ROM storing programs to be used, operating parameters, etc., and a RAM temporarily storing parameters and the like that are changed according to circumstances.

[0103] For example, the control unit 202 functions as an arithmetic processing device and a control device, and controls the overall operation of the information processing apparatus 111 , and executes various processes according to various programs.

[0104] For example, the control unit 202 implements the information processing unit 211 by executing an XR application that can edit user experience and virtual objects in the information processing device 111 and the XR space. The information processing unit 211 includes a recognition unit 221, an operation control unit 222, a space control unit 223, an audio control unit 224, a tactile presentation control unit 225, and a learning unit 226. That is, the recognition unit 221, the operation control unit 222, the space control unit 223, the audio control unit 224, the tactile presentation control unit 225, and the learning unit 226 are implemented by executing the XR application using the control unit 202. In addition, the input and output of each component of the information processing unit 211 (i.e., the recognition unit 221, the operation control unit 222, the space control unit 223, the audio control unit 224, the tactile presentation control unit 225, and the learning unit 226) are performed via the XR application.

[0105] The identification unit 221 identifies the state of the information processing device 111, the state of the terminal device 112, the state of the surroundings of the terminal device 112, the state of the controller device 113, the state of the user, the user operation, the state of the XR space, etc. based on at least one of the operation input signal from the operation input unit 201, the information from the control unit 202, the information from the display unit 203, the information from the communication unit 205, the sensing data sent from the terminal device 112, the controller signal sent from the controller device 113, the information from the operation control unit 222, or the information from the space control unit 223.

[0106] For example, the state of the information processing device 111 to be identified includes at least one of the state of each component of the information processing device 111, the state of each application including the XR application, the communication state between the information processing device 111 and another device, or various types of setting information (such as setting values ​​of various setting items, etc.). For example, the state of each component of the information processing device 111 includes at least one of the operating state of each component, the presence or absence of an abnormality, or the abnormal content. For example, the state of each application includes at least one of the start, end, or operating state of the application, the presence or absence of an abnormality, or the abnormal content. For example, the communication state between the information processing device 111 and another device includes the communication state with the terminal device 112 and the communication state with the controller device 113 via the terminal device 112.

[0107] For example, the state of the terminal device 112 to be identified includes at least one of the position, posture, or behavior of the terminal device 112 or various types of setting information (e.g., setting values ​​of various setting items, etc.). Note that, for example, in the case where the terminal device 112 is worn by a user, the position, posture, and behavior of the terminal device 112 indirectly indicate the position, posture, and behavior of a body part of the user wearing the terminal device 112.

[0108] For example, the state of the surroundings of the terminal device 112 to be recognized includes at least one of the type, position, posture, behavior, size, shape, appearance, or feature value of a real object around the terminal device 112 (user).

[0109] For example, the state of the controller device 113 to be recognized includes at least one of the position, posture, or behavior of the controller device 113 or various types of setting information (eg, setting values ​​of various setting items, etc.).

[0110] For example, the state of the user to be recognized includes at least one of the user's position, posture, overall behavior, behavior of a body part, or sight direction.

[0111] For example, the user operation to be recognized includes at least one of an operation input through the operation input unit 201, an operation input through the controller device 113, an operation input through a user's gesture, or an operation input through a virtual tool in an XR space, etc.

[0112] For example, the state of the XR space to be recognized includes at least one of the type, position, posture, behavior, size, shape, appearance, or feature value of the virtual object in the XR space.

[0113] The recognition section 221 provides information on the recognition result to each component of the information processing device 111 .

[0114] The recognition section 221 also transmits information about the recognition result to the terminal device 112 via the communication unit 205, and transmits the information to the controller device 113 via the communication unit 205 and the terminal device 112. For example, in the case where the recognition section 221 detects a change or an abnormality in the state of the terminal device 112 or the input device 113, the recognition section 221 transmits information indicating the result of the detection to the terminal device 112 via the communication unit 205, or transmits information to the controller device 113 via the communication unit 205 and the terminal device 112. For example, in the case where the recognition section 221 detects a change (e.g., start, stop, etc.) or an abnormality in the state of an application such as an XR application, the recognition section 221 transmits information indicating the result of the detection to the terminal device 112 via the communication unit 205, or transmits information to the controller device 113 via the communication unit 205 and the terminal device 112.

[0115] Note that, for example, any method such as image recognition, object recognition, etc. may be used for the recognition processing of various recognition targets by the recognition section 221 .

[0116] In addition, for example, in the case where a plurality of users share an XR space, for example, the recognition unit 221 performs recognition processing on each user. For example, the recognition unit 221 recognizes the state of each user's terminal device 112, the state of the area around each user's terminal device 112, the state of each user's controller device 113, the state of each user, and the user operation performed by each user. For example, by sending the result of the recognition processing of each user to each user's terminal device 112 or controller device 113, the result can be shared between users.

[0117] The operation control section 222 controls the operation process of the controller device 113 based on at least one of the recognition result from the recognition section 221 or the controller signal transmitted from the controller device 113 .

[0118] For example, the operation control section 222 controls the operation process of the controller device 113 based on at least one of the position and posture of the controller device 113 or the controller signal. For example, the operation control section 222 controls the enabling or disabling of each operating member included in the controller device 113, the function assigned to each operating member, the method for operating the function assigned to each operating member, etc., based on the installation method, holding method, use method, etc. of the controller device 113.

[0119] The operation control section 222 provides information on control of the operation process of the controller device 113 to each unit of the information processing apparatus 111 .

[0120] The space control unit 223 controls the presentation of the two-dimensional space or the three-dimensional space by the display unit 203 and the presentation of the XR space by the terminal device 112 based on at least a subset of the recognition results from the recognition unit 221 .

[0121] For example, the space control section 223 generates a display object to be displayed in a two-dimensional space or a three-dimensional space based on at least a subset of the recognition results from the recognition section 221, and performs various arithmetic operations required for construction, display, etc. of the two-dimensional space or the three-dimensional space including the behavior of the display object. The space control section 223 generates display control information for controlling the display of the two-dimensional space or the three-dimensional space based on the result of the arithmetic operation, and provides the display control information to the display unit 203 to control the display of the two-dimensional space or the three-dimensional space by the display unit 203. Note that, for example, the display control information may include information for using the two-dimensional space or the three-dimensional space (e.g., an operation menu, guidance, a message, etc.) and information for notifying the status of the information processing device 111 (e.g., setting information, remaining battery power, error display, etc.).

[0122] For example, the space control section 223 generates a virtual object to be displayed in the XR space based on at least a subset of the recognition results from the recognition section 221, and performs various arithmetic operations required for the construction, display, etc. of the XR space including the behavior of the virtual object. For example, the recognition result from the recognition section 221 includes the operation content of the controller device 113a recognized by the recognition section 221 based on the controller signal, etc. (including the operation input signal from the controller device 113a). The space control section 223 generates display control information for controlling the display of the XR space based on the result of the arithmetic operation, and sends the display control information to the terminal device 112 via the communication unit 205 to control the display of the XR space by the terminal device 112. Note that, for example, the display control information may include information for using the XR space (e.g., operation menus, guidance, messages, etc.) and information for notifying the status of the XR system 101 (e.g., setting information, remaining battery power, error display, etc.).

[0123] The space control section 223 provides each component of the information processing device 111 with information on the two-dimensional space, the three-dimensional space, and the XR space.

[0124] The audio control unit 224 controls the output of sound by the terminal device 112 based on at least one of the recognition result from the recognition unit 221 or the information from the space control unit 223. For example, the space control unit 223 generates audio control information for outputting sound in the terminal device 112. For example, the audio control information includes information about at least one of the type, content, frequency, amplitude, or waveform of the sound to be output. The audio control unit 224 controls the output of sound by the terminal device 112 by sending the audio control information to the terminal device 112 via the communication unit 205.

[0125] The tactile presentation control unit 225 controls the presentation of tactile stimulation to the user based on at least one of the recognition result from the recognition unit 221 or the information from the space control unit 223. For example, the tactile presentation control unit 225 generates tactile control information for presenting the tactile stimulation in the controller device 113. For example, the tactile control information includes information about at least one of the type, mode, intensity, or length of the tactile sensation to be presented. The tactile presentation control unit 225 controls the presentation of the tactile stimulation by the controller device 113 by sending the tactile control information to the controller device 113 via the communication unit 205 and the terminal device 112.

[0126] The learning unit 226 performs learning processing related to the processing of the XR system 101 based on at least one of the recognition result from the recognition unit 221 or the learning data given from the outside. For example, the learning unit 226 learns the taste, action pattern, etc. of the user, and adjusts various processing and parameters of the XR system 101 based on the learning result so as to appropriately respond to the taste, action pattern, etc. of the user. For example, the learning unit 226 learns the difference between the XR space and the real space, and adjusts the design data based on the learning result so that the characteristics, behaviors, etc. of the virtual objects in the XR space are closer to the characteristics, behaviors, etc. of the real objects.

[0127] For example, the learning section 226 stores information indicating the learning result (eg, a learning model, etc.) in the storage unit 204 .

[0128] Note that the control unit 202 can execute not only the XR application but also other applications.

[0129] For example, the storage unit 204 includes a read only memory (ROM) that stores programs, operating parameters, etc. for processing by the control unit 202, and a random access memory (RAM) that temporarily stores parameters and the like that vary depending on circumstances.

[0130] The communication unit 205 communicates with an external device to send and receive data. For example, the communication unit 205 communicates with the terminal device 112 to send and receive data. For example, the communication unit 205 sends display control information, audio control information, and tactile control information to the terminal device 112. For example, the communication unit 205 receives sensing data and controller signals from the terminal device 112.

[0131] The communication method of the communication unit 205 may be wired or wireless, and, for example, uses wired LAN, wireless LAN, Wi-Fi, Bluetooth, etc. Furthermore, the communication unit 205 may support two or more types of communication methods.

[0132] The terminal device 112 includes an operation input unit 251 , a sensing unit 252 , a control unit 253 , a display unit 254 , an audio output unit 255 , and a learning section 226 .

[0133] For example, the operation input unit 251 includes an operation input device such as a button. The operation input unit 251 receives the user's operation input and provides an operation input signal indicating the content of the user's operation input to the control unit 253. For example, the operation input unit 251 receives the user's operation input, such as turning on or off the terminal device 112 and adjusting the brightness of the display unit 254.

[0134] The sensing unit 252 includes various sensors for sensing the terminal device 112, the surroundings of the terminal device 112, and the status of the user. For example, the sensing unit 252 includes a camera or a depth sensor for capturing an image of the surroundings of the terminal device 112. For example, the sensing unit 252 includes a camera or a depth sensor for capturing an image of the user's eyes. For example, the sensing unit 252 includes an inertial measurement unit (IMU) for detecting the acceleration, angular velocity, etc. of the terminal device 112. For example, the sensing unit 252 includes a global navigation satellite system (GNSS) receiver for detecting the current position of the terminal device 112 (user). The sensing unit 252 provides sensing data indicating the detection result of at least one of the sensors to the control unit 253.

[0135] For example, the control unit 253 includes an electronic circuit such as a CPU or a microprocessor. In addition, the control unit 253 may include a ROM for storing programs to be used, operating parameters, etc., and a RAM for temporarily storing parameters that vary according to circumstances.

[0136] For example, the control unit 253 functions as an arithmetic processing device and a control device, and controls the overall operation of the terminal device 112 based on the operation input signal from the operation input unit 251, the sensing data from the sensing unit 252, the display control information and the audio control information from the information processing device 111, the controller signal from the controller device 113, etc., and performs various processes according to various programs. For example, the control unit 253 controls the display unit 254 to display the XR space, etc. based on the display control information. For example, the control unit 253 controls the audio output unit 255 to output sound based on the audio control information.

[0137] The display unit 254 includes various display devices. For example, in the case where the terminal device 112 is an HMD, the display unit 254 includes a display fixed relative to the left eye and right eye of the user, and displays the left eye image and the right eye image. For example, the display includes a display panel such as a liquid crystal display or an organic electroluminescent (EL) display, or a laser scanning display such as a retinal scanning display. In addition, for example, the display unit 254 may include an imaging optical system that enlarges and projects the display screen to form an enlarged virtual image with a predetermined viewing angle on the user's pupil. For example, the display unit 254 displays an XR space including a virtual object under the control of the control unit 253.

[0138] For example, the audio output unit 255 includes an audio output device such as a headphone, an earphone, or a speaker. The audio output unit 255 outputs sound under the control of the control unit 253.

[0139] The communication unit 256 communicates with an external device to send and receive data. For example, the communication unit 256 communicates with the terminal device 112 and the controller device 113 to send and receive data. For example, the communication unit 256 sends sensing data and a controller signal to the information processing device 111. For example, the communication unit 256 receives display control information, audio control information, and tactile control information from the information processing device 111. For example, the communication unit 256 sends tactile control information to the controller device 113. For example, the communication unit 256 receives a controller signal from the controller device 113.

[0140] The communication method of the communication unit 256 may be wired or wireless, and for example, a wired LAN, a wireless LAN, Wi-Fi, Bluetooth, etc. may be used. In addition, the communication unit 256 may support two or more types of communication methods. Moreover, the communication unit 256 may communicate with the information processing device 111 and the controller device 113 using different communication methods.

[0141] The following is an example of processing performed by the information processing device 111 using the XR application.

[0142] For example, the communication unit 205 receives input information indicating at least one of the state of the terminal device 112, the state of the surroundings of the terminal device 112, the state of the user, the behavior of the user, or an operation input from the terminal device 112 or the controller device 113 to the input device 113 via the terminal device 112, and provides the input information to the control section 221. The control section 221 executes the XR application based on the input information, generates output information for controlling the display of a virtual object (including CAD information about CAD in the XR space), and outputs the output information to the terminal device 112. The communication unit 205 sends the output information to the terminal device 112.

[0143] Furthermore, for example, the control section 221 executes the XR application and outputs output information indicating a change or abnormality in the state of the XR application to the terminal device 112 or the controller apparatus 113. The communication unit 205 transmits the output information to the terminal device 112 or the controller apparatus 113 via the terminal device 112.

[0144] On the other hand, for example, the terminal device 112 notifies the change or abnormality of the state of the XR application through an image, message, sound, vibration, etc. based on the output information. For example, the controller device 113 notifies the change or abnormality of the state of the XR application through vibration, etc. based on the output information.

[0145] Note that in the following description, in the case where each component of the information processing device 111 communicates with the outside via the communication unit 205, the description of the communication unit 205 may be omitted. For example, in the case where the space control unit 223 of the information processing device 111 communicates with the terminal device 112 via the communication unit 205, it may be simply described that the space control unit 223 of the information processing device 111 communicates with the terminal device 112.

[0146] In the following description, in the case where each component of the terminal device 112 communicates with the outside via the communication unit 256, the description of the communication unit 256 may be omitted. For example, in the case where the control unit 253 of the terminal device 112 communicates with the information processing device 111 via the communication unit 256, it may be simply described that the control unit 253 of the terminal device 112 communicates with the information processing device 111.

[0147] For example, in the XR system 101, the space control section 223 of the information processing device 111 generates display control information and transmits the display control information to the terminal device 112 via the communication unit 205, and the control unit 253 of the terminal device 112 receives the display control information via the communication unit 256 and controls the display unit 254 based on the display control information. In the following description, the description of a series of processes may be simplified, and, for example, it may be described that the space control section 223 of the information processing device 111 controls the display unit 254 of the terminal device 112.

[0148] For example, in the XR system 101, the audio control section 224 of the information processing device 111 generates audio control information and transmits the audio control information to the terminal device 112 via the communication unit 205, and the control unit 253 of the terminal device 112 receives the audio control information via the communication unit 256, and controls the audio output unit 255 based on the audio control information. In the following description, the description of a series of processes may be simplified, for example, it may be described that the audio control section 224 of the information processing device 111 controls the audio output unit 255 of the terminal device 112.

[0149] For example, in the XR system 101, the haptic presentation control unit 225 of the information processing device 111 generates haptic control information, and transmits the haptic control information to the controller device 113 via the communication unit 205 and the terminal device 112, and the controller device 113 presents haptic stimulation based on the haptic control information. In the following description, the description of a series of processes may be simplified, for example, the haptic presentation control unit 225 of the information processing device 111 may be described as controlling the controller device 113 via the terminal device 112.

[0150] <Configuration Example of Controller Device 113a>

[0151] Next, we will refer to Figures 5 to 14 describe Figure 3 An example configuration of the controller device 113a.

[0152] Figure 5 A configuration example of the appearance of the controller device 113 a is shown. Figure 5 A is a left side view of the controller device 113a. Figure 5 B is a front view of the controller device 113a. Figure 5 C is a bottom view of the controller device 113a. Figure 5 D is a stereoscopic diagram of the controller device 113a when viewed obliquely from the right front.

[0153] Notice, Figure 5 The upward direction in A is defined as the upward direction of the controller device 113a, and Figure 5 The downward direction in A is defined as the downward direction of the controller device 113a. Figure 5 The right direction in A is the front direction of the controller device 113a, and Figure 5 The left direction in A is the rear direction of the controller device 113a.

[0154] The controller device 113a has a symmetrical shape regardless of the direction (i.e., front, rear, left, right, top, or bottom) from which the controller device 113a is viewed. In addition, in the controller device 113a, the shape of the front surface viewed from the front is similar to the shape of the rear surface viewed from the rear, and the shape of the right side surface viewed from the right is similar to the shape of the left side surface viewed from the left.

[0155] The controller device 113a is roughly divided into three parts, namely, a ring-shaped part 301, an operating part 302a and a holding part 302b.

[0156] like Figure 5As shown in A, the annular portion 301 extends upward from near the center of gravity of the left side surface 314b. When viewed from the direction of the side surface of the annular portion 301 (for example, the left side surface 314b of the controller device 113a), the operating portion 302a and the retaining portion 302b have a symmetrical shape with respect to the annular portion 301. The operating portion 302a extends forward and obliquely downward from near the center of gravity of the left side surface 314b (near the lower end of the annular portion 301). The retaining portion 302b extends backward and obliquely downward from near the center of gravity of the left side surface 314b (near the lower end of the annular portion 301), symmetrical with the operating portion 302a. When the top of the annular portion 301, the top of the operating portion 302a, and the top of the retaining portion 302b are connected to each other, an isosceles triangle with the top of the annular portion 301 as the vertex is formed. The angle between the annular portion 301 and the operating portion 302a, the angle between the annular portion 301 and the holding portion 302b, and the angle between the operating portion 302a and the holding portion 302b are each about 120 degrees, and the isosceles triangle is substantially an equilateral triangle.

[0157] The top of the side surface of the annular portion 301 extends linearly, and the root extends in a curved shape. The top of the side surface of the operating portion 302a extends linearly, and the root extends in a curved shape. The top of the side surface of the holding portion 302b extends linearly, and the root extends in a curved shape. The boundary portion between the annular portion 301 and the operating portion 302a, the boundary portion between the annular portion 301 and the holding portion 302b, and the boundary portion between the operating portion 302a and the holding portion 302b are all curved.

[0158] like Figure 5 As shown in B, a hole 301A is formed in the annular portion 301, which is through in the front-rear direction. The outer periphery of the annular portion 301 gradually expands toward the top, and the top is curved. Similarly, the hole 301A gradually expands toward the top, and the top and the end are curved.

[0159] like Figure 5 As shown in FIG. 1B , the operation portion 302a tapers toward the top, and the top is curved. The upper surface 312a of the operation portion 302a is inclined forward and obliquely downward. A shallow groove that is curved in the lateral direction and extends in the front-to-back direction is formed in the upper surface 312a of the operation portion 302a. The top of the upper surface 312a of the operation portion 302a is slightly recessed relative to the top of the operation portion 302a. Therefore, in the case where the user's finger is inserted into the hole 301A of the annular portion 301 from the back to the front, the inserted finger can be easily placed on the upper surface 312a of the operation portion 302a.

[0160] The holding portion 302b has the same shape as the operating portion 302a, and forms an upper surface 312b (not shown) having the same shape as the upper surface 312a.

[0161] like Figure 5 As shown in FIG. 3C, a bottom surface 313 curved in the front-rear direction is formed by the lower surface of the operating portion 302a and the lower surface of the holding portion 302b. A shallow groove curved in the lateral direction and extending in the front-rear direction is formed in the bottom surface 313.

[0162] For example, a rubber-like material such as silicone or elastomer is used for the inner peripheral surface 311, the upper surface 312a, the upper surface 312b, and the bottom surface 313 of the controller device 113a. For example, for the other parts of the controller device 113a, an IR transmissive resin is used.

[0163] Figures 6 to 8 An example of a method for maintaining the controller device 113a is shown.

[0164] For example, Figure 6 As shown in A, the index finger of the right hand is inserted into the ring portion 301 from back to front, the tip of the index finger is placed near the top of the upper surface 312a of the operating portion 302a, and the operating portion 302a can be operated by the index finger. Since the size of the hole 301A of the ring portion 301 is slightly larger than the thickness of the index finger, the index finger can be easily inserted. The tip of the thumb of the right hand is gently placed near the top of the side surface of the operating portion 302a, and the holding portion 302b is gently grasped and held by the palm of the right hand.

[0165] For example, Figure 6 As shown by the arrow in A of FIG. 1 , when the top of the operating portion 302a is pressed downward by the index finger, as shown in FIG. Figure 6 As shown in FIG. 1B , the top of the holding portion 302b is in contact with the palm, and the controller device 113a is prevented from rotating in the pressing direction. Therefore, the vicinity of the top of the operating portion 302a is prevented from shaking in space, and the user can reliably press the vicinity of the top of the operating portion 302a in a state where the direction of the top of the operating portion 302a is stable.

[0166] Furthermore, as described above, the shapes of the controller device 113a viewed from the front and rear are similar to each other, and the shapes of the controller device 113a viewed from the right and left are similar to each other. Therefore, the user can hold the controller device 113a without worrying about the front and rear. That is, Figure 7 As shown in FIG. 1A , the user can hold the controller device 113 a so that the operation portion 302 a faces the fingertip direction and the right side surface 314 a faces the thumb direction. Alternatively, as Figure 7As shown in FIG. 1B , the user may hold the controller device 113 b so that the holding portion 302 b faces the fingertip direction and the left side surface 314 b faces the thumb direction.

[0167] Note that Figure 7 As shown in FIG. 1A , holding the controller device 113a with the operation portion 302a facing the fingertips is hereinafter referred to as holding the controller device 113a forward. Figure 7 As shown in B, holding the controller device 113a with the holding portion 302b facing the fingertips will be referred to as holding the controller device 113a backwards hereinafter.

[0168] When the controller device 113a is held backward, the roles of the operation portion 302a and the holding portion 302b are reversed. That is, the holding portion 302b serves as an operation portion that can be operated by the index finger of the right hand, and the operation portion 302a serves as a holding portion held by the palm of the right hand.

[0169] In addition, if Figure 8 As shown, even if the user releases his hand from the controller device 113a, the ring portion 301 is caught by the index finger and the controller device 113a does not fall. This prevents the user from accidentally dropping the controller device 113a when no strap or the like is provided.

[0170] Fig. 9 An arrangement example of operating members of the controller device 113 a is shown. Fig. 9 A is a stereoscopic view of the controller device 113a when viewed obliquely from the upper right. Fig. 9 B is a stereoscopic view of the controller device 113a when viewed obliquely from the upper left. Fig. 9 C is a stereoscopic view of the controller device 113a when viewed obliquely from the rear and below.

[0171] The respective operating members are symmetrically arranged about the annular portion 301 in the front-rear direction and the left-right direction of the controller device 113 a .

[0172] For example, the operating member 331 is provided at the lower end of the inner peripheral surface 311 (the hole 301A) of the annular portion 301. For example, the user bends his or her index finger and operates the operating member 331 with the fingertip of the index finger.

[0173] The operating member 332a is provided near the top of the upper surface 312a of the operating portion 302a. The operating member 332b is provided near the top of the upper surface 312b of the holding portion 302b. For example, the user operates the operating member 332a or the operating member 332b with the tip of his or her index finger.

[0174] The operating member 333a and the operating member 333b are respectively provided near the front and rear ends of the bottom surface 313. For example, the user operates the operating member 333a or the operating member 333b with the fingertip of his / her ring finger or little finger.

[0175] The operation member 334 is provided at the center of the bottom surface 313 in the front-rear direction. For example, the user operates the operation member 334 with the tip of his / her thumb, ring finger, or little finger.

[0176] For example, any type of operating member such as a button, a touch pad, or a joystick may be used as each of the operating member 331, the operating member 332a, the operating member 332b, the operating member 333a, the operating member 333b, and the operating member 334. However, the same type of operating member is used as the operating member 332a and the operating member 332b arranged at symmetrical positions with respect to the annular portion 301. Similarly, the same type of operating member is used as the operating member 333a and the operating member 333b arranged at symmetrical positions with respect to the annular portion 301.

[0177] For example, any function can be assigned to each of the operating member 331, the operating member 332a, the operating member 332b, the operating member 333a, the operating member 333b, and the operating member 334. However, the same function is assigned to the operating member 332a and the operating member 332b arranged at the symmetrical position with respect to the annular portion 301. Similarly, the same function is assigned to the operating member 333a and the operating member 333b arranged at the symmetrical position with respect to the annular portion 301.

[0178] Specifically, for example, the function of calling the main menu screen is assigned to the operating member 331. For example, the function of selecting a virtual object is assigned to the operating member 332a and the operating member 332b. For example, functions other than the selection functions of the operating member 332a and the operating member 332b are assigned to the operating member 333a and the operating member 333b. For example, the function of calling the submenu screen is assigned to the operating member 334.

[0179] Note that, for example, different functions may be assigned to the operating member 332a and the operating member 332b, and these functions may be switched according to the direction in which the controller device 113a is held. Similarly, for example, different functions may be assigned to the operating member 333a and the operating member 333b, and these functions may be switched according to the direction in which the controller device 113a is held.

[0180] Therefore, regardless of whether the user holds the controller device 113a forward or backward, the user can perform similar operations.

[0181] Note that, although it is assumed that the index finger is inserted into the annular portion 301 , for example, the middle finger or the ring finger may be inserted and used.

[0182] Note that, in the case where it is not necessary to distinguish the operating member 332a and the operating member 332b from each other, they are hereinafter simply referred to as the operating member 332. In the case where it is not necessary to distinguish the operating member 333a and the operating member 333b from each other, they are hereinafter simply referred to as the operating member 333.

[0183] <Example of layout of mark>

[0184] For example, a marker such as an IR light emitting element may be provided for the controller device 113a. Then, the recognition unit 221 of the information processing device 111 may detect the marker of the controller device 113a based on an image or the like sensed by the sensing unit 252 of the terminal device 112, and recognize the relative position and relative posture of the terminal device 112 and the controller device 113 based on the position of the detected marker.

[0185] Fig.10 An example of arrangement of the marks 351 of the controller device 113a is shown. Each mark 351 is indicated by a solid circle.

[0186] For example, Fig.10 As shown in FIG. 1A , the marks 351 are arranged on the right side surface 314a and the left side surface 314b in the vertical direction so as to surround the outer periphery of the annular portion 301. For example, the marks 351 are provided near the top ends of both side surfaces of the operating portion 302a and near the top ends of both side surfaces of the holding portion 302b. For example, the marks 351 are provided near the front end and the rear end of the bottom surface 313.

[0187] Therefore, if Fig.11 As shown in A to D of FIG. 1 , in any posture of the controller device 113 a , at least a subset of the markers 351 become visible without being obscured by the user's hand.

[0188] On the other hand, for example, it is assumed that the terminal device 112 includes a plurality of camera devices 401, such as Fig.12 Each camera device 401 constitutes the sensing unit 252 ( Figure 4 ). Each camera 401 captures an image of the controller device 113a. The terminal device 112 transmits sensing data including captured image data obtained by image capturing to the information processing device 111.

[0189] On the other hand, the control unit 202 of the information processing device 111 receives the sensing data. The recognition section 221 of the control unit 202 recognizes the position and posture of the controller device 113a relative to the terminal device 112 based on the light emission pattern of the marker 351 of the controller device 113a.

[0190] Note that, for example, Fig.10 As shown in FIG. 1B , the marks 351 may be arranged in two rows in the transverse direction so as to surround the outer circumference of the annular portion 301 . Alternatively, for example, Fig.10 As shown in FIG. 3C , the marks 351 may be arranged in three rows in the lateral direction so as to surround the outer circumference of the annular portion 301 .

[0191] By providing the mark 351 on the outer periphery of the ring portion 301 like this, the size of the controller device 113a can be reduced.

[0192] <Example of Internal Structure of Controller Device 113a>

[0193] Next, we will refer to Fig.13 An example of the internal structure of the controller device 113 a is described.

[0194] The controller device 113 a includes a haptic device 371 , a haptic device 372 a , a haptic device 372 b , a board 373 , and a battery 374 .

[0195] Each of the haptic devices 371, 372a, and 372b includes, for example, a device that presents (transmits) tactile stimulation such as vibration, i.e., a linear resonant actuator (LRA), an eccentric rotating mass (ERM), a piezoelectric element, etc.

[0196] The haptic device 371 is provided near the lower end of the inner peripheral surface 311 of the annular portion 301 (near the operation member 331 ( Fig. 9 )), and presents tactile stimulation near the lower end of the inner circumferential surface 311.

[0197] The tactile device 372a is provided near the top of the operating portion 302a (near the operating member 332a ( Fig. 9 )), and transmits tactile stimulation to the vicinity of the top of the operating portion 302a.

[0198] The tactile device 372b is provided near the top end of the holding portion 302b (near the operating member 332b ( Fig. 9 )), and transmits tactile stimulation to the vicinity of the top of the retaining portion 302b.

[0199] The board 373 is a board for controlling the controller device 113 a , and is disposed approximately at the center of the controller device 113 a and below the haptic device 371 .

[0200] The battery 374 is disposed in the controller device 113a, below the board 373, and powers each component of the controller device 113a.

[0201] For example, as Fig.14 shown, when the user holds the controller device 113a with the right hand forward, the haptic device 371 presents a haptic stimulus near the base of the thumb. The haptic device 372a presents haptic stimuli near the tip of the thumb and near the tip of the index finger. The haptic device 372b presents haptic stimuli near the base of the thumb and near the palm.

[0202] Note that the haptic device 371, the haptic device 372a, and the haptic device 372b are arranged at symmetric positions with respect to the annular portion 301 in the front - rear direction of the controller device 113a. Therefore, similar haptic stimuli are presented to the user's hand regardless of whether the user holds the controller device 113a forward or backward.

[0203] <Processing of the XR system 101>

[0204] Next, the processing of the XR system 101 will be described with reference to Figures 15 to 23 the following.

[0205] <Process for controlling the operation member>

[0206] First, the process for controlling the operation member executed by the XR system 101 will be described with reference to Fig.15 the flowchart below.

[0207] For example, this process is executed when the user holds or re - holds the controller device 113a.

[0208] In step S1, the information processing device 111 performs hand recognition through hand tracking.

[0209] For example, the control unit 253 of the terminal device 112 sends sensing data to the information processing device 111, and the sensing data includes captured image data indicating the images captured by each imaging device 401.

[0210] On the other hand, the control unit 202 of the information processing device 111 receives the sensing data. The recognition unit 221 of the control unit 202 performs hand recognition through hand tracking based on the captured image data included in the sensing data. Thus, for example, the recognition unit 221 tracks the hand of the user holding the controller device 113a based on the marker 351 provided for the controller device 113a.

[0211] In step S2, the recognition section 221 determines whether the hand holding the controller device 113a has been recognized based on the result of the processing in step S1. If it is determined that the hand holding the controller device 113a has not been recognized, the process returns to step S1.

[0212] Thereafter, the processing in steps S1 and S2 is repeatedly performed until it is determined in step S2 that the hand holding the controller device 113a has been recognized.

[0213] On the other hand, if it is determined in step S2 that the hand holding the controller device 113a has been identified, the process proceeds to step S3.

[0214] In step S3, the recognition section 221 recognizes the light emission pattern of the controller device 113a based on the captured image data. That is, the recognition section 221 recognizes the light emission pattern of the mark 351 in the controller device 113a that is not blocked by the user's hand.

[0215] In step S4, the recognition section 221 determines whether the holding direction of the controller device 113a has been recognized. Specifically, the recognition section 221 attempts to recognize the holding direction of the controller device 113a based on the recognition result of the hand of the user holding the controller device 113a and the recognition result of the light emission pattern of the controller device 113a. If it is subsequently determined that the holding direction of the controller device 113a has not been recognized, the process returns to step S3.

[0216] Thereafter, the processing in steps S3 and S4 is repeatedly performed until it is determined in step S4 that the holding direction of the controller device 113a has been identified.

[0217] On the other hand, if it is determined in step S4 that the holding direction of the controller device 113a has been identified, the process proceeds to step S5.

[0218] In step S5, the operation control unit 222 disables the operation member on the palm side. Fig.16 As shown, in the case where the controller device 113a is held forward, the operation member 332b on the palm side is disabled. Thereafter, for example, the recognition section 221 and the operation control section 222 ignore the operation input signal from the operation member 332b.

[0219] On the other hand, in the case where the controller device 113a is held backward, for example, the operation member 332a on the palm side is disabled.

[0220] Thereafter, the process for controlling the operating member ends.

[0221] Therefore, the operation member 332 is prevented from being erroneously operated by the palm of the user.

[0222] As described above, the hand holding the controller device 113 a and the holding direction are recognized, and the operability of the controller device 113 a does not change regardless of the holding direction of the controller device 113 a .

[0223] Therefore, if Fig.17 As shown in A and B of , for example, even if no special setting is made on the terminal apparatus 112 side, the user can use the controller device 113a with his or her dominant hand regardless of which hand the user's dominant hand is.

[0224] Note that, for example, Fig.18 As shown in A and B, the user can wear another controller device 113b, such as a ring device, on the side of his non-dominant hand to use the controller device 113b.

[0225] In addition, for example, Fig.19 As shown, the user can wear the controller device 113a on both hands to use the controller device 113a.

[0226] <Process for controlling tactile feedback>

[0227] Next, we will refer to Fig. 20 The flowchart of describes the process performed by XR system 101 to control haptic feedback.

[0228] For example, this process starts when the information processing device 111 is turned on, and ends when the information processing device 111 is turned off.

[0229] In step S51 , the information processing device 111 recognizes the state of the terminal device 112 , the state around the terminal device 112 , and the like.

[0230] Specifically, the sensing unit 252 of the terminal device 112 senses the state of the terminal device 112 and the state of the surroundings of the terminal device 112, and provides sensing data indicating the result of the sensing to the control unit 253. The control unit 253 transmits the sensing data to the information processing device 111.

[0231] On the other hand, the control unit 202 of the information processing device 111 receives the sensing data.

[0232] The controller device 113 a transmits a controller signal including an operation input signal indicating the operation content of each operation member to the information processing apparatus 111 via the terminal device 112 .

[0233] On the other hand, the control unit 202 of the information processing device 111 receives the controller signal.

[0234] The recognition unit 221 of the control unit 202 recognizes the state of the terminal device 112, the state of the surroundings of the terminal device 112, the state of the controller device 113, the state of the user, the user operation, etc. based on the sensing data and the controller signal. For example, the recognition unit 221 recognizes the position and posture of the terminal device 112. For example, the recognition unit 221 recognizes the direction of the user's line of sight. For example, the recognition unit 221 recognizes the position and posture of the controller device 113a relative to the terminal device 112. For example, the recognition unit 221 recognizes the operation content of the controller device 113a.

[0235] In step S52, the space control unit 223 of the information processing device 111 controls the XR space. Specifically, the space control unit 223 generates a virtual object to be displayed in the XR space based on at least a subset of the recognition results from the recognition unit 221, and performs various arithmetic operations required for the construction, display, etc. of the XR space including the behavior of the virtual object. The space control unit 223 generates display control information for controlling the display of the XR space based on the result of the arithmetic operation, and sends the display control information to the terminal device 112 via the communication unit 205 to control the display of the XR space by the terminal device 112.

[0236] The recognition unit 221 recognizes the type, position, posture, and the like of the virtual object around the terminal device 112 (user) based on information and the like from the space control unit 223 .

[0237] In step S53, the tactile presentation control section 225 determines whether it is the timing to present tactile feedback based on at least one of the recognition result from the recognition section 221 or the information from the space control section 223. If it is determined not to be the timing to present tactile feedback, the process returns to step S51.

[0238] Thereafter, the processing in step S51 to step S53 is repeatedly performed until it is determined in step S53 that it is the timing to present the tactile feedback.

[0239] On the other hand, if it is determined in step S53 that it is the timing to present tactile feedback, the process proceeds to step S54.

[0240] In step S54 , the information processing device 111 controls the presentation of tactile feedback. Specifically, the tactile presentation control unit 225 generates tactile control information for causing the controller device 113 a to present tactile stimulation. The tactile presentation control unit 225 sends a tactile control signal to the controller device 113 a via the terminal device 112 .

[0241] On the other hand, the controller device 113a receives haptic control information. Each haptic device of the controller device 113a presents haptic stimulation based on the haptic control information.

[0242] Thereafter, the process returns to step S51, and the processing in step S51 and subsequent steps is performed.

[0243] The controller device 113a thus appropriately presents tactile stimulation to the user.

[0244] Here, we will refer to Figure 21 to Figure 23 An example of a method of presenting tactile feedback performed by the controller device 113a is described.

[0245] For example, an operating member 332a ( Fig. 9 ) includes a touch pad and the tip of the index finger is as Fig.21 When the operation member 332a is slid in the front-rear direction as shown in FIG. 1A, the haptic device 372a ( Fig.13 ) presents tactile stimulation to the tip of the index finger.

[0246] For example, Fig.21 As shown in FIG. 1B, when the user touches the button 431 in the XR space with the top of the operation part 302a of the controller device 113a, the haptic device 372a ( Fig.13 ) presents tactile stimulation to the tip of the index finger.

[0247] For example, in a case where a hand of the controller device 113a is held or the controller device 113a collides with a virtual object in the XR space, an impact due to the collision is expressed using each haptic device of the controller device 113a.

[0248] For example, Fig. 22 FIG. 4A shows an example of a case where the top end of the operation portion 302a of the controller device 113a collides with the virtual object 441 in the XR space from above. For example, in this case, the haptic device 372a ( Fig.13 ) exhibits an upward vibration, and the haptic device 372b ( Fig.13 ) presents downward vibration. Therefore, the user can feel an upward rotation force (torque) relative to the controller device 113a.

[0249] For example, Fig. 22 FIG. 1B shows an example of a case where the top end of the operation portion 302a of the controller device 113a collides with the virtual object 441 in the XR space from below. For example, in this case, the haptic device 372a ( Fig.13 ) exhibits downward vibration, and the haptic device 372b ( Fig.13) presents an upward vibration. Therefore, the user can feel a downward rotation force (torque) relative to the controller device 113a.

[0250] For example, Fig.23 113a collides with the virtual object 441 in the XR space from the front. For example, in this case, by making the haptic device 371 ( Fig.13 ) vibrates to vibrate the entire controller device 113a. Therefore, the user can feel the reaction force from the virtual object 441 to the controller device 113a.

[0251] The operability of the controller device 113a is thus improved. Therefore, the operability with respect to the XR space is improved.

[0252] <Method for specifying virtual objects>

[0253] Next, we will refer to Figures 24 to 33 An example of a method of designating a desired virtual object in an XR space using the controller device 113 a as a pointing device is described.

[0254] For example, when a large number of virtual objects are assembled together in the XR space, it is sometimes difficult to specify a desired virtual object from the virtual objects. For example, it is sometimes difficult to specify a desired component from a collection of assembled components of multiple virtual objects (hereinafter referred to as a component).

[0255] In view of this, an example of a method for specifying a component to be activated from among virtual components constituting a virtual assembly will be described below.

[0256] Here, indicating a component means, for example, simply indicating a component using the controller device 113a. For example, specifying a component means defining (selecting) a component as a target (e.g., an operation target, an attention target, etc.). For example, a component is specified by performing a predetermined operation in a state where the component is indicated using the controller device 113a. For example, activating a component means making the component an operation target.

[0257] To simplify the description, in the following, Fig.24 As shown, an example will be described in which a desired component is designated and activated from the assembly 1001 including three virtual components 1011a to 1011c.

[0258] Note that, for example, it is assumed that when the HMD application is activated, the entire assembly 1001 (all components 1011 ) is activated.

[0259] In the case where it is not necessary to distinguish each of the components 1011 a to 1011 c , the components 1011 a to 1011 c will be simply referred to as the component 1011 .

[0260] <First Embodiment for Controlling Activation Process>

[0261] Here, we will refer to Fig.25 The flowchart of describes a first embodiment of a process performed by XR system 101 for controlling activation.

[0262] In step S101 , the information processing apparatus 111 recognizes the position and posture of the controller device 113 a relative to the component 1001 .

[0263] For example, the sensing unit 252 of the terminal device 112 senses the state of the terminal device 112 and the state of the surroundings of the terminal device 112, and provides sensing data indicating the sensing result to the control unit 253. The control unit 253 transmits the sensing data to the information processing device 111.

[0264] On the other hand, the control unit 202 of the information processing device 111 receives the sensing data. The recognition unit 221 of the control unit 202 recognizes the position and posture of the controller device 113a relative to the terminal device 112 based on the sensing data. The recognition unit 221 recognizes the position and posture of the controller device 113a relative to the component 1001 in the XR space displayed by the terminal device 112 based on the position and posture of the controller device 113a relative to the terminal device 112.

[0265] In step S102 , the identification section 221 determines whether the controller device 113 a is instructing the component 1001 based on the result of the processing in step S101 .

[0266] For example, when the controller device 113a is held forward, the direction in which the top end of the operating portion 302a faces is the indicated direction of the controller device 113a. For example, when the controller device 113a is held backward, the direction in which the top end of the holding portion 302b faces is the indicated direction of the controller device 113a.

[0267] When the component 1001 exists in the pointing direction of the controller device 113a, that is, when the pointing direction of the controller device 113a intersects with the component 1001 in the XR space, the recognition section 221 determines that the controller device 113a is pointing to the component 1001, and the process proceeds to step S103.

[0268] Note that the top end of the operating portion 302a in the case where the controller device 113a is held forward and the top end of the holding portion 302b in the case where the controller device 113a is held backward will hereinafter be referred to as the top end of the controller device 113a.

[0269] In step S103, the recognition section 221 determines whether the indicated component is in an active state. Specifically, if the component 1011 of the assembly 1001 existing in the indicated direction of the controller device 113a is in an active state, the recognition section 221 determines that the indicated component is in an active state, and the process proceeds to step S104.

[0270] In step S104, the recognition section 221 determines whether the indicated component 1011 is within a predetermined distance range from the terminal device 112. For example, the recognition section 221 recognizes the distance of the indicated component 1011 relative to the terminal device 112 based on the sensing data received in the process of step S101 and the information about the XR space provided from the space control section 223. If the recognized distance is less than or equal to a predetermined threshold value (hereinafter referred to as a distance threshold), the recognition section 221 determines that the indicated component 1011 is within a predetermined distance range from the terminal device 112, and the process proceeds to step S105.

[0271] For example, the distance threshold is set to a range within which the user can reach the indicated component 1011 (eg, approximately 50 cm).

[0272] In step S105, the XR system 101 displays the designated point at a predetermined distance from the controller device 113a. Specifically, the space control unit 223 of the information processing device 111 controls the display unit 254 of the terminal device 112 in the following manner: the designated point is displayed at a predetermined distance from the top of the controller device 113a in the direction indicated by the controller device 113a in the XR space, and the designated point is a virtual bright spot. That is, the controller device 113a virtually outputs the designated point.

[0273] For example, Fig.26 As shown, a designated point P1 is displayed at a position a predetermined distance from the top end of the controller device 113a.

[0274] Therefore, a method of indicating a position in the XR space by the controller device 113 a (hereinafter referred to as a pointing method) is set to a designation point mode.

[0275] Thereafter, the process proceeds to step S107.

[0276] On the other hand, if the identified distance exceeds the distance threshold in step S104 , the recognition section 221 determines that the indicated component 1011 is beyond the predetermined distance range from the terminal device 112 , and the process proceeds to step S106 .

[0277] Furthermore, if it is determined in step S103 that the indicated component 1011 is in the inactive state, the process proceeds to step S106.

[0278] In addition, if the component 1001 does not exist in the pointing direction of the controller device 113 a in step S102 , the recognition section 221 determines that the controller device 113 a does not point to the component 1001 , and the process proceeds to step S106 .

[0279] In step S106, the XR system 101 displays the ray. Specifically, the space control unit 223 of the information processing device 111 controls the display unit 254 of the terminal device 112 in the following manner: the ray is displayed in the XR space, and the ray is a virtual light beam extending from the top of the controller device 113a in the direction indicated by the controller device 113a. That is, the ray is virtually output from the top of the controller device 113a.

[0280] For example, Fig. 27 As shown, a ray R1 extending from the top of the controller device 113a along the indicated direction of the controller device 113a is shown.

[0281] Therefore, the pointing method of the controller device 113a is set to the ray mode.

[0282] Note that, for example, Fig.28 As shown, in the case where the component 1021a and the component 1021b are displayed in an overlapping manner in the indication direction of the controller device 113a, the pointing method is controlled (selected) based on the distance between the terminal apparatus 112 and the component in the active state.

[0283] For example, in Fig.28 In case A, component 1021a is inactive and component 1021b is active. Therefore, in this case, the pointing method is selected based on the distance between the terminal device 112 and component 1021b. For example, when the distance between the terminal device 112 and component 1021b exceeds the distance threshold, the pointing method is set to the ray mode, and the controller device 113a virtually outputs the ray R2.

[0284] For example, in Fig.28In case B, component 1021a is in an active state and component 1021b is in an inactive state. Therefore, in this case, a pointing method is selected based on the distance between the terminal device 112 and component 1021a. For example, when the distance between the terminal device 112 and component 1021a is less than or equal to the distance threshold, the pointing method is set to the designated point mode, and the controller device 113a virtually outputs the designated point P2.

[0285] Note that, as shown in the figure, the designated point P2 is not necessarily displayed on the component 1021a.

[0286] Thereafter, the process proceeds to step S107.

[0287] In step S107 , the XR system 101 determines whether an activation operation is performed.

[0288] For example, in the case where the user wants to activate the desired component 1011, the user activates the desired component 1011 in a state where the desired component 1011 is indicated by a ray or a designated point (eg, Fig.26 or Fig. 27 As shown) performs a predetermined activation operation on the controller device 113a.

[0289] For example, in the case where the controller device 113a is held forward, double clicking on the operating member 332a is an activation operation, and in the case where the controller device 113a is held backward, double clicking on the operating member 332b is an activation operation.

[0290] In addition, for example, in the case where the user wants to activate all components 1011, the user performs an activation operation on the controller device 113a without using the controller device 113 to indicate the component 1001 (that is, when the controller device 113 does not indicate the component 1011), such as Fig.29 shown.

[0291] In the case where the activation operation is performed, the controller device 113 a transmits a controller signal including an operation input signal indicating the activation operation to the information processing apparatus 111 via the terminal device 112 .

[0292] On the other hand, the control unit 202 of the information processing device 111 receives the controller signal via the communication unit 205 .

[0293] On the other hand, if the controller signal including the operation input signal indicating the activation operation is not received, the recognition section 221 of the information processing apparatus 111 determines that the activation operation is not performed, and the process returns to step S101.

[0294] Thereafter, the processing in step S101 to step S107 is repeatedly performed until it is determined in step S107 that the activation operation has been performed.

[0295] On the other hand, if the recognition section 221 of the information processing apparatus 111 receives a controller signal including an operation input signal indicating an activation operation from the controller device 113a via the terminal apparatus 112 in step S107, it is determined that an activation operation has been performed, and the process proceeds to step S108.

[0296] In step S108, the recognition section 221 determines whether the component 1011 is being indicated. If it is determined that any component 1011 is being indicated using a designated point or a ray, the process proceeds to step S109.

[0297] In step S109 , the XR system 101 activates the indicated component 1011 .

[0298] Specifically, the recognition section 221 of the information processing device 111 recognizes that the currently indicated component 1011 has been designated as an activation target.

[0299] The space control section 223 of the information processing device 111 activates the components 1011 recognized as designated by the recognition section 221, and does not activate other components 1011. For example, the space control section 223 controls the display unit 254 of the terminal device 112 to set the activated components 1011 and the unactivated components 1011 to different display modes so that the activated components 1011 stand out.

[0300] Fig.30 A display example is shown in which the component 1011b is activated and the components 1011a and 1011c are not activated. For example, the display mode of the component 1011a or at least one of the components 1011a and 1011c is changed so that the activated component 1011b is prominent and the inactivated components 1011a and 1011c become inconspicuous. Specifically, for example, the activated component 1011a continues to be displayed in the same manner, while the inactivated components 1011a and 1011c become semi-transparent.

[0301] Furthermore, for example, the space control section 223 sets the activated component 1011 as an operation target and enables operation on the activated component 1011. For example, in the case where the recognition section 221 recognizes the operation on the activated component 1011, the space control section 223 performs control so that the corresponding process is performed.

[0302] On the other hand, the space control section 223 sets the unactivated components 1011 as non-operation targets and prohibits operation on the unactivated components 1011. For example, in the case where the recognition section 221 recognizes operation on one of the unactivated components 1011, the space control section 223 performs control so that the corresponding processing is not performed.

[0303] Thereafter, the process returns to step S101 , and the processing in step S101 and subsequent steps is performed.

[0304] On the other hand, if it is determined in step S108 that any component 1011 is not indicated using a designated point or ray, the process proceeds to step S110.

[0305] In step S110 , the XR system 101 activates all components 1011 .

[0306] Specifically, the recognition section 221 of the information processing apparatus 111 recognizes that the activation status of the components 1011 has been reset, and all the components 1011 have been designated as targets to be activated.

[0307] The space control section 223 of the information processing device 111 activates all components 1011 and controls the display unit 254 of the terminal device 112 in such a manner as to display all components 1011 in a display mode indicating activation. The space control section 223 sets all components 1011 as operation targets and enables operation on all components 1011.

[0308] Thereafter, the process returns to step S101 , and the processing in step S101 and subsequent steps is performed.

[0309] The operability of operations related to indicating a position in the XR space is thus improved.

[0310] For example, in a 3D model (such as an assembly file) in which a large number of parts are mixed, the required parts can be easily indicated and activated.

[0311] Furthermore, since long-distance pointing by rays and short-distance pointing by specifying points are automatically and instantly switched, short-distance accurate pointing is possible while achieving long-distance pointing.

[0312] For example, the user can indicate the desired component 1011 using a ray from a position far from the component 1001, and indicate the desired component 1011 using a designated point at a position near the component 1001. Since the pointing method is automatically switched according to the distance from the component 1011, the user can easily indicate and activate the desired component 1011.

[0313] Note that, for example, in the case where the recognition section 221 of the information processing apparatus 111 recognizes a predetermined gesture instead of an activation operation using the controller device 113a, the space control section 223 may activate the indicated component 1011. Fig.31As shown, component 1011 can be activated when the position indicated by the ray (or specified point) is swung laterally on component 1011 a predetermined number of times or more (e.g., twice or more) with an amplitude of a predetermined length or less (e.g., 1 cm or less) within a predetermined time (e.g., within 1 second).

[0314] This saves time and effort in performing activation operations on the controller device 113a.

[0315] <Second Embodiment for Controlling Activation Process>

[0316] Next, we will refer to Fig.32 The flowchart of describes a second embodiment of a process performed by XR system 101 for controlling activation.

[0317] In the second embodiment, activation of component 1011 is further controlled based on the user's line of sight direction.

[0318] In steps S151 to S158, the Fig.25 The processing is similar to the processing in steps S101 to S108 in .

[0319] If it is determined in step S158 that a component is being indicated, the process proceeds to step S159.

[0320] In step S159, the recognition section 221 of the information processing device 111 determines whether the indicated component 1011 is in the user's line of sight. Specifically, the recognition section 221 estimates the user's line of sight direction based on the sensing data received in the process in step S151. If the recognition section 221 determines that the indicated component 1011 is located in the user's line of sight based on the result of the estimation of the user's line of sight direction, the process proceeds to step S160.

[0321] In step S160, Fig.25 The indicated component 1011 is activated as in the processing in step S109 in .

[0322] For example, Fig.33 The range V1 in indicates the estimated range of the user's line of sight. Since the indicated component 1011c exists within the range V1, the component 1011c is then activated.

[0323] Thereafter, the process returns to step S151, and the processing in step S151 and subsequent steps is performed.

[0324] On the other hand, if it is determined in step S159 that the indicated component 1011 is not in the user's sight, the process returns to step S151, and the processing in step S151 and subsequent steps is performed. That is, in this case, the indicated component 1011 is not activated.

[0325] On the other hand, if it is determined in step S158 that no component 1011 is being indicated, the process proceeds to step S161.

[0326] In step S161, Fig.25 As in the process of step S110 in , all components 1011 are activated.

[0327] Thereafter, the process returns to step S151, and the processing in step S151 and subsequent steps is performed.

[0328] Therefore, it is possible to prevent unintended components 1011 from being designated and activated outside the user's sight.

[0329] <<2. Modifications>>

[0330] Modifications of the above-described embodiments of the present technology will be described below.

[0331] <Modification Example Related to Controller Device 113a>

[0332] Although an example in which the controller device 113 a may be held forward or backward is described in the above description, it is also possible to hold the controller device 113 a only forward, for example.

[0333] In this case, the operating portion 302a and the holding portion 302b do not necessarily have symmetrical shapes with respect to the annular portion 301, and for example, the operating portion 302a and the holding portion 302b may have different shapes. In addition, the operating member 332b and the operating member 333b of the holding portion 302b may be removed.

[0334] For example, a material other than resin such as metal may be used for the controller device 113a.

[0335] <Modification example regarding sharing of processing>

[0336] For example, part of the processing of the information processing device 111 may be performed by the terminal device 112 .

[0337] For example, the terminal device 112 may perform all or part of the processing of the information processing unit 211 of the information processing device 111. For example, the terminal device 112 may independently present an XR space without being controlled by the information processing device 111. For example, the information processing device 111 and the terminal device 112 may independently share and perform processing such as constructing an XR space.

[0338] <Modification Example Related to Method for Indicating and Specifying Position in XR Space>

[0339] Although an example of specifying a component to be activated is described in the above description, the present technology can be applied to a case of specifying a virtual object in an XR space regardless of the purpose of the specification.

[0340] The present technology can also be applied to a case where a position in the XR space is indicated or specified using an input device other than the controller device 113 a , for example.

[0341] The designated point is not necessarily a virtual bright spot, and may instead be, for example, a virtual spot that does not emit light.

[0342] A ray is not necessarily a virtual beam (ray of light), and may instead be a non-luminous virtual line.

[0343] For example, the display mode of a virtual object indicated or specified in the XR space may be different from the display mode of other virtual objects, whether activated or not.

[0344] <Modification example related to switching conditions of pointing method>

[0345] For example, the user can set a distance threshold, which is a switching condition for the pointing method.

[0346] For example, the space control unit 223 of the information processing device 111 may adjust the distance threshold based on the state of the surroundings of the terminal device 112 (user) identified by the identification unit 221. For example, the space control unit 223 may adjust the distance threshold based on the space where the user is located.

[0347] Specifically, for example, when the user is in a wide space (such as a warehouse) where the user can move around, the space control unit 223 can increase the distance threshold. For example, since the user can move in a wide space and indicate a virtual object, it can be considered to improve the convenience of the user by expanding the range in which the virtual object can be indicated using the designated point.

[0348] As the distance from the component for determining the switching of the pointing method, for example, the distance based on the controller device 113 a can be used.

[0349] For example, as the distance for determining the switching of the pointing method, a distance other than the distance to the activated component may be used. For example, the distance to the nearest virtual object may be used.

[0350] <Modification example related to gesture for activation>

[0351] For example, as mentioned above Fig.31 As described above, in the case of using the gesture activation component 1011, the learning section 226 of the information processing device 111 can adjust the determination criteria of the gesture for activation based on the user's operation history.

[0352] For example, as described above, in the case where component 1011 is activated when the user laterally swings the indicated position on component 1011 two or more times with an amplitude of 1 cm or less within 1 second and the user tends to laterally swing the indicated position on component 1011 for an average of 1.1 seconds, the learning unit 226 can change the determination condition of the gesture for activation to within 1.2 seconds.

[0353] Furthermore, in the case where a similar gesture is performed in a state where any component 1011 is not indicated, all components may be activated.

[0354] Furthermore, for example, in the case where the recognition section 221 of the information processing apparatus 111 recognizes a predetermined gesture of a user independently of the controller device 113 a , the space control section 223 may activate the indicated component 1011 .

[0355] <Other Modifications>

[0356] For example, the controller device 113 a can be used not only for operations in the XR space but also for operations in two-dimensional space and three-dimensional space in games and the like.

[0357] <<3. Others>>

[0358] <Computer Configuration Example>

[0359] The above series of processing can be performed by hardware or by software. In the case where the series of processing is performed by software, the program constituting the software is installed in the computer. Here, examples of the computer include a computer incorporated in dedicated hardware, and a general-purpose personal computer that can perform various functions by installing various programs, for example.

[0360] Fig.34 : is a block diagram showing a configuration example of hardware of a computer that executes the above-described series of processes according to a program.

[0361] In a computer 2000 , a central processing unit (CPU) 2001 , a read only memory (ROM) 2002 , and a random access memory (RAM) 2003 are connected to one another via a bus 2004 .

[0362] An input / output interface 2005 is further connected to the bus 2004. An input unit 2006, an output unit 2007, a storage unit 2008, a communication unit 2009, and a drive 2010 are connected to the input / output interface 2005.

[0363] The input unit 2006 includes an input switch, a button, a microphone, an image sensor, etc. The output unit 2007 includes a display, a speaker, etc. The storage unit 2008 includes a hard disk, a nonvolatile memory, etc. The communication unit 2009 includes a network interface, etc. The drive 2010 drives a removable medium 2011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0364] In the computer 2000 configured as described above, for example, a program stored in the storage unit 2008 is loaded into the RAM 2003 via the input / output interface 2005 and the bus 2004 by the CPU 2001 and executed, thereby performing the above-described series of processing.

[0365] The program executed by the computer 2000 (CPU 2001) can be provided by being recorded on the removable medium 2011 as, for example, a package medium, etc. Furthermore, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0366] In the computer 2000, the program can be installed into the storage unit 2008 by mounting the removable medium 2011 onto the drive 2010 via the input / output interface 2005. In addition, the program can be received by the communication unit 2009 via a wired or wireless transmission medium and installed into the storage unit 2008. In addition, the program can be installed in the ROM 2002 or the storage unit 2008 in advance.

[0367] Note that the program executed by the computer may be a program that executes processing in time series in the order described in this specification, or may be a program that executes processing in parallel or at necessary timing such as when a call is made.

[0368] In addition, in this specification, a system is intended to mean an assembly of multiple components (devices, modules (components), etc.), and it is not important whether all the components are in the same housing. Therefore, multiple devices housed in independent housings and connected to each other via a network and a device that houses multiple modules in one housing are both systems.

[0369] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications may be made without departing from the scope of the present technology.

[0370] For example, the present technology may be implemented in cloud computing where multiple devices share and execute functions via a network.

[0371] Furthermore, each step described in the above flowchart may be performed by one device, or may be shared and performed by a plurality of devices.

[0372] Furthermore, in the case where a plurality of processes are included in one step, the plurality of processes included in one step may be executed by one device, or may be executed by a plurality of devices in a shared manner.

[0373] <Configuration combination example>

[0374] The present technology may also be configured as follows. (1)

[0376] An information processing device, comprising:

[0377] an identification unit that identifies a distance to a virtual object in a cross-reality (XR) space; and

[0378] A space control unit controls a method for indicating a position in the XR space using an input device based on a distance from the virtual object. (2)

[0380] The information processing device according to (1), wherein:

[0381] When the distance to the virtual object is less than or equal to a predetermined threshold, the space control unit indicates the position in the XR space using a point virtually output from the input device; and when the distance to the virtual object exceeds the threshold, the space control unit indicates the position in the XR space using a line virtually output from the input device. (3)

[0383] The information processing device according to (2), wherein:

[0384] In a case where a predetermined operation or gesture is performed in a state where the virtual object is indicated using the point or the line, the recognition unit recognizes that the virtual object has been designated. (4)

[0386] The information processing device according to (3), wherein:

[0387] The space control unit controls display of the XR space so that the designated virtual object and other virtual objects are displayed in different display modes. (5)

[0389] The information processing device according to (3) or (4), wherein:

[0390] The space control unit activates the designated virtual object and does not activate other virtual objects. (6)

[0392] The information processing device according to (5), wherein:

[0393] The space control section controls a method for indicating a position in the XR space using the input device based on a distance from the activated virtual object. (7)

[0395] The information processing device according to any one of (3) to (6), wherein:

[0396] The recognition unit estimates the direction of the user's line of sight, and recognizes that the virtual object has been designated when the operation or the gesture is performed in a state where the virtual object is indicated using the point or the line within the estimated range of the user's line of sight. (8)

[0398] The information processing device according to any one of (3) to (7), further comprising:

[0399] A learning unit that adjusts a determination criterion for the gesture based on an operation history of the user. (9)

[0401] The information processing device according to any one of (2) to (8), wherein:

[0402] In a case where a predetermined operation or gesture is performed in a state where none of the virtual objects is indicated using the point or the line, the recognition unit recognizes that all of the virtual objects have been designated. (10)

[0404] The information processing device according to (9), wherein:

[0405] The space control unit activates all of the designated virtual objects. (11)

[0407] The information processing device according to any one of (2) to (10), wherein:

[0408] The recognition unit recognizes the state of the user's surroundings, and

[0409] The space control unit adjusts the threshold value based on a state of the user's surroundings. (12)

[0411] The information processing device according to any one of (2) to (11), wherein:

[0412] The input device comprises:

[0413] an annular portion into which the finger is inserted;

[0414] an operating portion operable by a finger inserted into the annular portion in a first direction; and

[0415] a holding portion which is held in a palm with the operating portion being operated by the finger, and

[0416] The point or the line is virtually output in the direction indicated by the top end of the operation portion. (13)

[0418] The information processing device according to any one of (1) to (12), wherein:

[0419] The space control section controls a method for indicating a position in the XR space using the input device based on a distance between a terminal apparatus worn by a user and displaying the XR space and the virtual object. (14)

[0421] An information processing method executed by an information processing device, the information processing method comprising:

[0422] Identifying the distance to a virtual object in XR space; and

[0423] Based on the distance to the virtual object, a method for indicating a position in the XR space using an input device is controlled.

[0424] Note that the effects described in this specification are merely examples and are not limiting, and other effects may also be produced.

[0425] Reference numerals list

[0426] 101 XR System

[0427] 111 Information processing device

[0428] 112 Terminal Device

[0429] 113, 113a, 113b Controller device

[0430] 202 Control unit

[0431] 203 Display unit

[0432] 211 Information Processing Unit

[0433] 221 Identification Department

[0434] 222 Operation Control Department

[0435] 223 Space Control Department

[0436] 224 Audio Control Unit

[0437] 225 Tactile presentation control unit

[0438] 226 Learning Department

[0439] 252 sensor units

[0440] 253 Control Unit

[0441] 254 Display units

[0442] 255 audio output unit

[0443] 301 Ring part

[0444] 301A hole

[0445] 302a Operation section

[0446] 302b Keep part

[0447] 312a, 312b upper surface

[0448] 313 bottom surface

[0449] 331 to 334 Operating components

[0450] 351 Mark

[0451] 371 to 372b Haptic devices

[0452] 401 Camera Device

Claims

1. An information processing apparatus, comprising: an identification unit that identifies a distance to a virtual object in a cross-reality (XR) space; and a space control unit that controls a method for indicating a position in the XR space using an input device based on the distance to the virtual object.

2. The information processing apparatus according to claim 1, wherein when the distance to the virtual object is less than or equal to a predetermined threshold, the space control unit indicates a position in the XR space using a point virtually output from the input device; and when the distance to the virtual object exceeds the threshold, the space control unit indicates a position in the XR space using a line virtually output from the input device.

3. The information processing apparatus according to claim 2, wherein when a predetermined operation or gesture is performed in a state of indicating the virtual object using the point or the line, the identification unit identifies that the virtual object has been designated.

4. The information processing apparatus according to claim 3, wherein the space control unit controls the display of the XR space such that the designated virtual object and other virtual objects are displayed in different display modes.

5. The information processing apparatus according to claim 3, wherein the space control unit activates the designated virtual object and does not activate other virtual objects.

6. The information processing apparatus according to claim 5, wherein the space control unit controls a method for indicating a position in the XR space using the input device based on the distance to the activated virtual object.

7. The information processing apparatus according to claim 3, wherein the identification unit estimates the user's line of sight direction, and when the operation or the gesture is performed in a state of indicating the virtual object within the estimated range of the user's line of sight using the point or the line, identifies that the virtual object has been designated.

8. The information processing apparatus according to claim 3, further comprising: a learning unit that adjusts a determination criterion of the gesture based on the user's operation history.

9. The information processing apparatus according to claim 2, wherein when a predetermined operation or gesture is performed in a state of not indicating any of the virtual objects using the point or the line, the identification unit identifies that all of the virtual objects have been designated.

10. The information processing apparatus according to claim 9, wherein the space control unit activates all of the designated virtual objects.

11. The information processing apparatus according to claim 2, wherein the identification unit identifies a state around the user, and the space control unit adjusts the threshold based on the state around the user.

12. The information processing apparatus according to claim 2, wherein the input device includes: a ring portion into which a finger is inserted; an operation portion that can be operated by a finger inserted into the ring portion in a first direction; and a holding portion that is held in the palm when the operation portion is operated by the finger, and Virtually output the point or the line in the direction indicated at the top of the operation section.

13. The information processing apparatus according to claim 1, wherein the space control unit controls a method for indicating a position in the XR space using the input device based on the distance between a terminal device worn by a user and displaying the XR space and the virtual object.

14. An information processing method executed by an information processing apparatus, the information processing method comprising: identifying a distance to a virtual object in the XR space; and controlling a method for indicating a position in the XR space using an input device based on the distance to the virtual object.

Images