Information processing apparatus and information processing method

Abstract

The present technology relates to an information processing device and an information processing method that improve workability in an XR (cross-reality) space. The information processing apparatus includes: an identification unit that identifies a state of an element related to XR; and a space control unit that controls the physical law applied to the virtual object in the XR space on the basis of the state of the element related to the XR. The present technology can be applied to, for example, an XR system.

Description

Technical Field

[0001] The present technology relates to an information processing apparatus and an information processing method, and more particularly, to an information processing apparatus and an information processing method capable of improving workability in an XR (Cross Reality) space. Background Art

[0002] Generally, a technology for detecting the direction of gravity in the real space and reflecting the detected direction of gravity in the virtual space has been proposed (for example, see Patent Document 1).

[0003] Citation List

[0004] Patent Document

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2011-197777 Summary of the Invention

[0006] Problems to be Solved by the Invention

[0007] For example, in the case of performing a production line design of a factory using 3D CAD, in order to consider the arrangement of parts as virtual objects, a user may want to place virtual parts at arbitrary positions in the virtual space (for example, in the air, etc.). On the other hand, in the invention described in Patent Document 1, the direction of gravity is reflected in the virtual space, and thus the workability of the user may be quite poor.

[0008] The present technology has been completed in view of such a situation, and aims to improve workability in the XR space.

[0009] Solutions to the Problems

[0010] An information processing apparatus according to one aspect of the present technology includes: an identification unit that identifies the state of an element related to XR; and a space control unit that controls physical laws applied to virtual objects in the XR space based on the state of the element related to XR.

[0011] In an operation control method according to one aspect of the present technology, an information processing apparatus identifies the state of an element related to XR, and controls physical laws applied to virtual objects in the XR space based on the state of the element related to XR.

[0012] In one aspect of the present technology, the state of an element related to XR is identified, and physical laws applied to virtual objects in the XR space are controlled based on the state of the element related to XR. Brief Description of the Drawings

[0013] Figure 1 is a diagram showing an embodiment of an XR system to which the present technology is applied.

[0014] Figure 2 It is a diagram showing a display example of an XR system.

[0015] Figure 3 It is a diagram showing a display example of an XR system.

[0016] Figure 4 It is a block diagram showing a configuration example of an information processing device and a terminal device.

[0017] Figure 5 It is an external view showing a configuration example of a controller device.

[0018] Figure 6 It is a diagram showing a method of grasping a controller device.

[0019] Figure 7 It is a diagram showing a method of grasping a controller device.

[0020] Figure 8 It is a diagram showing a method of grasping a controller device.

[0021] Figure 9 It is a diagram showing an arrangement example of operation members of a controller device.

[0022] Figure 10 It is a diagram showing an arrangement example of marks of a controller device.

[0023] Figure 11 It is a diagram showing an example of how marks are presented on a controller device.

[0024] Figure 12 It is a diagram for explaining a method of identifying the position and orientation of a controller device.

[0025] Figure 13 It is a diagram showing an example of the internal configuration of a controller device.

[0026] Figure 14 It is a diagram showing an arrangement example of a tactile device of a controller device.

[0027] Figure 15 It is a flowchart for explaining an operation member control process executed by an XR system.

[0028] Figure 16 It is a diagram for explaining an operation member control process executed by an XR system.

[0029] Figure 17 It is a diagram showing an example of a method of grasping a controller device.

[0030] Figure 18 It is a diagram showing an example of a method of grasping a controller device.

[0031] Figure 19 It is a diagram showing an example of a method for a grasping controller device.

[0032] Figure 20 It is a flowchart for explaining the haptic feedback control process performed by an XR system.

[0033] Figure 21 It is a diagram for explaining an example of haptic feedback.

[0034] Figure 22 It is a diagram for explaining an example of haptic feedback.

[0035] Figure 23 It is a diagram for explaining an example of haptic feedback.

[0036] Figure 24 It is a schematic diagram of a production line site in the real world.

[0037] Figure 25 It is a diagram showing an example of the behavior of a virtual object in an XR space.

[0038] Figure 26 It is a flowchart for explaining the virtual object behavior control process performed by an XR system.

[0039] Figure 27 It is a table for comparing the functions of the physical law off mode and the physical law on mode.

[0040] Figure 28 It is a diagram showing an example of how to hold a controller device.

[0041] Figure 29 It is a diagram showing an example of how to hold a controller device.

[0042] Figure 30 It is a diagram showing an example of how to hold a controller device.

[0043] Figure 31 It is a diagram showing an example of the arrangement of touch sensors and buttons of a controller device.

[0044] Figure 32 It is a diagram showing an example of how to hold a controller device.

[0045] Figure 33 It is a diagram showing an example of how to hold a controller device.

[0046] Figure 34 It is a diagram showing an example of how to hold a controller device.

[0047] Figure 35 It is a diagram showing an example of the arrangement of touch sensors and buttons of a pen-shaped controller device.

[0048] Figure 36 It is a diagram for illustrating an example of a method for switching behavior patterns.

[0049] Figure 37 It is a diagram for illustrating an example of a method for presenting a virtual object.

[0050] Figure 38 It is a diagram for illustrating an example of applying this technology to product design.

[0051] Figure 39 It is a diagram for illustrating an example of applying this technology to surgical simulation.

[0052] Figure 40 It is a block diagram showing an example of the configuration of a computer. Detailed implementation manners

[0053] Hereinafter, the modes for implementing this technology will be described. The description will be given in the order hereinafter.

[0054] 1. Embodiment

[0055] 2. Application example

[0056] 3. Modification

[0057] 4. Others

[0058] <<1. Embodiment>>

[0059] The embodiments of this technology will be described with reference to Figures 1 to 23 the following.

[0060] <Example of the configuration of XR system 101>

[0061] Figure 1 An example of the configuration of an XR (Cross Reality) system 101 as an embodiment of an information processing system to which this technology is applied is shown.

[0062] The XR system 101 is a system that realizes XR, which is a technology for merging the real world and the virtual world, such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and substitute reality (SR). The XR system 101 is a system that presents to a user a space obtained by merging a real space and a virtual space (hereinafter referred to as an XR space). For example, the XR system 101 can present to a user an unrealistic virtual object (hereinafter referred to as a virtual object or a virtual entity), such as a model created by computer-aided design (CAD) (hereinafter referred to as a CAD model), as if the virtual object exists at that location.

[0063] The XR system 101 includes an information processing device 111, a terminal device 112, and a controller device 113.

[0064] The information processing device 111 and the terminal device 112 can communicate with each other wirelessly or wiredly, and can send and receive data to and from each other. The terminal device 112 and the controller device 113 can communicate with each other wirelessly or wiredly, and can send and receive data to and from each other. The information processing device 111 and the controller device 113 communicate with each other via the terminal device 112, and send and receive data to and from each other.

[0065] For example, the information processing device 111 can independently receive a user's operation and present various types of information such as visual information and auditory information to the user.

[0066] In addition, for example, the information processing device 111 executes a predetermined application (hereinafter referred to as an XR application) to control the terminal device 112 and control the XR space presented to the user by the terminal device 112. For example, the information processing device 111 executes the XR application to control the output of various types of information such as visual information and auditory information in the terminal device 112, and constructs the XR space presented by the terminal device 112.

[0067] Figure 1 An example is shown in which the information processing device 111 includes a personal computer (PC), and the personal computer (PC) includes an operation input unit, and the operation input unit includes a mouse and a keyboard. For example, the information processing device 111 can include other information processing devices such as a smart phone or a tablet terminal. For example, the information processing device 111 can include a plurality of information processing devices. For example, the information processing device 111 can include a system constructed by cloud computing via a network.

[0068] The terminal device 112 is a device that presents an XR space to the user.

[0069] Figure 1 An example is shown in which the terminal device 112 is a head-mounted display device that can be mounted on the user's head and includes a head-mounted display (HMD), and the head-mounted display is a device that presents an XR space to the user. More specifically, an example is shown in which the terminal device 112 is a non-transmissive HMD that covers the user's field of view.

[0070] For example, the terminal device 112 includes a video see-through type HMD that has an imaging function capable of imaging the real space based on the user's viewpoint, and is capable of presenting a combined image obtained by combining a real image obtained by imaging the real space with an image of a virtual space such as computer graphics (CG) (hereinafter referred to as a virtual image) to the user.

[0071] For example, the terminal device 112 includes a left imaging unit and a right imaging unit corresponding to the user's left and right eyes respectively, and a left display unit and a right display unit corresponding to the user's left and right eyes respectively.

[0072] For example, the left imaging unit and the right imaging unit constitute a stereoscopic imaging device, and capture images in the user's line of sight direction (hereinafter referred to as field-of-view images) from a plurality of viewpoints corresponding to the user's left and right eyes. That is, each of the left imaging unit and the right imaging unit captures an image of an object (hereinafter referred to as a real object or real entity) in the real space viewed from the user's viewpoint.

[0073] The left display unit and the right display unit can display different images for the left and right eyes respectively, and can present a three-dimensional virtual object by displaying images with parallax for the left and right eyes. For example, the left display unit and the right display unit display a left field-of-view image and a right field-of-view image captured by the left imaging unit and the right imaging unit respectively.

[0074] Note that the terminal device 112 may include other XR terminal devices such as AR glasses or a smartphone equipped with goggles for use. In addition, for example, a display device such as a spatial reproduction display may be used instead of the terminal device 112.

[0075] The controller device 113 is used for operations and inputs (hereinafter referred to as operation inputs) of the XR space presented to the user by the terminal device 112. For example, the user can perform various operations on the virtual object displayed by the terminal device 112 by using the controller device 113.

[0076] For example, the controller device 113 detects at least one of the user's operation input or the user's behavior (e.g., gesture) through at least one of operation members such as buttons or sensors. The controller device 113 sends a signal (hereinafter referred to as a controller signal) including at least one of an operation input signal indicating the user's operation input or a behavior signal indicating the user's behavior to the information processing device 111 via the terminal device 112.

[0077] In addition, for example, the controller device 113a includes a tactile device that presents a tactile stimulus such as vibration, and presents a tactile stimulus to the user under the control of the information processing device 111 or the terminal device 112.

[0078] The controller device 113 includes, for example, one or more types of input devices such as a controller, a ring input device, a pointing device, and a six-degree-of-freedom (6DoF) input device.

[0079] The controller is, for example, an input device grasped by a user's hand. The controller may include operation members such as buttons that can be operated by the user. For example, the user can perform selection operations, determination operations, scrolling operations, etc. on virtual objects displayed on the terminal device 112 by pressing the buttons of the controller. In addition, the controller may include, for example, a touch sensor and a motion sensor.

[0080] Note that the controller is not limited to being grasped by the user's hand and may, for example, be worn on a part of the user's body such as the elbow, arm, knee, ankle, or thigh.

[0081] The ring device is a ring-shaped input device worn on the user's finger. The ring device may include operation members such as buttons that can be operated by the user. For example, the user can change the position and orientation of a virtual object (e.g., a three-dimensional model) in the XR space in six degrees of freedom (6DoF) by operating the ring device.

[0082] The pointing device is an input device capable of indicating any position in the XR space. For example, the information processing device 111 identifies the position and orientation of the six degrees of freedom (6DoF) of the pointing device via the terminal device 112 by a tracking method such as a bright spot tracking method, a porcelain tracking method, or an ultrasonic tracking method.

[0083] The 6DoF input device is, for example, an input device capable of operating in six degrees of freedom (6DoF).

[0084] For example, the user can perform operation inputs using the controller device 113 while viewing various objects (display objects) displayed on the information processing device 111 or the terminal device 112.

[0085] Note that the type and number of the controller device 113 are not particularly limited. For example, as the controller device 113, an input device other than the above types may be used, or an input device obtained by combining multiple types of input devices may be used.

[0086] For example, the XR system 101 can be applied to various fields such as the manufacturing field and the medical field.

[0087] For example, the XR system 101 can perform product design support and assembly support in the manufacturing field. For example, in the product design stage, the user can freely edit a three-dimensional object as a virtual object by using the XR system 101, or can pre-grasp the design result and design before trial production by comparing with the real world.

[0088] For example, the XR system 101 can support surgeries and education in the medical field. For example, users can use the XR system 101 to display the internal state of the body on the surface of a patient, so as to pre-understand the surgical site or conduct training.

[0089] Note that, for example, in the case where the XR space is shared by multiple users, such as in the XR system 101, the terminal device 112 and the controller device 113 are provided for each user.

[0090] <Display example of the XR system 101>

[0091] Herein, reference will be made to Figure 2 and Figure 3 to describe the display example of the display object in the XR system 101.

[0092] Figure 2 and Figure 3 show the display example of the display object in the XR system 101 in the case of creating a CAD model.

[0093] For example, as shown in A of Figure 2 , a two-dimensional CAD model is displayed by the information processing device 111, and the user can edit the two-dimensional CAD model.

[0094] For example, as shown in B of Figure 2 , a three-dimensional CAD model is displayed by the terminal device 112, and the user can edit the three-dimensional CAD model.

[0095] For example, as shown in C of Figure 2 , a two-dimensional object such as a design drawing or specifications is displayed by the terminal device 112, and the user can confirm the design drawing, specifications, etc.

[0096] Figure 3 shows the display example of the XR space by the terminal device 112.

[0097] The display 151, keyboard 152, mouse 153, and table 154 of the information processing device 111 are displayed as a video perspective of a real image obtained by imaging the real space. On the other hand, the two-dimensional image of the terminal device 112 is superimposed on the display 151 serving as a virtual monitor. For example, a two-dimensional CAD model to be designed is displayed on the virtual monitor. For example, from the viewpoints of high accuracy of position detection and ease of position holding, the two-dimensional CAD model displayed by the virtual monitor is preferably operated by the keyboard 152 and the mouse 153.

[0098] In addition, in this example, a three-dimensional CAD model 155 to be designed is displayed in front of the display 151 by the terminal device 112.

[0099] The CAD model 155 is operated, for example, by a controller device 113a grasped by the user's dominant hand (the right hand in this example) and a controller device 113b that is an annular device worn on the index finger of the user's non-dominant hand (the left hand in this example).

[0100] For example, the information processing device 111 identifies the positions, orientations, and behaviors of the hand grasping the controller device 113a and the hand of the user wearing the controller device 113b by performing hand tracking based on an image captured by an imaging unit included in the terminal device 112. Further, for example, the information processing device 111 receives controller signals from the controller device 113a and the controller device 113b via the terminal device 112, and identifies operations on the CAD model 155 based on the controller signals.

[0101] For example, the user can use the controller device 113a or the controller device 113b to grasp, release the CAD model 155, or move and rotate the CAD model 155 in 6DoF.

[0102] Note that, for example, when the hand wearing the controller device 113a or the controller device 113b moves without picking up the CAD model 155, the CAD model 155 may not be moved, or the CAD model 155 may move along with the movement of a virtual point.

[0103] For example, the user can point at any point, line, surface, etc. of the CAD model 155 by using the controller device 113a via light (virtual light beam), etc. For example, the user can use the controller device 113a to perform line drawing to draw a line on the CAD model 155.

[0104] For example, the user can edit (e.g., model, wireframe, disassemble, etc.) the CAD model 155 by using the controller device 113a or the controller device 113b.

[0105] <Configuration example of the information processing device 111 and the terminal device 112>

[0106] Figure 4 is a block diagram showing a configuration example of the functions of the information processing device 111 and the terminal device 112 of the XR system 101.

[0107] The information processing device 111 includes an operation input unit 201, a control unit 202, a display unit 203, a storage unit 204, and a communication unit 205.

[0108] The operation input unit 201 includes input devices such as a keyboard and a mouse, for example. The operation input unit 201 receives an operation input from a user and provides an operation input signal indicating the content of the user's operation input to the control unit 202.

[0109] The control unit 202 includes electronic circuits such as a CPU and a microprocessor, for example. In addition, the control unit 202 may include a ROM that stores programs to be used, calculation parameters, etc., and a RAM that temporarily stores parameters that change as the case may be.

[0110] For example, the control unit 202 serves as an arithmetic processing device and a control device, and controls the overall operation of the information processing device 111 or executes various types of processing according to various programs.

[0111] For example, the control unit 202 implements the information processing unit 211 by executing the information processing device 111 and an XR application capable of performing user experience and editing virtual objects in the XR space. The information processing unit 211 includes an identification unit 221, an operation control unit 222, a space control unit 223, an audio control unit 224, a haptic rendering control unit 225, and a learning unit 226. That is, the identification unit 221, the operation control unit 222, the space control unit 223, the audio control unit 224, the haptic rendering control unit 225, and the learning unit 226 are implemented by the control unit 202 that executes the XR application. In addition, the input and output of each unit of the information processing unit 211, that is, the identification unit 221, the operation control unit 222, the space control unit 223, the audio control unit 224, the haptic rendering control unit 225, and the learning unit 226, are performed via the XR application.

[0112] The identification unit 221 identifies the state of the information processing device 111, the state of the terminal device 112, the state around the terminal device 112, the state of the controller device 113, the state of the user, the user operation, the state of the XR space, etc. based on at least one of an operation input signal from the operation input unit 201, information from the control unit 202, information from the display unit 203, information from the communication unit 205, sensed data transmitted from the terminal device 112, a controller signal transmitted from the controller device 113, information from the operation control unit 222, or information from the space control unit 223.

[0113] The states to be recognized of the information processing device 111 include at least one of, for example, the states of each unit in the information processing device 111, the states of each application such as an XR application, the communication state between the information processing device 111 and other devices, or various setting information (e.g., the setting values of various setting items, etc.). The state of each unit in the information processing device 111 includes at least one of, for example, the operation state of each unit, the presence or absence of an abnormality, or the content of the abnormality. The state of each application includes at least one of, for example, the start and end of each application, the operation state, the presence or absence of an abnormality, or the content of the abnormality. The communication state between the information processing device 111 and other devices includes, for example, the communication state with the terminal device 112 and the communication state with the controller device 113 via the terminal device 112.

[0114] The states to be recognized of the terminal device 112 include at least one of, for example, the position, orientation, or behavior of the terminal device 112 or various types of setting information (e.g., the setting values of various setting items, etc.). Note that, for example, when the terminal device 112 is worn by a user, the position, orientation, and behavior of the terminal device 112 indirectly indicate the position, orientation, and behavior of the part of the user wearing the terminal device 112.

[0115] The states around the terminal device 112 to be recognized include at least one of, for example, the type, position, orientation, behavior, size, shape, appearance, or feature amount of real objects around the terminal device 112 (user).

[0116] The states to be recognized of the controller device 113 include at least one of, for example, the position, orientation, or behavior of the controller device 113 or various types of setting information (e.g., the setting values of various setting items, etc.).

[0117] The states to be recognized of the user include at least one of, for example, the position, orientation, overall behavior, behavior of body parts, or line-of-sight direction of the user.

[0118] The user operations to be recognized include at least one of, for example, the operations input by the operation input unit 201, the operations input by the controller device 113, the operations input by the user's gestures, or the operations input by virtual tools in the XR space, etc.

[0119] The states to be recognized of the XR space include at least one of, for example, the type, position, orientation, behavior, size, shape, appearance, or feature amount of virtual objects in the XR space.

[0120] The recognition unit 221 provides information about the recognition result to each unit of the information processing device 111.

[0121] In addition, the recognition unit 221 transmits information regarding the recognition result to the terminal device 112 via the communication unit 205, or transmits information to the controller device 113 via the communication unit 205 and the terminal device 112. For example, in the case where a change or abnormality in the state of the terminal device 112 or the input device 113 is detected, the recognition unit 221 transmits information indicating the detected content to the terminal device 112 via the communication unit 205, or transmits information to the controller device 113 via the communication unit 205 and the terminal device 112. For example, when the recognition unit 221 detects a change (e.g., start, stop, etc.) or abnormality in the state of an application (e.g., an XR application), the recognition unit 221 transmits information indicating the detected content to the terminal device 112 via the communication unit 205, or transmits the information to the controller device 113 via the communication unit 205 and the terminal device 112.

[0122] Note that, for example, any method such as image recognition or object recognition can be used by the recognition unit 221 for the recognition processing of various recognition targets.

[0123] In addition, for example, in the case where the XR space is shared by multiple users, the recognition unit 221 performs recognition processing for each user, for example. For example, the recognition unit 221 recognizes the state of the terminal device 112 of each user, the state around the terminal device 112 of each user, the state of the controller device 113 of each user, the state of each user, and the user operations of each user. The results of the recognition processing for each user can be shared among the users, for example, by being sent to the terminal device 112 or the controller device 113 of each user.

[0124] The operation control unit 222 controls the operation processing of the controller device 113 based on at least one of the recognition result of the recognition unit 221 or the controller signal transmitted from the controller device 113.

[0125] For example, the operation control unit 222 controls the operation processing of the controller device 113 based on at least one of the position or orientation of the controller device 113 or the controller signal. For example, the operation control unit 222 controls the enabling or disabling of each operation member included in the controller device 113, the function given to each operation member, the operation method of the function given to each operation member, etc., based on the installation method, the grasping method, the usage method, etc., of the controller device 113.

[0126] The operation control unit 222 provides information regarding the control of the operation processing by the controller device 113 to each unit in the information processing device 111.

[0127] The spatial control unit 223 controls the presentation of the two-dimensional space or three-dimensional space by the display unit 203 and the presentation of the XR space by the terminal device 112 based on at least a part of the recognition result of the recognition unit 221.

[0128] For example, the spatial control unit 223 generates a display object to be displayed in the two-dimensional space or three-dimensional space based on at least a part of the recognition result of the recognition unit 221, and performs various calculations required for the construction, display, etc. of the two-dimensional space or three-dimensional space such as the behavior of the display object. The spatial control unit 223 generates display control information for controlling the display of the two-dimensional space or three-dimensional space based on the calculation result, and provides the display control information to the display unit 203, thereby controlling the display of the two-dimensional space or three-dimensional space by the display unit 203. Note that the display control information may include, for example, information for using the two-dimensional space or three-dimensional space (e.g., operation menu, guide, message, etc.) and information for notifying the state of the information processing device 111 (e.g., setting information, remaining battery power, error display, etc.).

[0129] For example, the spatial control unit 223 generates a virtual object to be displayed in the XR space based on at least a part of the recognition result of the recognition unit 221, and performs various calculations required for the construction, display, etc. of the XR space such as the behavior of the virtual object. Based on a controller signal including an operation input signal from the controller device 113a, etc., the recognition result of the recognition unit 221 includes, for example, the operation content for the controller device 113a recognized by the recognition unit 221. The spatial control unit 223 generates display control information for controlling the display of the XR space based on the calculation result, and transmits the display control information to the terminal device 112 via the communication unit 205, thereby controlling the display of the XR space by the terminal device 112. Note that the display control information may include, for example, information for using the XR space (e.g., operation menu, guide, message, etc.) and information for notifying the state of the XR system 101 (e.g., setting information, remaining battery power, error display, etc.).

[0130] The spatial control unit 223 provides information on the two-dimensional space, three-dimensional space, and XR space to each unit of the information processing device 111.

[0131] The audio control unit 224 controls the output of audio by the terminal device 112 based on at least one of the recognition results of the recognition unit 221 or the information from the spatial control unit 223. For example, the spatial control unit 223 generates audio control information for outputting audio in the terminal device 112. The audio control information includes, for example, information on at least one of the type, content, frequency, amplitude, or waveform of the audio to be output. The audio control unit 224 controls the output of audio by the terminal device 112 by sending the audio control information to the terminal device 112 via the communication unit 205.

[0132] The haptic rendering control unit 225 controls the rendering of haptic stimuli to the user based on at least one of the recognition results of the recognition unit 221 or the information from the spatial control unit 223. For example, the haptic rendering control unit 225 generates haptic control information for rendering haptic stimuli in the controller device 113. The haptic control information includes, for example, information on at least one of the type, pattern, intensity, or length of the haptic to be rendered. The haptic rendering control unit 225 sends the haptic control information to the controller device 113 via the communication unit 205 and the terminal device 112, thereby controlling the rendering of haptic stimuli by the controller device 113.

[0133] The learning unit 226 performs learning processing related to the processing of the XR system 101 based on at least one of the recognition results of the recognition unit 221 or the learning data given from the outside. For example, the learning unit 226 learns the user's preferences, action patterns, etc., and adjusts various processes and parameters of the XR system 101 based on the learning results to appropriately correspond to the user's preferences, action patterns, etc. For example, the learning unit 226 learns the differences between the XR space and the real space, and adjusts the design data, etc., based on the learning results so that the characteristics, behaviors, etc. of the virtual objects in the XR space are closer to the real objects.

[0134] For example, the learning unit 226 causes the storage unit 204 to store information (e.g., learning models, etc.) indicating the learning results.

[0135] Note that the control unit 202 can execute not only XR applications but also other applications.

[0136] The storage unit 204 includes, for example, a read-only memory (ROM) that stores programs, calculation parameters, etc. to be used in the processing of the control unit 202, and a random access memory (RAM) that temporarily stores parameters, etc. that change as the case may be.

[0137] The communication unit 205 communicates with an external device to transmit and receive data. For example, the communication unit 205 communicates with the terminal device 112 to transmit and receive data. For example, the communication unit 205 transmits display control information, audio control information, and tactile control information to the terminal device 112. For example, the communication unit 205 receives sensed data and a controller signal from the terminal device 112.

[0138] The communication method of the communication unit 205 can be wired or wireless, and for example, uses a wired LAN, a wireless LAN, Wi-Fi, Bluetooth, etc. In addition, the communication unit 205 can support two or more types of communication methods.

[0139] The terminal device 112 includes an operation input unit 251, a sensing unit 252, a control unit 253, a display unit 254, an audio output unit 255, and a learning unit 226.

[0140] The operation input unit 251 includes operation input devices such as buttons, for example. The operation input unit 201 receives a user's operation input and provides an operation input signal indicating the content of the user's operation input to the control unit 253. For example, the operation input unit 251 accepts operation inputs such as the user turning on or off the power of the terminal device 112 and adjusting the brightness of the display unit 254.

[0141] The sensing unit 252 includes various sensors for sensing the state of the terminal device 112, the periphery of the terminal device 112, and the user. For example, the sensing unit 252 includes a camera device or a depth sensor for imaging the periphery of the terminal device 112. For example, the sensing unit 252 includes a camera device or a depth sensor for imaging the user's both eyes. For example, the sensing unit 252 includes an inertial measurement unit (IMU) for detecting the acceleration, angular velocity, etc. of the terminal device 112. For example, the sensing unit 252 includes a global navigation satellite system (GNSS) receiver for detecting the current position of the terminal device 112 (user). The sensing unit 252 provides sensed data indicating the detection results of at least one or more of the sensors to the control unit 253.

[0142] The control unit 253 includes electronic circuits such as a CPU and a microprocessor, for example. In addition, the control unit 253 can include a ROM for storing programs, calculation parameters, etc. to be used, and a RAM for temporarily storing parameters that change as the case may be.

[0143] For example, the control unit 253 serves as an arithmetic processing device and a control device, and based on various programs, controls the overall operation of the terminal device 112 and executes various types of processing based on the operation input signal from the operation input unit 251, the sensing data from the sensing unit 252, the display control information and audio control information from the information processing device 111, the controller signal from the controller device 113, and the like. For example, the control unit 253 controls the display unit 254 to display an XR space or the like based on the display control information. For example, the control unit 253 controls the audio output unit 255 to output audio based on the audio control information.

[0144] The display unit 254 includes various display devices. For example, in the case where the terminal device 112 is an HMD, the display unit 254 includes displays fixed to the user's left and right eyes, and displays a left-eye image and a right-eye image. The display includes, for example, a display panel such as a liquid crystal display or an organic electroluminescence (EL) display, or a laser scanning display such as a retinal direct drawing display. In addition, the display unit 254 may include, for example, an imaging optical system that magnifies and projects a display screen to form a magnified virtual image with a predetermined viewing angle on the user's pupil. For example, the display unit 254 displays an XR space including virtual objects under the control of the control unit 253.

[0145] The audio output unit 255 includes, for example, audio output devices such as headsets, headphones, or speakers. The audio output unit 255 outputs audio under the control of the control unit 253.

[0146] The communication unit 256 communicates with an external device to send and receive data. For example, the communication unit 256 communicates with the terminal device 112 and the controller device 113 to send and receive data. For example, the communication unit 256 sends the sensing data and the controller signal to the information processing device 111. For example, the communication unit 256 receives the display control information, the audio control information, and the tactile control information from the information processing device 111. For example, the communication unit 256 sends the tactile control information to the controller device 113. For example, the communication unit 256 receives the controller signal from the controller device 113.

[0147] The communication method of the communication unit 256 may be wired or wireless, and for example, uses a wired LAN, a wireless LAN, Wi-Fi, Bluetooth, or the like. In addition, the communication unit 256 may support two or more types of communication methods. In addition, the communication unit 256 may perform communication between the information processing device 111 and the controller device 113 by using different communication methods.

[0148] The following is an example of the processing of the information processing device 111 using an XR application.

[0149] For example, the communication unit 205 receives input information indicating at least one of the state of the terminal device 112, the state around the terminal device 112, the state of the user, the behavior of the user, or an operation input to the input device 113 from the terminal device 112 or the controller device 113 via the terminal device 112, and provides the input information to the control unit 221. The control unit 221 executes an XR application based on the input information, generates output information for controlling the display of a virtual object including CAD information about CAD in the XR space, and outputs the output information to the terminal device 112. The communication unit 205 transmits the output information to the terminal device 112.

[0150] In addition, for example, the control unit 221 executes an XR application and outputs output information indicating a change or an abnormality in the state of the XR application to the terminal device 112 or the controller device 113. The communication unit 205 transmits the output information to the terminal device 122 or to the controller device 113 via the terminal device 112.

[0151] On the other hand, for example, the terminal device 112 notifies a change or an abnormality in the state of the XR application by an image, a message, audio, vibration, or the like based on the output information. For example, the controller device 113 notifies a change or an abnormality in the state of the XR application by vibration or the like based on the output information.

[0152] Note that hereinafter, when each unit of the information processing device 111 communicates with the outside via the communication unit 205, the description of the communication unit 205 may be omitted. For example, when the space control unit 223 of the information processing device 111 communicates with the terminal device 112 via the communication unit 205, it may be simply described as the space control unit 223 of the information processing device 111 communicating with the terminal device 112.

[0153] Hereinafter, when each unit of the terminal device 112 communicates with the outside via the communication unit 256, the description of the communication unit 256 may be omitted. For example, when the control unit 253 of the terminal device 112 communicates with the information processing device 111 via the communication unit 256, it may be simply described as the control unit 253 of the terminal device 112 communicating with the information processing device 111.

[0154] For example, in XR system 101, the spatial control unit 223 of the information processing device 111 generates display control information and sends the display control information to the terminal device 112 via the communication unit 205, and the control unit 253 of the terminal device 112 receives the display control information via the communication unit 256 and controls the display unit 254 based on the display control information. Hereinafter, the description of a series of processes can be simplified, and for example, it can be described as follows: The spatial control unit 223 of the information processing device 111 controls the display unit 254 of the terminal device 112.

[0155] For example, in XR system 101, the audio control unit 224 of the information processing device 111 generates audio control information and sends the audio control information to the terminal device 112 via the communication unit 205, and the control unit 253 of the terminal device 112 receives the audio control information via the communication unit 256 and controls the audio output unit 255 based on the audio control information. Hereinafter, the description of a series of processes is simplified, and for example, it can be described as follows: The audio control unit 224 of the information processing device 111 controls the audio output unit 255 of the terminal device 112.

[0156] For example, in XR system 101, the haptic rendering control unit 225 of the information processing device 111 generates haptic control information, and sends the haptic control information to the controller device 113 via the communication unit 205 and the terminal device 112, and the controller device 113 presents a haptic stimulus based on the haptic control information. Hereinafter, the description of a series of processes is simplified, and for example, it can be described as follows: The haptic rendering control unit 225 of the information processing device 111 controls the controller device 113 via the terminal device 112.

[0157] <Configuration example of controller device 113a>

[0158] Next, reference will be made to Figures 5 to 14 describe Figure 3 the configuration example of the controller device 113a.

[0159] Figure 5 Shows a configuration example of the appearance of the controller device 113a. Figure 5 A of Figure 5 is a left side view of the controller device 113a. Figure 5 B of Figure 5 is a front view of the controller device 113a.

[0160] Note that hereinafter, Figure 5The upward direction in A is defined as the upward direction of the controller device 113a, and Figure 5 the downward direction in A is defined as the downward direction of the controller device 113a. Figure 5 The right direction in A is defined as the front direction of the controller device 113a, and Figure 5 the left direction in A is defined as the rear direction of the controller device 113a.

[0161] The controller device 113a has a symmetric shape when observed from any direction among the front direction, rear direction, left direction, right direction, upward direction, and downward direction. In addition, in the controller device 113a, the shape of the front surface observed from the front is similar to the shape of the rear surface observed from the rear, and the shape of the right side surface observed from the right direction is similar to the shape of the left side surface observed from the left direction.

[0162] The controller device 113a is roughly divided into three parts: a ring portion 301, an operation portion 302a, and a holding portion 302b.

[0163] As Figure 5 shown in A, the ring portion 301 extends upward from near the center of gravity of the left side surface 314b. When observed from the direction of the side surface of the ring portion 301 (for example, the left side surface 314b of the controller device 113a), the operation portion 302a and the holding portion 302b have a shape symmetric about the ring portion 301. The operation portion 302a extends forward and obliquely downward from near the center of gravity of the left side surface 314b (near the lower end of the ring portion 301). The holding portion 302b extends symmetrically with the operation portion 302a from near the center of gravity of the left side surface 314b (near the lower end of the ring portion 301) backward and obliquely downward. When the top end of the ring portion 301, the top end of the operation portion 302a, and the top end of the holding portion 302b are connected, an isosceles triangle with the top end of the ring portion 301 as the vertex is formed. The angle between the ring portion 301 and the operation portion 302a, the angle between the ring portion 301 and the holding portion 302b, and the angle between the operation portion 302a and the holding portion 302b are each about 120 degrees, and the above isosceles triangle is a roughly equilateral triangle.

[0164] The top end of the side surface of the ring portion 301 extends linearly, and the root extends in a curved shape. The top end of the side surface of the operation portion 302a extends linearly, and the root extends in a curved shape. The top end of the side surface of the holding portion 302b extends linearly, and the root extends in a curved shape. The boundary portions between the ring portion 301 and the operation portion 302a, between the ring portion 301 and the holding portion 302b, and between the operation portion 302a and the holding portion 302b are curved.

[0165] As Figure 5As shown in B of , a hole 301A penetrating in the front-rear direction is formed in the ring portion 301. The outer periphery of the ring portion 301 gently expands toward the top end, and the top end is curved. Similarly, the hole 301A gently extends toward the top end, and the top end and the end are curved.

[0166] As Figure 5 As shown in B of , the operation portion 302a gradually thins toward the top end, and the top end is curved. The upper surface 312a of the operation portion 302a inclines forward and obliquely downward. A shallow groove that is curved in the lateral direction and extends in the front-rear direction is formed in the upper surface 312a of the operation portion 302a. The top end of the upper surface 312a of the operation portion 302a is slightly recessed with respect to the top end of the operation portion 302a. Therefore, the upper surface 312a of the operation portion 302a has a shape that can easily place the inserted finger when the user's finger is inserted from the rear along the front direction into the hole 301A of the ring portion 301.

[0167] The holding portion 302b has a shape similar to that of the operation portion 302a, and an upper surface 312b (not shown) having a shape similar to that of the upper surface 312a is formed.

[0168] As Figure 5 As shown in C of , a bottom surface 313 that is curved in the front-rear direction is formed by the lower surface of the operation portion 302a and the lower surface of the holding portion 302b. A shallow groove that is curved in the lateral direction and extends in the front-rear direction is formed in the bottom surface 313.

[0169] For the inner peripheral surface 311, the upper surface 312a, the upper surface 312b, and the bottom surface 313 of the controller device 113a, a rubber-like material such as silicon or an elastomer is used, for example. For other parts of the controller device 113a, an IR transmissive resin is used, for example.

[0170] Figures 6 to 8 An example of a method of gripping the controller device 113a is shown.

[0171] For example, as Figure 6 As shown in A of , the index finger of the right hand is inserted into the ring portion 301 from the rear along the front direction, the fingertip of the index finger is placed near the top end of the upper surface 312a of the operation portion 302a, and the operation portion 302a can be operated by the index finger. Since the size of the hole 301A of the ring portion 301 has a margin with respect to the thickness of the index finger, the index finger can be easily inserted. The fingertip of the thumb of the right hand is gently placed near the top end of the side surface of the operation portion 302a, and the holding portion 302b is gently grasped and held with the palm of the right hand.

[0172] For example, as Figure 6 As shown by the arrow in A of , when the index finger presses downward near the top end of the operation portion 302a, as Figure 6As shown in B of [reference], the top of the holding part 302b is brought into contact with the palm, thereby preventing the controller device 113a from rotating in the pressing direction. Accordingly, it is possible to prevent the vicinity of the top of the operating part 302a from wobbling in space, and in a state where the direction of the top of the operating part 302a is stable, the user can reliably press the vicinity of the top of the operating part 302a.

[0173] In addition, as described above, in the controller device 113a, the shape as viewed from the front is similar to the shape as viewed from the rear, and the shape as viewed from the right is similar to the shape as viewed from the left. Therefore, the user can hold the controller device 113a without worrying about the front and the rear. That is, as Figure 7 shown in A of [reference], the user can hold the controller device 113a such that the operating part 302a faces the fingertip direction and the right side surface 314a faces the thumb direction. In addition, as Figure 7 shown in B of [reference], the user can hold the controller device 113b such that the holding part 302b faces the fingertip direction and the left side surface 314b faces the thumb direction.

[0174] In addition, hereinafter, as Figure 7 shown in A of [reference], gripping the controller device 113a such that the operating part 302a faces the fingertip direction is referred to as gripping the controller device 113a in the forward direction. Hereinafter, as Figure 7 shown in B of [reference], gripping the controller device 113a such that the holding part 302b faces the fingertip direction is referred to as gripping the controller device 113a in the backward direction.

[0175] In the case of gripping the controller device 113a in the backward direction, the functions of the operating part 302a and the holding part 302b are switched. That is, the holding part 302b serves as an operating part that can be operated by the index finger of the right hand, and the operating part 302a serves as a holding part held by the palm of the right hand.

[0176] In addition, as Figure 8 shown in [reference], even if the user releases the hand from the controller device 113a, the ring part 301 is hooked by the index finger, and the controller device 113a does not fall. Therefore, without providing a strap or the like, it is possible to prevent the user from accidentally dropping the controller device 113a.

[0177] Figure 9 An arrangement example of the operation members of the controller device 113a is shown. Figure 9 A of [reference] is a perspective view of the controller device 113a as viewed from the upper right obliquely. Figure 9 B of [reference] is a perspective view of the controller device 113a as viewed from the upper left obliquely. Figure 9 C of [reference] is a perspective view of the controller device 113a as viewed from the lower rear obliquely.

[0178] The operating members are symmetrically arranged in the front - rear direction and the left - right direction of the controller device 113a with the ring portion 301 as the center.

[0179] For example, the operating member 331 is arranged at the lower end of the inner peripheral surface 311 (hole 301A) of the ring portion 301. For example, the user bends the index finger and operates the operating member 331 with the fingertip of the index finger.

[0180] The operating member 332a is arranged near the top end of the upper surface 312a of the operation part 302a. The operating member 332b is arranged near the top end of the upper surface 312b of the holding part 302b. For example, the user operates the operating member 332a or the operating member 332b with the fingertip of the index finger.

[0181] The operating members 333a and 333b are respectively arranged near the front end and the rear end of the bottom surface 313. For example, the user operates the operating member 333a or the operating member 333b with the fingertip of the ring finger or the little finger.

[0182] The operating member 334 is arranged along the front - rear direction at the center of the bottom surface 313. For example, the user operates the operating member 334 with the fingertip of the thumb, the ring finger or the little finger.

[0183] For example, any type of operating member such as a button, a touchpad or a joystick can be used as the operating member 331, the operating member 332a, the operating member 332b, the operating member 333a, the operating member 333b and the operating member 334. However, the same type of operating member is used for the operating member 332a and the operating member 332b arranged at symmetric positions with the ring portion 301 as the center. Similarly, the same type of operating member is used for the operating member 333a and the operating member 333b arranged at symmetric positions with the ring portion 301 as the center.

[0184] For example, the operating member 331, the operating member 332a, the operating member 332b, the operating member 333a, the operating member 333b and the operating member 334 can be given any function. However, the operating member 332a and the operating member 332b arranged at symmetric positions with the ring portion 301 as the center can be given similar functions. Similarly, the operating member 333a and the operating member 333b arranged at symmetric positions with the ring portion 301 as the center can be given similar functions.

[0185] Specifically, for example, the operation member 331 is given the function of calling the main menu screen. For example, the operation members 332a and 332b are given the function of selecting virtual objects. For example, the operation members 332a and 332b are given functions other than the selection function of the operation members 333a and 333b. For example, the operation member 334 is given the function of calling the sub-menu screen.

[0186] Note that, for example, the operation members 332a and 332b can be given different functions, and the functions of the two are switched according to the direction in which the control device 113a is held. Similarly, for example, the operation members 333a and 333b can be given different functions, and the functions of the two are switched according to the direction in which the control device 113a is held.

[0187] As described above, regardless of whether the user grasps the controller device 113a in the forward direction or the backward direction, the user can perform similar operations.

[0188] Note that although it is assumed that the index finger is inserted into the loop portion 301, for example, the middle finger or the ring finger can also be inserted for use.

[0189] Note that hereinafter, when it is not necessary to distinguish between the operation members 332a and 332b from each other, they are simply referred to as the operation member 332. Hereinafter, when it is not necessary to distinguish between the operation members 333a and 333b from each other, they are simply referred to as the operation member 333.

[0190] <Example of Mark Arrangement>

[0191] For example, marks such as IR light-emitting elements can be provided in the controller device 113a. Then, the recognition unit 221 of the information processing device 111 can detect the marks of the controller device 113a based on an image sensed by the sensing unit 252 of the terminal device 112 or the like, and identify the relative position and orientation between the terminal device 112 and the controller device 113 based on the positions of the detected marks.

[0192] Figure 10 An example of the arrangement of the marks 351 of the controller device 113a is shown. Each mark 351 is represented by a black circle.

[0193] For example, as Figure 10As shown in A of [FIGURE REFERENCE], the markers 351 are arranged vertically on the right surface 314a and the left surface 314b to surround the outer periphery of the ring portion 301. For example, the markers 351 are arranged near the tops of the two side surfaces of the operation portion 302a and near the tops of the two side surfaces of the holding portion 302b. For example, the markers 351 are arranged near the front end and the rear end of the bottom surface 313.

[0194] Therefore, as Figure 11 shown in A to D of [FIGURE REFERENCE], in any orientation of the controller device 113a, at least a part of the markers 351 becomes visible without being covered by the user's hand.

[0195] On the other hand, for example, as Figure 12 shown in [FIGURE REFERENCE], the terminal device 112 includes a plurality of imaging devices 401. Each imaging device 401 constitutes the sensing unit 252 of the terminal device 112 ( Figure 4 ). Each imaging device 401 captures an image of the controller device 113a. The terminal device 112 sends sensing data including the captured image data obtained by imaging to the information processing device 111.

[0196] On the other hand, the control unit 202 of the information processing device 111 receives the sensing data. The identification unit 221 of the control unit 202 identifies the position and orientation of the controller device 113a relative to the terminal device 112 based on the light-emitting pattern of the markers 351 of the controller device 113a.

[0197] Note that, for example, as Figure 10 shown in B of [FIGURE REFERENCE], the markers 351 may be arranged in two columns in the lateral direction to surround the outer periphery of the ring portion 301. Further, for example, as Figure 10 shown in C of [FIGURE REFERENCE], the markers 351 may be arranged in three columns in the lateral direction to surround the outer periphery of the ring portion 301.

[0198] In this way, by arranging the markers 351 on the outer periphery of the ring portion 301, the size of the controller device 113a can be reduced.

[0199] <Example of the internal structure of the controller device 113a>

[0200] Next, an example of the internal structure of the controller device 113a will be described with reference to Figure 13 [FIGURE REFERENCE].

[0201] The controller device 113a is built-in with a haptic device 371, haptic devices 372a and 372b, a substrate 373, and a battery 374.

[0202] Each of the haptic devices 371, 372a, and 372b includes a device that presents (delivers) haptic stimuli (such as vibrations of a linear resonant actuator (LRA), an eccentric rotating mass device (ERM), a piezoelectric element, etc.).

[0203] The haptic device 371 is arranged near the lower end of the inner peripheral surface 311 of the ring portion 301 (near the operating member 331( Figure 9 )) and presents haptic stimuli near the lower end of the inner peripheral surface 311.

[0204] The haptic device 372a is arranged near the top end of the operating portion 302a (near the operating member 332a( Figure 9 )) and sends haptic stimuli near the top end of the operating portion 302a.

[0205] The haptic device 372b is arranged near the top end of the holding portion 302b (near the operating member 332b( Figure 9 )) and sends haptic stimuli near the top end of the holding portion 302b.

[0206] The substrate 373 is a substrate for controlling the controller device 113a and is arranged at approximately the center of the controller device 113a and below the haptic device 371.

[0207] The battery 374 is arranged below the substrate 373 in the controller device 113a and supplies power to each unit of the controller device 113a.

[0208] For example, as Figure 14 shown, when the controller device 113a is grasped by the user's right hand in the forward direction, the haptic device 371 presents haptic stimuli near the root joint of the thumb. The haptic device 372a presents haptic stimuli near the tip of the thumb and near the tip of the index finger. The haptic device 372b presents haptic stimuli near the root of the thumb and near the palm.

[0209] Note that the haptic device 371, the haptic device 372a, and the haptic device 372b are arranged at symmetric positions centered on the ring portion 301 in the front - rear direction of the controller device 113a. Therefore, even if the user holds the controller device 113a in the forward or backward direction, similar haptic stimuli are presented to the user's hand.

[0210] <Processing of the XR System 101>

[0211] Next, the processing of the XR system 101 will be described with reference to Figures 15 to 23 .

[0212] <Operating Member Control Processing>

[0213] First, it will be described with reference toFigure 15 The flowchart is used to describe the operation component control process executed by the XR system 101.

[0214] This process is executed, for example, when the user holds or switches the controller device 113a.

[0215] In step S1, the information processing device 111 performs hand recognition through hand tracking.

[0216] For example, the control unit 253 of the terminal device 112 sends sensing data including captured image data indicating the images captured by each imaging device 401 to the information processing device 111.

[0217] On the other hand, the control unit 202 of the information processing device 111 receives the sensing data. The recognition unit 221 of the control unit 202 performs hand recognition through hand tracking based on the captured image data included in the sensing data. Therefore, for example, the recognition unit 221 tracks the hand of the user holding the controller device 113a based on the marker 351 provided in the controller device 113a.

[0218] In step S2, the recognition unit 221 determines whether the hand holding the controller device 113a is recognized based on the result of the process in step S1. If it is determined that the hand holding the controller device 113a is not recognized, the process returns to step S1.

[0219] After that, the processes of steps S1 and S2 are repeatedly executed until it is determined in step S2 that the hand holding the controller device 113a is recognized.

[0220] On the other hand, if it is determined in step S2 that the hand holding the controller device 113a is recognized, the process proceeds to step S3.

[0221] In step S3, the recognition unit 221 recognizes the light-emitting pattern of the controller device 113a based on the captured image data. That is, the recognition unit 221 recognizes the light-emitting pattern through the marker 351 in the controller device 113a that is not hidden by the user's hand.

[0222] In step S4, the recognition unit 221 determines whether the grasping direction of the controller device 113a is recognized. Specifically, the recognition unit 221 attempts to recognize the grasping direction of the controller device 113a based on the recognition result of the hand of the user holding the controller device 113a and the recognition result of the light-emitting pattern of the controller device 113a. Then, if it is determined that the grasping direction of the controller device 113a is not recognized, the process returns to step S3.

[0223] After that, the processes of steps S3 and S4 are repeatedly executed until it is determined in step S4 that the grasping direction of the controller device 113a has been recognized.

[0224] On the other hand, in the case where it is determined in step S4 that the grasping direction of the controller device 113a has been recognized, the process proceeds to step S5.

[0225] In step S5, the operation control unit 222 disables the operation member on the palm side. For example, as Figure 16 shown, in the case of grasping the controller device 113a in the forward direction, the operation member 332b on the palm side is disabled. After that, for example, the recognition unit 221 and the operation control unit 222 ignore the operation input signal of the operation member 332b.

[0226] On the other hand, for example, in the case of grasping the controller device 113a in the backward direction, the operation member 332a on the palm side is disabled.

[0227] After that, the operation member control process ends.

[0228] Therefore, it is possible to prevent the operation member 332 from being erroneously operated by the user's palm.

[0229] As described above, the hand grasping the controller device 113a and the grasping direction are recognized, and the operability of the controller device 113a does not change regardless of the grasping direction of the controller device 113a.

[0230] Therefore, for example, as Figure 17 shown in A and B of, even if no special setting is made on the terminal device 112 side, the user can grasp the controller device 113a with the dominant hand regardless of which hand of the user is dominant.

[0231] Note that, for example, as Figure 18 shown in A and B of, the user can wear another controller device 113b such as a ring device on the non-dominant hand side for use.

[0232] In addition, for example, as Figure 19 shown in, the user can wear the controller device 113a on both hands for use.

[0233] <Haptic feedback control process>

[0234] Next, the haptic feedback control process executed by the XR system 101 will be described with reference to the Figure 20 flowchart.

[0235] This process starts, for example, when the power of the information processing device 111 is turned on, and ends when the power is turned off.

[0236] In step S51, the information processing device 111 identifies the state of the terminal device 112, the surrounding state, and the like.

[0237] Specifically, the sensing unit 252 of the terminal device 112 senses the state of the terminal device 112 and the state around the terminal device 112, and provides sensing data indicating the sensing result to the control unit 253. The control unit 253 sends the sensing data to the information processing device 111.

[0238] On the other hand, the control unit 202 of the information processing device 111 receives the sensing data.

[0239] The controller device 113a sends a controller signal including an operation input signal indicating the operation content for each operation member to the information processing device 111 via the terminal device 112.

[0240] On the other hand, the control unit 202 of the information processing device 111 receives the controller signal.

[0241] The identification unit 221 of the control unit 202 identifies the state of the terminal device 112, the state around the terminal device 112, the state of the controller device 113, the state of the user, the user operation, etc. based on the sensing data and the controller signal. For example, the identification unit 221 identifies the position and orientation of the terminal device 112. For example, the identification unit 221 identifies the line-of-sight direction of the user. For example, the identification unit 221 identifies the position and orientation of the controller device 113a relative to the terminal device 112. For example, the identification unit 221 identifies the operation content for the controller device 113a.

[0242] In step S52, the space control unit 223 of the information processing device 111 controls the XR space. Specifically, the space control unit 223 generates a virtual object to be displayed in the XR space based on at least a part of the identification result of the identification unit 221, and performs various calculations required for the construction, display, etc. of the XR space such as the behavior of the virtual object. The space control unit 223 generates display control information for controlling the display of the XR space based on the calculation result, and sends the display control information to the terminal device 112 via the communication unit 205, thereby controlling the display of the XR space by the terminal device 112.

[0243] The identification unit 221 identifies the type, position, orientation, etc. of the virtual object around the terminal device 112 (user) based on the information from the space control unit 223 and the like.

[0244] In step S53, the haptic rendering control unit 225 determines whether it is the timing to render haptic feedback based on at least one of the recognition result of the recognition unit 221 or the information from the spatial control unit 223. In the case where it is determined that it is not the timing to render haptic feedback, the process returns to step S51.

[0245] After that, the processes of steps S51 to S53 are repeatedly executed until it is determined in step S53 that it is the timing to render haptic feedback.

[0246] On the other hand, in the case where it is determined in step S53 that it is the timing to render haptic feedback, the process proceeds to step S54.

[0247] In step S54, the information processing device 111 controls the rendering of haptic feedback. Specifically, the haptic rendering control unit 225 generates haptic control information for causing the controller device 113a to render a haptic stimulus. The haptic rendering control unit 225 sends a haptic control signal to the controller device 113a via the terminal device 112.

[0248] On the other hand, the controller device 113a receives the haptic control information. Each haptic device of the controller device 113a renders a haptic stimulus based on the haptic control information.

[0249] After that, the process returns to step S51, and the processes in step S51 and subsequent steps are executed.

[0250] As described above, the haptic stimulus is appropriately rendered to the user by the controller device 113a.

[0251] Here, an example of a method for rendering the haptic feedback of the controller device 113a will be described with reference to Figures 21 to 23 Description of an example of a method for rendering the haptic feedback of the controller device 113a.

[0252] For example, in the case where the operation member 332a ( Figure 9 ) near the top end of the operation unit 302a of the controller device 113a includes a touchpad, and as shown in A of Figure 21 , when the fingertip of the index finger slides in the front-rear direction on the operation member 332a, a haptic stimulus is rendered to the fingertip of the index finger by the haptic device 372a ( Figure 13 ) arranged near the operation member 332a.

[0253] For example, as shown in B of Figure 21 , in the case where the user touches the button 431 in the XR space with the top end of the operation unit 302a of the controller device 113a, a haptic stimulus is rendered to the fingertip of the index finger by the haptic device 372a ( Figure 13 ).

[0254] For example, in the case where the controller device 113a or the hand holding the controller device 113a collides with a virtual object in the XR space, each haptic device of the controller device 113a is used to represent the impact caused by the collision.

[0255] For example, Figure 22 A in shows an example of a case where the tip of the operation unit 302a of the controller device 113a collides with the virtual object 441 in the XR space from above. In this case, for example, an upward vibration is presented near the tip of the operation unit 302a by the haptic device 372a ( Figure 13 ), and a downward vibration is presented near the tip of the holding unit 302b by the haptic device 372b ( Figure 13 ). Therefore, the user can feel the rotational force (moment) in the upward direction with respect to the controller device 113a.

[0256] For example, Figure 22 B in shows an example of a case where the tip of the operation unit 302a of the controller device 113a collides with the virtual object 441 in the XR space from below. In this case, for example, a downward vibration is presented near the tip of the operation unit 302a by the haptic device 372a ( Figure 13 ), and an upward vibration is presented near the tip of the holding unit 302b by the haptic device 372b ( Figure 13 ). Therefore, the user can feel the rotational force (moment) in the downward direction with respect to the controller device 113a.

[0257] For example, Figure 23 shows an example of a case where the tip of the operation unit 302a of the controller device 113a collides with the virtual object 441 in the XR space from the front. In this case, for example, by vibrating the haptic device 371 ( Figure 13 ) near the center of the controller device 113a, the entire controller device 113a vibrates. Therefore, the user can feel the reaction force from the virtual object 441 on the controller device 113a.

[0258] As described above, the operability of the controller device 113a can be improved. Thereby, the operability in the XR space is improved.

[0259] <<2. Application Examples>>

[0260] Next, application examples of the present technology will be described with reference to Figures 24 to 37 . Specifically, an example of performing the production line design of a factory by using the XR system 101 through 3D CAD or the like will be described.

[0261] In the design of a production line in a factory using XR, for example, the two processes of design processing and confirmation processing are repeatedly executed. In the design processing, for example, a virtual object is used to consider the layout of each object (for example, production line, worker, product, part, tool, fixture, etc.). In the confirmation processing, for example, each virtual object is arranged according to the design, and the workability of product assembly, maintenance, etc. in the production line is confirmed.

[0262] Here, the required behavior of the virtual object in the XR space is different between the design processing and the confirmation processing.

[0263] For example, in the confirmation processing, it is desired that the virtual object exhibits behavior similar to that during production line operations in the real world.

[0264] For example, as Figure 24 schematically shown in, at the site of a production line in the real world, a tool such as a screwdriver 1001 is suspended by a wire 1002, and the movable range is limited. In addition, a screw 1003 is magnetically attracted to the tip of the screwdriver 1001. Therefore, in the confirmation processing, it is desired to reproduce the physical phenomena in the real world as much as possible, so that the workability close to the real world can be reproduced and confirmed as much as possible.

[0265] On the other hand, in the design processing, since the layout of each virtual object is often changed in the XR space, there are cases where it is desired that the virtual object exhibits behavior different from the physical phenomena in the real world.

[0266] For example, as Figure 25 shown, it is desired that tools such as a virtual screwdriver 1022 and parts such as a virtual screw 1023 for assembling a virtual product 1021 can move to or be arranged in any position such as in the air.

[0267] On the other hand, for example, the XR system 101 controls the physical laws applied to each virtual object so that each virtual object in the XR space exhibits the behavior expected by the user in the design processing and the confirmation processing.

[0268] Note that the physical laws applied to the virtual object are not all physical laws in the real world, but are limited to the range of physical laws that can be reproduced in the XR space, such as interference and friction, elasticity, magnetism, and gravity between virtual objects.

[0269] <Virtual Object Behavior Control Processing>

[0270] Here, the virtual object behavior control processing executed in the case of performing production line design in the XR system 101 will be described with reference to Figure 26 the flowchart of.

[0271] This process starts, for example, when starting the production line design operation using the XR system 101, and ends when the production line design operation ends.

[0272] In step S101, the XR system 101 sets application conditions for physical laws for virtual objects.

[0273] For example, the user performs an operation to set application conditions for physical laws on a virtual object in the XR space, or inputs information indicating application conditions for physical laws via the operation input unit 201 of the information processing device 111.

[0274] The space control unit 223 of the information processing device 111 sets application conditions for physical laws for virtual objects in the XR space based on the user's setting operation and the information input by the user.

[0275] Note that, for example, the space control unit 223 can obtain information for setting application conditions for physical laws for virtual objects in the XR space from the outside by downloading information from a server or the like.

[0276] For example, the space control unit 223 sets switching conditions for the behavior mode of the XR space.

[0277] Here, the behavior mode is a mode for controlling the behavior of virtual objects in the XR space. For example, the behavior mode includes two modes: a physical law off mode and a physical law on mode.

[0278] The physical law off mode is a mode in which physical laws are not applied to virtual objects in the XR space. The physical law on mode is a mode in which physical laws are applied to virtual objects in the XR space.

[0279] Figure 27 An example of a function comparing the physical law off mode and the physical law on mode is shown.

[0280] In the physical law off mode, transparent display of virtual objects is possible. For example, in the XR space, virtual objects can be made transparent and the content behind or inside the virtual objects can be displayed.

[0281] In the physical law off mode, transparent display of virtual objects is not possible.

[0282] In the physical law off mode, the behavior of virtual objects may not follow physical laws. For example, in the XR space, virtual objects may not operate according to physical laws such as gravity or magnetism. For example, the user can freely move virtual objects in the XR space and can freely change the position or orientation of virtual objects.

[0283] In the physical law enabled mode, the behavior of virtual objects follows physical laws. For example, in the XR space, virtual objects operate according to physical laws such as gravity and magnetism. For example, in the XR space, the user can move virtual objects within a restricted range based on physical laws and can change the position or orientation of virtual objects within a restricted range based on physical laws.

[0284] In the physical law disabled mode, invisible virtual objects can be selected. For example, the user can select a virtual object that is hidden behind other virtual objects etc. and is thus invisible in the XR space.

[0285] In the physical law enabled mode, invisible virtual objects cannot be selected. For example, the user cannot select a virtual object that is hidden behind other virtual objects etc. and is thus invisible in the XR space.

[0286] In the physical law disabled mode, the size of virtual objects can be changed. For example, the user can freely change the size while the virtual object is being displayed in the XR space.

[0287] In the physical law enabled mode, the size of virtual objects cannot be changed. For example, the size of the virtual object is fixed to a preset value, and the user cannot freely change the size of the virtual object in the XR space. For example, in order to change the size of the virtual object, operations such as stopping the display of the virtual object and changing the setting value are required.

[0288] Note that in the physical law enabled mode, physical laws do not necessarily need to be applied to all virtual objects. For example, in the physical law enabled mode, virtual objects to which physical laws are applied can be set. In this case, for example, they can be set individually, or virtual objects to which physical laws are applied can be set for each type of virtual object.

[0289] In addition, in the physical law enabled mode, not all physical laws will necessarily be applied. For example, the types of physical laws to be applied in the physical law enabled mode can be set. In this case, for example, the types of physical laws to be applied can be set for each individual virtual object, or the types of physical laws to be applied can be set for each type of virtual object.

[0290] In addition, in the physical law enabled mode, physical laws are not always applied. For example, in the physical law enabled mode, conditions for applying physical laws can be set. In this case, conditions for applying physical laws can be set for each individual virtual object, or conditions for applying physical laws can be set for each type of virtual object.

[0291] In step S102, the space control unit 223 of the information processing device 111 sets the behavior mode to the default mode. That is, the space control unit 223 sets the behavior mode to the mode that is default - set between the physical law off mode and the physical law on mode.

[0292] In step S103, the recognition unit 221 of the information processing device 111 recognizes the state of elements related to XR. For example, the recognition unit 221 recognizes the state of the terminal device 112, the state around the terminal device 112, the state of the controller device 113, the state of the user, the user operation, the state of the XR space, etc. based on at least one of the operation input signal from the operation input unit 201, the sensing data sent from the terminal device 112, the controller signal sent from the controller device 113, the information from the operation control unit 222, or the information from the space control unit 223. The state of the XR space includes the state of each virtual object in the XR space.

[0293] In step S104, the space control unit 223 of the information processing device 111 determines whether the mode - switching condition is satisfied based on the recognition result of the state of elements related to XR. If it is determined that the mode - switching condition is satisfied, the process proceeds to step S105.

[0294] In step S105, the space control unit 223 of the information processing device 111 switches the behavior mode. For example, if the current behavior mode is the physical law off mode, the mode is changed to the physical law on mode. For example, if the current behavior mode is the physical law on mode, the mode is changed to the physical law off mode.

[0295] After that, the process proceeds to step S106.

[0296] On the other hand, if it is determined in step S104 that the mode - switching condition is not satisfied, the process of step S105 is skipped, and the process proceeds to step S106.

[0297] In step S106, the space control unit 223 of the information processing device 111 controls the behavior of the virtual objects in the XR space according to the applied physical laws. For example, the space control unit 223 controls the display unit 254 of the terminal device 112 to control the behavior of each virtual object according to the physical laws applied to each virtual object in the XR space.

[0298] The following is an example of the behavior of virtual objects in the XR space when the physical law off mode is set.

[0299] For example, a virtual object can be moved to any position or located regardless of interference with another virtual object or real object, etc. Specifically, for example, virtual products, jigs, tools (such as screwdrivers, tweezers, hammers, etc.), parts (such as screws, etc.) can be positioned in the air without following gravity. For example, virtual products, jigs, tools, parts, etc. can be moved to any position to adjust the position and the movable range. For example, during consideration of the attachment position of a virtual seal and the placement of virtual tweezers, the virtual seal and the virtual tweezers can be moved to any position.

[0300] For example, a virtual screw is not attracted by a virtual screwdriver.

[0301] For example, virtual objects that are not used for consideration can be hidden.

[0302] Therefore, the user can perform design work in an environment suitable for production line design without applying the physical laws of the real world.

[0303] The following are examples of the behavior of virtual objects in the XR space when the physical law activation mode is set.

[0304] For example, a virtual object moves within a range where it does not interfere with other virtual objects or real objects. For example, a virtual screwdriver moves only within the range corresponding to a suspended virtual spring.

[0305] For example, the positions of a virtual product and a jig are fixed.

[0306] For example, a virtual object falls or rolls according to gravity.

[0307] For example, a virtual screw is attracted by magnetic force to a virtual screwdriver.

[0308] For example, a virtual screw can be inserted only in an appropriate direction relative to a virtual assembly object.

[0309] For example, in the case where a seal is attached to a virtual target to be assembled, after the attachment starts, due to the adhesive force, fine adjustment of the position becomes difficult.

[0310] By applying these physical laws, the user can perform operations such as confirmation of assemblability and maintainability in an environment close to the real world.

[0311] Note that the space control unit 223 controls the behavior of virtual objects based on, for example, the recognition result of the state of the XR space (the state of the virtual space and the real part). For example, when gravity is applied to a virtual object, the space control unit 223 recognizes the position or range of the floor based on the state of the XR space, and controls the action of the virtual object based on the recognition result so that it lands on the floor.

[0312] After that, the process returns to step S103, and the processes of steps S103 to S106 are repeatedly executed.

[0313] <Specific example of virtual object behavior control processing>

[0314] Next, each specific example of the virtual object behavior control processing described above will be described with reference to Figure 26 the flowchart shown in.

[0315] <Specific example of mode switching condition>

[0316] First, specific examples of behavior mode switching conditions will be described.

[0317] For example, the behavior mode is switched based on at least one of the state of the XR space or the state of the user using the XR space. The state of the XR space includes, for example, at least one of the state or attributes of virtual objects in the XR space. The state of a virtual object includes, for example, at least one of the operation method for the virtual object or the position of the virtual object in the XR space. The operation method for a virtual object includes, for example, at least one of the type, attributes, or usage method of a real or virtual tool for operating the virtual object (e.g., how to hold it).

[0318] <Example of switching behavior mode based on the tool for operating a virtual object>

[0319] Specifically, for example, the behavior mode can be switched based on the tool for operating a virtual object.

[0320] For example, as Figures 28 to 30 schematically shown in, the behavior mode can be switched based on the usage method of the controller device 113, and more specifically, based on how to hold the controller device 113.

[0321] For example, in the design process, it is assumed that a virtual object is operated by using an indicating device such as a laser pointer.

[0322] On the other hand, for example, as Figure 28 shown in, in the case of grasping the controller device 113 in a way that imitates the holding manner of a laser pointer, in other words, in the case of grasping the controller device 113 as if holding a laser pointer, the behavior mode can be set to the physical law off mode. In this case, for example, a light ray as a virtual beam can be output from the tip of the controller device 113, and the virtual object can be operated using this light ray.

[0323] For example, in the confirmation process, it is assumed that an input device that imitates a real tool such as a screwdriver or tweezers is used to operate a virtual object.

[0324] On the other hand, for example, when the controller device 113 is grasped in a manner that mimics the way of holding a real tool, in other words, when the controller device 113 is grasped as if holding a real tool, the behavior mode can be set to the physical law on mode. For example, in the case of grasping the controller device 113c as if holding a screwdriver as shown in Figure 29 or grasping the controller device 113c as if holding tweezers as shown in Figure 30 , the behavior mode can be set to the physical law on mode.

[0325] Next, specific examples of the behavior mode switching conditions will be described taking the more specific controller device 113 as an example.

[0326] First, examples of the behavior mode switching conditions in the case of using the controller device 113a described above with reference to Figures 31 to 34 will be described with reference to Figure 5 and the like.

[0327] Figure 31 FIG. shows an example of the arrangement of the capacitive touch sensors and buttons of the controller device 113a.

[0328] For example, a touch sensor is arranged in the front-rear direction at a position P1a near the center of the right surface of the operation unit 302a. A touch sensor and a button are arranged in the front-rear direction at a position P1b (only the reference numeral is shown) on the opposite side of the position P1a at the center of the left surface of the operation unit 302a. A touch sensor and a button are arranged in the front-rear direction at a position P2 near the center of the upper surface 312a of the operation unit 302a. A touch sensor is arranged in the front-rear direction at the center of the right surface of the holding unit 302b.

[0329] For example, when the touch sensor at the position P1b is activated, the recognition unit 221 of the information processing device 111 recognizes that the controller device 113a is grasped as if holding tweezers as shown in Figure 32 . That is, it is recognized that the controller device 113a is grasped in such a way that the side surface of the operation unit 302 is sandwiched between the thumb and index finger of the right hand without inserting the finger into the ring portion 301. In this case, the behavior mode is set to the physical law off mode. In addition, the operation control unit 222 of the information processing device 111 assigns a determination function to the button at the position P1b and operates it with the index finger.

[0330] For example, when the touch sensor at the position P1a is activated and the touch sensor at the position P1b is not activated, the recognition unit 221 recognizes that the controller device 113a is grasped in the normal holding manner as shown in Figure 33 . That is, it is recognized that it is grasped in the same way as described above with reference toFigure 6 Grip the controller device 113a with the right hand in a manner similar to the described manner along the forward direction. In this case, the behavior mode is set to the physical law on mode. In addition, the operation control unit 222 assigns a determination function to the button at the position P2, and operates the determination function with the index finger.

[0331] For example, in the case where the touch sensor at the position P3 reacts while the touch sensors at the positions P1a and P1b do not react, the recognition unit 221 recognizes that the controller device 113a is gripped as if Figure 34 shown in, for example, gripping a screwdriver. That is, it is recognized that the controller device 113a is gripped in a manner of holding the holding portion 302b with the right hand. In this case, the behavior mode is set to the physical law on mode.

[0332] For example, in the case where the touch sensors at the positions P1a, P1b, and P3 do not react, the recognition unit 221 recognizes that the controller device 113a is not gripped.

[0333] Next, an example of the behavior mode switching condition in the case of using the controller device 113c as a pen-shaped input device will be described with reference to Figure 35 FIG.

[0334] Figure 35 FIG. shows an example of the arrangement of the capacitive touch sensors and buttons of the controller device 113c.

[0335] For example, a touch sensor is arranged at a position P11 near the end on the upper side (upper surface) of the side surface of the controller device 113c. For example, a touch sensor is arranged at a position P2 at the top end of the controller device 113c. For example, a touch sensor and a button are arranged at a position P13 near the top on the left side (left side surface) of the side surface of the controller device 113c.

[0336] For example, in the case where the touch sensor at the position P11 reacts, the recognition unit 221 of the information processing device 111 recognizes that the controller device 113c is gripped as if gripping a screwdriver. That is, it is assumed that in the case of gripping a screwdriver or the like, the controller device 113c is only gripped near the end. In this case, the behavior mode is set to the physical law on mode.

[0337] For example, in the case where the touch sensors at the positions P12 and P13 react while the touch sensor at the position P11 does not react, the recognition unit 221 recognizes that the controller device 113c is gripped as if gripping a laser pointer. In this case, the behavior mode is set to the physical law off mode.

[0338] For example, when the touch sensor at position P13 responds while the touch sensors at positions P11 and P12 do not respond, the recognition unit 221 recognizes that the controller device 113c is grasped as if holding tweezers. In this case, the behavior mode is set to the physical law enabled mode.

[0339] For example, the behavior mode can be switched based on the attributes (such as shape, type, etc.) of the tool used to operate the virtual object.

[0340] For example, when the controller device 113 is deformable, the behavior mode can be switched based on the shape of the controller device 113. For example, when the shape of the controller device 113 becomes similar to the shape of a real tool, the behavior mode can be set to the physical law enabled mode. Specifically, for example, when the shape of the controller device 113 is like tweezers, the behavior mode is set to the physical law enabled mode. For example, when the controller device 113 extends like a pointing stick, the behavior mode can be set to the physical law disabled mode.

[0341] Note that when using a virtual tool instead of the controller device 113, the behavior mode can be switched in the same way based on the shape of the virtual tool.

[0342] For example, when operating a virtual object with a real tool (such as a screwdriver, tweezers, hammer, etc.), the behavior mode can be set to the physical law enabled mode.

[0343] Note that, for example, when using a real tool, by setting a marker in the tool and registering information about the marker in the information processing device 111, the recognition unit 221 of the information processing device 111 can recognize the presence, position, and orientation of the real tool based on the marker.

[0344] <Example of switching behavior mode based on user state>

[0345] For example, the behavior mode can be switched based on the user's state.

[0346] Specifically, for example, assume the cases of performing CAD design using the XR system 101, using the information processing device 111 alone, using the operation input unit 201 of the information processing device 111 and the terminal device 112, and using the terminal device 112 and the controller device 113.

[0347] On the other hand, for example, in CAD design using the operation input unit 201 of the information processing device 111 and the terminal device 112, the behavior mode can be switched based on the user's gestures by using hand tracking.

[0348] For example, as Figure 36As shown in A of , in the case of operating a virtual object using light, the behavior mode can be set to the physical law off mode. For example, in the case of moving the virtual screw 1101 by the ray 1102, after the virtual screw 1101 is released, the virtual screw 1101 continues to float in the air.

[0349] For example, as Figure 36 As shown in B of , in the case of directly operating a virtual object by the user's hand, the behavior mode can be set to the physical law on mode. For example, in the case of moving the virtual screw 1101 by hand, after the virtual screw 1101 is released, the virtual screw 1101 drops like in the real world.

[0350] For example, the behavior mode can be switched based on the user's actions. For example, in the case of the user performing an operation of assembling a product on an actual production line, the behavior mode can be set to the physical property on mode. Specifically, for example, in the case of the user standing up from a chair or squatting on the floor, the behavior mode can be set to the physical property on mode.

[0351] For example, the behavior mode can be switched based on whether the user operates with the dominant hand. For example, in the case of meeting the conditions for setting the physical law on mode, when the user performs an operation with the hand opposite to the dominant hand due to an injury or the like, considering the operability of the user, the behavior mode can be set to the physical law off mode.

[0352] Note that, for example, the user can preset the dominant hand in advance, or the learning unit 226 of the information processing device 111 can learn the hand that the user often uses and automatically identify the dominant hand.

[0353] For example, the behavior mode can be switched based on the position where the user uses XR.

[0354] For example, in the case of using XR at the place where the user is assumed to perform a design process (such as an office, etc.), the behavior mode can be set to the physical law on mode. On the other hand, for example, in the case of using XR at the place where the user is assumed to perform a confirmation process (such as a factory production line, an aircraft hangar, etc.), the behavior mode can be set to the physical law off mode.

[0355] <Examples of Switching the Behavior Mode Based on the State of the Virtual Object>

[0356] For example, the behavior mode can be switched based on the state of the virtual object in the XR space.

[0357] Specifically, for example, the behavior mode can be switched based on the placement position of the virtual design object in the XR space.

[0358] For example, when the virtual camera device in the design is placed on a table, it can be speculated that the user is confirming the assemblability of the camera device, and thus the behavior mode can be set to the physical law on mode.

[0359] For example, when the virtual vehicle in the design is placed on the floor and the user is in the driver's seat or the passenger seat, it can be speculated that the user is confirming the assemblability of the vehicle, and thus the behavior mode can be set to the physical law on mode.

[0360] <Examples of background switching behavior modes based on XR (VR)>

[0361] For example, when the user is using VR, the behavior mode can be switched based on the VR background (virtual background).

[0362] For example, when the VR background is a place where it is assumed that the user is performing a design process (e.g., an office, etc.), the behavior mode can be set to the physical law on mode. On the other hand, for example, when the VR background is a place where it is assumed that the user is performing a confirmation process (e.g., a factory production line, an aircraft hangar, etc.), the behavior mode can be set to the physical law off mode.

[0363] <Examples of manually setting the behavior mode>

[0364] For example, the user can set the behavior mode.

[0365] Note that in this case, for example, the space control unit 223 of the information processing device 111 can set the physical law on mode every predetermined time period. Thus, for example, the user can be prompted to perform a confirmation process instead of only performing a design process.

[0366] For example, when the behavior mode switching condition is satisfied, the space control unit 223 can prompt the user to switch the behavior mode without automatically switching the behavior mode. For example, the space control unit 223 can display a window for selecting whether to change the behavior mode, a button for changing the behavior mode, etc. in the XR space, and prompt the user to switch the behavior mode. For example, the space control unit 223 can prompt the user to switch the behavior mode through an audio message.

[0367] Then, the space control unit 223 can control the switching of the behavior mode according to the user's selection. For example, the space control unit 223 can switch the behavior mode when the user selects the switching of the behavior mode, and continue the current behavior mode when the user does not select the switching of the behavior mode.

[0368] For example, in the case of prompting a switching of the behavior mode, the spatial control unit 223 may set the timing of switching the behavior mode without immediately switching the behavior mode. For example, the spatial control unit 223 may be able to set the time until the behavior mode is switched (e.g., how many minutes from now, etc.) or the moment when the behavior mode is switched.

[0369] For example, the spatial control unit 223 may prompt the user to switch the behavior mode within a predetermined period (e.g., one month) after enabling the behavior mode switching function, without automatically switching the behavior mode. Then, for example, the spatial control unit 223 may automatically switch the behavior mode at a certain point in time after the predetermined period, when the probability of following the prompt of the behavior mode is equal to or greater than a predetermined threshold.

[0370] For example, the automatic switching and manual switching of the behavior mode can be switched according to conditions. For example, the behavior mode can be automatically switched when preset conditions are met, and cannot be automatically switched when the preset conditions are not met.

[0371] <Specific examples of the application conditions of physical laws>

[0372] Next, an example of controlling the application of physical laws for each virtual object in the physical law activation mode will be described.

[0373] For example, similar to the case of switching the behavior mode described above, the application of physical laws is controlled for each virtual object based on at least one of the state of the XR space or the state of the user using the XR space.

[0374] <Example of controlling the application of physical laws based on the attributes of virtual objects>

[0375] For example, the type of physical laws applied can be controlled based on the attributes of the virtual object (e.g., type, material, shape, size, coefficient of friction, etc.).

[0376] For example, a magnetic force can be applied when the virtual object includes metal, and a magnetic force cannot be applied when the virtual object does not include metal. Specifically, for example, a magnetic force can be applied when the virtual screw includes metal, and a magnetic force cannot be applied when the virtual screw does not include metal.

[0377] For example, whether to apply magnetic force or gravity can be switched based on the size of a virtual object. For example, magnetic force can be applied when the size of a virtual screw is less than a predetermined threshold, and magnetic force cannot be applied when the size of the virtual screw is equal to or greater than the predetermined threshold. For example, gravity can be applied when the size of the virtual object is equal to or greater than a predetermined threshold (e.g., one side is 1 m or more), and gravity cannot be applied when the size of the virtual object is less than the predetermined threshold.

[0378] For example, whether to apply physical laws can be switched based on the display size of a virtual object in the XR space. For example, physical laws can be applied to the virtual object when the difference between the display size of the virtual object and the size of the corresponding real object is within a predetermined range. On the other hand, for example, physical laws cannot be applied to the virtual object when the difference between the display size of the virtual object and the size of the corresponding real object exceeds the predetermined range.

[0379] <Example of controlling the application of physical laws based on the operation method for a virtual object>

[0380] For example, whether to apply physical laws can be switched based on the operation method of the user for the virtual object.

[0381] For example, when the user carefully handles the virtual object (e.g., when gently placing it on a table, etc.), physical laws can be applied to the virtual object. On the other hand, for example, when the user roughly handles the virtual object (e.g., when throwing a ball), physical laws cannot be applied to the virtual object.

[0382] This is because, for example, it is assumed that when the user wants to move the virtual object to one side during the design process, the user will quickly move the virtual object.

[0383] For example, whether to apply physical laws to the virtual object can be switched based on the gesture of the user for the virtual object. For example, when the user performs a grasping gesture or a gesture of grasping a door handle on the virtual object, physical laws can be applied to the virtual object.

[0384] <Display example of a virtual object>

[0385] For example, the space control unit 223 of the information processing apparatus 111 can distinguish the display mode of the virtual object to which physical laws are applied and the display mode of the virtual object to which physical laws are not applied in the XR space. For example, the space control unit 223 can change the transparency of the virtual object to which physical laws are applied and the transparency of the virtual object to which physical laws are not applied. For example, as Figure 37 shown, the virtual screw 1121 to which physical laws are not applied can slowly rotate in the air.

[0386] Therefore, the user can easily distinguish between virtual objects to which physical laws are applied and virtual objects to which physical laws are not applied, and the operability is improved.

[0387] For example, in the physical law off mode, the space control unit 223 can change the display mode of virtual objects to which physical laws are applicable and the display mode of virtual objects to which physical laws are not applicable in the XR space. For example, the space control unit 223 can change the transparency of virtual objects to which physical laws are applicable and the transparency of virtual objects to which physical laws are not applicable.

[0388] Therefore, before being set to the physical law on mode, the user can easily distinguish between virtual objects to which physical laws are applicable and virtual objects to which physical laws are not applicable, and can use the results for determining, such as switching of the behavior mode.

[0389] For example, as Figure 37 shown, an area A1 for displaying virtual tools such as a virtual screwdriver 1122 and a virtual tweezer 1123 can be displayed in the XR space. Then, for example, when the user holds a virtual tool in the area A1, the physical law corresponding to the tool held by the user (the tool used by the user) can be applied to the virtual object in the XR space.

[0390] For example, when the user holds and operates the virtual screwdriver 1122, the virtual screw 1121 is magnetically attracted toward the screwdriver 1122, but when the user holds and operates the virtual tweezer 1123, the virtual screw 1121 may not be attracted toward the tweezer 1123.

[0391] <Learning process for application conditions of physical laws>

[0392] For example, the learning unit 226 of the information processing device 111 can learn the application conditions of physical laws for virtual objects based on the user's usage history and the like.

[0393] For example, the learning unit 226 can learn the user's preference for physical laws applied to virtual tools based on the user's usage history of virtual tools.

[0394] For example, in the case where magnetism is applied to the virtual screwdriver by default, or in the case where it is recommended to apply magnetism to the virtual screwdriver by default, when the user selects not to apply magnetism to the virtual screwdriver up to or more than a predetermined number of times, or selects with a probability equal to or greater than a predetermined threshold, the learning unit 226 learns that the user prefers a non-magnetic screwdriver. Then, for example, the space control unit 223 does not apply magnetism to the user's virtual screwdriver based on the learning result. Therefore, the operability is improved for users who prefer non-magnetic screwdrivers.

[0395] For example, the spatial control unit 223 can apply the learning result of applying physical laws to a certain virtual tool to another virtual tool. For example, when the learning unit 226 learns that gravity is not applied to the virtual screwdriver, the spatial control unit 223 can also not apply gravity to another tool (such as tweezers, etc.).

[0396] For example, the learning unit 226 can learn the physical laws to be applied to virtual tools for each group instead of for each user.

[0397] Note that the classification method of groups is not particularly limited. For example, groups are classified by workplace, department, school, and class. In addition, for example, it is assumed that groups are formed via a network.

[0398] In this case, when a new user joins a group, physical laws are applied to the virtual tools used by the new user based on the learning results in the group. For example, when it is set in the group that no magnetic force is applied to the virtual screwdriver, this setting is also applied to the new user.

[0399] <Example of Application of Physical Laws in the Case where the XR Space is Shared by Multiple Users>

[0400] For example, when multiple users share the XR space and operate the same virtual object, when the physical laws applied to the virtual object are different for each user, the behavior of the virtual object is different for each user. Therefore, for example, a phenomenon where different videos are seen among users in the same XR space may occur. For example, there may be a situation where the virtual object for a certain user falls to the floor according to physical laws, while the virtual object for another user floats in the air without following physical laws.

[0401] On the other hand, when multiple users operate the same virtual object, the physical laws applied to the virtual object can be common among users. For example, when the switching conditions are met for all users, changes can be made to whether to apply physical laws and the type of physical laws to be applied. For example, only a specific user can set whether to apply physical laws and the type of physical laws to be applied, and other users can follow the settings of that user. For example, the priorities of users can be set, and users with low priorities can follow the settings of users with high priorities.

[0402] As described above, appropriate physical laws are applied to each virtual object in the XR space at appropriate timings. For example, in a scenario where it is estimated that the user intends to perform a design process, physical laws are not applied to the virtual object, while in a scenario where it is estimated that the user intends to perform a confirmation process, physical laws are applied to the virtual object. Further, the application of physical laws seamlessly switches between the design process and the confirmation process, and thus, the behavior of the virtual object is seamlessly switched. As a result, the operability of the user is improved.

[0403] <<3. Modification>>

[0404] Hereinafter, modifications of the above-described embodiments of the present technology will be described.

[0405] <Modification Regarding the Controller Device 113a>

[0406] In the above description, an example has been described in which the controller device 113a can be grasped in the forward direction or the backward direction. However, for example, it can also be set to grasp the controller device only in the forward direction.

[0407] In this case, the operation unit 302a and the holding unit 302b do not necessarily have a shape that is symmetric about the ring unit 301, and for example, the operation unit 302a and the holding unit 302b can have different shapes. Further, the operation members 332b and 333b of the holding unit 302b can be removed.

[0408] For example, the controller device 113a can use a material other than resin, such as metal.

[0409] <Modification Regarding the Sharing of Processing>

[0410] For example, a part of the processing of the information processing device 111 can be executed by the terminal device 112.

[0411] For example, the terminal device 112 can execute all or part of the processing of the information processing unit 211 of the information processing device 111. For example, the terminal device 112 can independently present the XR space without being controlled by the information processing device 111. For example, the information processing device 111 and the terminal device 112 can independently share and execute processing such as the construction of the XR space.

[0412] <Modification Regarding the Virtual Object Behavior Control Processing>

[0413] For example, the physical law activation mode can be classified into two or more modes, and the type of physical law applied to each mode or the type of virtual object to which the physical law is applied can be switched.

[0414] For example, in the case where no behavior pattern is provided, the space control unit 223 of the information processing apparatus 111 can control the physical laws to be applied for each virtual object or for each type of virtual object.

[0415] In the above description, although an example in which the virtual object behavior control process described above is applied to the production line design using 3D CAD has been described, the virtual object behavior control process can also be applied to other applications using 3D CAD. Figure 26 For example, in the case of developing a product using 3D CAD, similar to the case of the above circuit design, the behavior pattern can be switched between the design process of designing the product and the confirmation process of confirming the assemblability and maintainability of the product.

[0416] For example,

[0417] For example, Figure 38 Schematically shows a case where a product 1201 (in this example, a chair) is developed using 3D CAD in the XR space.

[0418] For example, in the design process, the virtual part 1202 combined with the virtual product 1201 is displayed floating in the air without following gravity. Therefore, the user can quickly combine the product 1201 and the part 1202, and the design work becomes efficient.

[0419] On the other hand, for example, in the confirmation process, the virtual parts 1203 to 1206 combined with the virtual product 1201 fall to the ground without following gravity. Therefore, the user can easily find problems that occur in reality, such as not being able to find parts when assembling the product 1201.

[0420] In addition, the virtual object behavior control process can be applied to, for example, an application in the XR space that mixes a scene that needs to follow physical laws and a scene that does not need to follow physical laws, or an application in the XR space that mixes a space that needs to follow physical laws and a space that does not need to follow physical laws.

[0421] For example, as Figure 39 shown, the virtual object behavior control process can be used for the simulation of a surgery using XR.

[0422] For example, in the XR space, an area A11 where physical laws are not applied is set beside the user U1 who is a surgeon. For example, in the case where virtual surgical tools such as an electric scalpel or forceps are placed in the area A11, the virtual surgical tools float in the air, or the wiring is not drawn.

[0423] The area A11 is set to an image of a surgical nurse. For example, in the real world, it is assumed that a surgical nurse follows the instructions of the user U1 to hand a surgical tool to the user U1 or return the surgical tool received from the user U1 to a predetermined position.

[0424] On the other hand, for example, the space control unit 223 of the information processing device 111 can store the order of surgical tools to be used according to the surgical procedure, and when performing a simulation of the surgery, arrange the virtual surgical tools in the area A11 according to the stored order in accordance with the progress state of the surgery.

[0425] In areas other than the area A11, physical laws are applied to the virtual surgical tools, or wiring is considered. Therefore, the user U1 can confirm whether the surgical tool can be inserted into the patient's body as envisioned, taking into account the size, orientation, wiring, etc. of the surgical tool, similar to a surgery in the real world.

[0426] Note that, for example, only some of the physical laws may not be applied in the area A11.

[0427] <Other Modifications>

[0428] For example, the controller device 113a can be used not only for operations in the XR space but also for operations in two-dimensional and three-dimensional spaces such as games.

[0429] <<4. Others>>

[0430] <Configuration Example of a Computer>

[0431] The series of processes described above can be executed by hardware or by software. When the series of processes is executed by software, the program constituting the software is installed on a computer. Here, for example, the computer includes a computer integrated in dedicated hardware, a general-purpose personal computer capable of executing various functions by installing various programs, and the like.

[0432] Figure 40 is a block diagram showing a configuration example of the hardware of a computer that executes the above series of processes using a program.

[0433] In the computer 2000, a central processing unit (CPU) 2001, a read-only memory (ROM) 2002, and a random access memory (RAM) 2003 are interconnected via a bus 2004.

[0434] An input / output interface 2005 is also connected to the bus 2004. An input unit 2006, an output unit 2007, a storage unit 2008, a communication unit 2009, and a drive 2010 are connected to the input / output interface 2005.

[0435] The input unit 2006 includes an input switch, buttons, a microphone, an image sensor, etc. The output unit 2007 includes a display, a speaker, etc. The storage unit 2008 includes a hard disk, a non-volatile memory, etc. The communication unit 2009 includes a network interface, etc. The drive 2010 drives a removable medium 2011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0436] In the computer configured as described above, the above series of processes are executed by, for example, loading a program stored in the storage unit 2008 into the RAM 2003 via the input / output interface 2005 and the bus 2004 by the CPU 2001 and executing the program.

[0437] A program executed by the computer 2000 (CPU 2001) can be provided by being recorded on a removable medium 2011 such as a packaged medium, etc. In addition, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0438] In the computer 2000, by mounting the removable medium 2011 on the drive 2010, the program can be installed in the storage unit 2008 via the input / output interface 2005. In addition, the program can be received by the communication unit 2009 via a wired or wireless transmission medium and installed in the storage unit 2008. In addition to the above, the program can also be pre-installed in the ROM 2002 or the storage unit 2008.

[0439] Note that the program executed by the computer can be a program in which the processes are executed in time series in the order described herein, or can be a program in which the processes are executed in parallel or at necessary timings such as when a call is made, etc.

[0440] In addition, in this document, a system refers to a group of multiple components (devices, modules (parts), etc.), and it is not important whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network and a single device in which a plurality of modules are housed in one housing are both systems.

[0441] In addition, the embodiments of the present technology are not limited to the above embodiments, and various modifications can be made without departing from the gist of the present technology.

[0442] For example, the present technology can be configured as cloud computing, in which functions are shared by multiple devices via a network to process together.

[0443] In addition, each step described in the above flowchart can be executed by one device, or can be executed in a shared manner by multiple devices.

[0444] In addition, in the case where multiple processes are included in one step, the multiple processes included in one step can be executed by one device or, in a shared manner, by multiple devices.

[0445] <Example of Configuration Combination>

[0446] The present technology may also have the following configurations. (1)

[0448] An information processing apparatus, comprising:

[0449] An identification unit that identifies the state of elements related to XR (Cross Reality); and

[0450] A space control unit that controls the physical laws applied to virtual objects in the XR space based on the state of elements related to the XR. (2)

[0452] The information processing apparatus according to (1), wherein

[0453] The state of elements related to the XR includes at least one of the state of the XR space and the state of a user using the XR space, and

[0454] The space control unit controls the physical laws applied to the virtual objects based on at least one of the state of the XR space and the state of the user. (3)

[0456] The information processing apparatus according to (2), wherein

[0457] The state of the XR space includes at least one of the state and attributes of the virtual objects in the XR space and the virtual background in the XR space, and

[0458] The space control unit controls the physical laws applied to the virtual objects based on at least one of the state and attributes of the virtual objects in the XR space and the virtual background. (4)

[0460] The information processing apparatus according to (3), wherein

[0461] The state of the virtual object includes at least one of the operation method for the virtual object and the position of the virtual object in the XR space, and

[0462] The space control unit controls the physical laws applied to the virtual objects based on at least one of the operation method for the virtual object and the position of the virtual object in the XR space. (5)

[0464] The information processing apparatus according to (4), wherein

[0465] The method of operating the virtual object includes at least one of the type, attribute, or usage method of the tool for operating the virtual object, and

[0466] The space control unit controls the physical laws applied to the virtual object based on at least one of the type, attribute, and usage method of the tool. (6)

[0468] The information processing apparatus according to (5), wherein

[0469] The tool is a real tool or a virtual tool. (7)

[0471] The information processing apparatus according to (6), wherein

[0472] The real tool includes a real input device,

[0473] The usage method of the tool includes the way of holding the input device, and

[0474] The space control unit controls the physical laws applied to the virtual object based on the way of holding the input device. (8)

[0476] The information processing apparatus according to (7), wherein

[0477] The input device includes:

[0478] A ring portion into which a finger is inserted;

[0479] An operation portion that can be operated by the finger inserted into the ring portion; and

[0480] A holding portion that is held by the palm when the operation portion is operated by the finger. (9)

[0482] The information processing apparatus according to any one of (6) to (8), wherein

[0483] The space control unit applies the physical laws corresponding to the virtual tool to the virtual object. (10)

[0485] The information processing apparatus according to any one of (6) to (9) further includes:

[0486] A learning unit that, based on the user's usage history of the virtual tool, enables the user to learn physical laws applicable to the virtual tool. (11)

[0488] The information processing apparatus according to any one of (2) to (10), wherein

[0489] The state of the user includes at least one of the user's position, movement, and gesture, and

[0490] The space control unit controls the physical laws applied to the virtual object based on at least one of the user's position, movement, and gesture. (12)

[0492] The information processing apparatus according to any one of (1) to (11), wherein

[0493] The space control unit switches between a mode in which physical laws are applied to the virtual object and a mode in which physical laws are not applied to the virtual object based on the state of elements related to the XR. (13)

[0495] The information processing apparatus according to (12), wherein

[0496] When the conditions for switching the mode are met, the space control unit prompts the user to switch the mode and controls the switching of the mode according to the user's selection. (14)

[0498] The information processing apparatus according to any one of (1) to (13), wherein

[0499] The space control unit differentiates between the display mode of the virtual object to which physical laws are applied and the display mode of the virtual object to which physical laws are not applied in the XR space. (15)

[0501] The information processing apparatus according to any one of (1) to (14), wherein

[0502] The space control unit does not apply at least a part of the physical laws to the virtual object in a predetermined area of the XR space. (16)

[0504] The information processing apparatus according to any one of (1) to (15), wherein

[0505] When the XR space is shared by multiple users, the space control unit makes the physical laws applied to the virtual object shared among the users. (17)

[0507] The information processing apparatus according to any one of (1) to (16), wherein,

[0508] The space control unit sets whether to apply physical laws, the type of physical laws to be applied, or the conditions for applying physical laws for each of the virtual objects or each type of the virtual objects. (18)

[0510] The information processing apparatus according to any one of (1) to (17), wherein,

[0511] The space control unit controls the behavior of the virtual object based on the physical laws to be applied in the XR space. (19)

[0513] An information processing method, including, by an information processing apparatus:

[0514] Identifying the states of elements related to XR; and

[0515] Controlling the physical laws applied to virtual objects in the XR space based on the states of the elements related to the XR.

[0516] Note that the effects described herein are merely examples and not limitations, and other effects may be provided.

[0517] List of reference numerals

[0518] 101 XR system

[0519] 111 Information processing apparatus

[0520] 112 Terminal device

[0521] 113, 113a, 113b Controller device

[0522] 202 Control unit

[0523] 203 Display unit

[0524] 211 Information processing unit

[0525] 221 Identification unit

[0526] 222 Operation control unit

[0527] 223 Space control unit

[0528] 224 Audio control unit

[0529] 225 Tactile rendering control unit

[0530] 226 learning unit

[0531] 252 sensing unit

[0532] 253 control unit

[0533] 254 display unit

[0534] 255 audio output unit

[0535] 301 ring part

[0536] 301A hole

[0537] 302a operation part

[0538] 302b holding part

[0539] 312a, 312 upper surface

[0540] 313 bottom surface

[0541] 331 to 334 operating members

[0542] 351 mark

[0543] 371 to 372b tactile devices

[0544] 401 imaging device

Claims

1. An information processing device, comprising: an identification unit that identifies the state of elements related to XR (Cross Reality); and a space control unit that controls the physical laws applied to virtual objects in the XR space based on the state of elements related to the XR.

2. The information processing device according to claim 1, wherein the state of elements related to the XR includes at least one of the state of the XR space and the state of a user using the XR space, and the space control unit controls the physical laws applied to the virtual objects based on at least one of the state of the XR space and the state of the user.

3. The information processing device according to claim 2, wherein the state of the XR space includes at least one of the state and attributes of the virtual objects in the XR space and the virtual background in the XR space, and the space control unit controls the physical laws applied to the virtual objects based on at least one of the state and attributes of the virtual objects in the XR space and the virtual background.

4. The information processing device according to claim 3, wherein the state of the virtual object includes at least one of the operation method for the virtual object and the position of the virtual object in the XR space, and the space control unit controls the physical laws applied to the virtual objects based on at least one of the operation method for the virtual object and the position of the virtual object in the XR space.

5. The information processing device according to claim 4, wherein the operation method for the virtual object includes at least one of the type, attributes, and usage method of a tool for operating the virtual object, and the space control unit controls the physical laws applied to the virtual objects based on at least one of the type, attributes, and usage method of the tool.

6. The information processing device according to claim 5, wherein the tool is a real tool or a virtual tool.

7. The information processing device according to claim 6, wherein the real tool includes a real input device, the usage method of the tool includes the holding manner of the input device, and the space control unit controls the physical laws applied to the virtual objects based on the holding manner of the input device.

8. The information processing device according to claim 7, wherein the input device includes: a ring portion into which a finger is inserted; an operation portion that can be operated by the finger inserted into the ring portion; and a holding portion that is held by the palm when the operation portion is operated by the finger.

9. The information processing device according to claim 6, wherein the space control unit applies the physical laws corresponding to the virtual tool to the virtual objects.

10. The information processing device according to claim 6, further comprising: a learning unit that learns, for the user, the physical laws applied to the virtual tool based on the user's usage history of the virtual tool.

11. The information processing apparatus according to claim 2, wherein, the state of the user includes at least one of the user's position, action, and gesture, and the spatial control unit controls the physical laws applied to the virtual object based on at least one of the user's position, action, and gesture.

12. The information processing apparatus according to claim 1, wherein, the spatial control unit switches between a mode in which physical laws are applied to the virtual object and a mode in which physical laws are not applied to the virtual object based on the state of the elements related to the XR.

13. The information processing apparatus according to claim 12, wherein, when the conditions for switching the mode are satisfied, the spatial control unit prompts the user to switch the mode and controls the switching of the mode according to the user's selection.

14. The information processing apparatus according to claim 1, wherein, the spatial control unit differentiates between the display mode of the virtual object to which physical laws are applied and the display mode of the virtual object to which physical laws are not applied in the XR space.

15. The information processing apparatus according to claim 1, wherein, the spatial control unit does not apply at least a part of the physical laws to the virtual object in a predetermined area of the XR space.

16. The information processing apparatus according to claim 1, wherein, when the XR space is shared by multiple users, the spatial control unit makes the physical laws applied to the virtual object shared among the users.

17. The information processing apparatus according to claim 1, wherein, the spatial control unit sets whether to apply physical laws, the type of physical laws to be applied, or the conditions for applying physical laws for each virtual object or each type of virtual object.

18. The information processing apparatus according to claim 1, wherein, the spatial control unit controls the behavior of the virtual object based on the physical laws to be applied in the XR space.

19. An information processing method, including, by an information processing apparatus: identifying the state of the elements related to the XR; and controlling the physical laws applied to the virtual object in the XR space based on the state of the elements related to the XR.

Images