Information processing device, information processing system, and information processing method

Abstract

An information processing device according to one embodiment of the present disclosure comprises: a setting unit that sets a surface of an object in a real space as a reference surface or sets a point in the object as a reference point; a processing unit that converts a three-dimensional position of a pointing device present within an operating range based on the reference surface or the reference point in the real space into a three-dimensional position of a pointer in a virtual space; and a control unit that causes a display device to display the virtual space including the pointer.

Description

Information processing device, information processing system, and information processing method

[0001] The present disclosure relates to an information processing device, an information processing system, and an information processing method.

[0002] Displays capable of stereoscopic display (displays capable of stereoscopic viewing), such as SRDs (Spatial Reality Displays) and HMDs (Head Mounted Displays), are becoming increasingly popular. For example, a user can use an operating device to move a pointer or the like in a three-dimensional space displayed on a display capable of stereoscopic display, thereby performing an input operation in the three-dimensional space.

[0003] Japanese Patent Application Laid-Open No. 2005-50120

[0004] However, when a user performs an input operation in a three-dimensional space (3D space), the user needs to move the operation device three-dimensionally, which requires the user to operate the device with their hand in the air, which makes the user feel fatigued and reduces operability and stability.

[0005] Therefore, the present disclosure proposes an information processing device, an information processing system, and an information processing method that can improve the operability and stability of three-dimensional input operations.

[0006] An information processing device according to one embodiment of the present disclosure includes a setting unit that sets a surface of an object in real space as a reference surface or a point of the object as a reference point, a processing unit that converts the three-dimensional position of an instruction device that exists within an operation range based on the reference surface or the reference point in the real space into the three-dimensional position of a pointer in a virtual space, and a control unit that displays the virtual space including the pointer on a display device.

[0007] An information processing system according to one embodiment of the present disclosure includes a display device, a setting unit that sets a surface of an object in real space as a reference surface or a point of the object as a reference point, a pointing device that is present within an operation range based on the reference surface or the reference point, a detection device that detects the three-dimensional position of the pointing device, a processing unit that converts the three-dimensional position of the pointing device into the three-dimensional position of a pointer in a virtual space, and a control unit that causes the display device to display the virtual space including the pointer.

[0008] An information processing method according to one embodiment of the present disclosure includes setting a surface of an object in real space as a reference surface or a point on the object as a reference point, converting the three-dimensional position of a pointing device present within an operation range based on the reference surface or the reference point in the real space into the three-dimensional position of a pointer in a virtual space, and displaying the virtual space including the pointer on a display device.

[0009] FIG. 1 is a diagram illustrating a configuration example of an information processing system according to an embodiment of the present disclosure. FIG. 2 is a diagram illustrating a detailed configuration example of an information processing system according to an embodiment of the present disclosure. FIG. 3 is a diagram illustrating a pointer display according to an embodiment of the present disclosure. FIG. 4 is a diagram illustrating a first modified example of a detailed configuration example of an information processing system according to an embodiment of the present disclosure. FIG. 5 is a diagram illustrating a second modified example of a detailed configuration example of an information processing system according to an embodiment of the present disclosure. FIG. 6 is a diagram illustrating a first example embodiment of an information processing system according to an embodiment of the present disclosure. FIG. 7 is a diagram illustrating a first example embodiment of an information processing system according to an embodiment of the present disclosure. FIG. 10 is a diagram illustrating a seventh example of an information processing system according to an embodiment of the present disclosure. FIG. 11 is a diagram illustrating a twelfth example of an information processing system according to an embodiment of the present disclosure. FIG. 12 is a diagram illustrating a thirteenth example of an information processing system according to an embodiment of the present disclosure. FIG. 13 is a diagram illustrating a thirteenth example of an information processing system according to an embodiment of the present disclosure. FIG. 14 is a diagram illustrating a thirteenth example of an information processing system according to an embodiment of the present disclosure.Fig. 10 is a diagram for explaining Example 15 of an information processing system according to an embodiment of the present disclosure. Fig. 11 is a flowchart showing the flow of Processing Example 1 of the information processing system according to an embodiment of the present disclosure. Fig. 12 is a flowchart showing the flow of Processing Example 2 of the information processing system according to an embodiment of the present disclosure. Fig. 13 is a diagram showing an example of the hardware configuration of a computer.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the embodiments do not limit the systems, devices, methods, etc. according to the present disclosure. Furthermore, in the following embodiments, essentially the same components are designated by the same reference numerals, and redundant explanations will be omitted. The following embodiments also include examples and modified examples.

[0011] One or more of the following embodiments can be implemented independently. However, at least a portion of the following embodiments may be implemented in appropriate combination with at least a portion of another embodiment. These embodiments may include novel features that are different from one another. Therefore, each embodiment may contribute to solving different purposes or problems, and may provide different effects. The effects of each embodiment are merely examples and are not intended to be limiting, and other effects may also be provided.

[0012] The present disclosure will be described in the following order: 1. Embodiment 1-1. Configuration example of information processing system 1-2. Detailed configuration example of information processing system 1-3. Modification of detailed configuration example of information processing system 1-4. Implementation example of information processing system 1-5. Processing example of information processing system 1-6. Actions and effects 2. Other embodiments 3. Hardware configuration example 4. Supplementary notes

[0013] <1. Embodiment> <1-1. Configuration Example of Information Processing System> An example of the configuration of an information processing system 1 according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the information processing system 1 according to this embodiment. The information processing system 1 according to this embodiment functions as a stereoscopic display system.

[0014] 1, an information processing system 1 according to this embodiment includes an information processing device 10, a display device 20, a detection device 30, and an instruction device 40. Each of the information processing device 10, the display device 20, the detection device 30, and the instruction device 40 is configured to be capable of communicating (transmitting and receiving) various types of information.

[0015] The information processing device 10 controls the display device 20. For example, the information processing device 10 transmits various images to the display device 20. As the information processing device 10, for example, a dedicated or general-purpose computer, a desktop PC (personal computer), a notebook PC, a tablet device, a smartphone, or the like can be used.

[0016] The various images include a stereoscopic image based on three-dimensional model data. The stereoscopic image is, for example, an image that includes a three-dimensional virtual space A1 and three-dimensionally shows various elements that make up the virtual space A1. The various elements include, for example, a background and an object. The stereoscopic image includes, for example, a video and a still image. The three-dimensional model data has information for expressing, for example, the three-dimensional shape, surface, interior, cross section, etc. of the background, object, etc.

[0017] For example, the information processing device 10 provides a function for editing a stereoscopic image, i.e., three-dimensional model data. For example, the information processing device 10 edits the three-dimensional model data based on input information supplied from the instruction device 40. The input information includes, for example, information such as position (coordinates), movement amount, and movement direction. Editing the three-dimensional model data includes, for example, displaying, checking, moving, and changing the three-dimensional model.

[0018] The information processing device 10 also provides a function of displaying a pointer C1 in the virtual space A1, for example. For example, the information processing device 10 controls the display of the pointer C1 in the virtual space A1 based on position information of the pointing device 40 supplied from the detection device 30. The position information of the pointing device 40 includes, for example, three-dimensional coordinates of the pointing device 40. The pointer C1 is also sometimes called a marker.

[0019] The display device 20 displays various images such as stereoscopic images under the control of the information processing device 10. The display device 20 may be either an active or passive display. The display device 20 may be, for example, a stationary stereoscopic display such as an SRD or a light field display, a wearable stereoscopic display such as an HMD, or a flat display that presents a stereoscopic image (three-dimensional space) using visual cues.

[0020] For example, the display device 20 displays a three-dimensional virtual space A1 based on three-dimensional model data together with a movable pointer C1 under the control of the information processing device 10. The pointer C1 is a geometric tool for specifying a position, an area, etc. in the three-dimensional virtual space A1.

[0021] The detection device 30 detects the position of the pointing device 40 operated by the user (e.g., the position of the tip C2 of the pointing device 40), and transmits position information regarding the detected position of the pointing device 40 to the information processing device 10. The position of the pointing device 40 is a position in a three-dimensional real space A2 (e.g., three-dimensional coordinates).

[0022] The instruction device 40 accepts an input operation (instruction operation) from a user and supplies input information relating to the input corresponding to the input operation to the information processing device 10. For example, the instruction device 40 is held by the user's hand. This instruction device 40 functions as an input device. For example, a pen-shaped device or a device that can be operated with one hand can be used as the instruction device 40.

[0023] For example, when the pointing device 40 is a pen-type device, the user performs an input operation by moving the tip C2 of the pointing device 40 while touching the top surface of an object T1, such as a desk or a support stand, with part of the hand holding the pointing device 40. At this time, the tip C2 of the pointing device 40 may be separated from or in contact with the top surface of the object T1.

[0024] <1-2. Detailed Configuration Example of Information Processing System> A detailed configuration example of the information processing system 1 according to this embodiment will be described with reference to Fig. 2 and Fig. 3. Fig. 2 is a diagram showing a detailed configuration example of the information processing system 1 according to this embodiment. Fig. 3 is a diagram for explaining the display of the pointer C1 according to this embodiment.

[0025] (Information Processing Apparatus) As shown in FIG. 2 , the information processing apparatus 10 includes a setting unit 11 , a processing unit 12 , a control unit 13 , and a communication unit 14 .

[0026] 3, the setting unit 11 sets a surface of the object T1 in the real space A2 as a reference plane T1a or a point on the object T1 in the real space A2 as a reference point T1b. The reference plane T1a is a physical surface in the real space A2, and the reference point T1b is a physical point in the real space A2. The reference plane T1a or the reference point T1b functions as a reference position that serves as a reference for the operation range B2 in the real space A2.

[0027] The operational range B2 is a range within the real space A2 that is determined based on the reference plane T1a or the reference point T1b. For example, the operational range B2 is a range within which the tip C2 of the pointing device 40 can move by moving the fingers when part of the hand holding the pointing device 40 is in contact with the surface of the object T1 (see FIG. 1). In this case, the operational range B2 is a fairly narrow range. For example, when part of the hand holding the pointing device 40 is in contact with the surface of the object T1, the operational range B2 is automatically determined to be near the reference plane T1a or the reference point T1b. Typically, when the user moves the hand holding the pointing device 40 from its current position, the operational range B2 also moves in accordance with the movement.

[0028] 3, the reference plane T1a is the top surface of the object T1, such as a desk or a support stand, but it may also be any other surface of the object T1, such as a pedestal, a ruler, a floor, etc. Furthermore, the reference point T1b is a corner of the top surface of the object T1, such as a desk or a support stand, but it may also be a physical feature other than the corner.

[0029] 3 , the processing unit 12 converts the three-dimensional position (e.g., XYZ coordinates) of the tip C2 of the pointing device 40 that exists within the operation range B2 of the real space A2 into the three-dimensional position of the pointer C1 within the movement range B1 of the virtual space A1. That is, the processing unit 12 reflects the position of the tip C2 of the pointing device 40 in the position of the pointer C1 based on the position information of the tip C2 of the pointing device 40 transmitted from the detection device 30.

[0030] The movement range B1 is a range within the virtual space A1 that is determined based on the spatial size of the virtual space A1. For example, the movement range B1 may be the entire virtual space A1, or may be a part of the virtual space A1. This movement range B1 may be set in advance, or may be changed by a user's input operation. The movement range B1 is a three-dimensional spatial range within the virtual space A1, and the operation range B2 is a three-dimensional spatial range within the real space A2 that is narrower than the movement range B1.

[0031] The three-dimensional position of the tip C2 of the pointing device 40 functions as the pointing position of the pointing device 40. The three-dimensional position of the tip C2 of the pointing device 40 is reflected in the three-dimensional position of the pointer C1, but the three-dimensional position of a part of the pointing device 40 other than the tip C2 may also be reflected in the three-dimensional position of the pointer C1.

[0032] 2 , the control unit 13 executes various processes such as communication processes with other devices and display processes on the display device 20. For example, the control unit 13 causes the display device 20 to display a virtual space A1 (stereoscopic image) within a movement range B1 including the pointer C1. The control unit 13 acquires the three-dimensional position of the pointer C1 from the processing unit 12, and executes display processes to display the pointer C1 within the virtual space A1 (for example, within the movement range B1) based on the three-dimensional position of the pointer C1.

[0033] The communication unit 14 is an interface that enables communication between the display device 20, the detection device 30, and the instruction device 40 and the information processing device 10. As the communication unit 14, for example, a serial interface such as USB (Universal Serial Bus) or HDMI (High-Definition Multimedia Interface) (registered trademark), or a wireless interface such as Bluetooth (registered trademark) or Wi-Fi (registered trademark) can be used. Note that communication may be realized either wired or wirelessly, or may be realized by a combination of wired and wireless communication.

[0034] Here, each functional unit, such as the setting unit 11, processing unit 12, and control unit 13, may be configured with both hardware and software, or one of them. The configuration is not particularly limited. For example, each functional unit may be realized by a computer, such as a CPU (Central Processing Unit) or an MPU (Micro Control Unit), executing a program pre-stored in a ROM (Read Only Memory) using a RAM (Random Access Memory) as a work area. Furthermore, each functional unit may be realized with an integrated circuit, such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

[0035] (Display Device) The display device 20 includes a display unit 21 and a communication unit 22 .

[0036] The display unit 21 displays various images such as a stereoscopic image. For example, the display unit 21 displays a stereoscopic image or a pointer C1 transmitted from the information processing device 10 on a display screen. As the display unit 21, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display can be used.

[0037] The communication unit 22 is an interface that enables communication between the display device 20 and the information processing device 10. For example, a serial interface such as USB or HDMI (registered trademark) can be used as the communication unit 22. Note that communication may be realized either wired or wirelessly, or may be realized by a combination of wired and wireless communication.

[0038] (Detection Device) The detection device 30 includes a detection unit 31 , a processing unit 32 , and a communication unit 33 .

[0039] The detection unit 31 detects the three-dimensional position (pointing position) of the tip C2 of the pointing device 40. As the detection unit 31, for example, a stereo camera or a ToF (Time-of-Flight) sensor can be used.

[0040] The processing unit 32 executes various processes such as communication with other devices and information generation. For example, the processing unit 32 generates position information regarding the three-dimensional position (e.g., XYZ coordinates) of the tip C2 of the pointing device 40 based on the three-dimensional position of the tip C2 of the pointing device 40 detected by the detection unit 31. The position information is transmitted to the information processing device 10 via the communication unit 33. For example, an integrated circuit such as an ASIC or an FPGA can be used as the processing unit 32.

[0041] The communication unit 33 is an interface that enables communication between the detection device 30 and the information processing device 10. For example, a serial interface such as USB or HDMI (registered trademark) can be used as the communication unit 33. Note that communication may be realized either wired or wirelessly, or may be realized by a combination of wired and wireless communication.

[0042] (Instruction Device) The instruction device 40 includes an input unit 41 , a light source 42 , a processing unit 43 , and a communication unit 44 .

[0043] The input unit 41 accepts input operations from the user. For example, the user operates the input unit 41 to perform a click operation. The input unit 41 may be, for example, a button pressed by the user. Input information such as click information (for example, a click signal) is transmitted to the information processing device 10 via the communication unit 44.

[0044] The light source 42 is provided at the tip C2 of the pointing device 40. The light source 42 emits light, making it easier for the detection device 30 to detect the three-dimensional position of the tip C2 of the pointing device 40. As the light source 42, for example, a light-emitting unit such as an LED (Light Emitting Diode) can be used.

[0045] The processing unit 43 executes various processes such as communication with other devices and light emission by the light source 42. For example, the processing unit 43 executes a process to realize a specific function in response to a user's input operation on the input unit 41. For example, if the input unit 41 is a button, the processing unit 43 executes a process to realize the specific function when the user presses and holds the button. For example, an integrated circuit such as an ASIC or an FPGA can be used as the processing unit 43.

[0046] The communication unit 44 is an interface that enables communication between the instruction device 40 and the information processing device 10. For example, a wireless interface such as Bluetooth (registered trademark) or Wi-Fi (registered trademark) can be used as the communication unit 44. Note that communication may be realized either wired or wirelessly, or may be realized by a combination of wired and wireless methods. To reduce the user's sense of fatigue, it is preferable that communication be realized wirelessly.

[0047] The indicator 40 may include, for example, an inertial measurement unit (IMU). The IMU includes various sensors such as an acceleration sensor, a gyro sensor, and a magnetic sensor.

[0048] The instruction device 40 may also have, for example, a feedback mechanism. The feedback mechanism is, for example, a mechanism that realizes haptic feedback. Haptics makes it possible to generate a tactile three-dimensional object (e.g., a switch or button) in the virtual space A1. The feedback mechanism may use, for example, an eccentric motor, a linear vibrator, or a piezoelectric element.

[0049] Furthermore, the pointing device 40 may be, for example, a remote control device that uses infrared rays or other radio waves. In this case, the detection device 30 detects the infrared rays or other radio waves and detects the three-dimensional position of the pointing device 40 based on the detection result.

[0050] <1-3. Modifications of detailed configuration example of information processing system> Modifications 1 and 2 of the detailed configuration example of the information processing system 1 according to this embodiment will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a diagram showing Modification 1 of the detailed configuration example of the information processing system 1 according to this embodiment. Fig. 5 is a diagram showing Modification 2 of the detailed configuration example of the information processing system 1 according to this embodiment.

[0051] 4 , input information (e.g., click information) from the instruction device 40 may be transmitted to the information processing device 10 via the detection device 30. In this case, the click information is transmitted from the instruction device 40 to the detection device 30 via the communication units 44 and 33, and is transmitted from the detection device 30 to the information processing device 10 via the communication units 33 and 14.

[0052] (Variation 2) As shown in Fig. 5 , the pointing device 40 may incorporate a detection unit 31 that is part of the detection device 30. In this case, the pointing device 40 also functions as the detection device 30. The detection unit 31 may be, for example, an IMU, an acceleration sensor, a gyro sensor, a magnetic sensor, a camera, or a combination of these. When a camera is used as the detection unit 31, the processing unit 43 may estimate the self-position of the pointing device 40 based on an image obtained by capturing an external image of the pointing device 40 (for example, capturing an image of the area around the pointing device 40).

[0053] <1-4. Examples of Information Processing System> Examples 1 to 15 of the information processing system 1 according to this embodiment will be described with reference to Fig. 6 to Fig. 28. Fig. 6 to Fig. 28 are diagrams for explaining each example (Example 1 to Example 15) of the information processing system 1 according to this embodiment.

[0054] (Example 1: Figures 6-12) As described above, the information processing device 10 detects the three-dimensional position of the tip C2 of the pointing device 40 within the operation range B2 based on the reference plane T1a or reference point T1b, converts the detected three-dimensional position of the tip C2 into the three-dimensional position of the pointer C1 in the virtual space A1, and displays the virtual space A1 including the pointer C1 on the display device 20.

[0055] Here, when the processing unit 12 of the information processing device 10 converts the three-dimensional position (indication position) of the tip C2 of the pointing device 40 into the three-dimensional position of the pointer C1 in the virtual space A1, the processing unit 12 may set and convert a three-dimensional coordinate system as necessary.

[0056] For example, as shown in Fig. 6, the processing unit 12 may determine the three-dimensional position of the pointer C1 in the virtual space A1 based on the distance of the pointing device 40 from an origin D1 in the real space A2. The origin D1 in the real space A2 is conveniently determined based on a reference plane T1a. Note that the origin D1 in the real space A2 may also be conveniently determined based on a reference point T1b. The origin D1 may also be, for example, the reference point T1b.

[0057] For example, as shown in Fig. 7, the processing unit 12 may determine the three-dimensional position of the pointer C1 in the virtual space A1 based on a value obtained by subtracting a certain offset amount (offset distance) from the distance between the origin D1 and the tip C2 of the pointing device 40. In the example of Fig. 7, different offset amounts are assigned depending on the X, Y, and Z directions of the three-dimensional coordinate system.

[0058] The origin D1 of the real space A2 is determined based on the reference plane T1a or the reference point T1b, but instead of the origin D1 of the real space A2, the origin plane of the real space A2 may be determined based on the reference plane T1a or the reference point T1b.

[0059] The processing unit 12 of the information processing device 10 determines, based on the output value of the detection device 30, that the tip C2 of the pointing device 40 has reached a surface defined by pre-calibration, and identifies the reference surface T1a or the reference point T1b. The detection unit 31 of the detection device 30 may be, for example, an image sensor such as a stereo camera. The operation mode of the information processing device 10 may be switched to a mode for determining the reference surface T1a or the reference point T1b.

[0060] 8, the user defines the reference plane T1a by operating the pointing device 40 and touching any of multiple (e.g., three) points within the plane that will become the reference plane T1a with the tip C2 of the pointing device 40. The detection device 30 detects the three-dimensional position of the tip C2 of the pointing device 40 at each point and calculates the coordinate values ​​of each point. The processing unit 12 of the information processing device 10 records the coordinate values ​​of each point and defines the plane that includes these points as the reference plane T1a.

[0061] 9, the user operates the pointing device 40 to define the reference plane T1a by drawing multiple (e.g., two) arbitrary straight lines on a surface that will become the reference plane T1a. The detection device 30 sequentially detects the three-dimensional position of the tip C2 of the pointing device 40 as it moves along each straight line, and calculates the coordinate values ​​of each straight line. The processing unit 12 of the information processing device 10 records the coordinate values ​​of each straight line, and defines the plane that includes these straight lines as the reference plane T1a.

[0062] 10, the user defines the reference plane T1a by operating the pointing device 40 to draw a figure E1, such as a rectangle, on a surface that will serve as the reference plane T1a. The detection device 30 sequentially detects the three-dimensional position of the tip C2 of the pointing device 40 as it moves along the figure E1, and determines the coordinate values ​​(line coordinate values) of the figure E1. The processing unit 12 of the information processing device 10 records the coordinate values ​​of the figure E1 and defines the surface that includes the figure E1 as the reference plane T1a.

[0063] 11, the user operates the pointing device 40 to place multiple recognizable markers E2a, E2b, E2c, and E2d on a surface that will become the reference surface T1a, thereby defining the reference surface T1a. The detection device 30 detects each of the markers E2a to E2d and determines the individual coordinate values ​​of each of the markers E2a to E2d. The processing unit 12 of the information processing device 10 records the coordinate values ​​of the markers E2a to E2d and defines the surface that includes the markers E2a to E2d as the reference surface T1a.

[0064] For example, as shown in FIG. 12 , the user operates the pointing device 40 and touches an arbitrary point on a plane that will become the reference plane T1a or the reference point T1b with the tip C2 of the pointing device 40, thereby defining the reference plane T1a or the reference point T1b. In the example of FIG. 12 , a contact sensor 45 is attached to the tip C2 of the pointing device 40 or the reference plane T1a. The detection device 30 detects the three-dimensional position of the tip C2 of the pointing device 40 at the arbitrary point and calculates the coordinate value of that point. The processing unit 12 of the information processing device 10 records the coordinate value and defines the plane containing that point as the reference plane T1a or the point as the reference point T1b. Note that the normal direction of the reference plane T1a is predetermined.

[0065] According to the first embodiment, by defining a physical reference plane T1a or reference point T1b, it is possible to improve the operability and stability of three-dimensional input operations. Typically, three-dimensional operations require input operations in the air, which is more likely to cause fatigue and hand shake than two-dimensional input operations. However, according to the first embodiment, the user can perform operations in a narrow operation range B2 with part of their hand in contact with the reference plane T1a or reference point T1b, thereby improving the operability and stability of three-dimensional input operations.

[0066] Furthermore, for example, the operation range B2 can be limited to the vicinity of the reference plane T1a or the reference point T1b, thereby narrowing the operation range B2. This enables three-dimensional input operations with small movements of a part of the user's hand. Furthermore, by operating the pointing device 40 while touching a part of the body, such as the wrist, with the reference plane T1a or the reference point T1b, hand shake of the pointing device 40 can be suppressed. Furthermore, by placing the pointing device 40 or the hand holding the pointing device 40 on the reference plane T1a or the reference point T1b that is within the operation range B2, frequent breaks can be realized. Using an offset from the reference plane T1a or the reference point T1b enables operation in a posture that is easy to operate (e.g., less prone to hand shake, easier to make fine movements) depending on the individual and the situation.

[0067] (Example 2: Figures 13-14) In Example 1, the processing unit 12 limits the movement range B1 of the pointer C1 in the virtual space A1 to a partial range of the entire space B3 of the virtual space A1, as shown in Figure 13. This makes it possible for the pointer C1 to move only within a partial range of the virtual space A1.

[0068] For example, if the pointer C1 moves beyond the movement range B1, the amount of movement of the pointer C1 in the direction beyond the movement range B1 will not be reflected in the pointer C1. In other words, if the pointer C1 moves beyond the movement range B1, the processing unit 12 performs processing to prevent the amount of movement of the pointer C1 in the direction beyond the movement range B1 from being reflected in the pointer C1.

[0069] The presence or absence of movement restrictions and the range thereof may be the same for all X, Y, and Z directions of the three-dimensional coordinate system, or may differ depending on the X, Y, and Z directions of the three-dimensional coordinate system. The presence or absence of movement restrictions and the range thereof may be set in advance, for example. That is, the movement range B1 of the pointer C1 is set in advance, but may also be freely set by a user's input operation or may be changed by a user's input operation or the like.

[0070] For example, as shown in Fig. 14, the movement range B1 of the pointer C1 may be a range in which the shape of the display device 20 provides a clue. In the example of Fig. 14, the display device 20 is a three-dimensional display such as an SRD, and the shape of the display device 20 is a triangular prism (e.g., a right-angled triangular prism), and the screen 20a of the display device 20 is tilted. In this case, the movement range B1 of the pointer C1 may be set so that the pointer C1 can only move within the triangular prism formed by cutting a rectangular parallelepiped or cube with the screen 20a of the display device 20. The screen 20a of the display device 20 is located on a diagonal line of the rectangular parallelepiped or cube.

[0071] This movement restriction function may be permanently activated or temporarily deactivated. For example, during a drag operation in which the pointer C1 is consciously moved and is unlikely to be lost, this movement restriction may be deactivated to provide greater freedom of operation.

[0072] According to the second embodiment, it is possible to prevent the user from losing sight of the position of the pointer C1 in the virtual space A1. Typically, in a three-dimensional virtual space A1, unlike a two-dimensional (2D) virtual space, display in the depth direction is possible and the degree of spatial freedom is high, but inevitably the pointer C1 is hidden in the shadow of a displayed object, and the user is likely to lose sight of the pointer C1. Even in such a case, according to the second embodiment, the user is unlikely to lose sight of the position of the pointer C1 in the virtual space A1.

[0073] For example, if the display device 20 has a distinctive shape and display method that evokes a three-dimensional range, such as an SRD in which the space is intuitively easy to understand, such as a rectangular prism or cube, then by using that three-dimensional range as the movement range B1, the learning cost is low and the movement range B1 can be intuitively identified.

[0074] Also, by restricting the movement range B1 during normal times when the pointer C1 can be moved randomly and releasing the movement restriction during intentional dragging, it is possible to easily achieve both ease of finding the pointer C1 and a feeling of free and unconstrained operation.

[0075] (Example 3: Fig. 15) In Example 2, the processing unit 12 can move or change the movement range B1 of the pointer C1 in the virtual space A1 as shown in Fig. 15. That is, the processing unit 12 can change any or all of the size, position, and shape of the movement range B1 of the pointer C1.

[0076] The method for changing the size, position, shape, etc. of the movement range B1 of the pointer C1 may be a user input operation on the pointing device 40, or a user input operation on another input device such as a keyboard or mouse, or it may be a method in which the pointer C1 moves automatically.

[0077] According to Example 3, by cutting out any space (for example, a space of any size, any position, any shape, etc.) from the entire virtual space A1 and setting it as the movement range B1, it is possible to increase the freedom of input operations while maintaining the characteristic of making it difficult to lose sight of the pointer C1.

[0078] (Example 4: Figure 16) In Example 2, as shown in Figure 16, when the pointer C1 is moved beyond the movement range B1 of the pointer C1 in the virtual space A1, the processing unit 12 readjusts the offset amount (offset distance) from the origin D1 of the operation range B2 in the real space A2.

[0079] For example, when the pointer C1 is about to exceed the movement range B1, the processing unit 12 readjusts the offset amount so that the position of the pointer C1 falls within the movement range B1, and moves the operation range B2 in the real space A2.

[0080] The presence or absence and degree of readjustment may be the same in the X, Y, and Z directions of the three-dimensional coordinate system, or may differ depending on the X, Y, and Z directions of the three-dimensional coordinate system. For example, the presence or absence and degree of readjustment may be set in advance, but may also be changed by a user's input operation, etc.

[0081] According to the fourth embodiment, the offset amount can be easily and freely changed, enabling the user to perform operations in a position and orientation of their choice. Furthermore, even if the pointer C1 is about to exceed the movement range B1, the operation range B2 immediately moves in the direction beyond the movement range B1 without the need to return the pointing device 40 to within the predetermined operation range B2 in the real space A2, thereby reducing the effort required for input operations.

[0082] (Example 5: Fig. 17) In Example 4, the processing unit 12 sets a limit to a changeable range B4 of the offset amount of the operation range B2 in the real space A2, as shown in Fig. 17. The changeable range B4 is a settable range in the real space A2 within which the operation range B2 can be set.

[0083] For example, when the pointing device 40 is a certain distance away from the operation range B2, the processing unit 12 does not readjust the offset amount beyond the certain distance. As a result, when the pointing device 40 is a certain distance away from the reference plane T1a or the reference point T1b, movement of the operation range B2 to a position that is further away than the certain distance is prohibited.

[0084] According to the fifth embodiment, the user naturally returns the pointing device 40 to within the operation range B2 for operation, which can prevent the pointing device 40 from moving out of the detection range of the detection device 30. Furthermore, as the user becomes more tired, the operation range B2 can be prevented from moving away from the reference plane T1a or the reference point T1b.

[0085] (Example 6: FIG. 18) In Examples 3 and 4, as shown in FIG. 18, the processing unit 12 adjusts the position of the movement range B1 of the pointer C1 in the virtual space A1 in accordance with a change in the offset of the operation range B2 in the real space A2.

[0086] For example, when the position of the pointer C1 exceeds the movement range B1 in the virtual space A1, the processing unit 12 readjusts the offset amount of the operation range B2 in the real space A2 so that the position of the pointer C1 falls within the movement range B1, and moves the movement range B1 of the pointer C1 in the virtual space A1. In other words, the processing unit 12 resets the movement range B1 of the pointer C1 in the virtual space A1 in conjunction with the resetting of the operation range B2.

[0087] Methods for resetting the movement range B1 of the pointer C1 include, for example, scrolling and moving the range displayed on the display device 20 such as an SRD or HMD, or moving the operation range B2 clearly displayed on the display device 20.

[0088] According to Example 6, by resetting the movement range B1 within the virtual space A1 as needed, the pointer C1 can be moved to the desired location without intentionally resetting the movement range B1.

[0089] (Example 7: Figures 19-20) In Example 2, as shown in Figure 19, when the pointer C1 reaches the limit of the movement range B1 of the pointer C1 in the virtual space A1, the control unit 13 displays a message indicating that the pointer C1 has reached the limit of the movement range B1.

[0090] For example, the control unit 13 displays that the pointer C1 has reached the limit (e.g., the farthest surface) of the movement range B1 of the virtual space A1, as shown in Fig. 19 and Fig. 20. In the example of Fig. 20, when the pointer C1 has reached the limit of the movement range B1, the reached surface is clearly indicated by a colored border F1 (e.g., a yellow border).

[0091] The presence or absence and degree of indication may be the same in the X, Y, and Z directions of the three-dimensional coordinate system, or may differ depending on the X, Y, and Z directions of the three-dimensional coordinate system. For example, the presence or absence and degree of indication may be set in advance, but may also be changed by a user's input operation, etc.

[0092] As a display method, when the pointer C1 reaches the limit of the movement range B1, it is possible to use a method of explicitly expressing the interface, a method of changing the display method of the interface that has been displayed in advance, a method of displaying a message or an image, a method of changing the display of the pointer C1, a method of activating a specific visual effect, a method of generating sound or vibration, etc. Note that any method may be used as long as it allows the user to recognize that the pointer C1 has reached the limit of the movement range B1.

[0093] According to the seventh embodiment, since the limit of the movement range B1 is clearly indicated, the user intuitively stops moving the pointing device 40 beyond the interface, which reduces the likelihood of unnecessary actions being induced and improves operability. Furthermore, it is possible to prevent the operation range B2 in the real space A2 from moving too far beyond what is necessary. Furthermore, if the user loses sight of the pointer C1, the user can find the pointer C1 by randomly moving the pointing device 40 and perceiving the interface where the pointer C1 collided.

[0094] (Example 8: Fig. 21) In Example 1, as shown in Fig. 21, the processing unit 12 enlarges the amount of movement (movement distance) of the pointing device 40 within the operation range B2 in the real space A2, and moves the pointer C1 based on the enlarged amount of movement. The amount of movement of the pointer C1 is greater than the amount of movement of the pointing device 40. In other words, the processing unit 12 enlarges small movements of the pointing device 40 in the real space A2 and reflects them in the movement of the pointer C1 in the virtual space A1.

[0095] The amount of enlargement may be the same in the X, Y, and Z directions of the three-dimensional coordinate system, or may differ depending on the X, Y, and Z directions of the three-dimensional coordinate system. The amount of enlargement may be set in advance, but may also be changed by a user's input operation, for example.

[0096] According to Example 8, the movement amount of the pointer C1 is greater than the movement amount of the pointing device 40, so it is possible to suppress the movement (movement amount) of the user's hand (e.g., fingers, etc.), thereby reducing the user's sense of fatigue.

[0097] (Example 9) The processing unit 12 can change the enlargement ratio, which is an example of the enlargement amount, in Example 8. For example, the processing unit 12 changes the enlargement ratio in response to an input operation by the user.

[0098] The method for changing the magnification ratio may be a user input operation on the instruction device 40, or a user input operation on another input device such as a keyboard or mouse, or may be a method for automatically changing the magnification ratio.

[0099] According to the ninth embodiment, it is possible to change the magnification ratio in response to the user's input operation, etc., and therefore it is possible to provide an operation feeling that meets the personal preferences of the user, etc.

[0100] (Example 10: Fig. 22) In Example 1, the processing unit 12 changes the operation range B2 in the real space A2 while the movement of the pointer C1 is stopped, as shown in Fig. 21. That is, the processing unit 12 moves the operation range B2 in the real space A2 while the movement of the pointer C1 is stopped.

[0101] For example, in response to a user's input operation (see, for example, Example 14), the processing unit 12 stops the movement of the pointer C1 and moves the operation range B2 of the real space A2 in accordance with the movement of the indicating device 40 so that when the movement of the pointer C1 is resumed, the pointer C1 starts moving from the same position.

[0102] According to the tenth embodiment, by moving the operation range B2 in the real space A2 while keeping the pointer C1 stationary in the virtual space A1, it is possible to set a comfortable operation range B2 while maintaining the position of the pointer C1 in the virtual space A1, thereby maintaining a comfortable operation feel while reducing the risk of losing sight of the pointer C1. For example, it is possible to perform an operation in which the pointing device 40 is moved to a location that is easy to use and then used.

[0103] (Example 11: Fig. 23) In Example 1, the processing unit 12 aligns the physical reference plane T1a with a specific plane in the virtual space A1, as shown in Fig. 23. For example, the processing unit 12 aligns the floor surface of the virtual space A1 with the reference plane T1a without applying an offset only in the up-down direction.

[0104] In the example of Figure 23, the top surface of object T1 in real space A2 corresponds to the floor surface of virtual space A1, but a reference surface T1a (e.g., a flat surface, a curved surface, etc.) other than the top surface of object T1 may also correspond to a specific plane in virtual space A1.

[0105] According to the eleventh embodiment, when the pointer C1 is to be moved to a specific plane in the virtual space A1, the pointing device 40 always comes into contact with the reference plane T1a. This eliminates the need for the user to consciously bring the pointing device 40 into contact with the reference plane T1a, allowing the user to enjoy a comfortable operation feel. For example, when an operation is made to operate a button on a specific plane in the virtual space A1, an operation state that is less susceptible to camera shake can be realized (see, for example, the thirteenth embodiment).

[0106] (Example 12: Fig. 24) In Example 1, the processing unit 12 always sets the reference plane T1a or the reference point T1b as the lower limit of the operation range B2 of the real space A2, as shown in Fig. 24. For example, the processing unit 12 sets the reference plane T1a as the lower limit, and sets a certain range above the reference plane T1a as the operation range B2.

[0107] According to Example 12, the reference plane T1a or the reference point T1b is the lower limit of the operation range B2 in the real space A2, so that it can be used in the same way as writing on a desk, the learning cost is low, and the user can intuitively grasp the downward limit of the operation range B2.

[0108] 25 to 27, in Example 1, the processing unit 12 initiates a specific action when a specific positional relationship is established between the reference surface T1a and the pointing device 40. For example, when the positional relationship between the reference surface T1a and the pointing device 40 satisfies a specific condition, the processing unit 12 initiates, as the specific action, any or all of a change in the input action of the pointing device 40, a change in the processing action of the pointing position of the pointing device 40, and a change in the display action of the pointer C1.

[0109] The conditions for initiating a certain action may be a combination of a specific positional relationship and a specific operation. For example, the conditions for initiating a certain action may be that the tip C2 of the pointing device 40 touches the reference surface T1a while pressing a key such as a shift key.

[0110] 25, when the pointing device 40 reaches the reference plane T1a, the processing unit 12 changes the operation mode to a two-dimensional mouse mode like a pen tablet (changes the input operation of the pointing device 40). When the pointing device 40 is in contact with the reference plane T1a, the user operates only within a specific plane, similar to a normal mouse.

[0111] 26, when the pointing device 40 reaches the reference surface T1a, the processing unit 12 continuously moves the pointer C1 and the movement range B1 of the pointer C1 downward (changing the processing operation of the pointing position of the pointing device 40). As a result, when the pointing device 40 is in contact with the reference surface T1a, the pointer C1 continuously moves downward. For example, it is conceivable that there may be a case where the pointer C1 is desired to move beyond the movement range B1, such as during dragging.

[0112] 27, when the pointing device 40 reaches the reference surface T1a, the processing unit 12 changes the operation mode to a UI mode in which a UI (user interface) button G1 in the virtual space A1 can be operated (changing the display operation of the pointer C1). As a result, when the pointing device 40 touches the reference surface T1a, the UI button G1 on the floor (bottom) of the virtual space A1 can be operated. Note that when the pointing device 40 is not touching the reference surface T1a, the UI button G1 may be hidden or displayed semi-transparently to explicitly indicate that it cannot be operated.

[0113] According to the thirteenth embodiment, a desired action can be achieved with fewer instruction actions. Furthermore, the user's own action state can be easily understood. For example, a single input system can be used to switch between three-dimensional and two-dimensional operations. Furthermore, even if the pointing device 40 reaches the reference plane T1a, which is the physical movement limit, the pointer C1 can be further moved within the virtual space A1. Therefore, the operation range B2 of the pointer C1 within the virtual space A1 can be widened while limiting the amount of movement of the pointing device 40 in the real space A2.

[0114] Furthermore, when the user operates the pointing device 40 without touching the reference surface T1a, it is difficult to click the fine UI button G1 due to hand shake. Therefore, when performing an operation that requires precision, such as clicking the fine UI button G1, the user inevitably has to touch the tip C2 of the pointing device 40 to the reference surface T1a, which provides frictional resistance and a comfortable operation feeling that is less susceptible to hand shake.

[0115] Fourteenth Embodiment In the first embodiment, the processing unit 12 activates a specific function in response to an input operation by a user.

[0116] User input operations include operations on the input unit 41 of the instruction device 40, operations on a touch sensor, etc., operations on the UI of the virtual space A1, operations to move the pointer C1 to a specific area in the virtual space A1, and operations on peripheral devices such as a keyboard or mouse.

[0117] Specific functions include "changing the magnification rate of the movement of the pointing device 40," "changing the shape and color of the pointer C1," "changing the offset," "switching individual functions of each embodiment ON / OFF," "stopping and resuming the movement of the pointer C1," "restricting the movement of the pointer C1 (for example, allowing movement only within a specific plane or straight line)," and "starting, stopping, changing the operation content, and switching targets for operations on objects displayed in the virtual space A1."

[0118] According to the fourteenth embodiment, a function can be activated in accordance with the user's intention.

[0119] (Example 15: Fig. 28) In Example 1, the processing unit 12 suppresses camera shake in the process of reflecting the position of the pointing device 40 in the real space A2 in the position of the pointer C1, as shown in Fig. 28. In the process of suppressing camera shake, the degree of camera shake suppression is changed according to, for example, the distance between the pointing device 40 and the reference plane T1a or the reference point T1b.

[0120] For example, the processing unit 12 changes the degree of camera shake correction (e.g., the amount of correction) depending on the distance between the reference plane T1a or the reference point T1b and the pointing device 40. The degree of camera shake correction also includes the magnification rate of the movement for correcting camera shake. In other words, the processing unit 12 changes the magnification rate and filter processing method of the signal related to the movement of the pointer C1 for suppressing camera shake. For example, the processing unit 12 changes the filter processing conditions for camera shake suppression (e.g., the amount of suppression of high-frequency movement, the degree to which small movement is invalidated, the degree to which prediction processing is performed, etc.) depending on the positional relationship between the reference plane T1a and the pointing device 40.

[0121] The correction amount (change amount) may be the same in the X, Y, and Z directions of the three-dimensional coordinate system, or may differ depending on the X, Y, and Z directions of the three-dimensional coordinate system. The correction amount may be set in advance, or may be changed by a user's input operation, for example. Furthermore, when a combination of multiple detection methods is used as the detection device 30, the combination of methods may be changed.

[0122] According to the fifteenth embodiment, it is possible to suppress hand shake caused by three-dimensional operation while achieving quick response. For example, when the wrist is lifted in the air away from the reference plane T1a and is prone to hand shake, or when the wrist is lifted from the detection device 30 and is in a position where noise is likely to be introduced into the detection value, the amount of correction for the filtering process is increased to suppress shaking of the pointer C1.

[0123] Generally, the larger the correction amount, the more delay occurs before the movement of the pointer C1 is reflected. Therefore, contrary to the above example, in a state where a part of the hand, such as the wrist, is fixed in contact with the reference plane T1a, or in a state where the pointing device 40 is located in a position where the detecting device 30 can detect it with high accuracy, by reducing the correction amount, it is possible to provide a sense of operation with a fast response.

[0124] <1-5. Processing Examples of Information Processing System> Processing examples 1 and 2 of the information processing system 1 according to this embodiment will be described with reference to Fig. 29 and Fig. 30. Fig. 29 is a flowchart showing the flow of processing example 1 of the information processing system 1 according to this embodiment. Fig. 30 is a flowchart showing the flow of processing example 2 of the information processing system 1 according to this embodiment.

[0125] 29, in step S11, the information processing system 1 performs INIT (initialization). In step S12, the detection device 30 detects the position of the pointing device 40.

[0126] In step S13, the information processing device 10 determines whether or not the position of the pointing device 40 has been detected, based on the position information from the detection device 30. If the information processing device 10 determines that the position of the pointing device 40 has been detected (Yes in step S13), then in step S14, the information processing device 10 determines whether or not an offset has been set.

[0127] If the information processing device 10 determines in step S14 that an offset has been set (Yes in step S14), it calculates an input value minus the offset (a value obtained by subtracting the offset from the position of the pointing device 40 described above) in step S15, and proceeds to step S16. On the other hand, if the information processing device 10 determines that an offset has not been set (No in step S14), it proceeds to step S16.

[0128] In step S16, the information processing device 10 calculates the coordinates (three-dimensional coordinates) of the pointer C1 in the virtual space A1 from the input values. In step S17, the display device 20 displays the pointer C1 in the virtual space A1 based on the calculated coordinates of the pointer C1 in the virtual space A1, and returns the process to step S12.

[0129] On the other hand, if the information processing device 10 determines in step S13 that the position of the pointing device 40 cannot be detected (No in step S13), the display device 20 displays in step S18 that the position of the pointing device 40 cannot be detected, and the process proceeds to step S17. In this case, in step S17, for example, the pointer C1 remains displayed in the same position.

[0130] In the display in step S18, for example, one or both of the shape and color of the pointer C1 may be changed, or a message may be displayed indicating that the position of the pointing device 40 has not been detected.

[0131] 30, in step S21, the information processing system 1 performs INIT (initialization). In step S22, the information processing device 10 executes processing when the pointing device 40 receives a specific key / UI operation. In step S23, the detection device 30 detects the position of the pointing device 40.

[0132] In step S24, the information processing device 10 determines whether or not the position of the indicating device 40 has been detected, based on the position information from the detection device 30. If the information processing device 10 determines that the position of the indicating device 40 has been detected (Yes in step S24), then in step S25, the information processing device 10 changes the processing method according to the position of the indicating device 40. On the other hand, if the information processing device 10 determines that the position of the indicating device 40 cannot be detected (No in step S24), then in step S39, the display device 20 displays that the position of the indicating device 40 cannot be detected, and the process proceeds to step S38.

[0133] In step S26, the information processing device 10 determines whether the movement of the pointer C1 is valid. If the information processing device 10 determines that the movement of the pointer C1 is valid (Yes in step S26), the information processing device 10 determines whether an offset has been set in step S27. On the other hand, if the information processing device 10 determines that the movement of the pointer C1 is not valid (No in step S26), the information processing device 10 changes the offset to match the position of the pointing device 40 in step S40, and proceeds to step S38.

[0134] In step S27, if the information processing device 10 determines that an offset has been set (Yes in step S27), in step S28, it calculates an input value minus the offset (a value obtained by subtracting the offset from the position of the aforementioned pointing device 40), and in step S29, it calculates the coordinates (three-dimensional coordinates) of the pointer C1 in the virtual space A1 from the aforementioned input value.

[0135] In step S30, the information processing device 10 determines whether the tip C2 of the pointing device 40 is in contact with the reference surface T1a. If the information processing device 10 determines that the tip C2 of the pointing device 40 is in contact with the reference surface T1a (Yes in step S30), in step S31, the information processing device 10 readjusts the offset so that the pointer C1 moves to a specific plane in the virtual space A1, and in step S32, moves the pointer C1 to the specific plane in the virtual space A1.

[0136] In step S33, the information processing device 10 determines whether the pointer C1 is located outside the movement range B1 of the virtual space A1. If the information processing device 10 determines that the pointer C1 is located outside the movement range B1 of the virtual space A1 (Yes in step S33), the information processing device 10 scrolls the virtual space A1 in the direction in which the pointer C1 extends beyond the movement range B1 in step S34. On the other hand, if the information processing device 10 determines that the pointer C1 is not located outside the movement range B1 of the virtual space A1 (No in step S33), the process proceeds to step S38.

[0137] In step S35, the information processing device 10 moves the pointer C1 to the interface of the movement range B1 in the direction in which the pointer C1 is projecting from the movement range B1, and in step S36, displays a guardian at the position of the pointer C1 or on the interface, and in step S37, readjusts the offset so that the pointer C1 fits within the movement range B1. In step S38, the display device 20 displays the pointer C1 in the virtual space A1, and the process returns to step S22.

[0138] On the other hand, if the information processing apparatus 10 determines in step S27 that an offset has not been set (No in step S27), it calculates the coordinates of the pointer C1 in the virtual space A1 from the input values ​​in step S41.

[0139] In step S42, the information processing device 10 determines whether the pointer C1 is located outside the movement range B1 of the virtual space A1. If the information processing device 10 determines that the pointer C1 is located outside the movement range B1 of the virtual space A1 (Yes in step S42), the information processing device 10 scrolls the virtual space A1 in the direction in which the pointer C1 extends beyond the movement range B1 in step S43. On the other hand, if the information processing device 10 determines that the pointer C1 is not located outside the movement range B1 of the virtual space A1 (No in step S42), the process proceeds to step S38.

[0140] In step S44, the information processing device 10 moves the pointer C1 to the interface of the movement range B1 in the direction in which the pointer C1 is projecting out of the movement range B1, and in step S45, displays a guardian at the position or interface where the pointer C1 is located, and proceeds to step S38.

[0141] Here, the specific key / UI operation in step S22 (see Example 14) includes, for example, operations to instruct "changing the magnification ratio," "changing the shape and color of the pointer C1," "changing the offset," "switching between determining whether or not to determine each item in steps S26, S27, S30, S33, and S42," etc. Also, the processing method in step S25 (see Example 15) includes, for example, "changing the filter coefficient," "changing the movement magnification," etc.

[0142] The determinations in steps S26, S27, S30, S33, and S42 can be independently set to apply to each of the X, Y, and Z coordinates with the reference plane T1a or the reference point T1b as the XZ plane or the XZ origin. In step S27, when the reference plane T1a is the XZ plane, an offset is set in each of the X and Z directions, but not in the Y direction, so that the reference plane T1a coincides with a specific plane in the virtual space A1.

[0143] <1-6. Actions and Effects> As described above, the information processing device 10 according to this embodiment includes a setting unit 11 that sets a surface of an object T1 in a real space A2 as a reference plane T1a or a point on the object T1 as a reference point T1b, a processing unit 12 that converts the three-dimensional position of the pointing device 40 (e.g., the tip C2 of the pointing device 40) present within an operation range B2 based on the reference plane T1a or the reference point T1b in the real space A2 into the three-dimensional position of a pointer C1 in a virtual space A1, and a control unit 13 that displays the virtual space A1 including the pointer C1 on the display device 20 (see FIG. 2, etc.). This allows the user to perform three-dimensional input operations with part of their hand in contact with the reference plane T1a or the reference point T1b, thereby improving the operability and stability of three-dimensional input operations.

[0144] Furthermore, the processing unit 12 may limit the movement range B1 of the pointer C1 in the virtual space A1 to a part of the range of the virtual space A1 (see FIGS. 13 and 14). This makes it difficult for the user to lose sight of the pointer C1, thereby improving operability.

[0145] The processing unit 12 may be able to change any or all of the size, position, and shape of the movement range B1 (see FIG. 15 ), which allows for the change of any or all of the size, position, and shape of the movement range B1, thereby improving the degree of freedom in operation.

[0146] Furthermore, the processing unit 12 may convert the three-dimensional position of the pointing device 40 into the three-dimensional position of the pointer C1 based on the offset amount of the operation range B2 from the reference plane T1a or the reference point T1b (see FIG. 7 ). This allows the position of the operation range B2 to be changed, thereby improving the degree of freedom of operation.

[0147] Furthermore, the processing unit 12 may change the offset amount when the pointer C1 moves beyond the movement range B1 (see FIG. 16 ). This makes it possible to change the offset amount and realize operation in a desired position and orientation, thereby improving operability.

[0148] Furthermore, the processing unit 12 may set a limit to the changeable range B4 of the offset amount (see FIG. 17 ), which makes it possible to determine the range within the real space A2 in which the operation range B2 can be set, thereby preventing a decrease in the operational feel.

[0149] Furthermore, the processing unit 12 may adjust the position of the movement range B1 in accordance with the change in the offset amount (see FIG. 18 ), which makes it possible to simultaneously move the operation range B2 and the movement range B1, thereby providing a comfortable operational feel.

[0150] Furthermore, when the pointer C1 reaches the limit of the movement range B1, the control unit 13 may cause the display device 20 to display a message indicating that the pointer C1 has reached the limit of the movement range B1 (see FIGS. 19 and 20). This makes it possible to limit the movement range B1, making it difficult for the user to lose sight of the pointer C1, thereby improving operability.

[0151] Furthermore, when moving the pointer C1 in response to the movement of the instruction device 40, the processing unit 12 may increase the amount of movement of the instruction device 40 and move the pointer C1 based on the increased amount of movement (see FIG. 21 ). This reduces the amount of movement of the instruction device 40 and reduces the user's sense of fatigue, thereby ensuring improved operability and stability.

[0152] Furthermore, the processing unit 12 may be able to change the amount of enlargement (for example, the enlargement factor) of the movement amount of the pointing device 40. This makes it possible to change the amount of enlargement, thereby providing a comfortable operational feel.

[0153] Furthermore, the processing unit 12 may change the operation range B2 while the movement of the pointer C1 is stopped (see FIG. 22 ). This makes it possible to set a comfortable operation range B2 while maintaining the position of the pointer C1 in the virtual space A1, thereby providing a comfortable operation feel while reducing the risk of losing sight of the pointer C1.

[0154] Furthermore, the processing unit 12 may align the reference plane T1a with a specific plane (e.g., the floor) within the virtual space A1 (see FIG. 23). This allows the reference plane T1a to align with the specific plane within the virtual space A1, thereby improving operability. For example, this allows the user to intuitively grasp the position of the pointer C1 within the virtual space A1, making it easier to operate.

[0155] Furthermore, the processing unit 12 may set the reference plane T1a or the reference point T1b as the lower limit of the operation range B2 (see FIG. 24). This allows the reference plane T1a or the reference point T1b to become the lower limit of the operation range B2, enabling use in the same way as writing on a desk, thereby improving operability.

[0156] Furthermore, the processing unit 12 may initiate a specific action when a specific positional relationship is established between the reference plane T1a or the reference point T1b and the pointing device 40 (see FIGS. 25-27). This makes it possible to initiate a specific action depending on the specific positional relationship between the reference plane T1a or the reference point T1b and the pointing device 40, thereby improving operability.

[0157] Furthermore, the processing unit 12 may activate a specific function in response to an input operation by the user, thereby improving operability since it is possible to activate a specific function in response to an input operation by the user.

[0158] Furthermore, the processing unit 12 may suppress shaking by varying the degree to which shaking of the pointer C1 caused by hand shake of the pointing device 40 is suppressed, depending on the distance between the reference plane T1a or the reference point T1b and the pointing device 40 (see FIG. 28 ). This allows the user to operate the pointing device 40 without feeling any hand shake, thereby providing a stable operational feel.

[0159] Furthermore, the operation range B2 may be a three-dimensional spatial range that is narrower than the movement range B1 of the pointer C1 in the virtual space A1 (see, for example, FIG. 3 ). This makes it possible to reduce the user's fatigue by narrowing the operation range B2, thereby reliably achieving improved operability and stability.

[0160] The display device 20 may also be a display device capable of stereoscopic display, thereby enabling the display device 20 to display images such as stereoscopic images.

[0161] 2. Other Embodiments The processes according to the above-described embodiments (or modifications) may be implemented in various different forms (modifications) other than the above-described embodiments. For example, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method. In addition, the process procedures, specific names, and information including various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0162] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0163] Furthermore, the above-described embodiments (or variations) can be combined as appropriate within the scope of not causing any contradiction in the processing content. Furthermore, the effects described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0164] Furthermore, in the above-described embodiment (or modified example), a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0165] Furthermore, in the above-described embodiment (or modified example), a cloud computing configuration can be adopted in which one function is shared and processed jointly by multiple devices via a network. Furthermore, each step described in the above-described process flow (e.g., a flowchart) can be executed by a single device or shared and executed by multiple devices. Furthermore, when a single step includes multiple processes, the multiple processes included in that single step can be executed by a single device or shared and executed by multiple devices.

[0166] 3. Example of Hardware Configuration The information processing device 10 described above may be realized by, for example, a computer 1000 configured as shown in Fig. 31. Fig. 31 is a diagram showing an example of the hardware configuration of the computer 1000.

[0167] 31 , the computer 1000 includes a CPU (Central Processing Unit) 1100, a RAM (Random Access Memory) 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0168] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0169] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0170] The HDD 1400 is a recording medium readable by the computer 1000, which non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the HDD 1400 is a recording medium that records program data 1450.

[0171] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0172] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0173] For example, when the computer 1000 functions as the information processing device 10, the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200 to realize the functions of each unit of the information processing device 10. The information processing program and various data are stored in the HDD 1400. The CPU 1100 reads and executes program data 1450 from the HDD 1400, but as another example, the CPU 1100 may obtain these programs from another device via an external network 1550.

[0174] <4. Supplementary Notes> The present technology may also be configured as follows. (1) An information processing device comprising: a setting unit that sets a surface of an object in real space as a reference surface or a point on the object as a reference point; a processing unit that converts a three-dimensional position of a pointing device that exists within an operation range based on the reference surface or the reference point in the real space into a three-dimensional position of a pointer in a virtual space; and a control unit that displays the virtual space including the pointer on a display device. (2) The information processing device according to (1), wherein the processing unit limits a movement range of the pointer in the virtual space to a partial range of the virtual space. (3) The information processing device according to (2), wherein the processing unit is capable of changing any or all of a size, a position, and a shape of the movement range. (4) The information processing device according to (2) or (3), wherein the processing unit converts the three-dimensional position of the pointing device into a three-dimensional position of the pointer based on an offset amount of the operation range from the reference surface or the reference point. (5) The information processing device according to (4), wherein the processing unit changes the offset amount when the pointer moves beyond the movement range. (6) The information processing device according to (5), wherein the processing unit sets a limit to a changeable range of the offset amount. (7) The information processing device according to (5) or (6), wherein the processing unit adjusts the position of the movement range in accordance with a change in the offset amount. (8) The information processing device according to any one of (2) to (7), wherein, when the pointer reaches the limit of the movement range, the control unit causes the display device to display that the pointer has reached the limit of the movement range. (9) The information processing device according to any one of (1) to (8), wherein, when moving the pointer in accordance with movement of the pointing device, the processing unit increases the amount of movement of the pointing device and moves the pointer based on the increased amount of movement. (10) The information processing device according to (9), wherein the processing unit is capable of changing the amount of increase in the amount of movement of the pointing device. (11) The information processing device according to any one of (1) to (10), wherein the processing unit changes the operation range in a state where movement of the pointer is stopped.(12) The information processing device according to any one of (1) to (11), wherein the processing unit matches the reference plane with a specific plane in the virtual space. (13) The information processing device according to any one of (1) to (12), wherein the processing unit sets the reference plane or the reference point to a lower limit of the operation range. (14) The information processing device according to any one of (1) to (13), wherein the processing unit invokes a specific action when the reference plane or the reference point and the pointing device have a specific positional relationship. (15) The information processing device according to any one of (1) to (14), wherein the processing unit invokes a specific function in response to a user's input operation. (16) The information processing device according to any one of (1) to (15), wherein the processing unit suppresses shaking of the pointer caused by hand shaking of the pointing device by changing a degree of suppression of shaking in the pointer caused by hand shaking of the pointing device in response to a separation distance between the reference plane or the reference point and the pointing device. (17) The information processing device according to any one of (1) to (16), wherein the operation range is a three-dimensional spatial range narrower than a movement range of the pointer in the virtual space. (18) The information processing device according to any one of (1) to (17), wherein the display device is a display device capable of stereoscopic display. (19) An information processing system comprising: a display device; a setting unit that sets a surface of an object in real space as a reference plane or a point on the object as a reference point; a pointing device that is present within an operation range based on the reference plane or the reference point; a detection unit that detects a three-dimensional position of the pointing device; a processing unit that converts the three-dimensional position of the pointing device into a three-dimensional position of a pointer in virtual space; and a control unit that causes the display device to display the virtual space including the pointer. (20) An information processing method including: setting a surface of an object in real space as a reference surface or a point on the object as a reference point; converting the three-dimensional position of an indication device present within an operation range based on the reference surface or the reference point in the real space into the three-dimensional position of a pointer in a virtual space; and displaying the virtual space including the pointer on a display device.(21) An information processing system having any or all of the components included in the information processing device according to any one of (1) to (18). (22) An information processing method using any or all of the components included in the information processing device according to any one of (1) to (18).

[0175] REFERENCE SIGNS LIST 1 Information processing system 10 Information processing device 11 Setting unit 12 Processing unit 13 Control unit 14 Communication unit 20 Display device 20a Screen 21 Display unit 22 Communication unit 30 Detection device 31 Detection unit 32 Processing unit 33 Communication unit 40 Pointing device 41 Input unit 42 Light source 43 Processing unit 44 Communication unit 45 Contact sensor A1 Virtual space A2 Real space B1 Movement range B2 Operation range B3 Whole space B4 Changeable range C1 Pointer C2 Tip D1 Origin E2a Marker E2b Marker E2c Marker E2d Marker F1 Colored border G1 UI button T1 Object T1a Reference plane T1b Reference point

Claims

1. An information processing device comprising: a setting unit that sets a surface of an object in real space as a reference surface or a point of the object as a reference point; a processing unit that converts the three-dimensional position of a pointing device that exists within an operation range based on the reference surface or the reference point in the real space into the three-dimensional position of a pointer in a virtual space; and a control unit that displays the virtual space including the pointer on a display device.

2. The information processing device according to claim 1, wherein the processing section limits a movement range of the pointer in the virtual space to a part of the virtual space.

3. The information processing device according to claim 2, wherein the processing unit is capable of changing any or all of the size, position and shape of the movement range.

4. The information processing device according to claim 2, wherein the processing unit converts the three-dimensional position of the pointing device into a three-dimensional position of the pointer based on an offset amount of the operation range from the reference plane or the reference point.

5. The information processing device according to claim 4, wherein the processing unit changes the offset amount when the pointer moves beyond the movement range.

6. The information processing device according to claim 5, wherein the processing unit sets a limit to a changeable range of the offset amount.

7. The information processing device according to claim 5, wherein the processing unit adjusts the position of the movement range in response to a change in the offset amount.

8. The information processing device according to claim 2, wherein the control unit, when the pointer reaches a limit of the movement range, causes the display device to display a message indicating that the pointer has reached the limit of the movement range.

9. The information processing device according to claim 1, wherein, when moving the pointer in response to movement of the pointing device, the processing unit magnifies the amount of movement of the pointing device and moves the pointer based on the magnified amount of movement.

10. The information processing device according to claim 9, wherein the processing unit is capable of changing an amount of enlargement of the amount of movement of the instruction device.

11. The information processing device according to claim 1, wherein the processing unit changes the operation range in a state where the movement of the pointer is stopped.

12. The information processing device according to claim 1, wherein the processing unit matches the reference plane with a specific plane in the virtual space.

13. The information processing device according to claim 1, wherein the processing unit sets the reference plane or the reference point as a lower limit of the operation range.

14. The information processing device according to claim 1, wherein the processing unit initiates a specific operation when the reference plane or the reference point and the pointing device have a specific positional relationship.

15. The information processing device according to claim 1, wherein the processing unit activates a specific function in response to an input operation by a user.

16. The information processing device according to claim 1, wherein the processing unit suppresses shaking by varying the degree to which shaking of the pointer caused by shaking of the pointing device is suppressed depending on the distance between the reference plane or the reference point and the pointing device.

17. The information processing device according to claim 1, wherein the operation range is a three-dimensional spatial range narrower than a movement range of the pointer in the virtual space.

18. The information processing device according to claim 1, wherein the display device is a display device capable of stereoscopic display.

19. An information processing system comprising: a display device; a setting unit that sets a surface of an object in real space as a reference plane or a point of the object as a reference point; a pointing device that is present within an operation range based on the reference plane or the reference point; a detection device that detects a three-dimensional position of the pointing device; a processing unit that converts the three-dimensional position of the pointing device into a three-dimensional position of a pointer in a virtual space; and a control unit that displays the virtual space including the pointer on the display device.

20. An information processing method comprising: setting a surface of an object in real space as a reference surface or a point of the object as a reference point; converting the three-dimensional position of a pointing device existing within an operation range based on the reference surface or the reference point in the real space into the three-dimensional position of a pointer in a virtual space; and displaying the virtual space including the pointer on a display device.