Information processing system, information processing method, and program

Abstract

An information processing system, an information processing method, and a program according to one embodiment of the present technology comprise an acquisition unit and a correction unit. The acquisition unit acquires input position information based on an input operation for a three-dimensional position using a body. The correction unit corrects an error caused by a structure of the body and / or a physiological phenomenon of the body, in the acquired input position information. This makes it possible to provide high usability for a user who performs the input operation for the position.

Description

Information processing system, information processing method, and program

[0001] The present technology relates to an information processing system, an information processing method, and a program applicable to position input by a user.

[0002] Patent Document 1 discloses a naked-eye stereoscopic image display device that can display content in a stereoscopic manner without using special glasses.

[0003] Patent document 2 discloses an information processing device that aims to recognize that a touch state is continuing even when it becomes difficult to detect at least one of the multiple instruction positions that make up the operation being input due to a part of the operating object being in the blind spot of the imaging means.

[0004] International Publication No. 2018 / 116580 Japanese Patent Application Laid-Open No. 2017-84307

[0005] There is a demand for technology that can provide high usability when users use their own bodies to input positions.

[0006] In view of the above circumstances, an object of the present technology is to provide an information processing system, an information processing method, and a program that can provide high usability to a user who performs a position input operation.

[0007] To achieve the above object, an information processing system according to one embodiment of the present technology includes an acquisition unit and a correction unit. The acquisition unit acquires input position information based on an input operation of a three-dimensional position using a body. The correction unit corrects an error in the acquired input position information caused by at least one of a structure of the body and a physiological phenomenon of the body.

[0008] In this information processing system, errors caused by at least one of the structure of the body and physiological phenomena of the body are corrected for input position information input by the user using the body, thereby enabling high usability for users who perform input operations for three-dimensional positions.

[0009] The input operation may include an operation of moving an input device for inputting the three-dimensional position in three-dimensional space, or an operation of moving a predetermined part of the body in three-dimensional space.

[0010] The input operation may include an operation of moving the input device held in a hand while a part of the hand is in contact with a specified object, or an operation of moving a specified fingertip while a part of the hand is in contact with the specified object.

[0011] The input device may have a pen shape.

[0012] The correction unit may correct an arc-shaped trajectory acquired as the input position information so that the arc-shaped trajectory becomes a linear trajectory.

[0013] The correction unit may invalidate a change in position that falls within a predetermined range acquired as the input position information.

[0014] The input operation may be an operation of moving the input device in the three-dimensional space, and the input device may have a clickable button. In this case, the correction unit may invalidate a change in the position for a predetermined period based on a timing when the button of the input device is clicked.

[0015] The correction unit may correct the change in position acquired as the input position information to a change in position multiplied by a predetermined magnification.

[0016] The correction unit may correct a change in position in a predetermined direction acquired as the input position information to a change in position obtained by multiplying a magnification corresponding to the predetermined direction.

[0017] The correction unit may correct the change in position acquired as the input position information to a change in position obtained by multiplying the change in position by a magnification corresponding to a speed of the change in position.

[0018] The acquisition unit may set a reference plane or a reference direction for acquiring the input position information corresponding to the input operation with respect to the three-dimensional space in which the input operation is performed.

[0019] The information processing system may further include a setting unit that sets a mode related to the correction by the correction unit. In this case, the correction unit may perform the correction based on the set mode. The correction mode may include an execution mode in which the correction is performed and a non-execution mode in which the correction is not performed.

[0020] The correction mode may include a plurality of modes that define the degree of the correction.

[0021] The input operation may be an operation of moving the input device in the three-dimensional space. In this case, the input device may have a sensor that detects information regarding the orientation of the input device. The acquisition unit may acquire the input position information based on a detection result of the sensor.

[0022] The acquisition unit may switch between a three-dimensional position input mode in which three-dimensional position information is acquired as the input position information based on the input operation, and a two-dimensional position input mode in which two-dimensional position information is acquired as the input position information based on the input operation.

[0023] The input operation may be an input operation of a three-dimensional position relative to a three-dimensional virtual space that is displayed on a stereoscopic display and is visible to the naked eye.

[0024] According to one aspect of the present technology, there is provided an information processing method executed by a computer system, the information processing method including: acquiring input position information based on an input operation of a three-dimensional position using a body; and correcting an error in the acquired input position information caused by at least one of a structure of the body and a physiological phenomenon of the body.

[0025] A program according to an embodiment of the present technology causes a computer system to execute the information processing method.

[0026] 10 is a diagram illustrating an example configuration of a three-dimensional position input system according to an embodiment of the present technology. It is a schematic diagram illustrating a basic example configuration of a stereoscopic display. It is a flowchart illustrating a basic operation of a control device. It is a schematic diagram for explaining an example of correction of a body-induced error. It is a schematic diagram for explaining another example of correction of a body-induced error. It is a diagram schematically illustrating an inputtable range in a virtual space. It is a schematic diagram for explaining another example of correction of a body-induced error. It is a schematic diagram for explaining another example of correction of a body-induced error. It is a block diagram illustrating another example configuration of a control device. It is a flowchart illustrating an example operation of the control device shown in FIG. It is a diagram for explaining setting of a magnification corresponding to a speed of change of position. It is a schematic diagram showing an example setting of a reference plane and a reference direction. It is a block diagram illustrating an example hardware configuration of a computer that can be used as a control device.

[0027] Hereinafter, embodiments of the present technology will be described with reference to the drawings.

[0028] [Three-dimensional position input system] Fig. 1 is a schematic diagram showing a configuration example of a three-dimensional position input system according to an embodiment of the present technology. The three-dimensional position input system 1 shown in Fig. 1 functions as an embodiment of an information processing system according to the present technology, and is capable of providing a user 2 with a novel input interface that has not been available in the past.

[0029] As shown in FIG. 1, the three-dimensional position input system 1 includes a stereoscopic display 3, a stereo camera 4, an input device 5, a control device 6, and a database (DB) 7.

[0030] The 3D display 3, the stereo camera 4, the input device 5, and the control device 6 are connected to each other so that they can communicate with each other. The connection form of each device is not limited, and it is possible to use, for example, wireless LAN communication such as Wi-Fi or short-range wireless communication such as Bluetooth (registered trademark). Of course, a wired connection may also be used.

[0031] The 3D display 3 is a device capable of displaying content three-dimensionally without using special glasses, and may also be called a naked-eye 3D image display device or a naked-eye 3D display. For example, the 3D display 3 can be realized using the technology disclosed in Patent Document 1.

[0032] Fig. 2 is a schematic diagram showing an example of the basic configuration of the stereoscopic display 3. The stereoscopic display 3 has a display surface 9 that displays content three-dimensionally, and an exterior unit 10 that holds the display surface 9. In Fig. 2, the exterior unit 10 is schematically shown by dotted lines, and has a bottom surface 10a that is placed on a placement surface 11, a back surface 10b, and a front surface 10c that is arranged so as to be inclined at a predetermined angle with respect to the bottom surface 10a (placement surface 11).

[0033] The display surface 9 is installed on the front surface 10c and configured to be inclined at a predetermined angle with respect to the bottom surface 10a (the placement surface 11). By viewing the display surface 9, the user 2 can view various virtual objects 12 that exist in a three-dimensional virtual space VS.

[0034] Hereinafter, the planar direction of the placement surface 11 on which the three-dimensional display 3 is placed is referred to as the horizontal direction. Two directions that are parallel to the horizontal direction and perpendicular to each other are referred to as the X direction and the Y direction. In this embodiment, as shown in Figures 1 and 2, the left-right direction of the three-dimensional display 3 is referred to as the X direction, and the positive side of the X axis is referred to as the left side, and the negative side is referred to as the right side. The depth direction of the three-dimensional display 3 is referred to as the Y direction, and the positive side of the Y axis is referred to as the front side, and the negative side is referred to as the back side.

[0035] Furthermore, the vertical direction perpendicular to the horizontal direction (XY plane direction) is defined as the Z direction, and the positive side of the Z axis is described as the upper side, and the negative side is described as the lower side. Of course, the application of the present technology is not limited to the orientation in which the three-dimensional display 3 is disposed.

[0036] 1 , a virtual space VS is constructed extending in the depth direction (Y direction) beyond the display surface 9 of the stereoscopic display 3. Four virtual objects 12, namely, a cylinder, a rectangular prism, a cube, and a rectangular parallelepiped, are placed at positions on the far side of the virtual space VS. Four virtual objects 12, each a rectangular parallelepiped, are placed at positions on the near side of the virtual space VS.

[0037] In the example shown in Fig. 2, a virtual object 12 consisting of a fictional character is displayed on the display surface 9. As shown in Fig. 2, the head of the character is displayed so as to protrude from the display surface 9, and the body of the character is displayed so as to recede into the display surface 9. In this way, it is also possible to control the position at which the virtual object 12 is perceived by the user 2. In other words, it is possible to appropriately control whether the virtual object 12 is perceived in front of or behind the display surface 9.

[0038] In this embodiment, the horizontal and vertical directions in the three-dimensional real space RS in which the stereoscopic display 3 is placed correspond to the horizontal and vertical directions in the three-dimensional virtual space VS displayed by the stereoscopic display 3. Therefore, the XY plane direction is the horizontal direction for both the real space RS and the virtual space VS. The Z direction is the vertical direction for both the real space RS and the virtual space VS.

[0039] 1 and 2, as in the real space RS, the X direction in the virtual space VS indicates the left-right direction, with the positive side of the X axis being referred to as the left side and the negative side as the right side. The Y direction indicates the depth direction, with the positive side of the Y axis being referred to as the front side and the negative side as the back side. The Z direction indicates the up-down direction, with the positive side of the Z axis being referred to as the top side and the negative side as the bottom side.

[0040] Of course, the application of the present invention is not limited to horizontal or vertical correspondence between the real space RS and the virtual space SV. For example, in response to a position input by the user 2 along the left-right direction (X direction), position information along another direction, such as up-down, in the virtual space SV may be calculated.

[0041] The stereo camera 4 can calculate distance information for each pixel of the captured image. The specific configuration of the stereo camera 4 and the specific algorithm for calculating the distance are not limited, and any configuration and algorithm may be used.

[0042] Furthermore, the distance measurement sensor is not limited to the stereo camera 4, and other distance measurement sensors may be used. Of course, it is also possible to use a combination of a camera and a distance measurement sensor. As the distance measurement sensor, various types of distance measurement sensors can be used, such as an optical laser distance measurement sensor, an ultrasonic distance measurement sensor, a ToF (Time of Flight) sensor, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), or a structured light distance measurement sensor.

[0043] The input device 5 is used by the user 2 to input a three-dimensional position using the body. As shown in Fig. 1, the input device 5 has a pen shape. That is, in this embodiment, a pen-type input device 5 is used.

[0044] An infrared LED 13 is provided at the tip (pen tip) of the pen-type input device 5. The pen-type input device 5 also has a plurality of clickable buttons 14.

[0045] 1, the input device 5 is held by a hand 15 of a user 2. In the present disclosure, the hand includes the wrist.

[0046] The user 2 holds the pen-shaped input device 5 in the same manner as holding a normal pen. Then, the user moves the input device 5 while touching the ball of the little finger (the part of the base of the little finger near the wrist) or part of the wrist to the operation pad 16 arranged on the placement surface 11, as if writing characters with the pen. Specifically, the user moves the hand to move the infrared LED 13 at the tip of the input device 5 not only on the operation pad 16 but also in the air while floating above the operation pad 16. This allows for input of a three-dimensional position.

[0047] In this embodiment, the input operation of the user 2 using the body to input a three-dimensional position is realized by moving the input device 5 for inputting the three-dimensional position in three-dimensional space (real space RS). Specifically, the input of the three-dimensional position is realized by moving the input device 5 held in the hand 15 of the user 2 while a part of the hand 15 (such as the ball of the little finger) is in contact with a predetermined object (the operation pad 16).

[0048] Of course, it is also possible to move the input device 5 while the hand 15 is floating above the operation pad 16. It is also possible to move the input device 5 by sliding the part of the hand 15 that is in contact with the operation pad 16 over the operation pad 16.

[0049] On the other hand, by contacting a part of the hand 15 (such as the ball of the little finger) with the operation pad 16, it is possible to improve the stability of the infrared LED 13 at the pen tip when moving it in the air, thereby improving the accuracy of inputting three-dimensional positions. It is also possible to reduce the sense of fatigue felt by the user 2.

[0050] 1, a pointer 17 is displayed in a virtual space VS displayed on the display surface 9 of the stereoscopic display 3. For example, the user 2 can move the infrared LED 13 at the tip of the pen in the real space RS, thereby moving the pointer 17 in the virtual space VS.

[0051] Of course, the processing in response to the input of a three-dimensional position is not limited to the movement of the pointer 17. For example, the present technology can be applied to various processing such as the movement of a specified virtual object 12, the selection of an item from a displayed menu, and the like.

[0052] The control device 6 has hardware necessary for a computer, such as a processor such as a CPU, a GPU, or a DSP, a memory such as a ROM or a RAM, and a storage device such as an HDD (see FIG. 13 ). The processor loads a program according to the present technology stored in the storage unit or memory into the RAM and executes it, thereby executing the information processing method (three-dimensional position input method) according to the present technology. The control device 6 can also be called a three-dimensional input device.

[0053] The DB 7 stores various information and data related to the present three-dimensional position input system 1. For example, various information is stored, such as information for constructing the virtual space VS, information for displaying the virtual object 12 and the pointer 17, information related to the stereo camera 4 and the input device 5, and information related to the user 2 who uses the present three-dimensional position input system 1.

[0054] 1, the DB 7 is configured by a storage device or the like separate from the control device 6 and is connected to the control device 6. However, the configuration is not limited to this, and the DB 7 may be configured by a storage unit (see FIG. 13) of the control device 6. Furthermore, a configuration may be adopted in which the DB 7 is constructed on a network and is accessible via the network.

[0055] 1, in this embodiment, the processor of the control device 6 executes a predetermined program to configure an acquisition unit 18 and a correction unit 19 as functional blocks. Of course, dedicated hardware such as an IC (integrated circuit) may be used to realize each functional block.

[0056] The program is installed in the control device 6 via, for example, various recording media. Alternatively, the program may be installed via the Internet or the like. There are no limitations on the type of recording medium on which the program is recorded, and any computer-readable recording medium may be used. For example, any computer-readable non-transitory storage medium may be used.

[0057] 3 is a flowchart showing the basic operation of the control device 6. The acquisition unit 18 acquires input position information based on an input operation of a three-dimensional position using the body of the user 2 (step 101).

[0058] In this disclosure, obtaining information (data) includes both generating the information (data) by executing a predetermined algorithm on one's own and receiving the information (data) from another device.

[0059] In this embodiment, the acquisition unit 18 generates input position information based on an input operation by the user 2. Specifically, first, the captured image (stereo image) captured by the stereo camera 4 and distance information for each pixel of the captured image are transmitted to the control device 6.

[0060] The acquisition unit 18 executes a recognition process based on the captured image and recognizes the infrared LED 13 at the pen tip of the input device 5 within the captured image. Then, based on distance information for each pixel of the captured image, position information in the three-dimensional real space RS of the infrared LED 13 at the pen tip of the input device 5 is calculated. In this embodiment, the position information of the infrared LED 13 is calculated as the position information of the input device 5.

[0061] As position information in the three-dimensional real space RS, for example, coordinate information (X, Y, Z) is used. As a coordinate system for defining the coordinate information (X, Y, Z), an absolute coordinate system (world coordinate system) may be used, or a relative coordinate system with a predetermined point as the reference (origin) may be used. In this embodiment, the coordinate information (X, Y, Z) in the XYZ coordinate system defined in Figures 1 and 2 is calculated.

[0062] Of course, the position information of the input device 5 (position information of the infrared LED 13) may be calculated using polar coordinates. For example, a reference point may be set on a part of the hand (such as the ball of the little finger) that is in contact with the operation pad 16, and the position information of the input device 5 (position information of the infrared LED 13) may be calculated using polar coordinates based on the set reference point.

[0063] The specific algorithm for recognizing the infrared LED 13 and calculating the coordinate information (X, Y, Z) in the real space RS based on the captured image taken by the stereo camera 4 and the distance information for each pixel is not limited, and any algorithm may be adopted.

[0064] Next, the acquisition unit 18 calculates position information in the three-dimensional virtual space VS based on the position information of the input device 5 (position information of the infrared LED 13). In this embodiment, coordinate information (XV, YV, ZV) in the XYZ coordinate system defined in Figures 1 and 2 is calculated as the position information in the virtual space VS. Of course, polar coordinates may also be used.

[0065] There are no specific limitations on the algorithm for calculating coordinate information (XV, YV, ZV) in the virtual space VS from coordinate information (X, Y, Z) in the real space RS, and any algorithm may be adopted.

[0066] For object recognition by the acquisition unit 18, calculation of coordinate information (X, Y, Z) in the real space RS, and calculation of coordinate information (XV, YV, ZV) in the virtual space VS, an algorithm (machine learning algorithm) using a machine learning model constructed using, for example, a neural network may be used.

[0067] For example, any machine learning algorithm may be used, such as a deep neural network (DNN), a recurrent neural network (RNN), or a convolutional neural network (CNN). For example, by using an AI (artificial intelligence) that performs deep learning, it is possible to perform processing with high accuracy.

[0068] The application of a machine learning algorithm may be performed for any process within the present disclosure, i.e., any process described within the present disclosure may be subjected to a process using machine learning.

[0069] In this embodiment, the coordinate information (XV, YV, ZV) in the virtual space VS calculated in step 101 corresponds to input position information based on an input operation of a three-dimensional position using the body of the user 2. The coordinate information (XV, YV, ZV) that is the input position information is calculated at a predetermined frame rate.

[0070] The correction unit 19 shown in FIG. 1 corrects the acquired input position information, i.e., the coordinate information (XV, YV, ZV) in the virtual space VS, for errors caused by at least one of the structure of the body of the user 2 and the physiological phenomenon of the body of the user 2 (step 102).

[0071] Correction of coordinate information (XV, YV, ZV) in virtual space VS includes correction of one piece of coordinate information (XV, YV, ZV) calculated in a predetermined frame. Correction of coordinate information (XV, YV, ZV) in virtual space VS also includes correction of a trajectory or the like made up of multiple pieces of coordinate information (XV, YV, ZV) calculated in multiple consecutive frames. In addition, correction of coordinate information (XV, YV, ZV) in virtual space VS also includes processing such as invalidating calculated coordinate information (XV, YV, ZV).

[0072] [Correction of Errors Due to Body Structure and Correction of Errors Due to Body Physiological Phenomena] Errors due to body structure include various errors that can occur due to the structure of the human body, such as the skeleton, muscles, skin, posture, range of motion (movement range), etc. Errors due to body physiological phenomena include various errors that can occur due to human physiological phenomena, such as breathing, body sway (phenomenon of not being able to stay completely still), convulsions, etc.

[0073] It should be noted that errors due to the structure of the body and errors due to physiological phenomena of the body are not clearly defined and a certain error that occurs may be included in errors due to the structure of the body and errors due to physiological phenomena of the body.

[0074] The error caused by the structure of the body can also be said to be an error caused by the structural constraints of the body, and the error caused by the physiological phenomena of the body can also be said to be an error caused by the physiological constraints of the body.

[0075] Hereinafter, errors due to the structure of the body and errors due to physiological phenomena of the body may be collectively referred to as "physical errors." Furthermore, correction of errors due to the structure of the body and correction of errors due to physiological phenomena of the body may be collectively referred to as "physical error correction."

[0076] 4 is a schematic diagram illustrating an example of correction of body-induced error. As shown in FIG. 4, the user 2 moves the position of the infrared LED 13 at the pen tip, including through the air, while keeping a part of the hand 15 (such as the ball of the little finger) in contact with the operation pad 16. For example, when it is desired to move the pointer 17 shown in FIG. 1 in each of the X, Y, and Z directions, the infrared LED 13 at the pen tip is moved in each of the X, Y, and Z directions.

[0077] In this case, due to the structure of the hand 15, when the user 2 wants to move the infrared LED 13 at the tip of the pen in a straight line along a specific direction, it is conceivable that the movement trajectory will be an arc-shaped trajectory rather than a straight line.

[0078] For example, when moving the infrared LED 13 at the tip of the pen along the left-right direction (X direction), along the depth direction (Y direction), or along the up-down direction (Z direction), due to the structure of the hand 15, the movement includes a rotational component centered on the ball of the little finger, etc., and the movement trajectory becomes arc-shaped.

[0079] 4, the movement locus of the infrared LED 13 in the left-right direction (X direction) is an arc-shaped locus as indicated by the arrow AX1. The movement locus of the infrared LED 13 in the depth direction (Y direction) is an arc-shaped locus as indicated by the arrow AY1. The movement locus of the infrared LED 13 in the up-down direction (Z direction) is an arc-shaped locus as indicated by the arrow AZ1.

[0080] As an input operation for a three-dimensional position, an arc-shaped locus as indicated by arrows AX1, AY1, and AZ1 is input. Then, in step 101 of Fig. 3, the coordinate information (XV, YV, ZV) in the virtual space VS calculated as input position information in a plurality of consecutive frames also forms an arc-shaped locus.

[0081] As a result, for example, in the virtual space VS, the pointer 17 shown in FIG. 1 does not move linearly along the X, Y, and Z directions, but moves along an arc-shaped locus.

[0082] In the present three-dimensional position input system 1, the input position information is corrected for body-induced errors by the correction unit 19. Specifically, it is possible to correct an arc-shaped trajectory acquired as input position information so that it becomes a linear trajectory.

[0083] As shown in Figure 4, in this embodiment, the coordinate information (XV, YV, ZV) of the virtual space VS calculated based on an input operation that forms an arc shape of the arrow AX1 is corrected to coordinate information (XV, YV, ZV) that forms a linear trajectory as shown by the arrow AX2.

[0084] In addition, the coordinate information (XV, YV, ZV) of the virtual space VS calculated based on the input operation that forms the arc of the arrow AY1 is corrected to coordinate information (XV, YV, ZV) that forms a linear trajectory as shown by the arrow AY2.

[0085] In addition, the coordinate information (XV, YV, ZV) of the virtual space VS calculated based on the input operation that forms the arc of arrow AZ1 is corrected to coordinate information (XV, YV, ZV) that forms a linear trajectory as shown by arrow AZ2.

[0086] In this way, it is possible to remove distortion that occurs when drawing a trajectory in the air with the present three-dimensional position input system 1. Specifically, it is possible to convert an arc-shaped trajectory that occurs when drawing a straight line centered on the ball of the little finger or the wrist into a linear trajectory and use it as input.

[0087] This makes it possible to realize movement of the pointer 17 in accordance with the user's 2 sense of linearly moving the pointer 17 along the X, Y, and Z directions, for example. As a result, high usability can be achieved for the user 2 performing a position input operation.

[0088] Note that the specific algorithm for correcting an arc-shaped trajectory to a linear trajectory is not limited, and any algorithm may be adopted. For example, a start point and an end point may be set for the arc-shaped trajectory, and a straight line connecting the set start point and end point may be calculated as the corrected position information. Alternatively, a straight line may be calculated by linear approximation (straight-line approximation) for the arc-shaped trajectory (coordinate information of each frame), and used as the corrected position information.

[0089] Alternatively, multiple people are asked to move the infrared LED 13 at the pen tip of the input device 5 in a straight line along a predetermined direction in a state as shown in FIG. 1. A machine learning model is trained using a pair of information on the arc-shaped trajectory obtained by the movement and information on the predetermined direction as training data. Information on the arc-shaped trajectory calculated as input position information is input to the machine learning model, and the output information on the straight line can be used as corrected position information. Any other method, such as pattern matching, may be used.

[0090] 5A and 5B are schematic diagrams illustrating another example of correction of body-induced errors. As shown in Fig. 5A, the typical range of motion for flexion of a human hand 15 is 0° to 25°, and the typical range of motion for ulnar flexion is 0° to 55°.

[0091] 5B, the typical range of motion for flexion from an extended state at the second joint of the middle finger of a human hand 15 is 0° to 100°. Also, as shown in FIG. 5C, the typical range of motion for dorsiflexion of a human hand 15 is 0° to 70°, and the typical range of motion for palmar flexion is 0° to 90°.

[0092] 5, it is assumed that the input operation of the three-dimensional position of the user 2 is performed by moving the fingertips of the middle and index fingers while keeping the wrist fixed. That is, in this example, the positions of the fingertips of the middle and index fingers in real space are calculated based on the detection results of the stereo camera 4. Then, it is assumed that the position in the virtual space VS is calculated as input position information based on the positions of the fingertips of the middle and index fingers in real space.

[0093] For example, input of a position along the X direction is performed by flexing and retracting the middle finger with the finger extended as shown in Fig. 5A. In this case, if the length of the hand 15 (the length from the wrist to the tip of the middle finger) is 178 mm, the range of motion RX of the middle fingertip along the X direction can be calculated using the following formula: RX = 178 mm × (sin(25°) + sin(25°)) From the above formula, the range of motion RX along the X direction is approximately 221 mm.

[0094] Furthermore, input of a position along the Y direction is performed by extending and bending the second joint of the index finger from an intermediate position where the bending angle is 50°, as shown in FIG. 5B . In this case, if the length from the tip of the index finger to the second joint is 46.3 mm, the range of motion RY of the index fingertip along the Y direction can be calculated using the following formula: RY = 46.3 mm × 2 sin (50°) From the above formula, the range of motion RY along the Y direction is calculated to be approximately 71 mm.

[0095] Furthermore, input of a position along the Z direction is performed by dorsiflexion and palmar flexion with the middle finger extended, as shown in Fig. 5C. In this case, if the length of the hand 15 (the length from the wrist to the tip of the middle finger) is 178 mm, the range of motion RZ of the tip of the middle finger along the Z direction can be calculated using the following formula: RZ = 178 mm × (sin(70°) + sin(90°)) From the above formula, the range of motion RZ along the Z direction is approximately 345 mm.

[0096] That is, in the three-dimensional position input operation shown in FIG. 5, the range of motion in each of the X, Y and Z directions differs due to the structure of the hand 15.

[0097] Fig. 6 is a diagram schematically illustrating an inputtable range in the virtual space VS. Fig. 6A is a diagram schematically illustrating an inputtable range corresponding to the range of motion of the fingertips of the middle finger and index finger in each of the X, Y, and Z directions in the three-dimensional position input operation shown in Fig. 5. Since the range of motion of the fingertips of the middle finger and index finger in each of the X, Y, and Z directions in the input operation differs, the inputtable range of the position in the virtual space VS also differs.

[0098] Specifically, the input range in the Y direction is smaller than the input range in the X direction, which corresponds to the difference between the range of motion in the X direction (approximately 221 mm) due to the flexion and contraction movements shown in Fig. 5A and the range of motion in the Y direction (approximately 71 mm) due to the extension and flexion movements shown in Fig. 5B.

[0099] The input range in the Z direction is larger than the input range in the X direction, which corresponds to the difference between the range of motion in the X direction (approximately 221 mm) due to the flexion and extension movements shown in Fig. 5A and the range of motion in the Z direction (approximately 345 mm) due to the dorsiflexion and palmar flexion movements shown in Fig. 5C.

[0100] In the case of the input range shown in FIG. 6A , for example, when user 2 moves pointer 17, the range in which pointer 17 can be moved in the Y direction is smaller than the range in which pointer 17 can be moved in the X direction, and user 2 may find it annoying that he or she has to bend and straighten his or her index finger repeatedly.

[0101] Furthermore, the movable range of the pointer 17 along the Z direction is larger than the movable range along the X direction, and the user may feel annoyed that the pointer 17 moves more than expected when performing the dorsiflexion and palmar flexion movements shown in Figure 5C.

[0102] In the present three-dimensional position input system 1, the correction unit 19 can correct a change in position acquired as input position information to a change in position obtained by multiplying a predetermined magnification. The correction unit 19 can also correct a change in position in a predetermined direction acquired as input position information to a change in position obtained by multiplying a magnification corresponding to the predetermined direction.

[0103] For example, no correction is made to a change in position along the X direction acquired as input position information (this can also be considered a correction with a magnification of 1). A magnification of 221 / 71 (=approximately 3.11) is applied to a change in position along the Y direction acquired as input position information. Furthermore, a magnification of 221 / 345 (=approximately 0.64) is applied to a change in position along the Z direction acquired as input position information.

[0104] As a result, as shown in Fig. 6B, it is possible to make the range of positions that can be input in each of the X, Y, and Z directions substantially equal in the three-dimensional position input operation shown in Fig. 5. In other words, it is possible to substantially widen the range of positions that can be input along the Y direction, and to substantially narrow the range of positions that can be input along the Z direction.

[0105] As a result, for example, when the user 2 moves the pointer 17, the user 2 can experience movement of the pointer 17 that is adapted to the sensation of moving the fingertips of the middle finger and index finger in each of the X, Y, and Z directions. That is, in each of the flexion and retraction movements shown in Fig. 5A, the extension and flexion movements shown in Fig. 5B, and the dorsiflexion and palmar flexion movements shown in Fig. 5C, the movement of the pointer 17 when moved within the maximum range of motion is approximately the same (i.e., the movable range is approximately the same in each of the X, Y, and Z directions). As a result, high usability can be achieved for the user 2 performing a position input operation.

[0106] In this way, the present three-dimensional position input system 1 can accommodate differences in the range of motion of the fingertips of the middle finger and index finger in each of the X, Y, and Z directions. Note that the present technology can also accommodate differences in the size and range of motion of the hands of multiple users 2.

[0107] For example, it is possible to reduce the magnification for a user 2 with very large hands and increase the magnification for a user 2 with small hands.

[0108] As a method for setting the magnification, for example, a GUI (Graphical User Interface) that allows the magnification to be adjusted using a slide bar or the like may be presented to the user 2, and the user 2 may set the magnification. For example, the user 2 sets the magnification in each of the X, Y, and Z directions, and then actually performs an input operation to move the pointer 17. This makes it possible to easily set a magnification that is easy for the user to operate. Of course, it is also possible to intentionally set different input ranges in each of the X, Y, and Z directions.

[0109] Alternatively, the control device 6 can detect the size of the hand of the user 2 and automatically set the magnification in each of the X, Y, and Z directions.

[0110] 5, the operation of inputting a three-dimensional position using the body of the user 2 includes an operation of moving a predetermined part of the body in three-dimensional space (real space RS) without using the input device 5. For example, the operation of inputting a three-dimensional position using the body of the user 2 includes an operation of moving a predetermined fingertip while a part (such as the ball of the little finger) of the hand 15 of the user 2 is in contact with a predetermined object (the operation pad 16).

[0111] Of course, even when the pen-type input device 5 shown in Figure 1 is used, the process of correcting the position change acquired as input position information to a position change multiplied by a predetermined magnification, and the process of correcting the position change in a predetermined direction acquired as input position information to a position change multiplied by a magnification corresponding to the predetermined direction are effective.

[0112] The user 2 moves the pointer 17 using the pen-type input device 5 while adjusting the magnification in each of the X, Y, and Z directions as appropriate, thereby enabling the user 2 to easily set a magnification that is easy for the user to operate.

[0113] 7 is a schematic diagram illustrating another example of correction of body-induced error. When the position of the infrared LED 13 at the pen tip is moved through the air in the three-dimensional real space RS, it is difficult to keep the infrared LED 13 at the pen tip completely still due to fluctuations, which are a physiological phenomenon of the body. Therefore, when moving the pointer 17, for example, the position of the pointer 17 changes and shakes, as shown in FIG. 7, making it difficult to keep the pointer 17 completely still.

[0114] 7 illustrates the change in the position of the pointer 17 in each of the X, Y, and Z directions in the virtual space VS. Of course, the change in the position of the pointer 17 due to bodily movement can occur in any direction.

[0115] In the present three-dimensional position input system 1, the correction unit 19 can invalidate position changes that fall within a predetermined range acquired as input position information. For example, in the illustration of the pointer 17 shown in Fig. 7, the pointer 17 displayed in gray represents a position change that falls within a predetermined range. The position change is invalidated, and the pointer 17 displayed in black is displayed in a stationary state.

[0116] For example, a virtual three-dimensional area having a spherical shape with a predetermined radius is set. Positional changes that fall within the three-dimensional area can be invalidated. Furthermore, as shown in FIG. 7 , ranges in which positional changes are invalid can be individually set for each of the X, Y, and Z directions in the virtual space VS.

[0117] By disabling position changes within a predetermined range, it is possible to prevent the position of the pointer 17 from changing minutely and shaking due to unconscious physiological body movements. In other words, it is possible to eliminate changes (shaking) in the position of the pointer 17 due to body movements, and it is possible to stop the pointer 17 at a predetermined position. As a result, it is possible to provide high usability for the user 2 who performs a position input operation.

[0118] 8 is a schematic diagram for explaining another example of correction of the body-induced error. The user 2 can input various instructions by clicking the buttons 14 provided on the input device 5.

[0119] For example, various instructions can be input by moving the pointer 17 displayed in the virtual space VS to a predetermined position and clicking the button 14. It is also possible to select the virtual object 12 by moving the pointer 17 to a position where it overlaps with the virtual object 12 and clicking the button 14. It is also possible to perform a drag-and-drop operation by clicking and holding down the button 14.

[0120] While holding the pen-type input device 5, the user clicks the button 14 using, for example, the index finger. At this time, due to the structure of the hand 15 (which can also be considered a physiological phenomenon), the position of the infrared LED 13 at the tip of the pen often changes slightly and becomes unstable.

[0121] For example, when the button 14 is clicked, the infrared LED 13 at the tip of the pen may move slightly in the direction in which the button 14 is clicked (the direction in which force acts on the button 14). Also, when the click is released, the infrared LED 13 at the tip of the pen may move slightly.

[0122] This may result in a phenomenon in which the position of the pointer 17 displayed in the virtual space VS changes slightly and becomes blurred every time the user 2 clicks the button 14 of the input device 5.

[0123] In the present three-dimensional position input system 1, the correction unit 19 can invalidate changes in position during a predetermined period based on the timing at which the button 14 of the input device 5 is clicked. The timing at which the click is clicked corresponds to the timing at which the click is detected.

[0124] 8, the state before the button 14 of the input device 5 is clicked is illustrated as the initial state, and the pointer 17 is stationary at a predetermined position. When the user 2 clicks the button 14, the acquisition unit 18 detects a change in the coordinate information, which is input position information, and at the same time detects that the user 2 has clicked the button 14.

[0125] Any change in position during a predetermined period based on the timing when the button 14 is clicked is invalidated, and the pointer 17 is returned to its initial position several frames prior. In other words, the movement of the pointer 17 at the time of the click is not reflected. This makes it possible to prevent the pointer 17 from slightly changing and shaking in the virtual space VS every time the button 14 is clicked. As a result, high usability can be achieved for the user 2 who performs a position input operation.

[0126] The predetermined period (e.g., the number of frames) based on the timing at which the button 14 is clicked may be set arbitrarily. For example, by disabling position changes for a predetermined period after the click, it is possible to prevent position changes when the click is released, for example.

[0127] In addition, in this three-dimensional position input system 1, the correction unit 19 can correct various errors caused by the structure of the body and various errors caused by physiological phenomena of the body in the three-dimensional input position information.

[0128] Fig. 9 is a block diagram showing another example configuration of the control device 6. The control device 6 shown in Fig. 9 includes a data acquisition unit 21, a device position detection unit 22, a device angle detection unit 23, a position information calculation unit 24, a mode setting unit 25, a position information correction unit 26, an input acceptance unit 27, and a display control unit 28. These functional blocks are realized, for example, by the processor of the control device 6 executing a predetermined program.

[0129] Of the functional blocks shown in Fig. 9, the data acquisition unit 21, the device position detection unit 22, the device angle detection unit 23, and the position information calculation unit 24 function as the acquisition unit 18 shown in Fig. 1. Furthermore, the position information correction unit 26 functions as the correction unit 19 shown in Fig. 1.

[0130] The mode setting unit 25 functions as an embodiment of a setting unit that sets a mode related to correction by the correction unit 19 according to the present technology.

[0131] Fig. 10 is a flowchart showing an example of the operation of the control device 6 shown in Fig. 9. First, the data acquisition unit 21 acquires the image captured by the stereo camera 4 and the distance information for each pixel of the captured image (step 101).

[0132] In this embodiment, an IMU (Inertial Measurement Unit) sensor is mounted on the pen-type input device 5 shown in Fig. 1. Then, in step 101, the data acquisition unit 21 acquires sensing data from the IMU sensor (also in step 201).

[0133] The IMU sensor functions as an embodiment of a sensor that detects information related to the orientation of an input device according to the present technology. Any sensor, such as a compass, an acceleration sensor, or a gyro sensor, may be used as the sensor.

[0134] The device position detection unit 22 calculates the position information of the input device 5 in three-dimensional real space of the infrared LED 13 at the pen tip of the input device 5 based on the captured image and distance information for each pixel (step 202).

[0135] The device angle detection unit 23 detects the angle of the input device 5 based on the sensing data of the IMU sensor (step 203). The angle information can be defined, for example, by vector information based on an XYZ coordinate system that defines a position in three-dimensional real space. Of course, the angle information is not limited to such data.

[0136] The position information calculation unit 24 calculates the position information in the virtual space VS based on the position information of the input device 5 and the angle of the input device 5 (step 204). In this way, in this embodiment, it is possible to acquire the input position information based on the detection result of the sensor (IMU sensor) mounted on the input device 5 that detects information regarding the attitude of the input device 5.

[0137] By calculating the angle (attitude) of the input device 5 based on sensing data from the IMU sensor, it becomes possible to use rotation information of the input device 5 as an input. This makes it possible to simultaneously input movement and rotation of the input device 5, and to realize movement and rotation of the pointer 17 and the like in accordance with the sense of the user 2 operating the input device 5. As a result, high usability can be achieved.

[0138] For example, the user 2 can switch between a movement mode and a rotation mode by performing an operation such as selecting a predetermined button 14. When the movement mode is set, the pointer 17 and the like are moved in response to the movement of the infrared LED 13. When the rotation mode is set, the pointer 17 and the like are rotated in response to the movement of the infrared LED 13 in a predetermined direction. Compared to when rotation is input by switching modes in this way, it is possible to achieve movement and rotation of the pointer 17 and the like that is closer to the sensation felt by the user 2.

[0139] It is also possible to correct the posture and position calculated from the values ​​of the IMU sensor, etc., using constraints related to the range of motion of a person that occurs when inputting to the three-dimensional position input system 1, such as when part of the hand (such as the ball of the little finger) is in contact with the operation pad 16. This makes it possible to improve the accuracy of the input posture (rotation) information. For example, it is also possible to prevent the value calculated by integrating the values ​​of the IMU sensor, etc., from diverging.

[0140] The position information correcting section 26 determines whether or not the correction execution mode is in effect (step 205).

[0141] In this embodiment, a correction execution mode in which correction of a body-induced error is executed and a correction non-execution mode in which correction of a body-induced error is not executed can be switchably set by the mode setting unit 25. The correction execution mode and the correction non-execution mode are embodiments of the correction mode and the non-correction mode according to the present technology.

[0142] The correction execution mode and the correction non-execution mode may be set by an instruction from the user 2. For example, the input receiving unit 27 displays a GUI for setting the correction execution mode and the correction non-execution mode on the three-dimensional display 3. The user 2 selects either the correction execution mode or the correction non-execution mode using, for example, the input device 5. Of course, voice input or the like may also be possible.

[0143] For example, there may be cases where the user 2 wants to use the input device 5 to draw an image including subtle curves, such as a picture or text. In such cases, it may be better not to perform the correction of body-induced errors according to the present technology. In this embodiment, it is possible to switch between a correction execution mode and a correction non-execution mode, so that processing can be performed according to the user 2's wishes. As a result, high usability can be achieved.

[0144] If the correction execution mode is set (Yes in step 205), the position information correction unit 26 corrects the body-induced error in the position information in the virtual space VS calculated in step 204 (step 206). Then, the display control unit 28 executes display control based on the corrected position information (step 207). For example, any processing may be executed, such as moving the pointer 17 shown in FIG. 1 or drawing a marker on text.

[0145] Note that correction filtering may be performed as the correction processing of the position information correction unit 26. For example, trajectory correction filtering, position change correction filtering, or the like may be performed.

[0146] If the correction execution mode is not set (No in step 205), the display control unit 28 executes display control based on the position information in the virtual space VS calculated in step 204. For example, any processing may be executed, such as moving the pointer 17 shown in FIG. 1 or drawing pictures or characters (step 207).

[0147] A plurality of modes that define the degree of correction may be set as modes related to correction by the mode setting unit 25 shown in Fig. 9. The degree of correction may also be referred to as the degree of correction or the strength of correction.

[0148] For example, it is possible to set a plurality of modes with gradually increasing degrees of correction, such as (corrected level 1), (corrected level 2), (corrected level 3), (corrected level 4), and (corrected level 5).

[0149] (Level 1 with correction) is the mode with the lowest degree of correction, and for example, an arc-shaped trajectory that is very close to a straight line is corrected to a straight line trajectory, but an arc-shaped trajectory with a large curvature is not corrected to a straight line trajectory. Also, the range in which position changes in the virtual space VS are invalid is set relatively small.

[0150] (Level 5 with correction) is the mode with the highest degree of correction, and for example, not only arc-shaped trajectories that are very close to straight lines, but also arc-shaped trajectories with a certain degree of curvature are corrected to straight trajectories. In addition, the range in which position changes are invalid in the virtual space VS is set relatively large.

[0151] Of course, the present invention is not limited to such mode settings, and any mode setting that specifies the degree of correction may be adopted. For example, by adopting a configuration that allows the user 2 to set the degree of correction himself, high usability can be achieved.

[0152] 10, the position information in the virtual space VS is corrected based on the mode set in step 206. Of course, if the non-correction mode is set, the process proceeds from No in step 205 to step 207.

[0153] The three-dimensional position input system 1 can further be equipped with new and effective functions, which will be described below.

[0154] For example, in the process of correcting the position change acquired as input position information to a position change multiplied by a predetermined magnification, as described with reference to Figures 5 and 6, it is also possible to correct the position change to a position change multiplied by a magnification corresponding to the speed of the position change.

[0155] 11A to 11C are schematic diagrams for explaining the setting of a magnification corresponding to the speed of change in position. The horizontal axis of the graphs shown in Fig. 11A to 11C represents the movement speed of the infrared LED 13 at the pen tip, which corresponds to the speed of change in input position. The vertical axis of each graph represents the set magnification.

[0156] 11A , if a constant magnification is set regardless of the movement speed of the infrared LED 13 at the pen tip, the magnification may be too high and the pointer 17 may move too quickly, which may be annoying when moving slowly and performing delicate work.Also, if the magnification is too low and the pointer 17 moves a large distance and the pointer 17 moves quickly, the pointer 17 may move too slowly and may be annoying.

[0157] 11B, the magnification is changed according to the movement speed of the pointer 17. Specifically, when the movement speed of the infrared LED 13 at the pen tip is slow, the magnification is set relatively low, and when the movement speed is fast, the magnification is set relatively high.

[0158] For example, as shown in Fig. 11B, a speed threshold (threshold value) is set for the movement speed of the pointer 17. Then, for slow movement below the speed threshold, a magnification ratio suited to fine work is set. For fast movement above the speed range, a magnification ratio suited to large movements is set. This makes it possible to eliminate the above-mentioned inconvenience and achieve high usability.

[0159] 11B, the magnification changes linearly for slow movement less than the speed range and for fast movement greater than the speed range. Of course, this is not limiting, and as shown in FIG. 11C, the magnification may be changed based on several functions using the speed range as a reference.

[0160] In the example shown in Figure 1, an operation pad 16 is placed in the area to the right of the three-dimensional display 3, and the operation pad 16 serves as the operation surface, with the infrared LED 13 at the pen tip of the input device 5 being moved within the operation range, which is a range from a few centimeters to a few tens of centimeters above the operation pad 16.

[0161] User 2 operates input device 5 with his / her right hand while viewing virtual space VS displayed on display surface 9. Of course, the operation surface and operation range can be freely set to a position that is easy for user 2 to operate, and as long as the input device 5 can be photographed by stereo camera 4, this three-dimensional position input system can be constructed.

[0162] In this way, by configuring the operation surface to be any plane other than the display surface 9 on which the virtual space VS is displayed, it is possible to accommodate individual differences in the position, posture, and range that are easy for each user 2 to operate. Furthermore, even when moving the pointer 17 by multiplying the actual movement distance of the infrared LED 13 at the pen tip by a magnification factor, even if a sensing delay or error occurs, it is difficult for the user 2 to perceive it, and high usability can be maintained. This also makes it possible to reduce the device / computing costs for the stereo camera 4 and the control device 6. Furthermore, the fact that the entire screen can be operated without large hand movements due to the magnification factor is very effective in reducing fatigue felt by the user 2.

[0163] Furthermore, when user 2 moves pointer 17 while viewing virtual space VS, the direction that is easiest for operation may differ from the actual left-right direction (X direction), depth direction (Y direction), and up-down direction (Z direction).

[0164] For example, when it is desired to move the pointer 17 along the X direction, it may be easier to operate by moving the infrared LED 13 in a direction slightly tilted from the actual left-right direction (X direction).

[0165] In this three-dimensional position input system 1, the acquisition unit 18 can set a reference plane or reference direction for acquiring input position information corresponding to the input operation in the three-dimensional space (real space RS) in which the input operation is performed.

[0166] For example, as a reference plane corresponding to the horizontal direction (XY plane direction) in the virtual space VS, it is possible to set a plane along a plane direction different from the actual horizontal direction (XY plane direction) in the real space RS as the reference plane.

[0167] Furthermore, three directions different from the actual X, Y, and Z directions in the real space RS can be set as reference directions, which correspond to three mutually orthogonal reference directions corresponding to the X, Y, and Z directions in the virtual space VS. For example, a virtual plane or a virtual direction can be set as a reference plane or a reference direction.

[0168] 12 is a schematic diagram showing an example of setting a reference plane and a reference direction. For example, in an operating posture in which part of the hand (such as the ball of the little finger) is in contact with the operation pad 16, the user 2 is asked to specify points in the air corresponding to the four corners of a rectangle. The plane formed by these four points is set as a reference plane corresponding to the horizontal direction (XY plane direction) in the virtual space VS. Furthermore, reference directions corresponding to the X and Y directions in the virtual space VS are set along opposing sides. Furthermore, a direction perpendicular to the reference plane is set as a reference direction corresponding to the Z direction in the virtual space VS. The reference directions can also be called operation axes.

[0169] Alternatively, it is possible to have the user 2 specify eight vertices of a cube, and set the directions along the opposing faces of the cube as three reference directions corresponding to the X, Y, and Z directions in the virtual space VS. Any other method or calibration method that can set reference faces and reference directions that are easy for the user 2 to operate may be adopted.

[0170] By making it possible to set the reference plane and reference direction for each user 2, it becomes possible to further absorb individual differences in the position, posture, and range that are easy for each user 2 to operate. As a result, it becomes possible to realize improved operability and intuitive operation.

[0171] 1, the magnification factor integrated with the change in position may be adjusted by operating the button 14 of the input device 5. This allows the user 2 to set a magnification factor that is easy to operate in real time while using the input device 5, and to adjust the speed of the pointer 17, etc.

[0172] Furthermore, the magnification may be set to 0 by operating the button 14. When the magnification is 0, the pointer 17 does not move or rotate even if the input device 5 is moved. Therefore, while the magnification is set to 0, it is possible to move the input device 5 to a position that is easy to operate, or to set it to a posture (angle) that is easy to operate.

[0173] The user 2 may be able to switch between a three-dimensional position input mode for inputting a three-dimensional position and a two-dimensional position input mode for inputting a two-dimensional position. In the three-dimensional position input mode, a three-dimensional position is input in accordance with the movement of the infrared LED 13 at the pen tip. In the two-dimensional position input mode, a three-dimensional position is input in accordance with the movement of the infrared LED 13 at the pen tip on the operation pad 16, for example.

[0174] In the present three-dimensional position input system 1, the acquisition unit 18 switches between a three-dimensional position input mode and a two-dimensional position input mode. In the three-dimensional position input mode, the acquisition unit 18 acquires three-dimensional position information as input position information based on an input operation. In the two-dimensional position input mode, the acquisition unit 18 acquires two-dimensional position information as input position information based on an input operation.

[0175] For example, when selecting an item from a menu displayed on the display surface 9, it may be more effective to move the pointer 17 two-dimensionally along the display surface 9. In such a case, the user 2 sets the two-dimensional position input mode. This improves operability and enables high usability.

[0176] Of course, the GUI for position input (3D input GUI / 2D input GUI) may be switched and displayed depending on whether the three-dimensional position input mode or the two-dimensional position input mode is selected. Furthermore, the shape of the pointer 17 may change depending on whether the three-dimensional position input mode or the two-dimensional position input mode is selected (3D input pointer / 2D input pointer).

[0177] Any method may be adopted as a method for switching between the three-dimensional position input mode and the two-dimensional position input mode by the user 2. For example, the mode may be switched by operating the button 14 of the input device 5, by voice input, or by moving the hand 15, for example.

[0178] Furthermore, for example, when the pen tip of the input device 5 comes into contact with a predetermined physical plane such as the operation pad 16, the input mode may be switched from the three-dimensional position input mode to the two-dimensional position input mode. This allows smooth access to a 2D operation target without, for example, having to change the device's grip. Furthermore, when inputting a two-dimensional position, camera shake is less likely to occur because the pen tip is in contact with the operation pad 16. For example, by consolidating UIs such as small menu buttons into a UI for two-dimensional position input, it is possible to provide a comfortable operating experience that is less susceptible to camera shake even in situations where fine operations are required.

[0179] Furthermore, for example, a new input device may be created that is an integrated input device that combines a pen-type input device for three-dimensional position input and a mouse-type input device for two-dimensional position input.

[0180] As such a novel input device, it is possible to adopt a shape that allows a mouse-type input device to be held while holding a pen-type input device. For example, when a mouse-type input device is held while holding a pen-type input device, a groove or the like that accommodates the pen-type input device is created in the mouse-type input device. This is effective in preventing the pen-type input device from interfering with holding the mouse-type input device.

[0181] A configuration in which the relative positional relationship between the pen-type input device and the mouse-type input device can be adjusted is also effective. Furthermore, when the pen-type input device and the mouse-type input device are located at less than a certain distance, the position of the pen-type input device may be adjustable. Furthermore, a ring-shaped structure for holding the pen-type input device with a finger may be added to the pen-type input device.

[0182] As described above, in the three-dimensional position input system 1 according to this embodiment, errors due to at least one of the structure of the body and physiological phenomena of the body are corrected for input position information input by the user 2 using his or her body. This makes it possible to provide high usability for the user 2 performing an input operation of a three-dimensional position.

[0183] By applying this technology, it becomes possible to deal with distortions in the operation trajectory and positional fluctuations caused by the structure of the human hand and physiological phenomena, which are characteristics of three-dimensional input, and to realize an appropriate device structure for actual usage environments.

[0184] In other words, by applying this technology, it is possible to realize an input device for three-dimensional position information, which is a device in which user 2 inputs position information into a calculator (computer) by changing his / her position three-dimensionally, and which enables arbitrary three-dimensional position input by eliminating the influence of the trajectory that user 2 attempts to input when moving the device in three-dimensional space being distorted due to structural constraints of the hand, and the influence of not coming to a complete stop when not moving, thereby realizing an input device and a control method thereof.

[0185] Other Embodiments The present technology is not limited to the above-described embodiments, and various other embodiments can be realized.

[0186] The above describes an example of a three-dimensional position input operation using the body by the user 2, in which the user 2 inputs a three-dimensional position into a three-dimensional virtual space VS that is visible with the naked eye and that is displayed on the stereoscopic display 3. Of course, the application of the present technology is not limited to the input of a three-dimensional position into a three-dimensional virtual space VS that is visible with the naked eye, and the present technology can be applied to the input of any other three-dimensional position.

[0187] In the above, the hand 15 is taken as an example of the body of the user 2. However, the present technology is not limited to this, and can also be applied to inputting a three-dimensional position using the foot, head, or the like of the user 2.

[0188] The control device 6 may perform motion detection to detect the motion of the user 2. When a motion that generates a body-induced error is performed, the body-induced error may be corrected. For example, when a motion that draws a straight line is detected, an arc-shaped trajectory is corrected to a straight trajectory. Such processing is also possible.

[0189] FIG. 13 is a block diagram showing an example of the hardware configuration of a computer 60 that can be used as the control device 6.

[0190] The computer 60 includes a CPU 61, a ROM 62, a RAM 63, an input / output interface 65, and a bus 64 interconnecting these components. The input / output interface 65 is connected to a display unit 66, an input unit 67, a storage unit 68, a communication unit 69, a drive unit 70, and other components. The display unit 66 is a display device using, for example, an LCD or EL display. The input unit 67 is a keyboard, a pointing device, a touch panel, or other operating device. If the input unit 67 includes a touch panel, the touch panel may be integrated with the display unit 66. The storage unit 68 is a non-volatile storage device such as a HDD, flash memory, or other solid-state memory. The drive unit 70 is a device capable of driving a removable storage medium 71 such as an optical storage medium or magnetic recording tape. The communication unit 69 is a modem, router, or other communication device connectable to a LAN, WAN, or the like for communicating with other devices. The communication unit 69 may communicate via either a wired or wireless connection. The communication unit 69 is often used separately from the computer 60. Information processing by the computer 60 having the above-described hardware configuration is realized by cooperation between software stored in the storage unit 68 or the ROM 62, etc. and the hardware resources of the computer 60. Specifically, the information processing method according to the present technology is realized by loading a program constituting the software stored in the ROM 62, etc., into the RAM 63 and executing it. The program is installed in the computer 60 via, for example, the recording medium 71. Alternatively, the program may be installed in the computer 60 via a global network, etc. Alternatively, any computer-readable, non-transitory storage medium may be used.

[0191] The information processing method (three-dimensional position input method) and program according to the present technology may be executed by cooperation between multiple computers connected to each other via a network or the like, thereby constructing an information processing system or information processing device according to the present technology. In other words, the information processing method and program according to the present technology can be executed not only in a computer system composed of a single computer, but also in a computer system in which multiple computers operate in conjunction with each other. In this disclosure, a "system" refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are contained in the same housing. Therefore, both multiple devices housed in separate housings and connected via a network and a single device housed in a single housing with multiple modules are systems.

[0192] The execution of the information processing method and program according to the present technology by a computer system includes both cases where, for example, acquisition of input position information, correction of body-induced errors in the input position information, setting of a correction mode, switching between 3D position input mode and 2D position input mode, etc. are performed by a single computer, and cases where each process is performed by a different computer. Furthermore, execution of each process by a specific computer also includes having another computer execute part or all of the process and obtaining the results. In other words, the information processing method and program according to the present technology can also be applied to a cloud computing configuration in which a single function is shared and processed collaboratively by multiple devices via a network.

[0193] The configurations of the three-dimensional position input system, three-dimensional display, input device, stereo camera, control device, and pointer, as well as the processing flows for obtaining input position information, correcting body-induced errors in the input position information, setting a correction mode, and switching between three-dimensional position input mode and two-dimensional position input mode, which have been described with reference to the drawings, are merely one embodiment and can be modified as desired without departing from the spirit of the present technology. In other words, any other configurations, algorithms, etc. for implementing the present technology may be adopted.

[0194] In this disclosure, terms such as "about," "approximately," "almost," and "roughly" may be used as appropriate to facilitate understanding of the description. However, there is no clear difference between using and not using terms such as "about," "approximately," "almost," and "approximately." In other words, in this disclosure, concepts that define shape, size, positional relationship, state, etc., such as "center," "middle," "uniform," and "equal," are concepts that include "substantially center," "substantially central," "substantially uniform," and "substantially equal." For example, states that fall within a predetermined range (e.g., a range of ±10%) based on "completely centered," "completely central," "completely uniform," and "completely equal" are also included. Therefore, even if terms such as "approximately," "almost," and "approximately" are not used, concepts expressed by adding "approximately," "almost," and "approximately" may be included. Conversely, states expressed by adding terms such as "approximately," "almost," and "approximately" do not necessarily exclude perfect states.

[0195] In the present disclosure, expressions using "than", such as "greater than A" and "smaller than A", are expressions that comprehensively include both concepts that include the case where it is equivalent to A and concepts that do not include the case where it is equivalent to A. For example, "greater than A" is not limited to cases that do not include equivalent to A, but also includes "A or greater". Furthermore, "smaller than A" is not limited to "less than A" but also includes "A or less". When implementing the present technology, specific settings and the like can be appropriately adopted from the concepts included in "greater than A" and "smaller than A" so that the effects described above can be achieved.

[0196] It is also possible to combine at least two of the features of the present technology described above. That is, the various features described in each embodiment may be arbitrarily combined without distinguishing between the embodiments. Furthermore, the various effects described above are merely examples and are not intended to be limiting, and other effects may also be achieved.

[0197] The present technology can also be configured as follows. (1) An information processing system including: an acquisition unit that acquires input position information based on an input operation of a three-dimensional position using the body; and a correction unit that corrects, for the acquired input position information, an error caused by at least one of the structure of the body and a physiological phenomenon of the body. (2) The information processing system described in (1), wherein the input operation includes an operation of moving an input device for inputting the three-dimensional position in three-dimensional space, or an operation of moving a predetermined part of the body in three-dimensional space. (3) The information processing system described in (2), wherein the input operation includes an operation of moving the input device held in a hand while a part of the hand is in contact with a predetermined object, or an operation of moving a predetermined fingertip while a part of the hand is in contact with the predetermined object. (4) The information processing system described in (2) or (3), wherein the input device has a pen shape. (5) The information processing system according to any one of (2) to (4), wherein the correction unit corrects an arc-shaped trajectory acquired as the input position information to become a linear trajectory. (6) The information processing system according to any one of (2) to (5), wherein the correction unit invalidates a change in position acquired as the input position information that falls within a predetermined range. (7) The information processing system according to any one of (2) to (6), wherein the input operation is an operation of moving the input device in the three-dimensional space, the input device has a clickable button, and the correction unit invalidates a change in the position within a predetermined period based on the timing when the button of the input device is clicked. (8) The information processing system according to any one of (2) to (7), wherein the correction unit corrects a change in position acquired as the input position information to a change in position multiplied by a predetermined magnification.(9) The information processing system according to (8), wherein the correction unit corrects a change in position in a predetermined direction acquired as the input position information to a change in position obtained by multiplying a magnification corresponding to the predetermined direction. (10) The information processing system according to (8) or (9), wherein the correction unit corrects a change in position acquired as the input position information to a change in position obtained by multiplying a magnification corresponding to a speed of the change in position. (11) The information processing system according to any one of (2) to (10), wherein the acquisition unit sets a reference plane or a reference direction for acquiring the input position information corresponding to the input operation with respect to the three-dimensional space in which the input operation is performed. (12) The information processing system according to any one of (2) to (11), further comprising a setting unit for setting a mode related to correction by the correction unit, wherein the correction unit performs the correction based on the set mode, and the mode related to correction includes an execution mode in which the correction is performed and a non-execution mode in which the correction is not performed. (13) The information processing system according to (12), wherein the correction mode includes a plurality of modes that define the degree of correction. (14) The information processing system according to any one of (1) to (13), wherein the input operation is an operation of moving the input device in the three-dimensional space, the input device has a sensor that detects information about the attitude of the input device, and the acquisition unit acquires the input position information based on the detection result of the sensor. (15) The information processing system according to any one of (1) to (14), wherein the acquisition unit switches between a three-dimensional position input mode in which three-dimensional position information is acquired as the input position information based on the input operation and a two-dimensional position input mode in which two-dimensional position information is acquired as the input position information based on the input operation.(16) The information processing system according to any one of (1) to (15), wherein the input operation is an input operation of the three-dimensional position in a three-dimensional virtual space visible to the naked eye and displayed on a stereoscopic display. (17) An information processing method in which a computer system executes the following: acquiring input position information based on an input operation of a three-dimensional position using the body, and correcting, for the acquired input position information, an error caused by at least one of the structure of the body and a physiological phenomenon of the body. (18) A program in which a computer system executes the following: acquiring input position information based on an input operation of a three-dimensional position using the body, and correcting, for the acquired input position information, an error caused by at least one of the structure of the body and a physiological phenomenon of the body.

[0198] RS...Real space VS...Virtual space 1...3D position input system 2...User 3...3D display 4...Stereo camera 5...Input device 6...Control device 12...Virtual object 13...Infrared LED 14...Button 15...Hand 16...Operation pad 17...Pointer 60...Computer

Claims

1. An information processing system comprising: an acquisition unit that acquires input position information based on a three-dimensional position input operation using the body; and a correction unit that corrects errors in the acquired input position information that are caused by at least one of the structure of the body and physiological phenomena of the body.

2. An information processing system according to claim 1, wherein the input operation includes an operation of moving an input device for inputting the three-dimensional position in three-dimensional space, or an operation of moving a specified part of the body in three-dimensional space.

3. An information processing system according to claim 2, wherein the input operation includes an operation of moving the input device held by a hand while a part of the hand is in contact with a specified object, or an operation of moving a specified fingertip while a part of the hand is in contact with the specified object.

4. An information processing system according to claim 2, wherein the input device has a pen shape.

5. An information processing system according to claim 2, wherein the correction unit corrects an arc-shaped trajectory acquired as the input position information so that it becomes a linear trajectory.

6. An information processing system according to claim 2, wherein the correction unit invalidates position changes that fall within a predetermined range acquired as the input position information.

7. An information processing system as described in claim 2, wherein the input operation is an operation of moving the input device in the three-dimensional space, the input device has a clickable button, and the correction unit invalidates changes in the position during a predetermined period based on the timing when the button on the input device is clicked.

8. An information processing system according to claim 2, wherein the correction unit corrects the change in position acquired as the input position information to a change in position multiplied by a predetermined magnification.

9. An information processing system according to claim 8, wherein the correction unit corrects a change in position in a predetermined direction acquired as the input position information to a change in position obtained by multiplying a magnification corresponding to the predetermined direction.

10. An information processing system according to claim 8, wherein the correction unit corrects the change in position acquired as the input position information to a change in position multiplied by a magnification corresponding to the speed of the change in position.

11. An information processing system according to claim 2, wherein the acquisition unit sets a reference plane or reference direction for acquiring the input position information corresponding to the input operation with respect to the three-dimensional space in which the input operation is performed.

12. An information processing system as described in claim 2, further comprising a setting unit that sets a mode related to correction by the correction unit, the correction unit executes the correction based on the set mode, and the mode related to correction includes an execution mode in which the correction is executed and a non-execution mode in which the correction is not executed.

13. An information processing system according to claim 12, wherein the correction mode includes a plurality of modes that define the degree of the correction.

14. An information processing system as described in claim 1, wherein the input operation is an operation of moving the input device in the three-dimensional space, the input device has a sensor that detects information regarding the orientation of the input device, and the acquisition unit acquires the input position information based on the detection result of the sensor.

15. An information processing system according to claim 1, wherein the acquisition unit switches between a three-dimensional position input mode in which three-dimensional position information is acquired as the input position information based on the input operation, and a two-dimensional position input mode in which two-dimensional position information is acquired as the input position information based on the input operation.

16. An information processing system according to claim 1, wherein the input operation is an input operation of the three-dimensional position relative to a three-dimensional virtual space visible to the naked eye and displayed on a stereoscopic display.

17. An information processing method in which a computer system acquires input position information based on a three-dimensional position input operation using the body, and corrects errors in the acquired input position information that are caused by at least one of the structure of the body and physiological phenomena of the body.

18. A program that causes a computer system to acquire input position information based on input operations of three-dimensional positions using the body, and correct errors in the acquired input position information that are caused by at least one of the structure of the body and physiological phenomena of the body.

Images