Operating device, information processing device, information processing system, information processing method, and information processing program

Abstract

An operating device (10) receives an operation by a user and comprises: a detection unit (21) that detects an operation; an acquisition unit (22) that acquires a first image showing the user's fingers; a characteristic-information generation unit (231) that generates characteristic information about the fingers on the basis of the first image; and a characteristic-information output unit (12) that outputs detection results by the detection unit and the characteristic information.

Description

Operation device, information processing device, information processing system, information processing method, and information processing program

[0001] The present invention relates to an operation device, an information processing device, an information processing system, an information processing method, and an information processing program.

[0002] Conventionally, there is known a technology for tracking the movements of a user's fingers for use in virtual reality (VR) and the like (see, for example, Patent Document 1). In the invention described in Patent Document 1, the positions of the user's fingertips are tracked using a glove equipped with sensors at positions corresponding to the user's fingertips. Information indicating the positions tracked by the glove is converted into various control signals and used for various purposes, such as generating and displaying an image that reproduces the user's fingers in a virtual space.

[0003] US Patent Application Publication No. 2016 / 024369

[0004] When generating the above-mentioned reproduced image in the virtual space, if there is a large visual difference between the actual user's fingers and the reproduced image, the user may feel uncomfortable. Therefore, there is a need to provide a better experience to the user by improving the realism and immersion of the experience by alleviating this sense of discomfort.

[0005] An operating device according to a first aspect of the present invention is an operating device that accepts an operation by a user and includes a detection unit that detects the operation, an acquisition unit that acquires a first image of the user's fingers, a feature information generation unit that generates feature information about the fingers based on the first image, and a feature information output unit that outputs the detection result by the detection unit and the feature information.An information processing device according to a second aspect of the present invention includes an operating device that has a detection unit that detects an operation by the user and an acquisition unit that acquires the detection result by the operating device and the first image of the user's fingers, a feature information generation unit that generates feature information about the fingers based on the first image, an image generation unit that generates a second image that reproduces the shape of the fingers based on the detection result by the detection unit and the feature information, and a display unit that displays the second image. An information processing device according to a third aspect of the present invention includes an acquisition unit that acquires a detection result from an operation device having a detection unit that detects an operation by a user and a first image capturing an image of the user's fingers, a feature information generation unit that generates feature information about the fingers based on the first image, an operation information generation unit that generates operation information for operating an operation target based on the detection result from the detection unit, an operation information adjustment unit that adjusts the operation information based on the feature information, and an operation information output unit that outputs the adjusted operation information to the operation target.An information processing method according to a fourth aspect of the present invention is an information processing method performed by an operation device that has a detection unit and accepts operations by a user, the information processing method including: an acquisition step of acquiring a first image capturing an image of the user's fingers, a feature information generation step of generating feature information about the fingers based on the first image, and a feature information output step of outputting the detection result from the detection unit and the feature information. An information processing program according to a fifth aspect of the present invention is an information processing program executed by an information terminal that has a detection unit and is connected to an operation device that accepts operations by a user, and causes the information terminal to execute an acquisition step of acquiring a first image of the user's fingers, a feature information generation step of generating feature information about the fingers based on the first image, and a feature information output step of outputting the detection result by the detection unit and the feature information.

[0006] 1 is a schematic diagram showing a general configuration of an information processing system according to a first embodiment; a perspective view showing a general configuration of an operation device according to the first embodiment; a diagram showing an example of a reference image; a diagram explaining feature information; a diagram showing an example of a model image; a diagram explaining adjustment of a 3D model based on feature information; a flowchart showing an example of a processing method according to the first embodiment; a schematic diagram showing a general configuration of an information processing system according to a second embodiment; a flowchart showing an example of a processing method according to the second embodiment;

[0007] First Embodiment A first embodiment of the present invention will now be described with reference to the drawings. [General Configuration of Information Processing System] Fig. 1 is a schematic diagram showing the general configuration of an information processing system according to the first embodiment. As shown in Fig. 1, the information processing system 1 according to the first embodiment includes an operation device 10, an information processing device 20, and an HMD unit 30. In the information processing system 1, a model image (second image) that reproduces the user's fingers in a virtual space is generated by the information processing device 20 in response to an operation on the operation device 10, and the model image (second image) is displayed on the HMD unit 30.

[0008] [Configuration of Operation Device] The operation device 10 includes a detection unit 11 and an output unit 12, and receives operations from a user. The detection unit 11 detects the position of the user's fingers (hand and fingers). The output unit 12 outputs the detection result by the detection unit 11 to the outside of the operation device 10 via a wireless or wired connection.

[0009] Fig. 2 is a perspective view that schematically shows the operating device 10. As shown in Fig. 2, the operating device 10 is worn on the user's fingers by a glove-shaped wearing member. The wearing member is preferably made of a stretchable material that can fit snugly against the user's hand.

[0010] The operation device 10 includes, as the detection unit 11, a plurality of sensors 111a, 111b, 111c, 111d, and 111e provided corresponding to each finger, as shown in FIG. 2, for example, and a sensor 112 provided corresponding to a position other than the fingers on the user's hand.

[0011] The sensors 111a, 111b, 111c, 111d, and 111e are provided at positions corresponding to the fingertips of the respective fingers and on the nail side of the respective fingers. Specifically, for example, as shown in FIG. 2 , the sensor 111a is provided corresponding to the first finger (thumb), and the position of the sensor 111a is detected as the position of the thumb tip. Similarly, the sensors 111b, 111c, 111d, and 111e are provided corresponding to the second finger (index finger), third finger (middle finger), fourth finger (ring finger), and fifth finger (pinky finger), respectively, and the position of each sensor is detected as the position of the fingertip of each finger. The sensor 112 is provided at a position corresponding to the back of the hand, and the position of the sensor 112 is detected as the position of the back of the hand.

[0012] In this embodiment, any known sensors may be used for the sensors 111a, 111b, 111c, 111d, and 111e, and the sensor 112. The positions of the sensors 111a, 111b, 111c, 111d, and 111e, and the sensor 112, are not limited to the example shown in FIG. 2 . For example, the sensors 111a, 111b, 111c, 111d, and 111e may be provided on the pads of the fingers, on the tips of the fingers, or so as to cover part or all of the fingers. In this embodiment, the output unit 12 is provided in a housing 121, as shown in FIG. 2 , which is provided at a position corresponding to the user's wrist. If the operating device 10 has a control unit that controls each unit, the control unit may also be provided in the housing 121. The housing 121 may be provided integrally or independently at the same position as the sensor 112, which is provided at a position corresponding to the back of the hand, or may be provided at another position.

[0013] 1, the information processing device 20 includes a communication unit 21 and a control unit 22. The communication unit 21 receives the detection result by the detection unit 11 output from the output unit 12 of the operation device 10, and is connected to the communication unit 32 of the HMD unit 30 wirelessly or via a wired connection.

[0014] The control unit 22 includes the functions of a feature information generation unit 221 and an image generation unit 222, which are realized by the processor operating in accordance with a program stored in the memory or received via a communication interface. The functions of each unit will be described later.

[0015] [Configuration of HMD Unit] The HMD unit 30 can be used by being worn on the user's head via a mounting unit (not shown). The HMD unit 30 also functions as a display device and an imaging device that captures images from the user's first-person perspective, i.e., from an inside-out perspective. The HMD unit 30 may be configured to be worn on the user's head via a mounting unit such as a band, or may have any other configuration, such as a helmet or glasses. When wearing the HMD unit 30, it is recommended that a tutorial or the like be displayed on the display unit of the HMD unit 30 to guide the user to wear the HMD unit 30 properly.

[0016] As shown in FIG. 1 , the HMD unit 30 includes an imaging unit 31, a communication unit 32, a control unit 33, and a display unit 34. The imaging unit 31 has an imaging element (not shown) and captures an image from a user's first-person perspective to generate an image. The imaging element may be an image sensor that generates an image including color information, or a depth sensor that generates an image including depth information. The imaging unit 31 may also be equipped with multiple imaging elements. For example, the imaging unit 31 may be equipped with both an image sensor and a depth sensor to generate an image including color information and an image including depth information, respectively. Alternatively, for example, the imaging unit 31 may be equipped with multiple image sensors to generate an image including color information and generate depth information from the multiple images. The imaging unit 31 captures a reference image (first image) of the user's fingers. The reference image is information used to generate a model image that reproduces the user's fingers in a virtual space. The communication unit 32 is connected to the communication unit 21 of the information processing device 20 wirelessly or via a wired connection. The control unit 33 controls each unit in the HMD unit 30 .

[0017] The display unit 34 includes a display element such as an LCD (Liquid Crystal Display) or an organic EL display, and an optical device such as a lens, and is capable of displaying an image to the user. The display element of the display unit 34 may be a transmissive display element or a non-transmissive display element. A terminal device such as a smartphone that can be attached to or detached from the housing of the HMD unit 30 may also be used as the display device. Furthermore, a wearable device such as AR (Augmented Reality) glasses or MR (Mixed Reality) glasses may also be used as the HMD unit 30.

[0018] Next, the functions of each unit in the control unit 22 of the information processing device 20 will be described. The feature information generation unit 221 generates feature information about fingers based on a reference image. Like the reference image described above, the feature information is information used to generate a model image and includes at least one of the following two types: (1) Position Information of Feature Points on Fingers The feature information generation unit 221 calculates position information of feature points, such as joint information on the user's fingers, from the reference image using, for example, a machine learning technique. Examples of feature points include the endpoints of each finger, each finger joint, and the wrist joint. A trained model used in machine learning can be constructed in advance by performing supervised learning using, for example, images of a person's fingers having multiple feature points as input data and coordinate information indicating the positions of the feature points on the person's fingers as correct answer data. Note that a detailed description of the specific machine learning technique will be omitted, as various known techniques can be used. The feature information generation unit 221 may also be configured to include a relationship learning unit, which learns the relationship between an image based on the input reference image and coordinate information indicating the positions of feature points on a person's fingers each time a reference image is generated, and updates the trained model. FIG. 3B illustrates an example of feature points based on a reference image. In the example of FIG. 3B , the end points of each finger, each finger joint, and the wrist joint are detected as feature points, and the position information of each feature point is calculated as feature information. By generating position information of the feature points on the fingers as feature information and determining the relative positional relationships of these feature points, the actual length of the user's fingers, the proportions of each part of the fingers, and the like can be accurately determined.

[0019] The feature information generating unit 221 may extract silhouettes of the user's fingers using a method such as skin color detection for a reference image including color information, and generate position information of feature points on the fingers based on the extracted silhouette. The feature information generating unit 221 may also generate position information of feature points on the fingers by estimating thickness around the finger joints based on depth information for a reference image including depth information.

[0020] (2) Texture Information of Finger Surfaces The feature information generation unit 221 calculates texture information indicating the surface condition of the user's fingers from the reference image using, for example, a method such as skin color detection. The texture information includes the color and texture of the user's skin, as well as irregularities on the finger surface such as scratches, moles, and wrinkles.

[0021] The image generation unit 222 generates a model image (second image) that reproduces the user's fingers in the virtual space based on the detection results received from the operation device 10 via the communication unit 21. An example of a model image is shown in FIG. 4A . The image generation unit 222 controls a 3D model of the fingers based on the detection results received from the operation device 10, adjusts the 3D model of the fingers based on the feature information generated by the feature information generation unit 221, and generates a model image based on the adjusted 3D model. More specifically, the image generation unit 222 uses, for example, known inverse kinetics (hereinafter referred to as "IK") technology to specify a start point (IK root), an end point (IK end), and a target point (IK goal) for each joint, thereby performing posture correction such as bending and stretching movements to control a 3D model of the skeleton (bones) of the fingers. In IK technology, when the tip of a joint structure is moved, correction calculations are performed by tracing the rotation of the joint in the reverse direction (parent direction). The image generating unit 222 then performs skinning processing to set the range of influence of the skeleton on the 3D model, thereby generating a model image.

[0022] When moving the intermediate joints of the fingers in a model image using IK technology, it is desirable that the skeleton of the 3D model used to generate the model image match the skeleton of the fingers based on the detection results received from the operation device 10. For example, if the hand of the 3D model is larger than the hand of the user operating the operation device 10, the sensors (sensors 111a, 111b, 111c, 111d, and 111e) will be recessed into the fingertips in the 3D model. Furthermore, if the hand of the 3D model is smaller than the hand of the user operating the operation device 10, the sensors (sensors 111a, 111b, 111c, 111d, and 111e) will be located at positions away from the fingertips. In such cases, because the skeleton of the 3D model used to generate the model image does not match the skeleton of the fingers based on the detection results received from the operation device 10, the bending of the intermediate joints will not be accurately reproduced in the model image generated based on the 3D model, resulting in unnatural movement and a sense of discomfort to the user.

[0023] Therefore, in order to eliminate or reduce the above-mentioned problems, the image generation unit 222 adjusts the 3D model as necessary based on the feature information generated by the feature information generation unit 221, and then generates a model image. The model image is generated according to the type of feature information. (1) Position Information of Feature Points on Fingers When the feature information is position information of feature points on fingers, the image generation unit 222 adjusts the length of the fingers in the 3D model and the proportions of each part of the fingers based on the feature information. More specifically, the image generation unit 222 adjusts the 3D model by scaling the fingers in the 3D model, lengthening or shortening the length of the fingers in the 3D model based on the feature information, or changing the proportions of each part of the fingers in the 3D model based on the feature information. Then, the image generation unit 222 generates a model image by performing a skinning process on the adjusted 3D model.

[0024] (2) Texture Information of the Surface of Fingers When the feature information is texture information of the surface of fingers, the image generation unit 222 uses the texture of the image of the finger part of the reference image as the texture for the part of the fingers in the model image, or generates a model image by reflecting the color and texture of the user's skin and surface irregularities of the fingers such as scratches, moles, and wrinkles based on the feature information. Note that when images of both the palm side and the back side of the user's fingers are captured as the reference image, the images of both sides are also adjusted in the model image.

[0025] FIG. 4B illustrates an example of a model image generated based on feature information. In the example of FIG. 4B, the balance of the lengths of the fingers in the 3D model, the thickness of each finger, and the like are adjusted, resulting in a model image that more closely resembles the user's actual fingers. Note that, when the feature information includes both (1) and (2) above, the image generation unit 222 may perform the image generation processes described in (1) and (2) in combination, or may perform only one of the image generation processes. Furthermore, the content of the image generation process may be set by the user.

[0026] [Information Processing Flow] Figure 5 is a flowchart showing the processing executed by the control unit 22. The control unit 22 determines whether a reference image has been acquired by the imaging unit 31 (step S101). If it is determined that a reference image has been acquired (step S101: YES), the feature information generation unit 221 generates feature information based on the reference image (step S102). The control unit 22 then determines whether a detection result by the operation device 10 has been acquired via the communication unit 21 (step S103). If it is determined that a detection result has been acquired (step S103: YES), the image generation unit 232 adjusts the 3D model image according to the feature information generated in step S102 (step S104) and generates a model image (step S105). Furthermore, the communication unit 21 outputs the generated model image to the HMD unit 30 (step S106), and the process returns to step S103. That is, once feature information is generated by the processes of steps S101 and S102, the processes of steps S103 to S106 are repeatedly executed thereafter every time a detection result is obtained by the operation device 10. Therefore, a model image is output to the HMD unit 30 every time a detection result is obtained by the operation device 10, and the model image displayed on the display unit 24 is sequentially updated, and at that time, a model image based on the reference image is always generated.

[0027] Effects of First Embodiment The information processing system 1 according to the first embodiment described above has the following effects: The information processing device 20 includes an acquisition unit (communication unit 21) that acquires a detection result from the operation device 10, which includes a detection unit 11 that detects an operation by a user, and a reference image that is a first image capturing an image of the user's fingers, a feature information generation unit 221 that generates feature information about the fingers based on the reference image, and an image generation unit 232 that generates a model image that is a second image that reproduces the shape of the fingers, based on the detection result from the detection unit 11 and the feature information.

[0028] With this configuration, by accurately scaling and corresponding to the shape of the actual user's fingers, it is possible to reduce the difference between the actual user's fingers and the user's fingers reproduced in the model image, thereby improving the sense of discomfort felt by the user. As a result, it is possible to improve the sense of realism and immersion and provide a better experience to the user.

[0029] The image generation unit 222 controls the 3D model of the fingers based on the detection results, adjusts the 3D model of the fingers based on the feature information, and generates a model image based on the adjusted 3D model. With this configuration, the skeleton of the 3D model from which the model image is generated is matched with the skeleton of the fingers based on the detection results received from the operation device 10, so that the movement of the fingers can be accurately reproduced in the model image.

[0030] The feature information generated by the feature information generating unit 221 includes position information of feature points on the fingers. With this configuration, the model image is adjusted based on the position information of feature points on the fingers, thereby accurately reproducing the shape of the actual user's fingers. Therefore, defects that occur in the model image can be eliminated or reduced, and the actual user's fingers and their movements can be accurately reproduced in the model image.

[0031] The feature information generated by the feature information generating unit 221 includes texture information of the surface of the fingers. With this configuration, the model image is adjusted based on the texture information of the surface of the fingers, thereby making it possible to favorably reproduce the atmosphere and appearance of the actual user's fingers. This is expected to improve realism and immersion.

[0032] In the HMD unit 30, which is an information processing device, the reference image captured by the imaging unit 31 is an image that includes at least one of color information and depth information. With this configuration, feature information required for adjusting a model image can be easily generated from an easily obtainable reference image.

[0033] Second Embodiment A second embodiment of the present invention will be described below with reference to the drawings. In the second embodiment, only differences from the first embodiment will be described, and descriptions of similar parts to the first embodiment will be omitted. Furthermore, in the second embodiment, components having substantially the same functional configuration as those in the first embodiment will be denoted by the same reference numerals. FIG. 6 is a schematic diagram illustrating an entire information processing system according to a second embodiment of the present invention. As shown in FIG. 6, the information processing system 2 according to the second embodiment includes an imaging device 40, an information processing device 50, and an operation target device 60, instead of the information processing device 20 and HMD unit 30 of the information processing system 1 of the first embodiment. The information processing system 2 generates operation information in the information processing device 50 in response to an operation on the operation device 10, and controls the operation target device 60 based on the operation information.

[0034] [Configuration of Imaging Device 40] As shown in FIG. 6 , the imaging device 40 includes an imaging unit 41 and an output unit 42. The imaging unit 41 has an imaging element (not shown) and captures an image from a user's third-person perspective, i.e., from an outside-in perspective, to generate an image. Details of the imaging element and the number of imaging elements are similar to those of the imaging unit 31 of the HMD unit 30 in the information processing system 1 of the first embodiment, and therefore will not be described here. Similarly to the imaging unit 31 of the HMD unit 30 in the information processing system 1 of the first embodiment, the imaging unit 41 captures a reference image (first image) of the user's fingers. The output unit 42 outputs the reference image captured by the imaging unit 41 to the outside of the imaging device 40 via wireless or wired connections.

[0035] [Configuration of Information Processing Device 50] As shown in Fig. 6, the information processing device 50 includes a communication unit 51 and a control unit 52. The communication unit 51 receives the detection result by the detection unit 11 output from the output unit 12 of the operation device 10 and the reference image output from the output unit 42 of the imaging device 40. Furthermore, the communication unit 51 outputs operation information generated by the control unit 52 to the outside of the information processing device 50 via wireless or wired communication. Details of the operation information will be described later.

[0036] The control unit 52 includes the functions of a feature information generation unit 521, an operation information generation unit 522, and an operation information adjustment unit 523, which are realized by the processor operating in accordance with a program stored in the memory or received via the communication interface. The functions of each unit will be further described below.

[0037] The feature information generating unit 521 generates feature information about the fingers based on the reference image. The feature information includes position information of feature points on the fingers. Similar to the feature information generating unit 221 in the information processing device 20 of the first embodiment, the feature information generating unit 521 calculates position information of feature points on the user's fingers from the reference image.

[0038] The operation information generation unit 522 generates operation information for operating the operation target device 60 based on the detection results received from the operation device 10 via the communication unit 51. The operation information may be the detection results themselves, or may be control information indicating the drive locations and drive amounts of the operation target device 60 calculated based on the detection results. Here, if there is a large difference between the actual user's fingers and the corresponding locations on the operation target device 60, malfunctions may occur in the operation of the operation target device 60. For example, if the size of the corresponding locations on the operation target device 60 is significantly different from the size of the actual user's fingers, the operation of the corresponding locations on the operation target device 60 may not be correctly reproduced. In other words, if there is no difference, or if the difference is sufficiently small, between the actual user's fingers and the corresponding locations on the operation target device 60, the movement of the actual user's fingers is correctly reproduced on the operation target device 60.

[0039] In order to eliminate or reduce the above-mentioned problems, the operation information adjustment unit 523 adjusts the operation information generated by the operation information generation unit 522 based on the feature information generated by the feature information generation unit 521. A specific method of adjustment will be described later.

[0040] [Configuration of Operation Target Device 60] The operation target device 60 is a device that is operated in response to an operation on the operation device 10. Examples of the operation target device 60 include a robot hand and a manipulator. As shown in FIG. 6 , the operation target device 60 includes a communication unit 61 and a drive unit 62. The communication unit 61 receives operation information output from the communication unit 51 of the information processing device 50. The drive unit 62 includes a plurality of motors and actuators, such as servo motors, depending on the shape of the robot hand and manipulator, and drives at least one of them. At this time, the drive unit 62 drives based on the operation information (e.g., a control signal for operating a servo motor) received via the communication unit 61.

[0041] It should be noted that the installation positions of the motors and actuators in the drive unit 62 may be fixed or movable. For example, if the operation target device 60 is a robot hand, the motors and actuators of the drive unit 62 installed in the joints of the robot hand may be fixed or movable. In either case, information indicating the installation positions of the motors and actuators in the drive unit 62 of the operation target device 60 is stored in advance in the control unit 52, or is configured to be obtainable from the operation target device 60 via the communication unit 51. The same applies when the operation target device 60 is a manipulator.

[0042] [Adjustment of Operation Information] When the installation positions of the motor and actuator in the drive unit 62 of the operation target device 60 are fixed, the operation information adjustment unit 523 adjusts the operation information based on the feature information. More specifically, the operation information adjustment unit 523 first finds a correlation between a feature point in the feature information and the installation position of the drive unit 62. Then, the operation information adjustment unit 523 adjusts the operation information in accordance with the correlation. In other words, by adjusting the operation information to the installation position of the drive unit 62 in the operation target device 60, the difference between the actual user's fingers and the installation positions of the motor and actuator in the drive unit 62 of the operation target device 60 is reduced.

[0043] On the other hand, when the installation positions of the motor and actuator in the drive unit 62 of the operation target device 60 are movable, the operation information adjustment unit 523 adjusts the operation information by adding to the operation information adjustment information for enlarging or reducing the installation positions of the motor and actuator based on the feature information, adjustment information for lengthening or shortening the length between the installation positions of the motor and actuator based on the feature information, adjustment information for changing the installation positions of the motor and actuator based on the feature information, etc. In other words, by matching the installation positions of the motor and actuator in the drive unit 62 of the operation target device 60 to the shape of the actual user's fingers, the difference between the installation positions of the actual user's fingers and the installation positions of the motor and actuator in the drive unit 62 of the operation target device 60 is reduced.

[0044] As described above, the information processing device 50 receives the detection result from the operation device 10 and also receives the reference image from the imaging device 40. Then, the information processing device 50 calculates feature information based on the reference image, generates operation information based on the detection result, and adjusts the operation information based on the feature information.

[0045] [Information Processing Flow] FIG. 7 is a flowchart showing processing executed by the control unit 52 of the information processing device 50. The control unit 52 determines whether a reference image has been acquired via the communication unit 51 (step S201). If it is determined that a reference image has been acquired (step S201: YES), the feature information generation unit 521 generates feature information based on the reference image (step S202). The control unit 52 then determines whether a detection result by the operation device 10 has been acquired via the communication unit 51 (step S203). If it is determined that the detection result has been acquired (step S203: YES), the operation information generation unit 522 generates operation information (step S204), and the operation information adjustment unit 523 adjusts the operation information based on the feature information generated in step S202 (step S205). Furthermore, the communication unit 51 outputs the adjusted operation information (step S206), and the process returns to step S203. That is, once the feature information is generated by the processes of steps S201 and S202, the processes of steps S203 to S206 are repeatedly executed thereafter every time a detection result is obtained by the operation device 10. Therefore, every time a detection result is obtained by the operation device 10, operation information is output, and at that time, the operation information is always adjusted based on the reference image.

[0046] Effect of Second Embodiment The information processing system 2 according to the second embodiment described above has the following effect. The information processing device 50 includes an acquisition unit (communication unit 51) that acquires a detection result from the operation device 10, which includes a detection unit 11 that detects an operation by a user, and a reference image that is a first image capturing the user's fingers, a feature information generation unit 521 that generates feature information about the fingers based on the reference image, an operation information generation unit 522 that generates operation information for operating the operation target device 60, which is an operation target, based on the detection result from the detection unit 11, and an operation information adjustment unit 523 that adjusts the operation information based on the feature information. The adjusted operation information is output to the operation target device 60 via the communication unit 51.

[0047] With this configuration, the operation information is adjusted based on the position information of the feature points of the fingers, so that the actual movements of the user's fingers can be accurately reproduced on the operation target. This makes it possible to eliminate or reduce problems that occur in the operation of the operation target. In particular, it is possible to absorb individual differences in the shape of the fingers that differ from user to user.

[0048] [Modifications of the Embodiments] The present invention is not limited to the above-described embodiments, and modifications and improvements within the scope of achieving the object of the present invention are included in the present invention.

[0049] In the first embodiment, the operation device 10 is attached to the user's fingers by a glove-shaped attachment member, and the position of the user's fingers (hand and fingers) is detected by the detection unit 11. However, the present invention is not limited to this, and can be similarly applied to an operation device 10 such as a stick-shaped operation device that the user holds with their fingers.

[0050] In the first embodiment described above, the information processing device 20 and the HMD unit 30 are used as examples of information processing devices. However, the present invention is not limited to this, and can be similarly applied to other information processing devices that include an internal or external imaging device, as in the second embodiment.

[0051] A configuration may be adopted in which some of the processes performed by the HMD unit 30, which is an information processing device in the first embodiment, are performed by the operation device 10. For example, a configuration may be adopted in which some or all of the functions of the feature information generating unit 221 and the image generating unit 222 in the HMD unit 30 in the first embodiment are performed by the operation device 10. Furthermore, the control unit 22 of the information processing device 20 may be provided in the HMD unit 30.

[0052] A configuration may be adopted in which part of the processing performed by the information processing device 50 in the second embodiment is performed by the operation device 10, or a configuration may be adopted in which part of the processing is performed by the operation target device 60. For example, a configuration may be adopted in which part or all of the functions of the feature information generation unit 521, the operation information generation unit 522, and the operation information adjustment unit 523 in the information processing device 50 of the second embodiment are performed by the operation device 10. Furthermore, for example, a configuration may be adopted in which part or all of the functions of the feature information generation unit 521, the operation information generation unit 522, and the operation information adjustment unit 523 in the information processing device 50 of the second embodiment are performed by the operation target device 60.

[0053] In the above-described embodiments, the operating device 10 is attached to the user's fingers by a glove-shaped attachment member. However, the present invention is not limited to this, and the attachment member may be in the form of a band. Furthermore, the operating device 10 may be a so-called exoskeleton controller.

[0054] [Summary of the Invention] The summary of the invention is as follows. [1] An operation device that accepts an operation by a user includes a detection unit that detects the operation, an acquisition unit that acquires a first image of the user's fingers, a feature information generation unit that generates feature information about the fingers based on the first image, and a feature information output unit that outputs the detection result by the detection unit and the feature information. [2] In the operation device described in [1], the feature information includes position information of feature points on the fingers. [3] In the operation device described in [1] or [2], the feature information includes texture information on the surface of the fingers. [4] In the operation device described in any of [1] to [3], the detection unit detects the position of the fingers. [5] In the operation device described in any of [1] to [4], the first image is an image that includes at least one of color information and depth information. [6] An information processing device comprising: an acquisition unit that acquires a detection result from an operation device having a detection unit that detects a user's operation and a first image that captures the user's fingers, a feature information generation unit that generates feature information about the fingers based on the first image, an image generation unit that generates a second image that reproduces the shape of the fingers based on the detection result from the detection unit and the feature information, and a display unit that displays the second image. [7] In the information processing device described in [6], the image generation unit controls a 3D model of the fingers based on the detection result, adjusts the 3D model of the fingers based on the feature information, and generates the second image based on the adjusted 3D model. [8] An information processing device comprising: an acquisition unit that acquires a detection result from an operation device having a detection unit that detects a user's operation and a first image that captures the user's fingers, a feature information generation unit that generates feature information about the fingers based on the first image, an operation information generation unit that generates operation information for operating an operation target based on the detection result from the detection unit, an operation information adjustment unit that adjusts the operation information based on the feature information, and an operation information output unit that outputs the adjusted operation information to the operation target. [9] An information processing system comprising an operating device having a detection unit that detects an operation by a user, and the information processing device according to [6] or [8].

[10] In the information processing system according to [9], the information processing device further comprises an imaging unit that has an imaging element and captures a first image.

[11] In the information processing system described in

[10] , the information processing device further includes a mounting unit that allows at least the image sensor to be mounted on a user's body.

[12] An information processing system including an operation device including a detection unit that detects an operation by a user, the information processing device described in [8], and an operation target that operates based on operation information derived from the detection result.

[13] An information processing method implemented by an operation device that includes a detection unit and accepts an operation by a user, the information processing method including: an acquisition step of acquiring a first image of the user's fingers; a feature information generation step of generating feature information related to the fingers based on the first image; and a feature information output step of outputting the detection result by the detection unit and the feature information.

[14] An information processing program executed by an information terminal that includes a detection unit and is connected to an operation device that accepts an operation by a user, the information processing program causing the information terminal to execute an acquisition step of acquiring a first image of the user's fingers, a feature information generation step of generating feature information related to the fingers based on the first image, and a feature information output step of outputting the detection result by the detection unit and the feature information.

[0055] 1, 2...information processing system, 10...operation device, 11...detection unit, 12, 42...output unit, 31, 41...imaging unit, 21, 32, 51, 61...communication unit, 22, 33, 52...control unit, 30...HMD unit, 34...display unit, 40, 50...information processing device, 60...operation target device, 62...drive unit, 111a to 111e, 112...sensor, 221, 521...feature information generation unit, 222...image generation unit, 522...operation information generation unit, 523...operation information adjustment unit.

Claims

1. An operation device that accepts an operation by a user, A detection unit that detects the operation; An acquisition unit that acquires a first image of the user's fingers; a feature information generating unit that generates feature information regarding the hand and fingers based on the first image; a feature information output unit that outputs a detection result by the detection unit and the feature information; An operating device comprising:

2. 2. The operating device according to claim 1, The operating device, characterized in that the feature information includes position information of feature points on the fingers.

3. 3. The operating device according to claim 1, The operating device, characterized in that the feature information includes texture information of the surface of the fingers.

4. 2. The operating device according to claim 1, The detection unit detects the position of the finger. An operating device characterized by:

5. 2. The operating device according to claim 1, An operating device, characterized in that the first image is an image including at least one of color information and depth information.

6. an acquisition unit that acquires a detection result by an operation device including a detection unit that detects an operation by a user and a first image capturing an image of the user's fingers; a feature information generating unit that generates feature information regarding the hand and fingers based on the first image; an image generating unit that generates a second image reproducing the shape of the hand and fingers based on a detection result by the detecting unit and the feature information; a display unit for displaying the second image; An information processing device comprising:

7. 7. The information processing device according to claim 6, The image generation unit controls the 3D model of the hand based on the detection result, adjusts the 3D model of the hand based on the feature information, and generates the second image based on the adjusted 3D model.

23. An information processing apparatus comprising:

8. an acquisition unit that acquires a detection result by an operation device including a detection unit that detects an operation by a user and a first image capturing an image of the user's fingers; a feature information generating unit that generates feature information regarding the hand and fingers based on the first image; an operation information generating unit that generates operation information for operating an operation target based on a detection result by the detecting unit; an operation information adjustment unit that adjusts the operation information based on the feature information; an operation information output unit that outputs the adjusted operation information to the operation target; An information processing device comprising:

9. an operation device including a detection unit that detects an operation by a user; The information processing device according to claim 6 or 8. An information processing system comprising:

10. 10. The information processing system according to claim 9, The information processing device further includes an imaging unit having an image sensor and configured to capture the first image. An information processing system comprising:

11. 11. The information processing system according to claim 10, The information processing device further includes a mounting unit capable of mounting at least the imaging element on a body of the user. An information processing system comprising:

12. an operation device including a detection unit that detects an operation by a user; The information processing device according to claim 8 . an operation target that operates based on operation information based on the detection result; An information processing system comprising:

13. An information processing method implemented by an operation device that includes a detection unit and accepts an operation by a user, comprising: An acquisition step of acquiring a first image of the user's finger; a feature information generating step of generating feature information regarding the hand and fingers based on the first image; a feature information output step of outputting a detection result by the detection unit and the feature information; 13. An information processing method comprising:

14. An information processing program executed by an information terminal connected to an operation device that includes a detection unit and accepts operations by a user, In the information terminal, An acquisition step of acquiring a first image of the user's finger; a feature information generating step of generating feature information regarding the hand and fingers based on the first image; An information processing program that causes execution of a feature information output step of outputting a detection result by the detection unit and the feature information.