Operation device, information processing device, information processing system, information processing method, and information processing program

Abstract

An operating device (10) receives an operation by a user and comprises: a detection unit (21) that detects an operation; an acquisition unit (22) that acquires a first image showing the user's fingers; a characteristic-information generation unit (231) that generates characteristic information about the fingers on the basis of the first image; and a characteristic-information output unit (12) that outputs detection results by the detection unit and the characteristic information.

Description

[Technical Field]

[0001] The present invention relates to an operation device, an information processing device, an information processing system, an information processing method, and an information processing program. [Background technology]

[0002] BACKGROUND ART Conventionally, a technique for tracking the movement of a user's fingers for use in VR (Virtual Reality) and the like has been known (see, for example, Patent Document 1). In the invention described in Patent Document 1, the positions of a user's fingertips are tracked using a glove equipped with sensors at positions corresponding to the user's fingertips. Information indicating the positions tracked by the glove is converted into various control signals and used for various purposes, such as generating and displaying images that reproduce the user's fingers in a virtual space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2016 / 024369 Summary of the Invention [Problem to be solved by the invention]

[0004] When generating the reproduction image in the virtual space described above, if there is a large visual difference between the actual user's fingers and the reproduction image, the user may feel uncomfortable. Therefore, there is a need to provide users with a better experience by improving this sense of discomfort and increasing realism and immersion. [Means for solving the problem]

[0005] An operating device according to a first aspect of the present invention is an operating device that accepts operations by a user, and includes a detection unit that detects the operation, an acquisition unit that acquires a first image of the user's fingers, a feature information generation unit that generates feature information about the fingers based on the first image, and a feature information output unit that outputs the detection result by the detection unit and the feature information. An information processing device according to a second aspect of the present invention includes an acquisition unit that acquires a detection result from an operating device having a detection unit that detects operations by a user and a first image of the user's fingers, a feature information generation unit that generates feature information about the fingers based on the first image, an image generation unit that generates a second image that reproduces the shape of the fingers based on the detection result from the detection unit and the feature information, and a display unit that displays the second image. An information processing device according to a third aspect of the present invention includes an acquisition unit that acquires a detection result from an operation device having a detection unit that detects an operation by a user and a first image of the user's fingers, a feature information generation unit that generates feature information about the fingers based on the first image, an operation information generation unit that generates operation information for operating an operation target based on the detection result from the detection unit, an operation information adjustment unit that adjusts the operation information based on the feature information, and an operation information output unit that outputs the adjusted operation information to the operation target. An information processing method according to a fourth aspect of the present invention is an information processing method implemented by an operating device that has a detection unit and accepts operations by a user, and includes an acquisition step of acquiring a first image of the user's fingers, a feature information generation step of generating feature information about the fingers based on the first image, and a feature information output step of outputting the detection result by the detection unit and the feature information. An information processing program according to a fifth aspect of the present invention is an information processing program executed by an information terminal that has a detection unit and is connected to an operation device that accepts operations by a user, and causes the information terminal to execute an acquisition step of acquiring a first image of a user's fingers, a feature information generation step of generating feature information about the fingers based on the first image, and a feature information output step of outputting the detection result by the detection unit and the feature information. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a schematic diagram showing a schematic configuration of an information processing system according to a first embodiment. [Figure 2] FIG. 1 is a perspective view schematically showing an operating device according to a first embodiment. [Figure 3A] FIG. 10 is a diagram showing an example of a reference image. [Figure 3B] FIG. 10 is a diagram illustrating feature information. [Figure 4A] FIG. 10 is a diagram showing an example of a model image. [Figure 4B] 10A to 10C are diagrams illustrating adjustment of a 3D model based on feature information. [Figure 5] 4 is a flowchart illustrating an example of a processing method according to the first embodiment. [Figure 6] FIG. 10 is a schematic diagram showing a schematic configuration of an information processing system according to a second embodiment. [Figure 7] 10 is a flowchart illustrating an example of a processing method according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0007] First Embodiment A first embodiment of the present invention will be described below with reference to the drawings. [Outline of information processing system] FIG. 1 is a schematic diagram showing a schematic configuration of an information processing system according to the first embodiment. 1, an information processing system 1 according to the first embodiment includes an operation device 10, an information processing device 20, and an HMD unit 30. The information processing system 1 generates a model image (second image) that reproduces the user's fingers in a virtual space in response to an operation on the operation device 10, and displays the model image (second image) on the HMD unit 30.

[0008] [Configuration of operation device] The operation device 10 includes a detection unit 11 and an output unit 12, and receives operations from a user. The detection unit 11 detects the position of the user's fingers (hand and fingers). The output unit 12 outputs the detection result by the detection unit 11 to the outside of the operation device 10 wirelessly or via a wired connection.

[0009] FIG. 2 is a perspective view schematically showing the operating device 10. As shown in FIG. The operating device 10 is worn on the user's fingers by a glove-shaped wearing member, as shown in Fig. 2. The wearing member is preferably made of a stretchable material that can fit snugly on the user's hand.

[0010] As shown in FIG. 2, the operation device 10 includes, as the detection unit 11, a plurality of sensors 111a, 111b, 111c, 111d, and 111e provided corresponding to the fingers, and a sensor 112 provided corresponding to a position other than the fingers on the user's hand.

[0011] Multiple sensors 111a, 111b, 111c, 111d, and 111e are provided at positions corresponding to the fingertips of each finger and on the nail side of each finger. Specifically, for example, sensor 111a is provided corresponding to the first finger (thumb) as shown in Fig. 2, and detects the position of sensor 111a as the position of the thumb tip. Similarly, sensors 111b, 111c, 111d, and 111e are provided corresponding to the second finger (index finger), third finger (middle finger), fourth finger (ring finger), and fifth finger (pinky finger), respectively, and detect the position of each sensor as the position of the fingertip of each finger. The sensor 112 is provided at a position corresponding to the back of the hand, and detects the position of the sensor 112 as the position of the back of the hand.

[0012] In this embodiment, any known sensors may be used for the sensors 111a, 111b, 111c, 111d, and 111e, and the sensor 112. The positions of the sensors 111a, 111b, 111c, 111d, and 111e, and the sensor 112 are not limited to the example in Fig. 2. For example, the sensors 111a, 111b, 111c, 111d, and 111e may be provided on the pads of the fingers, at the tips of the fingers, or so as to cover part or all of the fingers. In this embodiment, the output unit 12 is provided in a housing 121 provided at a position corresponding to the user's wrist, as shown in Fig. 2. If the operating device 10 has a control unit that controls each unit, the control unit may also be provided in this housing 121. Furthermore, the housing 121 may be provided integrally or independently at the same position as the sensor 112 provided at a position corresponding to the back of the hand described above, or may be provided at another position.

[0013] [Configuration of information processing device] As shown in FIG. 1, the information processing device 20 includes a communication unit 21 and a control unit 22. The communication unit 21 receives the detection result by the detection unit 11 output from the output unit 12 of the operation device 10, and is connected to the communication unit 32 of the HMD unit 30 wirelessly or via a wire.

[0014] The control unit 22 includes the functions of a feature information generation unit 221 and an image generation unit 222, which are realized by the processor operating in accordance with a program stored in the memory or received via a communication interface. The functions of each unit will be described later.

[0015] [HMD unit configuration] The HMD unit 30 can be used by being worn on the user's head via a mounting unit (not shown). The HMD unit 30 also functions as a display device and an imaging device that captures images from the user's first-person perspective, or so-called inside-out. The HMD unit 30 may be configured to be worn on the user's head via a mounting unit such as a band, or may be configured in any other way, such as a helmet or glasses. Furthermore, when wearing the HMD unit 30, it is advisable to display a tutorial or the like on the display device of the HMD unit 30 to guide the user to wear the HMD unit 30 properly.

[0016] As shown in FIG. 1, the HMD unit 30 includes an imaging unit 31, a communication unit 32, a control unit 33, and a display unit . The imaging unit 31 has an imaging element (not shown) and captures an image from a first-person viewpoint of a user to generate an image. The imaging element may be an image sensor that generates an image including color information, or a depth sensor that generates an image including depth information. The imaging unit 31 may also be provided with multiple imaging elements. For example, the imaging unit 31 may be provided with both an image sensor and a depth sensor, and may generate an image including color information and an image including depth information, respectively. For example, the imaging unit 31 may be provided with multiple image sensors, and may generate an image including color information and generate depth information from the multiple images. The imaging unit 31 captures a reference image (first image) of the user's fingers. The reference image is information used to generate a model image that reproduces the user's fingers in the virtual space. The communication unit 32 is connected to the communication unit 21 of the information processing device 20 wirelessly or via a wire. The control unit 33 controls each unit in the HMD unit 30 .

[0017] The display unit 34 includes a display element such as an LCD (Liquid Crystal Display) or an organic EL display, and an optical device such as a lens, and is capable of displaying an image to the user. The display element of the display unit 34 may be a transmissive display element or a non-transmissive display element. Also, a terminal device such as a smartphone that can be attached to or detached from the housing of the HMD unit 30 may be used as a display device. Furthermore, a wearable device such as AR (Augmented Reality) glasses or MR (Mixed Reality) glasses may be used as the HMD unit 30.

[0018] Next, the function of each unit in the control unit 22 of the information processing device 20 will be described. The feature information generating unit 221 generates feature information about fingers based on the reference image. The feature information is information used to generate a model image, similar to the reference image described above, and includes at least one of the following two types: (1) Position information of feature points on fingers The feature information generation unit 221 calculates position information of feature points, such as joint information of the user's fingers, from the reference image using, for example, a machine learning technique. Examples of feature points include the endpoints of each finger, each finger joint, and the wrist joint. A trained model used for machine learning can be constructed in advance by performing supervised learning using, for example, images of a person's fingers having multiple feature points as input data and coordinate information indicating the positions of the feature points on the person's fingers as ground truth data. Note that detailed explanations of specific machine learning techniques will be omitted since various known techniques can be used. Also, the feature information generation unit 221 may be configured to include a relationship learning unit, and each time a reference image is generated, the relationship between an image based on the input reference image and coordinate information indicating the positions of feature points on a person's fingers is learned and the learned model is updated. Figure 3B shows an example of feature points based on a reference image. In the example of Figure 3B, the endpoints of each finger, each finger joint, and the wrist joint are detected as feature points, and the position information of each feature point is calculated as feature information. By generating position information of feature points on the fingers as feature information and determining the relative positional relationships of these feature points, it is possible to accurately determine the actual length of the user's fingers, the proportions of each part of the fingers, etc.

[0019] In addition, the feature information generation unit 221 may extract the silhouette of the user's fingers using a technique such as skin color detection for a reference image including color information, and generate position information of feature points on the fingers based on the extracted silhouette. Furthermore, the feature information generating unit 221 may generate position information of feature points on fingers by, for example, estimating the thickness of the area around the finger joints based on the depth information of a reference image including the depth information.

[0020] (2) Texture information of the surface of the fingers The feature information generating unit 221 calculates texture information indicating the surface condition of the user's fingers from the reference image using, for example, a method such as skin color detection. The texture information includes the color and texture of the user's skin, as well as irregularities on the surface of the fingers, such as scratches, moles, and wrinkles.

[0021] The image generation unit 222 generates a model image (second image) that reproduces the user's fingers in the virtual space, based on the detection result received from the operation device 10 via the communication unit 21. An example of the model image is shown in FIG. 4A. The image generation unit 222 controls the 3D model of the fingers based on the detection results received from the above-mentioned operation device 10, adjusts the 3D model of the fingers based on the feature information generated by the feature information generation unit 221, and generates a model image based on the adjusted 3D model. More specifically, the image generation unit 222 uses, for example, known Inverse Kinetics (hereinafter referred to as "IK") technology to specify a start point (IK root), an end point (IK end), and a target point (IK goal) for each joint, thereby performing posture corrections such as bending and stretching movements to control a 3D model of the finger skeleton (bones). In IK technology, when the tip of a joint structure is moved, correction calculations are performed by tracing the rotation of the joint in the reverse direction (parent direction). The image generation unit 222 then generates a model image by performing skinning processing to set the range of influence of the skeleton for the 3D model.

[0022] When moving the middle joints of the fingers in a model image using IK technology, it is desirable that the skeleton of the 3D model used to generate the model image matches the skeleton of the fingers based on the detection results received from the operation device 10. For example, if the hand of the 3D model is larger than the hand of the user operating the operation device 10, the sensors (sensors 111a, 111b, 111c, 111d, and 111e) will be recessed into the fingertips in the 3D model. Also, if the hand of the 3D model is smaller than the hand of the user operating the operation device 10, the sensors (sensors 111a, 111b, 111c, 111d, and 111e) will be located at positions far away from the fingertips. In such a case, since the skeleton of the 3D model used to generate the model image does not match the skeleton of the fingers based on the detection results received from the operation device 10, the bending of the middle joints will not be accurately reproduced in the model image generated based on the 3D model, resulting in unnatural movement and a sense of discomfort to the user.

[0023] Therefore, in order to eliminate or reduce the above-mentioned problems, the image generation unit 222 generates a model image after adjusting the 3D model as necessary based on the feature information generated by the feature information generation unit 221. The model image is generated according to the type of feature information. (1) Position information of feature points on fingers When the feature information is position information of feature points on the fingers, the image generation unit 222 adjusts the length of the fingers in the 3D model and the ratio of each part of the fingers based on the feature information. More specifically, the image generation unit 222 adjusts the 3D model by enlarging or reducing the fingers in the 3D model, lengthening or shortening the length of the fingers in the 3D model based on the feature information, or changing the ratio of each part of the fingers in the 3D model based on the feature information. Then, the image generation unit 222 generates a model image by performing skinning processing on the adjusted 3D model.

[0024] (2) Texture information of the surface of the fingers If the feature information is texture information of the surface of the fingers, the image generation unit 222 uses the image of the finger part of the reference image as the texture for the part of the fingers in the model image, or generates a model image that reflects the color and texture of the user's skin, as well as surface irregularities of the fingers such as scratches, moles, and wrinkles, based on the feature information. Note that if images of both the palm side and the back side of the user's fingers are captured as the reference image, the images of both sides are also adjusted in the model image.

[0025] An example of a model image generated based on feature information is shown in Figure 4B. In the example of Figure 4B, the balance of the lengths of the fingers in the 3D model, the thickness of each finger, etc. have been adjusted, resulting in a model image that is closer to the user's actual fingers. If the feature information includes both (1) and (2) above, the image generation unit 222 may perform the image generation processes described in (1) and (2) in combination, or may perform only one of the image generation processes. In addition, the content of the image generation process may be set by the user.

[0026] [Information processing flow] FIG. 5 is a flowchart showing the processing executed by the control unit 22. The control unit 22 determines whether a reference image has been acquired by the imaging unit 31 (step S101), and if it determines that a reference image has been acquired (step S101 YES), the feature information generation unit 221 generates feature information based on the reference image (step S102). Then, the control unit 22 determines whether or not the detection result by the operation device 10 has been acquired via the communication unit 21 (step S103). If it is determined that the detection result has been acquired (step S103: YES), the image generation unit 232 adjusts the 3D model image in accordance with the feature information generated in step S102 (step S104), and generates a model image (step S105). Furthermore, when the communication unit 21 outputs the generated model image to the HMD unit 30 (step S106), the process returns to step S103. That is, once feature information is generated by the processes of steps S101 and S102, the processes of steps S103 to S106 are repeatedly executed thereafter every time a detection result is obtained by the operation device 10. Therefore, a model image is output to the HMD unit 30 every time a detection result is obtained by the operation device 10, and the model image displayed on the display unit 24 is sequentially updated, and at that time, a model image based on a reference image is always generated.

[0027] [Effects of the first embodiment] The information processing system 1 according to the first embodiment described above provides the following advantages. The information processing device 20 includes an acquisition unit (communication unit 21) that acquires the detection results from the operation device 10, which has a detection unit 11 that detects operations by the user, and a reference image, which is a first image capturing the user's fingers, a feature information generation unit 221 that generates feature information about the fingers based on the reference image, and an image generation unit 232 that generates a model image, which is a second image that reproduces the shape of the fingers, based on the detection results from the detection unit 11 and the feature information.

[0028] With this configuration, by accurately scaling and corresponding to the shape of the actual user's fingers, it is possible to reduce the difference between the actual user's fingers and the user's fingers reproduced in the model image, thereby improving the sense of discomfort felt by the user. As a result, it is possible to improve the sense of realism and immersion and provide a better experience to the user.

[0029] The image generation unit 222 controls the 3D model of the fingers based on the detection result, adjusts the 3D model of the fingers based on the feature information, and generates a model image based on the adjusted 3D model. With this configuration, the skeleton of the 3D model that is the basis for generating the model image is matched with the skeleton of the fingers based on the detection results received from the operation device 10, so that the movement of the fingers can be accurately reproduced in the model image.

[0030] The feature information generated by the feature information generating unit 221 includes position information of feature points on the fingers. With this configuration, the shape of the actual user's fingers can be accurately reproduced by adjusting the model image based on the position information of the feature points of the fingers. This eliminates or reduces defects that occur in the model image, and the actual user's fingers and their movements can be accurately reproduced in the model image.

[0031] The feature information generated by the feature information generating unit 221 includes texture information of the surface of the fingers. With this configuration, the model image can be adjusted based on the texture information of the surface of the fingers, making it possible to accurately reproduce the feel and appearance of the actual user's fingers, which is expected to improve realism and immersion.

[0032] In the HMD unit 30, which is an information processing device, the reference image captured by the imaging unit 31 is an image that includes at least one of color information and depth information. According to this configuration, feature information required for adjusting a model image can be easily generated from a reference image that can be easily obtained.

[0033] Second Embodiment A second embodiment of the present invention will be described below with reference to the drawings. In the second embodiment, only the parts that are different from the first embodiment will be described, and the parts that are the same as those in the first embodiment will not be described. In the second embodiment, components that have substantially the same functional configuration as those in the first embodiment will be assigned the same reference numerals. FIG. 6 is a schematic diagram showing the entire information processing system according to the second embodiment of the present invention. 6, the information processing system 2 according to the second embodiment includes an imaging device 40, an information processing device 50, and an operation target device 60, instead of the information processing device 20 and the HMD unit 30 of the information processing system 1 according to the first embodiment. The information processing system 2 generates operation information in the information processing device 50 in response to an operation on the operation device 10, and controls the operation target device 60 based on the operation information.

[0034] [Configuration of imaging device 40] 6, the imaging device 40 includes an imaging unit 41 and an output unit 42. The imaging unit 41 has an imaging element (not shown) and captures an image from a third-person viewpoint of the user, that is, from an outside-in perspective, to generate an image. Details of the imaging element and the number of imaging elements are the same as those of the imaging unit 31 of the HMD unit 30 in the information processing system 1 of the first embodiment, and therefore will not be described here. Furthermore, the imaging section 41, like the imaging section 31 of the HMD unit 30 in the information processing system 1 of the first embodiment, captures a reference image (first image) of the user's fingers. The output unit 42 outputs the reference image captured by the imaging unit 41 to the outside of the imaging device 40 via wireless or wired communication.

[0035] [Configuration of information processing device 50] As shown in FIG. 6, the information processing device 50 includes a communication unit 51 and a control unit 52. The communication unit 51 receives the detection result by the detection unit 11 output from the output unit 12 of the operation device 10 and the reference image output from the output unit 42 of the imaging device 40. Furthermore, the communication unit 51 outputs operation information generated by the control unit 52 to the outside of the information processing device 50 via wireless or wired communication. Details of the operation information will be described later.

[0036] The control unit 52 includes functions of a feature information generation unit 521, an operation information generation unit 522, and an operation information adjustment unit 523, which are realized by the processor operating in accordance with a program stored in the memory or received via the communication interface. The functions of each unit will be further described below.

[0037] The feature information generating unit 521 generates feature information about the fingers based on the reference image. The feature information includes position information of feature points on the fingers. The feature information generating unit 521 calculates position information of feature points on the user's fingers from the reference image, similar to the feature information generating unit 221 in the information processing device 20 of the first embodiment.

[0038] The operation information generating unit 522 generates operation information for operating the operation target device 60 based on the detection result received from the operation device 10 via the communication unit 51. The operation information may be the detection result itself, or may be control information that indicates the drive locations and drive amounts of the operation target device 60 calculated based on the detection result. Here, if there is a large difference between the actual user's fingers and the corresponding locations on the operation target device 60, problems may occur in the operation of the operation target device 60. For example, if the size of the corresponding locations on the operation target device 60 is significantly different from the size of the actual user's fingers, the operation of the corresponding locations on the operation target device 60 may not be reproduced correctly. That is, if there is no difference or if the difference is sufficiently small between the actual user's fingers and the corresponding parts on the operation target device 60, the movement of the actual user's fingers will be correctly reproduced on the operation target device 60.

[0039] In order to eliminate or reduce the above-mentioned problems, the operation information adjustment unit 523 adjusts the operation information generated by the operation information generation unit 522 based on the feature information generated by the feature information generation unit 521. A specific method of adjustment will be described later.

[0040] [Configuration of operation target device 60] The operation target device 60 is a device that is operated in response to an operation on the operation device 10. Examples of the operation target device 60 include a robot hand and a manipulator. As shown in FIG. 6, the operation target device 60 includes a communication unit 61 and a drive unit 62. The communication unit 61 receives the operation information output from the communication unit 51 of the information processing device 50 . The driving unit 62 includes a plurality of motors and actuators, such as servo motors, depending on the shape of the robot hand and manipulator, and drives at least one of them. At this time, the driving unit 62 drives based on operation information (for example, a control signal for operating a servo motor) received via the communication unit 61.

[0041] It should be noted that the installation positions of the motors and actuators in the drive unit 62 may be fixed or movable. For example, if the operation target device 60 is a robot hand, the motors and actuators of the drive unit 62 installed in the joints of the robot hand may be fixed or movable. In either case, information indicating the installation positions of the motors and actuators in the drive unit 62 of the operation target device 60 is stored in advance in the control unit 52, or is configured to be obtainable from the operation target device 60 via the communication unit 51. The same applies when the operation target device 60 is a manipulator.

[0042] [Adjustment of operation information] When the installation positions of the motor and the actuator in the drive unit 62 of the operation target device 60 are fixed, the operation information adjustment unit 523 adjusts the operation information based on the feature information. More specifically, the operation information adjustment unit 523 first obtains a correlation between the feature points in the feature information and the installation position of the drive unit 62. Then, the operation information adjustment unit 523 adjusts the operation information in accordance with the correlation. In other words, by adjusting the operation information to the installation position of the drive unit 62 in the operation target device 60, the difference between the actual user's fingers and the installation positions of the motor and the actuator in the drive unit 62 of the operation target device 60 is reduced.

[0043] On the other hand, when the installation positions of the motor and actuator in the drive unit 62 of the operation target device 60 are movable, the operation information adjustment unit 523 adjusts the operation information by adding to the operation information adjustment information for enlarging or reducing the installation positions of the motor and actuator based on the feature information, adjustment information for lengthening or shortening the length between the installation positions of the motor and actuator based on the feature information, adjustment information for changing the installation positions of the motor and actuator based on the feature information, etc. In other words, by matching the installation positions of the motor and actuator in the drive unit 62 of the operation target device 60 to the shape of the actual user's fingers, the difference between the installation positions of the actual user's fingers and the installation positions of the motor and actuator in the drive unit 62 of the operation target device 60 is reduced.

[0044] As described above, the information processing device 50 receives the detection result from the operation device 10 and also receives the reference image from the imaging device 40. Then, the information processing device 50 calculates feature information based on the reference image. Then, the information processing device 50 generates operation information based on the detection result and adjusts the operation information based on the feature information.

[0045] [Information processing flow] FIG. 7 is a flowchart showing the processing executed by the control unit 52 of the information processing device 50. The control unit 52 determines whether a reference image has been acquired via the communication unit 51 (step S201), and if it determines that a reference image has been acquired (step S201 YES), the feature information generation unit 521 generates feature information based on the reference image (step S202). Then, the control unit 52 determines whether or not the detection result by the operation device 10 has been acquired via the communication unit 51 (step S203). If it is determined that the detection result has been acquired (step S203: YES), the operation information generation unit 522 generates operation information (step S204), and the operation information adjustment unit 523 adjusts the operation information based on the feature information generated in step S202 (step S205). Furthermore, when the communication unit 51 outputs the adjusted operation information (step S206), the process returns to step S203. That is, once the feature information is generated by the processes of steps S201 and S202, the processes of steps S203 to S206 are repeatedly executed thereafter every time a detection result is obtained by the operation device 10. Therefore, every time a detection result is obtained by the operation device 10, operation information is output, and at that time, the operation information is always adjusted based on the reference image.

[0046] [Effects of the second embodiment] The information processing system 2 according to the second embodiment described above has the following advantages. The information processing device 50 includes an acquisition unit (communication unit 51) that acquires a detection result by the operation device 10 having a detection unit 11 that detects an operation by the user and a reference image that is a first image capturing the user's fingers, a feature information generation unit 521 that generates feature information about the fingers based on the reference image, an operation information generation unit 522 that generates operation information for operating an operation target device 60 that is an operation target based on the detection result by the detection unit 11, and an operation information adjustment unit 523 that adjusts the operation information based on the feature information. The adjusted operation information is output to the operation target device 60 via the communication unit 51.

[0047] With this configuration, the operation information is adjusted based on the position information of the feature points of the fingers, so that the actual movements of the user's fingers can be accurately reproduced on the operation target. This makes it possible to eliminate or reduce problems that occur in the operation of the operation target. In particular, it is possible to absorb individual differences in the shape of the fingers that differ from user to user.

[0048] [Modification of the embodiment] The present invention is not limited to the above-described embodiments, and modifications and improvements within the scope of achieving the object of the present invention are included in the present invention.

[0049] In the first embodiment, the operating device 10 is attached to the user's fingers by a glove-shaped attachment member, and the position of the user's fingers (hand and fingers) is detected by the detection unit 11. However, the present invention is not limited to this, and can be similarly applied to an operating device 10 in the shape of a stick or the like that is held by the user's fingers.

[0050] In the first embodiment, the information processing device is described by taking the information processing device 20 and the HMD unit 30 as examples of the information processing device. However, the present invention is not limited to this, and can be similarly applied to other information processing devices that include an internal or external imaging device, as in the second embodiment.

[0051] A configuration may be adopted in which part of the processing performed by the HMD unit 30, which is the information processing device in the first embodiment, is performed by the operation device 10. For example, a configuration may be adopted in which part or all of the functions of the feature information generation unit 221 and the image generation unit 222 in the HMD unit 30 of the first embodiment are performed by the operation device 10. Furthermore, the control unit 22 of the information processing device 20 may be provided in the HMD unit 30.

[0052] A configuration may be adopted in which part of the processing performed by the information processing device 50 in the second embodiment is performed by the operation device 10, or a configuration may be adopted in which part of the processing is performed by the operation target device 60. For example, a configuration may be adopted in which part or all of the functions of the feature information generation unit 521, the operation information generation unit 522, and the operation information adjustment unit 523 in the information processing device 50 of the second embodiment are performed by the operation device 10. Furthermore, for example, a configuration may be adopted in which part or all of the functions of the feature information generation unit 521, the operation information generation unit 522, and the operation information adjustment unit 523 in the information processing device 50 of the second embodiment are performed by the operation target device 60.

[0053] In the above-described embodiments, the operating device 10 is attached to the user's fingers by a glove-shaped attachment member. However, the attachment member may be a band-shaped attachment member. The operating device 10 may also be a so-called exoskeleton controller.

[0054] [Summary of the present invention] The present invention will be summarized below. [1] An operating device that accepts operations by a user, comprising: a detection unit that detects the operation; an acquisition unit that acquires a first image of the user's fingers; a feature information generation unit that generates feature information about the fingers based on the first image; and a feature information output unit that outputs the detection result by the detection unit and the feature information. [2] In the operation device described in [1], the feature information includes position information of feature points on the fingers. [3] In the operation device according to [1] or [2], the feature information includes texture information of the surface of the fingers. [4] In the operating device according to any one of [1] to [3], the detection unit detects the position of a finger. [5] In the operation device according to any one of [1] to [4], the first image is an image including at least one of color information and depth information. [6] An information processing device comprising: an acquisition unit that acquires a detection result from an operating device having a detection unit that detects operations by a user and a first image of the user's fingers; a feature information generation unit that generates feature information about the fingers based on the first image; an image generation unit that generates a second image that reproduces the shape of the fingers based on the detection result from the detection unit and the feature information; and a display unit that displays the second image. [7] In the information processing device described in [6], the image generation unit controls the 3D model of the fingers based on the detection result, adjusts the 3D model of the fingers based on the feature information, and generates a second image based on the adjusted 3D model. [8] An information processing device including: an acquisition unit that acquires a detection result from an operation device having a detection unit that detects an operation by a user and a first image of the user's fingers; a feature information generation unit that generates feature information about the fingers based on the first image; an operation information generation unit that generates operation information for operating an operation target based on the detection result from the detection unit; an operation information adjustment unit that adjusts the operation information based on the feature information; and an operation information output unit that outputs the adjusted operation information to the operation target. [9] An information processing system including an operation device having a detection unit that detects an operation by a user, and the information processing device according to [6] or [8].

[10] In the information processing system according to [9], the information processing device further includes an imaging unit having an imaging element and configured to capture the first image.

[11] In the information processing system described in

[10] , the information processing device further includes a mounting unit that allows at least the imaging element to be mounted on a user's body.

[12] An information processing system comprising an operation device having a detection unit that detects an operation by a user, the information processing device described in [8], and an operation target that operates based on operation information based on the detection result.

[13] An information processing method implemented by an operating device having a detection unit and accepting operations by a user, the information processing method including an acquisition step of acquiring a first image of a user's fingers, a feature information generation step of generating feature information about the fingers based on the first image, and a feature information output step of outputting the detection result by the detection unit and the feature information.

[14] An information processing program executed by an information terminal having a detection unit and connected to an operation device that accepts operations by a user, the information processing program causing the information terminal to execute an acquisition step of acquiring a first image of a user's fingers, a feature information generation step of generating feature information about the fingers based on the first image, and a feature information output step of outputting the detection result by the detection unit and the feature information. [Explanation of symbols]

[0055] 1,2...information processing system, 10...operation device, 11...detection unit, 12,42...output unit, 31,41...imaging unit, 21,32,51,61...communication unit, 22,33,52...control unit, 30...HMD unit, 34...display unit, 40,50...information processing device, 60...operation target device, 62...drive unit, 111a-111e,112...sensor, 221,521...feature information generation unit, 222...image generation unit, 522...operation information generation unit, 523...operation information adjustment unit.

Claims

1. An operation device that accepts an operation by a user, a detection unit that detects the operation; an acquisition unit that acquires a first image of the user's fingers; a feature information generating unit that generates feature information about the fingers based on the first image; an image generation unit that controls the 3D model of the fingers based on the detection result by the detection unit, adjusts the 3D model of the fingers based on the feature information, and generates a second image that reproduces the shape of the fingers based on the adjusted 3D model; An operating device comprising:

2. 2. The operating device according to claim 1, The operating device, wherein the feature information includes position information of feature points on the fingers.

3. 3. The operating device according to claim 1, The operation device, wherein the feature information includes texture information of the surface of the fingers.

4. 2. The operating device according to claim 1, The detection unit detects the position of the finger. An operating device characterized by:

5. 2. The operating device according to claim 1, An operating device, wherein the first image is an image including at least one of color information and depth information.

6. an acquisition unit that acquires a detection result by an operation device including a detection unit that detects an operation by a user and a first image capturing a finger of the user; a feature information generating unit that generates feature information about the fingers based on the first image; an image generation unit that controls the 3D model of the fingers based on the detection result by the detection unit, adjusts the 3D model of the fingers based on the feature information, and generates a second image that reproduces the shape of the fingers based on the adjusted 3D model; An information processing device comprising:

7. an acquisition unit that acquires a detection result by an operation device including a detection unit that detects an operation by a user and a first image capturing a finger of the user; a feature information generating unit that generates feature information about the fingers based on the first image; an operation information generating unit that generates operation information for operating an operation target based on a detection result by the detecting unit; an operation information adjustment unit that adjusts the operation information based on the feature information; an operation information output unit that outputs the adjusted operation information to the operation target; An information processing device comprising:

8. an operation device including a detection unit that detects an operation by a user; The information processing device according to claim 6 or 7, An information processing system comprising:

9. 9. The information processing system according to claim 8, The information processing device further includes an imaging unit that has an imaging element and captures the first image. An information processing system comprising:

10. 10. The information processing system according to claim 9, The information processing device further includes a mounting unit that can mount at least the imaging element on the user's body. An information processing system comprising:

11. an operation device including a detection unit that detects an operation by a user; The information processing device according to claim 7 ; an operation target that operates based on operation information based on the detection result; An information processing system comprising:

12. An information processing method implemented by an operating device that includes a detection unit that detects an operation by a user and that accepts the operation by the user, an acquisition step of acquiring a first image of the user's fingers; a feature information generating step of generating feature information about the fingers based on the first image; an image generation step of controlling the 3D model of the fingers based on the detection result by the detection unit, adjusting the 3D model of the fingers based on the feature information, and generating a second image that reproduces the shape of the fingers based on the adjusted 3D model; An information processing method comprising:

13. An information processing program executed by an operating device that includes a detection unit that detects an operation by a user and that accepts the operation by the user, The operating device includes: an acquiring step of acquiring a first image of the user's fingers; a feature information generating step of generating feature information about the fingers based on the first image; an image generation step of controlling the 3D model of the fingers based on the detection result by the detection unit, adjusting the 3D model of the fingers based on the feature information, and generating a second image that reproduces the shape of the fingers based on the adjusted 3D model.

Images