Information processing device, information processing method, and program

Abstract

An information processing device according to one aspect of the present technology has a light source unit, a detection unit, and a determination unit. The light source unit irradiates a body part of a user with light. The detection unit has a plurality of light detection parts, and outputs a plurality of detection signals based on reflected light reflected from the body part. The determination unit determines the motion of the user based on information on speckles generated by irradiating the body part with light contained in the plurality of detection signals.

Description

Technical Field

[0001] The present technology relates to an information processing device, an information processing method, and a program that can be applied to, for example, a wearable device. Background Art

[0002] Patent document 1 discloses a projection system including a wristband terminal and a smartphone. In the projection system, an image is transmitted from the smartphone to the wristband terminal. The transmitted image is projected onto the palm of the user through a projector including the wristband terminal. This can improve the visibility of the image. In addition, various GUIs projected onto the palm can be operated like operating a smartphone. This improves the usability of the sports terminal (for example, paragraphs

[0013] to

[0025] of the specification and the Figure 1 ).

[0003] Citation List

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2015-41052

[0006] Invention Disclosure

[0007] Technical issues

[0008] In the future, various systems using wearable devices and the like will be widely used, and a technology that can achieve high usability will be required.

[0009] In view of the above circumstances, an object of the present technology is to provide an information processing device, an information processing method, and a program that enable high usability to be achieved. Summary of the invention

[0010] In order to achieve the above-mentioned object, an information processing apparatus according to an embodiment of the present technology includes a light source section, a detector, and a determination section.

[0011] The light source portion irradiates light onto a body part of the user.

[0012] The detector includes a plurality of light detectors, and outputs a plurality of detection signals based on reflected light reflected from a body part.

[0013] The determination section determines the motion of the user based on information on speckles generated due to light being irradiated onto the body part, the information on the speckles being included in the plurality of detection signals.

[0014] In the information processing device, light is irradiated to the user's body part, and a plurality of detection signals are output based on its reflected light. Then, the user's motion is determined based on the information about speckle included in the plurality of detection signals. This enables high usability.

[0015] The light source portion can irradiate the laser onto the body part.

[0016] The plurality of light detectors may be a plurality of photodiodes.

[0017] The determination section may determine the motion of the user based on a speckle pattern included in the information about the speckle.

[0018] The determination section may determine the motion of the user based on a time-series change in the speckle pattern.

[0019] The determination section may determine the motion of the user based on the periodicity of the time-series change in the speckle pattern.

[0020] The body part may be a wrist. In this case, the determination section may determine the movement of the user's hand.

[0021] The determination section may determine at least one of which finger is bent, the bending amount of the bent finger, an interaction between fingers, or an interaction between a finger and another object.

[0022] The information processing apparatus may further include an execution section that executes a process corresponding to the determined motion.

[0023] The execution section may recognize an operation input by the user based on the determined motion, and may execute a process corresponding to the recognized operation.

[0024] The information processing device may further include a history information storage storing therein history information about operations input by the user at past times. In this case, the execution section may recognize the operation input by the user based on the stored history information.

[0025] The information processing device may further include a display section capable of displaying a specified graphical user interface (GUI). In this case, the execution section may recognize an operation input into the displayed specified GUI based on the determined motion.

[0026] The execution section may select a plurality of selection candidates selectable by the user based on the determined motion. In this case, the display section may display a selection image including the plurality of selected selection candidates.

[0027] The determination section may determine the user's motion according to a specified learning algorithm.

[0028] The information processing apparatus may further include an instruction section and a determination information memory.

[0029] The instruction portion instructs the user to perform the specified movement.

[0030] The determination information storage stores therein determination information including information on speckles, which is included in the plurality of detection signals and is information when a user performs a designated motion that has been instructed to be performed.

[0031] In this case, the determination section may determine the motion of the user based on the stored determination information.

[0032] The information processing device may further include a receiving section that receives true or false information about whether the result of the determination performed by the determination section is correct. In this case, the determination section may determine the user's motion based on the received true or false information.

[0033] The detector may include an image sensor. In this case, the plurality of light detectors may be a plurality of pixels of the image sensor.

[0034] An information processing device according to another embodiment of the present technology includes a receiver and a determination section.

[0035] The receiver receives a plurality of detection signals output based on reflected light that is reflected off the body part of the user due to light being irradiated onto the body part of the user.

[0036] The determination section determines the motion of the user based on information about speckles generated due to light being irradiated on the body part, the information about the speckles being included in the plurality of received detection signals.

[0037] An information processing method according to an embodiment of the present technology is a method performed by a computer system, and includes receiving a plurality of detection signals output based on reflected light reflected from a body part of a user due to light being irradiated thereon.

[0038] The motion of the user is determined based on information about speckles generated due to light being irradiated onto the body part, the information about the speckles being included in the plurality of received detection signals.

[0039] The program according to the embodiment of the present technology causes a computer system to execute processing including the following:

[0040] receiving a plurality of detection signals output based on reflected light reflected from a body part of the user due to the light being irradiated to the body part of the user; and

[0041] The motion of the user is determined based on information about speckles generated due to light being irradiated on the body part, the information about the speckles being included in the plurality of received detection signals.

[0042] Advantageous Effects of the Invention

[0043] As described above, the present technology enables high usability.Note that the effects described here are not necessarily restrictive, and any effects described in the present disclosure may be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The appearance of a wearable device according to an embodiment of the present technology is schematically shown.

[0045] Figure 2 The wearable device is schematically shown in a state where it is not attached to a user.

[0046] Figure 3 is a block diagram showing an example of a functional configuration of a wearable device.

[0047] Figure 4 is a schematic diagram describing the operation of the light source section and the PD array sensor.

[0048] Figure 5 A PD array sensor is schematically shown.

[0049] Figure 6 Examples of changes in the movement of the hand and changes in the intensity distribution of the detection signal are schematically shown.

[0050] Figure 7 is a schematic diagram describing decisions regarding interactions between fingers and interactions between fingers and other objects.

[0051] Figure 8 is a block diagram showing an example of a software configuration of a wearable device.

[0052] Fig. 9 is a flow chart illustrating an example of the operation of a wearable device.

[0053] Fig.10 is a table of examples of gesture patterns that can be determined.

[0054] Fig.11 is a table of examples of gesture patterns that can be determined.

[0055] Fig.12 is a schematic diagram describing an example of a virtual remote controller using the present technology.

[0056] Fig.13 is a diagram describing an example of a virtual dialog box using the present technology.

[0057] Fig.14 An example of an input UI is schematically shown.

[0058] Fig.15 is a diagram describing an example of a virtual keyboard using the present technology.

[0059] Fig.16 is a schematic diagram illustrating an example of a method for providing a virtual keyboard.

[0060] Fig.17 is a diagram describing an example of a virtual flick input performed using the present technology.

[0061] Fig.18 An example of selecting an image is schematically shown.

[0062] Fig.19 An example of a configuration related to generation of training data is schematically shown.

[0063] Fig. 20 An example of a configuration related to generation of training data is schematically shown.

[0064] Fig.21 Other examples of positions of fixed bodies are schematically shown. DETAILED DESCRIPTION

[0065] Embodiments according to the present technology will be described below with reference to the drawings.

[0066] Figure 1 The appearance of a wearable device according to an embodiment of the present technology is schematically shown. The wearable device 100 is a wristband wearable device and is used by being worn on the user's wrist 2. In this embodiment, the wearable device 100 corresponds to an embodiment of an identification device and an information processing device according to the present technology.

[0067] Figure 1 The wearable device 100 shown in A and B includes a main body 10 and an attachment band 11. The main body 10 is fixed by wrapping the attachment band 11 to be fastened on the wrist 2. In the present embodiment, the wearable device 100 is installed so that the main body 10 is located inside the user's wrist 2 (a part of the palm side).

[0068] Figure 2 The wearable device is schematically shown as viewed from the side in a state where it is not attached to a user. Figure 3 is a block diagram showing an example of the functional configuration of the wearable device 100.

[0069] The wearable device 100 includes a controller 12 , a speaker 13 , a projector 14 , a touch panel 15 , an operation button 16 , a communication part 17 , a sensor part 18 , and a memory 19 .

[0070] These blocks are included in the main body 10 of the wearable device 100 .

[0071] The speaker 13 can output sound. The speaker 13 outputs, for example, voice guidance or alarm sound. The projector 14 can project various images and a graphical user interface (GUI). For example, the image and GUI are projected onto a surface such as a table or a wall on which the user's hand is placed.

[0072] Alternatively, it is also possible to project the image, for example, onto the user's knee or onto the user's palm.

[0073] Various images and GUI are displayed on the touch panel 15. The user can input, for example, a designated instruction by performing a touch operation on the touch panel 15. The operation button 16 is provided to perform an operation different from the operation performed by the touch panel 15, for example, an on / off operation of a power source. In the present embodiment, the projector 14 and the touch panel 15 are used as a display section.

[0074] In the present embodiment, instruction input and the like can be easily performed using gestures of the hand (including, for example, fingers, joints of the hand, and the back of the hand). In other words, input operations (gesture operations) can be performed using gestures. Of course, gestures are concepts included in "motions". The input of gesture operations will be described in detail later.

[0075] The communication section 17 is a module for performing, for example, network communication, near field communication, and infrared communication with another device. For example, a wireless LAN module such as Wi-Fi or a communication module such as Bluetooth (registered trademark) is provided. Any infrared communication module may also be used.

[0076] The sensor part 18 includes a light source part 21, a photodiode (PD) array sensor 22, a camera 23, and a nine-axis sensor 24. The camera 23 is capable of capturing an image of the periphery of the wearable device 100. The camera 23 makes it possible to capture an image of, for example, a user's hand or face.

[0077] The nine-axis sensor 24 includes a three-axis acceleration sensor, a three-axis gyroscope, and a three-axis compass sensor. For example, the nine-axis sensor 24 makes it possible to detect acceleration, angular velocity, and azimuth on three axes of the wearable device 100.

[0078] like Figure 2 As shown, the light source section 21 and the PD array sensor 22 are arranged side by side on the rear surface 10 a (the surface facing the surface of the user's wrist 2 ) of the main body 10 .

[0079] The light source section 21 includes a laser light source and irradiates laser light L onto the wrist 2 of the user (see FIG. Figure 4). The type of laser light source is not limited, and for example, various laser sources such as semiconductor lasers, gas lasers, solid-state lasers, and liquid lasers can be used. In addition, a lens system or the like capable of adjusting, for example, a laser beam emitted from a laser source can be provided as the light source section 21.

[0080] The PD array sensor 22 includes a plurality of photodiodes (PDs) 26 arranged in two dimensions (see Figure 5 PD 26 is a photodetector that can output a detection signal corresponding to the amount of incident light.

[0081] PD 26 can detect light with a time resolution of tens of kilohertz and can detect changes in the amount of light in a short period of time. In this embodiment, 100 PDs 26 are used, but the number of PDs 26 is not limited and can be set arbitrarily. The specific configuration of PD 26 is not limited and can be designed arbitrarily.

[0082] In the present embodiment, the light source section 21 corresponds to a light source section that irradiates light onto a body part of the user. In addition, the wrist 2 is selected as the body part. The PD array sensor 22 corresponds to a light source section that includes a plurality of photodetectors and that irradiates light based on the reflected light L1 (see FIG. 1 ) reflected from the body part (wrist 2). Figure 4 ) A detector that outputs a plurality of detection signals. In the present embodiment, the plurality of photodetectors are a plurality of PDs 26, and the detection signals output from the plurality of PDs 26 correspond to the plurality of detection signals.

[0083] The memory 19 is a nonvolatile storage device, and a hard disk drive (HDD) is used, for example. The memory 19 stores therein a control program for controlling the entire operation of the wearable device 100. In addition, training data, historical information, etc. to be described later are stored in the memory 19. There is no limitation on the method for installing the control program, etc. in the wearable device 100.

[0084] The controller 12 controls the operation of each block included in the wearable device 100. The controller 12 is composed of hardware such as a CPU, a memory (RAM, ROM, etc.) required for a computer. Various processes are performed by loading a control program stored in the memory 19 into the RAM and executing the control program through the CPU.

[0085] For example, a programmable logic device (PLD) such as a field programmable gate array (FPGA) or other devices such as an application specific integrated circuit (ASIC) may be used as the controller 12 .

[0086] Will refer to Figures 4 to 7 An overview of a method for recognizing a gesture operation according to the present technology is described. Figure 41 is a schematic diagram for describing the operations of the light source section 21 and the PD array sensor 22 arranged on the rear surface 10 a of the main body 10 . Figure 5 A PD array sensor 22 is schematically shown.

[0087] Figure 5 A shows an example of arrangement of a plurality of PDs 26, and Figure 5 B schematically shows the intensity distribution of detection signals output from the plurality of PDs 26 .

[0088] In this embodiment, the wearable device 100 is worn close to the skin. Therefore, the light source section 21 and the PD array sensor 22 arranged on the rear surface 10a of the body 10 are arranged to face the skin 4 covering the blood vessels 3. A narrow gap is provided between the light source section 21 and the PD array sensor 22 and the surface 4a of the skin 4.

[0089] The light source section 21 irradiates the laser light L on the surface 4a of the skin 4. When the coherent light such as the laser light L is irradiated, the laser light L is diffused (scattered) due to the minute unevenness on the surface of the skin 4. Due to the interference of the diffused laser light L, speckles (light spots) are generated, and a speckle pattern including a plurality of speckles is generated. Therefore, a detection signal corresponding to the speckle pattern is output from each PD 26 of the PD array sensor 22.

[0090] For example, suppose Figure 5 The intensity of the detection signal of the white PD 26 shown in B is the highest, and the intensity of the detection signal of the black PD 26 is the lowest. In addition, the intensity of the detection signal of the gray PD 26 presents an intermediate value. Figure 5 Only two gray colors are schematically shown in B, but the intensity of the detection signal is of course not limited thereto. A detection signal having an intensity ranging from a minimum value to a maximum value is output according to the amount of incident light.

[0091] The intensity distribution of the plurality of detection signals output from the plurality of PDs 26 is an intensity distribution corresponding to the generated speckle pattern. In other words, the intensity distribution of the plurality of detection signals corresponds to the planar intensity pattern of the generated speckle.

[0092] The speckle pattern is a pattern corresponding to the shape of the surface 4a of the skin 4, and when the shape of the surface 4a changes, the speckle pattern also changes. For example, when the fingers move, the muscles and tendons connected to each other from the hand to the arm move, and the shape of the surface 4a of the skin 4 changes. In other words, when the fingers move, the speckle pattern generated by the irradiation laser L changes.

[0093] The inventors have noticed this and have newly discovered that the motion of the hand centered on the motion of the fingers is determined based on the motion of the muscles and tendons captured based on the speckle pattern. In other words, the inventors have newly discovered that the motion of the hand is determined based on, for example, the current speckle pattern and the time series changes in the speckle pattern.

[0094] As described above, the distribution of the intensity of the plurality of detection signals output from the PD array sensor 22 corresponds to the speckle pattern. Therefore, analysis of the intensity distribution of the plurality of detection signals output from the PD array sensor 22 enables capturing the movement of muscles and tendons and determining the movement of the hand.

[0095] Figure 6 An example of the change in hand movement and the change in the intensity distribution of the detection signal is schematically shown. Figure 6 As shown in A, bend the index finger of your left hand inward. Then, Figure 6 As shown in FIG. 1B , the muscles and tendons of the moving fingers are deformed, and the surface 4a of the skin 4 covering the muscles and tendons is deformed near the wrist 2. When the surface 4a of the skin 4 is deformed, the speckle pattern under the laser irradiation changes. Figure 6 As shown in C, the distribution of the intensities of the plurality of detection signals output from the PD array sensor 22 is changed. Note that Figure 6 C further schematically shows the distribution of the intensity of the detection signal.

[0096] Which parts of the muscles and tendons are deformed, and how the muscles and tendons are deformed vary depending on which finger is bent and how the finger is bent (the bending amount of the finger, the bending speed of the finger, etc.), so the way the speckle pattern changes is also different. Therefore, the distribution of the intensity of the multiple detection signals output from the PD array sensor 22 makes it possible to determine which finger is bent and how the finger is bent (the bending amount of the finger, the bending speed of the finger, etc.).

[0097] Figure 7 is a schematic diagram describing determinations regarding interactions between fingers and interactions between fingers and another object. Interactions between fingers include any interactions such as tapping fingers (colliding two fingers with each other) and rubbing fingers (rubbing fingers together). Interactions between fingers and another object (e.g., a table or a wall) include any interactions such as tapping an object with a finger or rubbing an object with a finger.

[0098] When there is interaction between fingers or interaction between fingers and an object, the muscles and tendons in the hand vibrate. Depending on the frequency of the vibration (vibration period), the shape of the surface 4a of the skin 4 changes, and the speckle pattern changes. For example, when the fingers tap the fingers to produce a sound, the muscles and tendons vibrate, and the speckle pattern changes at a frequency equal to the frequency of the sound. Of course, it is not limited to this.

[0099] For example, analysis of the periodicity of the time series changes in the speckle pattern makes it possible to determine the interaction between fingers or the interaction between fingers and an object. Figure 7 The analysis of the time series variation of the signal intensity schematically shown in makes it possible to determine the interaction between fingers or the interaction between fingers and an object. The analysis of the periodicity of the time series variation includes the analysis of any parameters related to the periodicity of the time series variation, such as the shape of the frequency distribution of the time series variation and the peak frequency of the period of the time series variation.

[0100] For example, Figure 7 As shown, frequency analysis is performed for the time series change of the signal intensity when the finger is at rest and the time series change of the signal intensity when the finger taps the table, etc. When comparing the power spectra, a peak is reached at a specific frequency when the finger taps the table, etc. The reason is that the muscles and tendons vibrate due to tapping with the finger, so the surface 4a of the skin 4 also vibrates, which causes vibration due to the vibration intensity.

[0101] The position between the muscles and tendons and the structure of the muscles and tendons depend on which fingers are interacted, the degree of bending of the fingers when interacting (e.g., the amount of bending), the speed of interaction, which finger is applied to the object, the degree of bending of the finger when it is applied to the object, the speed of the action, etc. Therefore, the vibration frequency of the muscles and tendons changes, and the frequency of the time series change of the speckle pattern (signal intensity), the peak frequency, the shape of the frequency distribution, etc. also change.

[0102] Therefore, analysis of the periodicity of the time series changes in signal intensity makes it possible to determine which fingers interact with each other, the degree of bending of the fingers when they interact (e.g., the amount of bending), the speed of the interaction, which finger is acting on the object, the degree of bending of the fingers when they act on the object, the speed of the movement, etc.

[0103] As described above, this embodiment enables determination of how a user moves his / her hand and what gesture the user makes. It is also possible to identify operations using gesture input. In other words, it is possible to identify gesture operations performed by a user with high accuracy. The identification operation may also be referred to as an estimation operation.

[0104] In the present embodiment, the distribution of the intensity of the detection signal output from the PD array sensor 22, the time series variation of the intensity of the detection signal, the periodicity of the time series variation (e.g., the frequency of the time series variation and the peak frequency of the period of the time series variation) are included in the “information about speckle generated due to light irradiation on a body part, which is included in a plurality of detection signals”. The information about speckle includes any information about the generated speckle, and includes, for example, spatial speckle pattern features and time series features that can be detected from the detection signal. The intensity of the detection signal itself is also included in the information about speckle, because the intensity of the detection signal itself is determined based on the speckle.

[0105] In the present embodiment, the determination of the user motion performed based on the distribution of the intensities of the plurality of detection signals is included in the determination of the user motion performed based on the speckle pattern. The determination of the user motion performed based on the time series changes in the plurality of detection signals is included in the determination of the user motion performed based on the time series changes in the speckle pattern. Furthermore, the determination of the user motion performed based on the periodicity of the time series changes in the plurality of detection signals is included in the determination of the user motion performed based on the periodicity of the time series changes in the speckle pattern.

[0106] Note that for the determination based on the time series change, the determination may be performed based on the time series change of some of the multiple detection signals. In other words, the determination based on the time series change in the speckle pattern (the peak frequency of the time series change) includes the determination based on the time series change in some of the multiple detection signals (the peak frequency of the time series change). In some cases, the determination of the user's motion is performed by focusing on the detection signal of a single PD 26.

[0107] Figure 8 is a block diagram showing an example of a software configuration of the wearable device 100. Fig. 9 is a flowchart illustrating an example of the operation of the wearable device 100. The detection signal output from the PD array sensor 22 may be referred to as a speckle signal hereinafter.

[0108] In the present embodiment, the speckle signal receiver 30, the speckle signal analyzer 31, the motion determination section 32, the operation recognition section 33, and the processing execution section 34 are implemented by the CPU of the controller 12 according to the program of the present technology. These blocks execute the information processing method according to the present technology. In order to implement each block, dedicated hardware can be used appropriately.

[0109] In addition, Figure 8 , the memory 19 is schematically shown in the controller 12 in order to easily understand the fact that the training data and the history information are read from the memory 19.

[0110] A plurality of detection signals (speckle signals) output from the PD array sensor 22 are received by the speckle signal receiver 30 (step 101). In the present embodiment, the speckle signal receiver 30 serves as a receiver.

[0111] The speckle signal analyzer 31 analyzes the speckle signal (step 102). In the present embodiment, the speckle pattern is analyzed by the speckle signal pattern analyzer 35. Furthermore, the time series variation of the speckle pattern over a plurality of times is analyzed by the time series speckle signal analyzer 36. As described above, the analysis result is included in the information on the speckle.

[0112] The motion determination section 32 determines the motion of the user's hand based on the analysis result performed by the speckle signal analyzer 31. In other words, the gesture made by the user is determined (step 103). In the present embodiment, the motion determination section 32 corresponds to a determination section that determines the motion of the user based on the information on speckles included in the plurality of detection signals.

[0113] In this embodiment, the user's motion (user's gesture) is determined according to a specified machine learning algorithm. For example, a machine learning algorithm using a neural network such as a recurrent neural network (RNN), a convolutional neural network (CNN), or a multilayer perceptron (MLP) is used. In addition, any machine learning algorithm that performs, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning can be used.

[0114] In the present embodiment, the memory 19 stores therein training data including correct answer labels (user's gestures) and results of analyzing speckle signals corresponding to the correct answer labels. The motion determination section 32 performs learning according to a specified machine learning algorithm using the stored training data. This makes it possible to improve the accuracy of determination of the user's gesture.

[0115] Fig.10 and 11 is a table of examples of gesture patterns that can be determined. Fig.10 As shown, the gesture of shaking fingers in the air can be determined as an independent movement of the fingers. The shaking movement of a single finger can be determined, and the shaking movement of multiple fingers can also be determined.

[0116] For example, the sensor area corresponding to the motion of each finger relative to the spatial direction information (i.e., the two-dimensional speckle pattern) can also be determined. In other words, the analysis target PD 26 and its detection signal can be appropriately selected from the plurality of PDs 26. For example, the sensor area corresponding to the motion of each finger can be set in advance based on the motion of the finger and the change of the speckle pattern. Of course, machine learning can be used to set the sensor area corresponding to the motion of each finger.

[0117] With respect to the time direction information, intensity changes occur in a period corresponding to the shaking, that is, time series changes of the detection signal output by each PD 26. Fig.10 In the example shown, the time change of the detection signal of the PD 26 in the sensor area corresponding to the movement of the index finger is analyzed. Then, shaking is determined based on the periodicity of the time change.

[0118] Fig.10 The features of the spatial direction information and the temporal direction information of the shaking shown are examples discovered by the inventors, and the determination of shaking is not limited to the case where these features are exhibited. For example, when other features are exhibited, shaking can also be determined based on other features. Of course, shaking can be determined by the input signal intensity and the time series change of the intensity without any change using machine learning.

[0119] Furthermore, based on the change in the intensity distribution of the detection signal, the sensor area with the largest change is calculated. The moving finger is determined based on the calculated sensor area, and the movement of the finger is determined based on the detection signal of the PD 26 included in the sensor area. This determination method is also possible.

[0120] In addition, a gesture of bending a finger can be determined as an independent movement of the finger. The movement of bending a single finger and the movement of bending multiple fingers can be determined. Fig.10 In the example shown, the temporal change of the detection signal of the PD 26 in the sensor area corresponding to the movement of the index finger is analyzed. Since the deformation amount of the surface 4a of the skin 4 increases due to the bending of the finger, the change in the intensity of the detection signal (the change in the intensity of the scattered light) increases. The bending motion can be determined based on such a feature. Of course, the determination of the bending motion is not limited to the case where such a feature is exhibited.

[0121] like Fig.10 As shown, the gesture of rubbing the fingers performed with the fingers and the action of tapping the fingers with the fingers can be determined as the movement performed by the fingers. Fig.10 In the example shown, the sensor area associated with the friction between the index finger and the thumb and the sensor area associated with the index finger tapping the thumb are set as analysis targets. For example, the sensor area associated with each movement can be set in advance based on each movement and the change of the speckle pattern. Of course, machine learning can be used to set the sensor area associated with each movement.

[0122] The PD 26 included in the sensor area associated with each movement and associated with the index finger, and the PD 26 included in the sensor area associated with the movement and associated with the thumb exhibit similar phenomena. In the present embodiment, the gesture is determined based on the fact that the peak frequency due to rubbing the fingers and the peak frequency due to tapping the fingers present unique values, respectively. In other words, the gesture of rubbing the fingers and the gesture of tapping the fingers are determined based on the peak frequency of the time series change in the detection signal of each PD 26. Of course, the determination of the gesture is not limited to the case where these features are exhibited.

[0123] exist Fig.10 In the example shown, a moving average (average value for each specified interval) of the intensity of the detection signal of the PD 26 is obtained as the "time direction information" of the "independent movement of the finger". The calculation of the average value for each specified interval makes it possible to perform smoothing on small time changes to macroscopically observe the trend of the signal. On the other hand, the change in the intensity of the detection signal of the PD 26 is obtained as the "time direction information" of the "movement performed with the finger" without any change. As described above, various methods can be adopted as a method for obtaining the "time direction information" from the detection signal.

[0124] like Fig.11 As shown, when a finger is used to perform a motion relative to an object, a gesture of rubbing the finger relative to the object can be determined. Rubbing with a single finger and rubbing with multiple fingers can be determined. Rubbing motions in different rubbing directions can also be determined separately.

[0125] exist Fig.11 In the example shown, one of the PDs 26 in the sensor area corresponding to the movement of the index finger is selected as the analysis object. The gesture of rubbing the finger against the object is determined based on the peak frequency of the time series change in the detection signal of the selected PD 26. Of course, the determination of the gesture is not limited to the case where such a feature is exhibited.

[0126] In addition, if Fig.11 As shown, the gesture of tapping an object with a finger can be determined as a movement performed with a finger relative to the object. Tap with a single finger and tap with multiple fingers can be determined.

[0127] exist Fig.11 In the example shown, one of the PDs 26 in the sensor area corresponding to the movement of the index finger is selected as the analysis object. The gesture of tapping the object with the finger is determined based on the peak frequency of the time series change in the detection signal of the selected PD 26. Of course, the determination of the gesture is not limited to the case where such a feature is exhibited.

[0128] Note that Fig.11As shown, the frequency and peak frequency of the time series changes in the detection signal vary depending on how the finger bends when the object is tapped with the finger (the amount of bending of the finger, the bending speed of the finger, etc.). Therefore, the analysis of the time series changes in the detection signal makes it possible to determine how the finger bends when the finger taps the object (the amount of bending of the finger, the bending speed of the finger, etc.).

[0129] Refer again Figure 6 The operation recognition section 33 recognizes the operation input by the user based on the determined gesture (step 104). In other words, the content of the input operation using the gesture is recognized.

[0130] For example, based on information such as the currently selected operation mode and the application being executed, it is determined what input operation the user's gesture corresponds to.

[0131] The process execution section 34 executes a process corresponding to the gesture determined by the motion determination section 32. Specifically, the process execution section 34 executes a process corresponding to the operation input by the user and recognized by the operation recognition section 33 (step 105). In the present embodiment, the execution section is realized by the operation recognition section 33 and the process execution section.

[0132] A specific example of recognizing a gesture operation and executing a process corresponding to the gesture operation is described below.

[0133] Fig.12 1 is a schematic diagram describing an example of a virtual remote controller using the present technology. Assume that a user is sitting in a room and the wearable device 100 according to the present technology is worn on the user's left hand, for example. Fig.12 As shown in A.

[0134] For example, the operation of the remote controller is assigned to gestures using the fingers of the left hand. This enables the TV, speakers, lighting, air conditioning, etc. in the room to be controlled using gestures of the fingers of the left hand.

[0135] For example, as shown below, gestures are associated with device operations.

[0136] -like Fig.12 As shown in B, a single finger gesture is performed to select a control target device by shaking it up and down.

[0137] -use Fig.12 Tap your finger on the C icon to select an action item

[0138] - Fig.12 D shows a finger-rubbing gesture for controlling an operation item.

[0139] The motion determination section 32 determines a gesture of shaking a single finger up and down. The operation recognition section 33 recognizes an input of an operation of selecting a control target device performed using a gesture. Here, it is assumed that a TV is selected as a control target device.

[0140] The motion determination section 32 determines the tapping of fingers against fingers. The operation recognition section 33 recognizes the input of the operation of selecting the operation item performed using the gesture. Here, it is assumed that the volume adjustment is selected.

[0141] The gesture of rubbing the finger is determined by the motion determination section 32. The operation recognition section 33 recognizes the input of the control operation item performed using the gesture. Assume, for example, that fast rubbing is designated as turning up the volume and slow rubbing is designated as turning down the volume. The operation recognition section 33 recognizes whether the input operation is instructing turning up the volume or turning down the volume.

[0142] The processing execution section 34 executes processing corresponding to the recognized input operation. In other words, when the input operation indicates turning up the volume, the processing execution section 34 sends a control signal for turning up the volume to the TV. When the input operation indicates turning down the volume, the processing execution section 34 sends a control signal for turning down the volume to the TV.

[0143] Use of the present technology as described above makes it possible to provide a virtual remote controller and achieve high usability. Allocation of remote control operations to gestures, control target devices, control methods, etc. can be arbitrarily set.

[0144] Fig.13 is a diagram illustrating an example of a virtual dialog box using the present technology. Fig.13 In the example shown, the wearable device 100 according to the present technology is worn on the right hand.

[0145] like Fig.13 As shown in A of FIG. 1 , the projector 14 of the wearable device 100 displays the rotating wheel UI 37 as a dialog box on a surface of a table, a wall, etc. For example, a gesture using the fingers of the right hand makes it possible to operate the displayed rotating wheel UI 37.

[0146] For example, the wheel to be rotated is selected according to the finger used, and the selected wheel is rotated using a gesture of rubbing the fingers. As a specific example, when a gesture of rubbing the surface with the thumb is determined, an operation of rotating the leftmost wheel is recognized, and the leftmost wheel is rotated. When a gesture of rubbing the surface with the index finger is determined, an operation of rotating the center wheel is recognized, and the center wheel is rotated. When a gesture of rubbing the surface with the middle finger is determined, an operation of rotating the rightmost wheel is recognized, and the rightmost wheel is rotated.

[0147] When an operation of tapping the surface with one finger is determined, an operation of inputting a value selected by rotating the wheel is recognized, and the selected value is input. For example, such a process is performed.

[0148] exist Fig.13 In the wearable device 100' shown in B, the display 38 serving as the display portion is provided on the outside of the wrist (back of the hand). For example, a gesture using the fingers of the right hand enables the rotation wheel UI 37' displayed on the display 38 to be operated.

[0149] As described above, gestures can be used to input operations on the GUI displayed on the projector 14 or the display 38. For example, high usability can be achieved by applying the present technology to a wearable device without a display portion, a wearable device with a small display portion, or a wearable device without an input portion.

[0150] This technology is not only applicable to the rotating wheel UI 37, but also to Fig.14 Various input UIs shown in A and B of . For example, the input target character or value is determined by which finger is used to select a column, and by how the finger is moved to select a row. Other gestures can be used if appropriate. In addition, the present technology can be applied to any GUI used as an input UI, such as a UI for selecting a desired selection label from a plurality of selection labels, or a UI for displaying a menu including a plurality of options when a drop-down button is selected.

[0151] Fig.15 is a diagram illustrating an example of a virtual keyboard using the present technology. Fig.15 In the example shown, the wearable device 100 is worn on the user's left hand.

[0152] Using ten fingers to tap on a table or the like, it is like an operation performed on an actual keyboard. This makes it possible to perform keyboard input. For example, a tap with a specified finger is determined, and the key for performing the operation input is identified based on which finger it is, how the finger is spread, how the finger is bent, etc. Then, input is performed on the corresponding key.

[0153] For example, the wearable device 100 according to the present technology is attached to each wrist of both hands so that it is possible to determine gestures using ten fingers and thus provide a virtual keyboard. This enables high usability.

[0154] Alternatively, if Fig.16 As shown, the camera 23 included in the wearable device 100 captures an image of a hand that is not wearing the wearable device 100. The movement of the hand can also be determined based on the captured image, and key input can be recognized. This method can be used to provide a virtual keyboard.

[0155] exist Fig.16In the example shown, the wearable device 100 is attached so that the main body 10 is located on the outside of the wrist (on the side of the back of the hand). As will be described later, the position where the main body 10 is fixed is not limited and can be set arbitrarily as long as it is arranged to face the skin.

[0156] Fig.17 3 is a diagram describing an example of a virtual flick input performed using the present technology. The flick input can be performed by using a gesture of a finger. For example, when it is determined that a certain portion is touched with a finger for a specified period of time or longer, a process of displaying a UI 39 for flick input around the portion is performed. Thereafter, when a gesture of rubbing the finger and a gesture of lifting the finger are determined, the selected character is identified based on how the finger is rubbed (rubbing direction), and the selected character is input.

[0157] Assume that the user selects a character by clicking on the UI for flick input. In this case, it is also possible to recognize which character is selected by detecting vibration or the like according to the tapping. This enables high usability.

[0158] like Figure 8 As shown, the recognition processing performed by the operation recognition section 33 can be performed using the history information stored in the memory 19. The history information is information about the operation input by the user in the past. The history information is not limited to information about the operation input using gestures, and may include information about the operation input by another method. In the present embodiment, the memory 19 in which the history information is stored corresponds to the history information memory.

[0159] For example, when executing Fig.15 When inputting keys of the virtual keyboard shown, the input data history of the past input is called historical information. For example, the input data is accumulated, and the next input is predicted based on the pattern of the input data. When the results of gesture-based recognition and prediction are combined to comprehensively recognize the input operation, this enables improved recognition accuracy.

[0160] Examples of prediction based on the pattern of input data include methods using n-grams. In n-grams, the frequencies of occurrence of N consecutive input data strings are stored, the probability of the input following N-1 input strings is obtained using the distribution of the frequencies of occurrence, and the highest is determined as the estimated result. For example, assume that "son" is input. Here, assume that the distribution of the frequencies of occurrence of the following inputs is as follows: sony is 7, song is 10, sona is 2, and sone is 1. Therefore, "g" is determined as the estimated result (with a probability of 1 / 2).

[0161] This results in being able to recognize the next input character with high accuracy when the estimation results obtained by gesture determination, distribution of the occurrence frequency of subsequent inputs, etc. are comprehensively determined. Note that the reliability of gesture determination, the reliability of the operation recognized based on gesture determination, etc. can be calculated, and these parameters can be used when performing comprehensive determination.

[0162] The historical information can also be used to identify the operation when, for example, an application other than the virtual keyboard is executed. The specific type of the historical information is not limited, and any information can be used as the historical information. In addition, the method is not limited to the method using N-gram, and any method can be used.

[0163] Fig.18 An example of a selection image is schematically shown. The selection image is an image including a plurality of selection candidates selectable by a user, and corresponds to a GUI for selecting candidates.

[0164] Based on the gesture determination, a plurality of operations can be identified as a plurality of candidates by the operation recognition section 33. For example, it is assumed that when key input using the virtual keyboard is performed, two operations, an operation of inputting "H" and an operation of inputting "J", are identified as candidates based on the gesture determination. In this case, a selection image 40 having "H" and "I" as selection candidates is displayed, as shown in FIG. Fig.18 It can also be said that the selection image 40 is used to inform the user that "H" and "I" have been identified as candidate operations.

[0165] Note that the selection image 40 also includes "Others", with which an input indicating that it is not a selection candidate can be performed. The user selects one of "H", "I" and "Others" by operating the selection image 40. Of course, gestures can also be used to input operations. When there are multiple candidates, the display of the selection image 40 also makes it easy to input operations. Moreover, even if the operation is incorrectly recognized, it is sufficient for the user to take only a single step of selecting "Others" instead of taking the two steps of "delete" and "reselect". This enables a high degree of usability.

[0166] Note that an input indicating that it is not a selection candidate may also be designated as a special operation instead of selecting “other.” For example, when two taps are input simultaneously, an input indicating that the taps are not a selection candidate in the selection image 40 is performed.

[0167] The selection image 40 may also be displayed when, for example, an application other than the virtual keyboard is executed. The selection image including a plurality of selection candidates for each application may be appropriately displayed.

[0168] Note that when the reliability of gesture determination, the reliability of the operation recognized based on gesture determination, etc. is calculated, the selection image is displayed when the reliability is not greater than a specified threshold. Such processing is also possible.

[0169] refer to Fig.19 and 20 An example of generating training data is described. Fig.19 As shown, the controller 12 includes an instruction processing execution section 45 and a training data generator 46. The instruction processing execution section 45 instructs the user to perform a designated movement (make a designated gesture).

[0170] The training data generator 46 generates training data including the result of analyzing the speckle signal (information about speckle) output from the PD array sensor 22 when the user makes a designated gesture that the user is instructed to make. In other words, the gesture that the user has been instructed to make is set as a correct answer label, the correct answer label is associated with the analysis result performed when the user makes the gesture, and the associated correct answer label and analysis result are generated as training data. In the present embodiment, the training data corresponds to the determination information, and the memory 19 in which the training data is stored corresponds to the determination information memory.

[0171] For example, the projector 14 of the wearable device 100 will Fig.14 The UI for input shown in B is displayed on a display surface of a table etc. Then, for example, an instruction of the following indication is given by displaying a text image or by outputting a sound from the speaker 13.

[0172] (1) Use your index finger to tap the display surface, just like tapping the number "4".

[0173] (2) Move the index finger upward and perform a tap as if tapping a "7" located at the next higher position.

[0174] (3) Move your index finger downward and perform a tap as if you were tapping the “1” located at the next lower position.

[0175] (4) Move the index finger further downward and perform tapping as if tapping the "0" located at the next lower position.

[0176] The instruction processing execution part 45 outputs information about the gesture corresponding to each of the instructions (1) to (4), that is, information related to the correct answer label, to the training data generator 46. The training data generator 46 generates training data by associating the received correct answer label with the analysis result performed when the gesture is made. In this example, the instruction part is implemented by the projector 14, the speaker 13 and the instruction processing execution part 45.

[0177] Note that there is no restriction on what gesture the user is to make, and it can be set arbitrarily. There is also no restriction on how to make the gesture, and any method can be adopted.

[0178] For example, when the wearable device 100 starts to be used or when a designated UI starts to be used, an instruction on the UI to perform an operation (make a gesture) is given as an initial calibration. Then, training data is generated, and the motion determination part 32 is made to perform learning. Thus, the influence of individual differences such as the size of the hand and the length of the fingers can be suppressed, thereby performing gesture determination and operation recognition suitable for each person.

[0179] exist Fig. 20 In the example shown, true-or-false information on whether a determination result obtained by performing gesture determination is correct is input through the user input section 47. The user input section 47 can be implemented with any configuration capable of inputting true-or-false information.

[0180] For example, by Figure 3 Operations are performed on the UI displayed by the projector 14 shown in the figure, or by Figure 3 Operations are performed on the touch panel 15 shown, or by Figure 3 By performing an operation on the operation button 16 shown, correct-or-wrong information can be input. For example, selection of the delete key "Del" displayed by the projector 14 or the touch panel 15, selection of a designated delete button included in the operation button 16, etc. can be performed. In addition, a microphone or the like can be included, and correct-or-wrong information can be input by voice input being performed by the user. Alternatively, correct-or-wrong information can be input by performing a special gesture operation.

[0181] True-or-false information about whether the determination result obtained by performing gesture determination is correct is generally information indicating whether the processing performed by the processing execution part is correct. When a process that the user does not want by the gesture operation is performed, information indicating this event is input as true-or-false information. For example, when "H" has been input to the virtual keyboard but "J" has actually been input, true-or-false information indicating that the determination result obtained by performing gesture determination is wrong is input. Other information may be used as true-or-false information.

[0182] Of course, the input of information indicating that the determination result is an incorrect answer is not restrictive, and information indicating that the determination result is a correct answer may be input. For example, after inputting the correct-or-wrong information of the incorrect answer, the process is performed again, and the information of the correct answer is input to perform the input indicating that the process is correct. Of course, this case is not restrictive.

[0183] The true-or-false information input by the user is received by a feedback receiver 48 included in the controller 12. The feedback receiver 48 outputs the true-or-false information to the training data generator 46. In the present embodiment, the feedback receiver 48 functions as a receiving section.

[0184] For example, the training data generator 46 generates training data based on the determination result performed by the motion determination section 32 and the analysis result output from the speckle signal analyzer 31, and stores the training data in the memory 19. Here, reference is made to the true-or-false information output from the feedback receiver 48. For example, when true-or-false information indicating a wrong answer is received, the training data generated when the process of receiving feedback of the wrong answer is performed is updated.

[0185] Specifically, thereafter, the process is performed again and feedback of the correct answer is received. In this case, the training data can be updated by updating the correct answer label to the gesture associated with the process performed again. If it is not possible to update the training data, the training data can be discarded.

[0186] This reception of true-or-false information from the user enables the training data to be updated, and enables the motion determination section 32 to accurately perform learning. Furthermore, the influence of individual differences between users can be suppressed, and gesture determination and operation recognition suitable for each person can be performed.

[0187] As described above, in the wearable device 100 according to the present embodiment, light is irradiated onto a body part of the user, and a plurality of detection signals are output based on the reflected light thereof. Then, based on the information about speckles included in the plurality of detection signals, that is, based on the result of analyzing the speckle signals, the user's gesture is determined. This results in being able to provide various applications described in the above examples, and achieving high usability.

[0188] For example, there is a method that includes scanning a laser beam from a watch-type wearable device and detecting the movement of a fingertip entering an irradiated area based on, for example, a change in reflected light. However, it is difficult to detect the movement of a hand wearing the wearable device by this method. Therefore, it is necessary to use the other hand for gesture operation, which is not very convenient for the user.

[0189] Furthermore, in the case of a method of determining a gesture using, for example, an acceleration sensor, it is difficult to perform correct determination unless the arm is greatly moved, and this results in a heavy burden on the user. Furthermore, it is impossible to determine a fine gesture made using a finger.

[0190] In addition, there is a method of determining a gesture by attaching a cap side sensor to a finger. However, the cap side sensor is uncomfortable to install, which puts a burden on the user. In addition, it is necessary to install finger caps according to the number of fingers desired to be sensed, and this leads to an increase in the cost of introducing the device.

[0191] In addition, there is a method of detecting the sound of finger friction using, for example, a microphone. However, in this method, a sensor such as a microphone is required, and this leads to an increase in the cost of introducing the device.

[0192] The wearable device 100 according to the present embodiment enables gestures to be determined with high accuracy based on information related to speckles generated by irradiating light onto a body part (i.e., a speckle pattern); time series changes in the speckle pattern; peak frequency of the speckle pattern; etc., and is capable of identifying input operations with high accuracy.

[0193] Therefore, gesture operations can be input using only the hand wearing the wearable device 100, and fine movements of fingers, for example, can be captured. This leads to providing very high usability. In addition, the tapping gesture of the hand and the rubbing gesture of the hand can be determined without using a device such as a microphone or a cap-type sensor, and this makes it possible to reduce the cost of introducing the device.

[0194] In addition, in the present embodiment, the PD array sensor 22 is used as a detector. Therefore, compared with using an image sensor such as a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor, a change in the amount of light can be detected in a short time. Thus, the user's gesture can be determined with high accuracy.

[0195] In addition, the PD array sensor 22 can be operated with lower power consumption than when using an image sensor, thereby driving the PD array sensor 22 for a longer time. In addition, since the PD array sensor 22 is sensitive, the power of the laser emitted to the body part can be reduced, and this provides an advantage of reducing power consumption. It is also possible to reduce the size of the condenser, etc., thereby reducing costs.

[0196] In addition, the influence of the brightness of the environment in which the wearable device 100 is used can be fully suppressed. For example, in the case of performing image capture of the movement of the hand using an image sensor, if the surrounding environment is dark, the recognition accuracy is reduced. In the present embodiment, the light source part 21 and the PD array sensor 22 are configured very close to the body part, and the intensity of the reflected light is detected. Therefore, it is possible to analyze the speckle pattern and the like with high accuracy without being affected by the surrounding brightness.

[0197] The present technology using an image sensor can also be provided. In other words, it is also possible to determine a user's gesture based on a speckle pattern and recognize an input operation, wherein the image sensor captures an image of the speckle pattern. In this case, the image sensor is used as a detector, and a plurality of pixels of the image sensor are used as a plurality of light detectors. In addition, a plurality of pixel signals output from a plurality of pixels correspond to a plurality of detection signals. The use of an image sensor enables improvement in spatial resolution.

[0198] Other Implementations

[0199] The present technology is not limited to the above-described embodiments, and various other embodiments can be implemented.

[0200] exist Figure 8 In the example of the configuration shown, the determination information memory and the history information memory are implemented by the memory 19, and motion determination processing using training data and operation recognition processing using history information are performed. This configuration is not limited to this, and in the embodiment of the information processing device according to the present technology, a configuration including only one of the determination information memory and the history information memory and a configuration not including the determination information memory or the history information memory are also included. In other words, motion determination processing may be performed without using determination information such as training data, or operation recognition processing may be performed without using history information. Similarly, for Fig.19 and Fig. 20 In the example of the configuration in, a configuration without history information storage can be adopted as an implementation of the information processing device according to the present technology.

[0201] In the above description, the determination information memory and the history information memory are implemented by the memory 19, but the determination information memory and the history information memory may be separately implemented by different memories or the like.

[0202] In the above description, based on the result of analyzing the speckle signal, machine learning is used to determine the user's motion (user's gesture). Then, the operation input by the user is recognized based on the result of the determination. The configuration is not limited thereto, and machine learning may be used to recognize the operation input by the user based on the result of analyzing the speckle signal. In other words, estimation of what operation is input may be performed without determining how the user moves.

[0203] In this case, for example Figure 8 The motion determination section 32 and the operation recognition section 33 shown are integrated to form an input estimator. Based on the result of analyzing the speckle signal, the input estimator can use machine learning to recognize the input operation. Such a configuration also corresponds to an embodiment of the recognition device and the information processing device according to the present technology. Such a configuration also enables processing to be performed similarly using historical information and training data.

[0204] In the above description, gesture determination and operation recognition are performed based on the analysis results of the speckle signal. Alternatively, gesture determination and operation recognition can be performed using, for example, machine learning based on the detection signal output from the PD array sensor. In other words, the analysis step can be omitted.

[0205] Gesture determination and operation recognition can also be performed without using machine learning. In other words, rule-based gesture determination and rule-based operation recognition can be performed on the basis of rules using, for example, table information. When machine learning is not used, it is also possible to appropriately use, for example, Figure 8 The data corresponding to the training data shown in FIG. 1 and the data are used as decision information. Of course, any learning algorithm other than machine learning can also be used.

[0206] An example using a laser light source as the light source section has been described above. Note that the present technology is also applicable when another coherent light source capable of emitting coherent light is used.

[0207] In the above description, a PD array sensor including a plurality of PDs arranged two-dimensionally is used. The configuration is not limited thereto, and a PD array sensor including a plurality of PDs arranged one-dimensionally may be used.

[0208] Fig.21 Other examples of positions where the subject is fixed are schematically shown. Fig.21 As shown in A of FIG. 1 , the wearable device 100 may be attached so that the main body 10 is located on the outside of the wrist (on the side of the back of the hand). Fig.21 As shown in A, the main bodies 10a and 10b can also be fixed on the inside and outside of the wrist, respectively. In other words, analysis of speckle patterns and the like can be performed at two locations, the inside and the outside of the wrist. The setting positions of the light source parts and the detectors, the number of light source parts, and the number of detectors are not limited and can be set arbitrarily.

[0209] The attachment state of the wearable device can be determined based on the analysis results of the speckle signal. For example, a process including detecting the positional deviation of the main body (light source part and detector) and prompting the user to confirm the installation state can also be performed. Gesture determination and operation recognition can also be performed with reference to information about the position of the main body (light source part and detection part).

[0210] Although a wristband wearable device has been described as an example, the application of the present technology is not limited to this. The present technology is applicable to various wearable devices, such as a bracelet type to be worn around the upper arm, a headband type worn on the head (head-mounted), a neckband type worn around the neck, a body type worn on the chest, a belt type worn on the waist, an ankle type worn around the ankle, a watch type, a ring type, a necklace type, an earring type, and an earring type. Of course, there is no limitation on the part irradiated with light, and the part irradiated with light can be selected arbitrarily.

[0211] The user's movements that can be determined using this technology are not limited to the user's movements. By appropriately setting the part to which the wearable device is attached, that is, the part irradiated with light, the movement of any part such as the foot (thigh, knee, calf, ankle, toe), neck, waist, hip, arm, head, face, and chest can be determined.

[0212] Use of the present technology also makes it possible to perform the processing shown below.

[0213] - Sign language interpretation: capture of hand, arm and finger movements and oral-signed representation (e.g. displayed on a smartphone).

[0214] - Behavior recognition: Capture the action of holding or gripping an object and recognize the user's behavior (e.g., spoon or chopstick movements, PC operation, driving a vehicle, and holding a belt)

[0215] - Recording by pen-based input: When writing with a pen, for example in a notebook, the movement of the pen in the electronic medium is recorded.

[0216] The user's motion may be determined by combining, for example, a PD array sensor and an acceleration sensor. For example, the motions determined separately based on the results obtained by each sensor may be integrated, or the user's motion may be determined using, for example, machine learning using the results obtained by each sensor as input.

[0217] The information processing method and program according to the present technology can be executed not only in a computer system including a single computer, but also in a computer system in which a plurality of computers operate in collaboration. Note that in the present disclosure, a system refers to a group of components (e.g., devices and modules (components), regardless of whether all components are in a single housing. Therefore, a plurality of devices housed in a single housing and connected via a network and a single device in which a plurality of modules are housed in a single housing are both systems.

[0218] For example, all or some functions of each module included in the controller 12 may be performed by a cloud server. In addition, the information processing method according to the present technology may be performed by cooperatively operating a plurality of computers capable of communicating with each other. In addition, executing each process by a designated computer includes causing another computer to execute some or all of the processes and obtain the results thereof.

[0219] The information processing method and program according to the present technology are also applicable to the configuration of cloud computing in which a plurality of devices share the task of a single function and work in cooperation to execute the single function through a network.

[0220] The corresponding configurations of the wearable device described with reference to the various figures, the control flow executed, etc. are merely implementations and can be modified arbitrarily without departing from the spirit of the present technology. In other words, for example, any other configuration or algorithm for practicing the present technology can be adopted.

[0221] It is also possible to combine at least two features of the present technology described above. In other words, the various features described in the various embodiments can be arbitrarily combined regardless of the embodiment. In addition, the various effects described above are not restrictive, but merely illustrative, and other effects may be provided.

[0222] Note that the present technology can also adopt the following configurations.

[0223] (1) An information processing device comprising: a light source portion for irradiating light thereto

[0224] The user's body part;

[0225] a detector including a plurality of light detectors and outputting a plurality of detection signals based on reflected light reflected from a body part; and

[0226] The determination section determines the motion of the user based on information about speckles generated due to light being irradiated on the body part, the information about the speckles being included in the plurality of detection signals.

[0227] (2) The information processing device according to (1), wherein

[0228] The light source part irradiates the laser onto the body part.

[0229] (3) The information processing device according to (1) or (2), wherein

[0230] The plurality of light detectors is a plurality of photodiodes.

[0231] (4) The information processing device according to any one of (1) to (3), wherein

[0232] The determination section determines the motion of the user based on the speckle pattern included in the information about the speckle.

[0233] (5) The information processing device according to (4), wherein

[0234] The determination section determines the motion of the user based on the time-series change of the speckle pattern.

[0235] (6) The information processing device according to (5), wherein

[0236] The determination section determines the motion of the user based on the periodicity of the time-series change of the speckle pattern.

[0237] (7) The information processing device according to any one of (1) to (6), wherein the body part is a wrist, and

[0238] The determination section determines the movement of the user's hand.

[0239] (8) The information processing device according to (7), wherein

[0240] The determination section determines at least one of which finger is bent, the bending amount of the bent finger, an interaction between fingers, or an interaction between a finger and another object.

[0241] (9) The information processing device according to any one of (1) to (8), further comprising:

[0242] An execution section is included that executes a process corresponding to the determined motion.

[0243] (10) The information processing device according to (9), wherein

[0244] The execution section recognizes an operation input by a user based on the determined motion, and executes a process corresponding to the recognized operation.

[0245] (11) The information processing device according to (10), further comprising a history information storage device in which history information about operations input by a user in the past is stored, wherein

[0246] The execution section recognizes an operation input by a user based on the stored history information.

[0247] (12) The information processing device according to (10) or (11), further comprising a display section capable of displaying a specified graphical user interface (GUI), wherein

[0248] The execution section recognizes an operation input into the displayed specified GUI based on the determined motion.

[0249] (13) The information processing device according to (12), wherein

[0250] The execution section selects a plurality of selection candidates that can be selected by the user based on the determined motion, and

[0251] The display section displays a selection image including the selected plurality of selection candidates.

[0252] (14) The information processing device according to any one of (1) to (13), wherein

[0253] The determination section determines the user's motion according to a specified learning algorithm.

[0254] (15) The information processing device according to any one of (1) to (14), further comprising:

[0255] an instruction portion, instructing the user to perform a specified movement; and

[0256] a determination information storage storing determination information including information on speckles, the information on speckles being included in the plurality of detection signals and being information when the user performs a designated motion that has been instructed to perform, wherein the determination section determines the motion of the user based on the stored determination information.

[0257] (16) The information processing device according to any one of (1) to (15), further comprising a receiving section that receives true-false information as to whether a result of determination performed by the determination section is correct, wherein

[0258] The determination section determines the user's motion based on the received correct information.

[0259] (17) The information processing device according to (1), wherein:

[0260] The detector includes an image sensor, and

[0261] The plurality of light detectors are the plurality of pixels of the image sensor.

[0262] (18) An information processing device comprising:

[0263] a receiver that receives a plurality of detection signals output based on reflected light reflected off the body part of the user due to the light being irradiated on the body part of the user; and

[0264] The determination section determines the motion of the user based on information about speckles generated due to light being irradiated on the body part, the information about the speckles being included in the plurality of received detection signals.

[0265] (19) An information processing method comprising:

[0266] receiving, by the computer system, a plurality of detection signals output based on reflected light that is reflected off a body part of the user as a result of light being irradiated on the body part of the user; and

[0267] The motion of the user is determined by the computer system based on information related to speckle generated by light being irradiated on the body part, the information related to the speckle being included in the received plurality of detection signals.

[0268] (20) A program for causing a computer system to execute a process, comprising:

[0269] receiving a plurality of detection signals output based on reflected light that is reflected off a body part of the user due to light being irradiated on the body part of the user; and

[0270] The motion of the user is determined based on information about speckles generated due to light being irradiated on the body part, the information about the speckles being included in the received plurality of detection signals.

[0271] Reference Mark List

[0272] L Laser

[0273] L1 reflected light

[0274] 2 Wrist

[0275] 10 Main Body

[0276] 11 Connecting belt

[0277] 12 Controller

[0278] 19 Memory

[0279] 21 Light source

[0280] 22 PD array sensor

[0281] 26 PD

[0282] 30 Speckle signal receiver

[0283] 31 Speckle Signal Analyzer

[0284] 32 Motion determination part

[0285] 33 Operation Identification Section

[0286] 34 Processing execution part

[0287] 35 Speckle Signal Pattern Analyzer

[0288] 36 Time Series Speckle Signal Analyzer

[0289] 40 Select Image

[0290] 45 Instruction processing execution part

[0291] 46 Training Data Generator

[0292] 100, 100' wearable device

Claims

1. An information processing device, comprising: a light source portion for irradiating light onto a body part of a user; a detector including a plurality of light detectors and outputting a plurality of detection signals based on reflected light reflected from the body part; as well as a determination section that determines the user's motion based on information about speckles generated due to the light being irradiated on the body part, the information about the speckles being included in the plurality of the detection signals, wherein The light source section irradiates laser light onto the body part, the determination section determines a motion of the user based on a speckle pattern included in the information about the speckle, and The change in the speckle pattern is based on deformation of the surface of the skin covering the muscles and tendons, and the intensity distribution of the plurality of the detection signals corresponds to the speckle pattern, and the change in the speckle pattern causes the distribution of the intensities of the plurality of the detection signals to be changed.

2. The information processing device according to claim 1, wherein The plurality of light detectors are a plurality of photodiodes.

3. The information processing device according to claim 1, wherein The determination section determines the motion of the user based on a time-series change of the speckle pattern.

4. The information processing device according to claim 3, wherein The determination section determines the motion of the user based on periodicity of a time-series change of the speckle pattern.

5. The information processing device according to claim 1, wherein The body part is a wrist, and The determination section determines a movement of a hand of the user.

6. The information processing device according to claim 5, wherein The determination section determines at least one of which finger is bent, an amount of bending of the bent finger, an interaction between fingers, or an interaction between a finger and another object.

7. The information processing apparatus according to claim 1, further comprising an execution section that executes a process corresponding to the determined motion.

8. The information processing device according to claim 7, wherein The execution section recognizes an operation input by the user based on the determined motion, and executes a process corresponding to the recognized operation.

9. The information processing apparatus according to claim 8, further comprising a history information storage in which history information about operations input by the user in the past is stored, wherein The execution section recognizes an operation input by the user based on the stored history information.

10. The information processing apparatus according to claim 8, further comprising a display section capable of displaying a specified graphical user interface, wherein The execution section recognizes an operation input into a displayed designated graphical user interface based on the determined motion.

11. The information processing device according to claim 10, wherein The execution section selects a plurality of selection candidates selectable by the user based on the determined motion, and The display section displays a selection image including the selected plurality of selection candidates.

12. The information processing device according to claim 1, wherein The determination section determines the motion of the user according to a specified learning algorithm.

13. The information processing device according to claim 1, further comprising: An instruction part, instructing the user to perform a specified movement; as well as a determination information storage storing determination information including information on speckles, the information on speckles being included in the plurality of detection signals and being information when the user performs the designated motion that has been instructed to perform, wherein The determination section determines the motion of the user based on the stored determination information.

14. The information processing apparatus according to claim 1, further comprising a receiving section that receives true-or-false information as to whether a result of determination performed by the determination section is correct, wherein The determination section determines the user's motion based on the received true-or-false information.

15. The information processing device according to claim 1, wherein: The detector includes an image sensor, and The plurality of light detectors are a plurality of pixels of the image sensor.

16. An information processing device, comprising: a receiver that receives a plurality of detection signals output based on reflected light reflected off the body part of the user due to the light being irradiated on the body part of the user; as well as a determination section that determines the motion of the user based on information about speckles generated due to the light being irradiated on the body part, the information about the speckles being included in the plurality of received detection signals, wherein The laser light is irradiated onto the body part by the light source section, the determination section determines the motion of the user based on the speckle pattern included in the information about the speckle, and The change in the speckle pattern is based on deformation of the surface of the skin covering the muscles and tendons, and the intensity distribution of the plurality of the detection signals corresponds to the speckle pattern, and the change in the speckle pattern causes the distribution of the intensities of the plurality of the detection signals to be changed.

17. An information processing method, comprising: receiving, by a computer system, a plurality of detection signals output based on reflected light that is reflected off a body part of a user as a result of light being irradiated on the body part of the user; as well as The computer system determines the movement of the user based on information related to speckle generated by the light being irradiated on the body part, the information related to the speckle being included in the received plurality of detection signals, wherein The laser is irradiated onto the body part by the light source portion, and the motion of the user is determined based on the speckle pattern included in the information about the speckle, and The change in the speckle pattern is based on deformation of the surface of the skin covering the muscles and tendons, and the intensity distribution of the plurality of the detection signals corresponds to the speckle pattern, and the change in the speckle pattern causes the distribution of the intensities of the plurality of the detection signals to be changed.

18. A storage medium comprising a program that, when executed, causes a computer system including the storage medium to perform a process comprising: receiving a plurality of detection signals output based on reflected light that is reflected off a body part of the user as a result of light being irradiated on the body part of the user; as well as The motion of the user is determined based on information related to speckle generated by the light being irradiated on the body part, the information related to the speckle being included in the received plurality of detection signals, wherein The laser is irradiated onto the body part by the light source portion, and the motion of the user is determined based on the speckle pattern included in the information about the speckle, and The change in the speckle pattern is based on deformation of the surface of the skin covering the muscles and tendons, and the intensity distribution of the plurality of the detection signals corresponds to the speckle pattern, and the change in the speckle pattern causes the distribution of the intensities of the plurality of the detection signals to be changed.

Images