Information processing device, information processing method, and program

Abstract

[Problem] To execute actions corresponding to gestures with more complex expressiveness. [Solution] An information processing device comprises: a recognition unit that, on the basis of a depth map of a photographic subject, recognizes gestures using a detection target based on the photographic subject; a calculation unit that calculates, on the basis of time change of the depth map, the velocity or acceleration of the detection target performing the gestures; and an execution unit that executes actions corresponding to the gestures with a strength of expression based on the velocity or acceleration that has been calculated.

Description

Information processing device, information processing method, and program

[0001] The present disclosure relates to an information processing device, an information processing method, and a program.

[0002] In recent years, development of gesture user interfaces (UIs) has been progressing, which recognize user gestures by tracking the movements of a user's hands and fingers and execute processes corresponding to the recognized gestures. By using the gesture UI, a user can cause an information processing device to execute processes corresponding to the gestures by intuitive movements of their hands or fingers.

[0003] For example, Patent Document 1 listed below discloses tracking the movement of a user's hand, finger, or other object using depth image data obtained from a depth sensor.

[0004] JP 2013-37675 A

[0005] However, with the technology disclosed in Patent Document 1, it is difficult to obtain information representing the sense of force of a gesture made by a user, and therefore, with the technology disclosed in Patent Document 1, it is difficult to express the sense of force of a gesture, and the expressive power of actions corresponding to gestures is limited.

[0006] Therefore, the present disclosure provides a new and improved information processing device, information processing method, and program that are capable of executing actions corresponding to gestures with more complex expressiveness.

[0007] According to the present disclosure, an information processing device is provided that includes a recognition unit that recognizes a gesture made by a subject using a detection target based on a depth map of the subject, a calculation unit that calculates the speed or acceleration of the detection target making the gesture based on changes over time in the depth map, and an execution unit that executes an action corresponding to the gesture using an intensity expression based on the calculated speed or acceleration.

[0008] In addition, according to the present disclosure, there is provided an information processing method by a computer, which includes recognizing a gesture made by a subject using a detection target based on a depth map of the subject, calculating a speed or acceleration of the detection target making the gesture based on changes in the depth map over time, and performing an action corresponding to the gesture using an intensity expression based on the calculated speed or acceleration.

[0009] In addition, according to the present disclosure, a program is provided for causing a computer to function as a recognition unit that recognizes a gesture made by a subject using a detection target based on a depth map of the subject, a calculation unit that calculates the speed or acceleration of the detection target making the gesture based on changes over time in the depth map, and an execution unit that executes an action corresponding to the gesture using an intensity expression based on the calculated speed or acceleration.

[0010] 10 is a block diagram showing a functional configuration of an information processing device according to an embodiment of the present disclosure. FIG. 11 is an explanatory diagram showing the state of each frame of a depth map acquired in time series by a ranging sensor unit. FIG. 2 is a graph showing a position change of a detection target in each frame shown in FIG. 2. FIG. 3 is a graph showing a velocity change of a detection target calculated from the position change shown in FIG. 3. FIG. 4 is a graph showing an acceleration change of a detection target calculated from the velocity change shown in FIG. 4. FIG. 5 is a flowchart showing the flow of a first operation example of the information processing device according to the embodiment. FIG. 6 is a flowchart showing the flow of a second operation example of the information processing device according to the embodiment. FIG. 7 is an explanatory diagram showing the configuration of an information processing system including the information processing device according to the embodiment. FIG. 12 is a block diagram showing an overview of a ToF sensor included in a ranging sensor unit. FIG. 13 is a schematic plan view showing the configuration of a two-dimensional pixel array of the ToF sensor. FIG. 14 is a circuit diagram showing an equivalent circuit of the two-dimensional pixel array shown in FIG. 10 is a timing chart showing the operation of the two-dimensional pixel array shown in FIG. 10 is a schematic plan view showing the configuration of a two-dimensional pixel array according to a modified example. FIG. 15 is a block diagram showing an example of the hardware configuration of the information processing device according to the embodiment.

[0011] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0012] The description will be given in the following order: 1. Information processing device 1.1. Configuration example 1.2. Operation example 1.3. Application example 2. Distance measurement sensor unit 2.1. Overview 2.2. Two-dimensional pixel array 2.3. Modification example 3. Hardware configuration

[0013] 1. Information Processing Apparatus> (1.1. Configuration Example) First, a configuration example of an information processing apparatus according to an embodiment of the present disclosure will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the functional configuration of an information processing apparatus 100 according to this embodiment.

[0014] As shown in FIG. 1, the information processing device 100 according to this embodiment includes a distance measurement sensor unit 110, a recognition unit 120, a calculation unit 130, an execution unit 140, a display unit 150, and a communication unit 160.

[0015] The information processing device 100 recognizes a gesture of a user, who is a subject, and executes an action corresponding to the recognized gesture. Specifically, the information processing device 100 recognizes the gesture of the subject using the recognition unit 120 based on a depth map of the subject acquired by the distance measurement sensor unit 110, and executes an action corresponding to the recognized gesture using the execution unit 140. The information processing device 100 also calculates the speed or acceleration of a detection target (e.g., a hand, finger, or arm) making a gesture based on a time change in the depth map of the subject using the calculation unit 130, and can control the intensity expression of the action based on the calculated speed or acceleration using the execution unit 140. By controlling the intensity expression of the action corresponding to the gesture, the information processing device 100 can execute the action corresponding to the gesture with more complex expressiveness.

[0016] The ranging sensor unit 110 acquires a depth map of the subject. Specifically, the ranging sensor unit 110 includes a time-of-flight (ToF) sensor and acquires a depth map that represents the depth to the subject by measuring the distance to the subject based on the time of flight of light emitted to the subject. For example, the ranging sensor unit 110 may acquire a depth map of the entire angle of view including the subject by emitting light over the entire angle of view including the subject and receiving reflected light of the emitted light using a two-dimensional pixel array.

[0017] The ranging sensor unit 110 can acquire a depth map of a subject at a high frame rate. Therefore, the ranging sensor unit 110 can detect temporal changes in the depth of the subject in the depth map in more detail. Details of the ToF sensor included in the ranging sensor unit 110 will be described later.

[0018] The ranging sensor unit 110 may output the entire depth map or may extract and output a portion of the depth map to the downstream recognition unit 120. For example, the ranging sensor unit 110 may extract a target region including a subject, a moving body, an object, or the like from the acquired depth map and output the extracted portion of the depth map to the downstream recognition unit 120. Furthermore, the ranging sensor unit 110 may output metadata obtained by image processing or AI (Artificial Intelligence) processing the acquired depth map to the downstream recognition unit 120.

[0019] The recognition unit 120 recognizes gestures made by a subject based on a depth map acquired by the ranging sensor unit 110. Specifically, the recognition unit 120 recognizes a detection target, such as a user's hand, finger, or arm, which is the subject, from all or part of the depth map input from the ranging sensor unit 110, and recognizes a gesture using the detection target. A gesture using the detection target is, for example, moving the detection target (such as the user's hand, finger, or arm) in a predetermined motion or assuming a predetermined pose with the detection target. For example, the recognition unit 120 may recognize the detection target from all or part of the depth map and recognize a gesture using the detection target by using a machine-learned neural network or the like. However, it goes without saying that the recognition unit 120 may also recognize the detection target and gesture based on predetermined rules.

[0020] The calculation unit 130 calculates the velocity or acceleration of the detection target making the gesture based on the temporal change in the depth map acquired by the distance measurement sensor unit 110. Specifically, the calculation unit 130 detects the movement direction and movement amount of the detection target making the gesture from the difference between each frame of the depth map acquired in time series, and divides the detected movement amount of the detection target by the time interval between each frame to calculate the velocity of the detection target. Furthermore, the calculation unit 130 further divides the calculated velocity change of the detection target by the time interval between each frame to calculate the acceleration of the detection target. Note that the calculation unit 130 may calculate the position change, velocity, and acceleration of the detection target as vectors or as scalar quantities.

[0021] Calculation of the velocity or acceleration of the detection target by the calculation unit 130 will be described in more detail with reference to Figures 2 to 5. Figure 2 is an explanatory diagram showing the state of each frame of a depth map acquired in time series by the distance measurement sensor unit 110. Figure 3 is a graph showing positional changes of the detection target in each frame shown in Figure 2. Figure 4 is a graph showing velocity changes of the detection target calculated from the positional changes shown in Figure 3. Figure 5 is a graph showing acceleration changes of the detection target calculated from the velocity changes shown in Figure 4.

[0022] 2, for example, assume that a user, who is a subject, is making a punch gesture by thrusting his / her fist toward the distance measurement sensor unit 110. In such a case, the distance measurement sensor unit 110 can acquire nine depth maps of the subject, frame numbers 1 to 9, in chronological order by sensing the subject at a high frame rate such as 120 fps. This allows the recognition unit 120 to recognize the user's punch gesture by recognizing the user's fist, who is the subject, as a detection target in each of the depth maps of frame numbers 1 to 9.

[0023] Meanwhile, the calculation unit 130 can detect the change in the position of the fist, which is the detection target, as shown in the graph diagram in FIG. 3 by detecting the difference between the depth maps of frame numbers 1 to 9. That is, the calculation unit 130 can detect the change in the position of the user's fist in the gesture of punching in more detail. This is because the ranging sensor unit 110 can acquire depth maps of the subject at a high frame rate, such as 120 fps. For example, the calculation unit 130 may detect a change in the position of the fist, which is the detection target, in the Z direction toward the ranging sensor unit 110 (i.e., the depth), or may detect a change in the three-dimensional position that also takes into account a change in the position in the XY plane perpendicular to the Z direction as the change in the fist position.

[0024] Furthermore, by dividing the amount of change in position of the fist, which is the detection target, by the time interval of each frame (i.e., by differentiating the amount of change in position with respect to time), the calculation unit 130 can detect a change in velocity of the fist, which is the detection target, as shown in Fig. 4. Furthermore, by dividing the change in velocity of the fist, which is the detection target, by the time interval of each frame (i.e., by differentiating the amount of change in velocity with respect to time), the calculation unit 130 can detect a change in acceleration of the fist, which is the detection target, as shown in Fig. 5.

[0025] The execution unit 140 executes an action corresponding to a gesture recognized by the recognition unit 120. Specifically, the execution unit 140 may associate gestures with actions in advance, and execute the action corresponding to the gesture when a gesture associated with the action is recognized by the recognition unit 120. Furthermore, the execution unit 140 may associate a permutation or combination of multiple consecutive gestures with an action in advance, and execute the action corresponding to the permutation or combination when a permutation or combination of multiple gestures associated with an action is recognized by the recognition unit 120.

[0026] Furthermore, the execution unit 140 executes an action corresponding to the gesture recognized by the recognition unit 120 with a strength expression based on the speed or acceleration calculated by the calculation unit 130. Specifically, the execution unit 140 may execute the action corresponding to the gesture with a stronger expression the higher the speed or acceleration calculated by the calculation unit 130. For example, the execution unit 140 may strengthen the visual or auditory effect when executing the action corresponding to the gesture, increase the speed when executing the action, or strengthen the result of the executed action, the higher the speed or acceleration calculated by the calculation unit 130.

[0027] The display unit 150 is an example of an output unit that executes an action corresponding to a gesture. For example, the display unit 150 may display an image of the action corresponding to the gesture using an intensity expression controlled by the execution unit 140. This allows the information processing device 100 to present to the user, as an image, the result of the action input by the user, who is the subject, using a gesture.

[0028] The communication unit 160 is a communication interface for connecting to a network 920 external to the information processing device 100. The communication unit 160 can transmit and receive data, for example, between the network 920 or an external communication device based on a predetermined protocol. The network 920 connected to the communication unit 160 is a wired or wireless network. The network 920 may be, for example, an Internet communication network, a home LAN, an infrared communication network, a radio wave communication network, or a satellite communication network. The information processing device 100 may cause an external device to execute an action corresponding to a gesture via the network 920.

[0029] According to the above configuration, the information processing device 100 calculates the speed or acceleration of a gesture made by a user who is a subject, and can control the intensity of an action corresponding to the gesture based on the calculated speed or acceleration. Therefore, the information processing device 100 can execute an action corresponding to a gesture with more complex expressiveness.

[0030] (1.2. Operation Example) Next, the flow of operations of the information processing device 100 according to this embodiment will be described with reference to FIGS. 6 and 7. FIG.

[0031] (First Operation Example) FIG. 6 is a flowchart showing the flow of a first operation example of the information processing device 100 according to this embodiment.

[0032] 6 , the information processing device 100 first acquires a depth map of the subject using the distance measurement sensor unit 110 (S101). Next, the information processing device 100 detects a detection target included in the depth map using the distance measurement sensor unit 110 (S103). The detection target is, for example, the user's hand, finger, or arm, which is the subject. The distance measurement sensor unit 110 cuts out an area including the detection target from the depth map and outputs it to the recognition unit 120 and calculation unit 130, which are provided downstream. The detection of the detection target from the depth map may be performed using, for example, a machine-learned neural network.

[0033] Next, the information processing device 100 recognizes a gesture using the detection target using the recognition unit 120 (S105), and calculates the velocity and acceleration of the detection target from the position change of the detection target using the calculation unit 130 (S107, S109). The recognition unit 120 may recognize the gesture using, for example, a machine-learned neural network. Furthermore, the calculation unit 130 may calculate the velocity and acceleration of the detection target by, for example, differentiating the position change of the detection target once or twice with respect to time. Note that the processes of steps S105, S107, and S109 may be performed in parallel with each other.

[0034] Thereafter, the information processing device 100 determines, in the execution unit 140, the execution of an action corresponding to the gesture (S111). The execution unit 140 may determine, for example, the execution of an action corresponding to the recognized gesture based on a correspondence relationship between gestures and actions that has been set in advance.

[0035] Furthermore, the information processing device 100 determines an intensity representation of the action to be executed based on the speed or acceleration of the detection target in the execution unit 140 (S113). For example, the execution unit 140 may determine that the higher the speed or acceleration of the detection target calculated by the calculation unit 130, the stronger the visual or auditory effect to be imparted when the action is executed, the faster the speed at which the action is executed, or the stronger the result of the action to be executed.

[0036] This allows the information processing device 100 to execute an action corresponding to the gesture with an intensity expression based on the speed or acceleration of the detection target (S115). The action executed with an intensity expression based on the speed or acceleration of the detection target is presented to the user, who is the subject, on the display unit 150, for example.

[0037] According to the above operations, the information processing device 100 can recognize a gesture made by a user, who is a subject, using a detection target, and can control the intensity expression of an action corresponding to the gesture based on the speed or acceleration of the detection target.

[0038] (Second Operation Example) FIG. 7 is a flowchart showing the flow of a second operation example of the information processing device 100 according to this embodiment.

[0039] 7 , similarly to the first operation example, the information processing device 100 first acquires a depth map of a subject using the ranging sensor unit 110 (S201). Next, the information processing device 100 detects a detection target included in the depth map using the ranging sensor unit 110 (S203).

[0040] Next, the information processing device 100 recognizes each gesture using the detection target using the recognition unit 120 (S205), and calculates the velocity and acceleration of the detection target from the position change of the detection target using the calculation unit 130 (S207, S209). The recognition unit 120 may recognize each gesture using, for example, a machine-learned neural network. The calculation unit 130 may calculate the velocity and acceleration of the detection target by, for example, differentiating the position change of the detection target once or twice with respect to time. The velocity and acceleration of the detection target calculated by the calculation unit 130 may be, for example, the velocity and acceleration of the detection target during a gesture, or the velocity and acceleration of the detection target between gestures. Note that the processes of steps S205, S207, and S209 may be performed in parallel with each other.

[0041] Thereafter, the information processing device 100 recognizes a permutation or combination of a plurality of consecutive gestures using the recognition unit 120 (S211), and determines execution of an action corresponding to the permutation or combination of the plurality of consecutive gestures using the execution unit 140 (S213). The execution unit 140 may determine execution of an action corresponding to the recognized permutation or combination of the plurality of gestures based on, for example, a correspondence relationship between a pre-set permutation or combination of the plurality of gestures and an action.

[0042] Furthermore, the information processing device 100 determines an intensity representation of the action to be executed based on the speed or acceleration of the detection target in the execution unit 140 (S215). For example, the execution unit 140 may determine that the higher the speed or acceleration of the detection target calculated by the calculation unit 130, the stronger the visual or auditory effect to be imparted when the action is executed, the faster the speed at which the action is executed, or the stronger the result of the action to be executed.

[0043] This allows the information processing device 100 to execute an action corresponding to the gesture with an intensity expression based on the speed or acceleration of the detection target (S217). The action executed with an intensity expression based on the speed or acceleration of the detection target is presented to the user, who is the subject, on the display unit 150, for example.

[0044] According to the above operations, the information processing device 100 can recognize multiple consecutive gestures made by the user, who is the subject, using the detection object, and can control the intensity expression of the action corresponding to the gesture based on the speed or acceleration of the detection object.

[0045] (1.3. Application Examples) Next, application examples of the information processing device 100 according to this embodiment will be described with reference to Fig. 8. Fig. 8 is an explanatory diagram showing the configuration of an information processing system 1 including the information processing device 100 according to this embodiment.

[0046] 8, the information processing system 1 includes a plurality of information processing devices 100 and an information processing server 200. The plurality of information processing devices 100 are connected to the information processing server 200 via a network 920.

[0047] The information processing server 200 is, for example, an application server that controls a competitive game or the like that is played between a plurality of information processing devices 100. The information processing server 200 can provide an interactive gaming experience between the plurality of information processing devices 100 by transmitting an action corresponding to a gesture recognized by one of the information processing devices 100 to the other information processing device 100.

[0048] As an example, a user playing a fighting game may make a punching gesture to input an intention to attack to the information processing device 100. At this time, the information processing device 100 may determine the effect or power of the attack input by the gesture based on the speed or acceleration of the detection target used in the gesture (i.e., the user's hand, finger, or arm), and may display the result of the attack input by the gesture on the display unit 150.

[0049] Furthermore, a user who has been attacked may input a response to the attack (e.g., evasion, defense, or counterattack) using a gesture to the information processing device 100. At this time, the information processing device 100 may determine the effectiveness of the input response based on the speed or acceleration of the detection target (i.e., the user's hand, finger, or arm) used in the gesture, and may display the result of the input response on the display unit 150.

[0050] As another example, a user playing a competitive game may input an intention to attack or defend to the information processing device 100 by successively performing multiple gestures that assume different predetermined poses. In this case, the information processing device 100 may determine the success or failure of the attack or defense based on whether or not it has successfully recognized the permutation or combination of the multiple consecutive gestures. The information processing device 100 may determine the effect or power of the input attack or defense based on the speed or acceleration of the detection target (i.e., the user's hand, finger, or arm) when the pose is assumed, and may display the result of the input attack or defense on the display unit 150.

[0051] <2. Distance Measuring Sensor Unit> (2.1. Overview) The information processing device 100 according to this embodiment can acquire a depth map of a subject at a high frame rate (e.g., 120 fps) using the distance measuring sensor unit 110. This enables the information processing device 100 to detect detailed position or depth changes of a gesture made by the subject, and accurately recognize each of a series of multiple gestures. Acquisition of a depth map at such a high frame rate can be performed, for example, using the distance measuring sensor unit 110 that includes a ToF sensor, which will be described below.

[0052] An overview of the ToF sensor included in the distance measurement sensor unit 110 will be described with reference to Fig. 9. Fig. 9 is a block diagram showing an overview of the ToF sensor 110a included in the distance measurement sensor unit 110.

[0053] As shown in FIG. 9, the ToF sensor 110 a includes a light emitting unit 111 , a light receiving unit 112 , and a distance measuring unit 113 .

[0054] The light emitting unit 111 is a light source that emits emitted light Tx toward the subject U. Specifically, the light emitting unit 111 emits phase-modulated emitted light Tx over the entire angle of view for acquiring a depth map. The light emitting unit 111 may be, for example, a semiconductor laser light source that emits near-infrared light (e.g., wavelength 800 nm to 1000 nm) having a longer wavelength than visible light toward the subject U.

[0055] The light receiving unit 112 is a pixel array that receives reflected light Rx from the subject U. Specifically, the light receiving unit 112 receives reflected light Rx that is formed when emitted light Tx, which is emitted from the light emitting unit 111 toward the subject U, is reflected by the subject U. The light receiving unit 112 may be, for example, a two-dimensional pixel array in which phase difference pixels capable of detecting the phase of the reflected light Rx are arranged in a matrix.

[0056] The ranging unit 113 derives the depth to the subject U based on the time of flight of the emitted light Tx emitted from the light emitting unit 111, reflected by the subject U, and returned as reflected light Rx. Specifically, the ranging unit 113 may derive the depth to the subject U based on a phase shift between the emitted light Tx and the reflected light Rx caused by the time of flight. In other words, the ranging unit 113 may derive the depth to the subject U using an iToF method (indirect ToF method).

[0057] The ToF sensor 110a having the above configuration can simultaneously emit emitted light Tx over the entire angle of view including the subject U, and receive reflected light Rx of the emitted light Tx using a two-dimensional pixel array, thereby simultaneously acquiring a depth map over the entire angle of view including the subject U.

[0058] (2.2. Two-dimensional pixel array) Next, the two-dimensional pixel array of the light receiving unit 112 will be described in more detail with reference to Figs. 10 to 12. Fig. 10 is a schematic plan view showing the configuration of a two-dimensional pixel array 1120. Fig. 11 is a circuit diagram showing an equivalent circuit of the two-dimensional pixel array 1120 shown in Fig. 10. Fig. 12 is a timing chart showing the operation of the two-dimensional pixel array 1120 shown in Fig. 10.

[0059] 10 , the two-dimensional pixel array 1120 of the light receiving unit 112 is configured by arranging first pixels 1121 and second pixels 1122 alternately in the row and column directions. The first pixels 1121 are pixels that acquire phase information of 0° and phase information of 180° by receiving reflected light Rx, and the second pixels 1122 are pixels that acquire phase information of 90° and phase information of 270° by receiving reflected light Rx.

[0060] Specifically, the first pixel 1121 is provided with two charge distribution transistors CSGa and CSGb. By applying voltages of opposite phases to the gates of the charge distribution transistors CSGa and CSGb, signal charges generated in the first pixel 1121 upon receiving reflected light Rx can be distributed to the memory units MEMa and MEMb, respectively. This allows the first pixel 1121 to store signal charges representing 0° phase information in the memory unit MEMa and signal charges representing 180° phase information in the memory unit MEMb. For example, the first pixel 1121 may apply a voltage of 0° phase to the charge distribution transistor CSGa and a voltage of 180° phase to the charge distribution transistor CSGb. The 0° phase voltage is a voltage that is modulated at the same frequency as the modulation frequency of the emitted light Tx emitted from the light emitting unit 111 and has a phase shift of 0° from the phase of the emitted light Tx. The 180° phase voltage is a voltage that is modulated at the same frequency as the modulation frequency of the emitted light Tx emitted from the light emitting unit 111 and has a phase shift of 180° from the phase of the emitted light Tx.

[0061] The second pixel 1122 is also provided with two charge distribution transistors CSGc and CSGd. By applying voltages of opposite phases to the gates of the charge distribution transistors CSGc and CSGd, signal charges generated in the second pixel 1122 upon receiving reflected light Rx can be distributed to the memory units MEMc and MEMd, respectively. This allows the second pixel 1122 to store signal charges representing 90° phase information in the memory unit MEMc and signal charges representing 270° phase information in the memory unit MEMd. For example, the second pixel 1122 may apply a 90° phase voltage to the charge distribution transistor CSGc and a 270° phase voltage to the charge distribution transistor CSGd. The 90° phase voltage is a voltage that is modulated at the same frequency as the modulation frequency of the emitted light Tx emitted from the light emitting unit 111 and has a phase shift of 90° from the phase of the emitted light Tx. The 270° phase voltage is a voltage that is modulated at the same frequency as the modulation frequency of the emitted light Tx emitted from the light emitting unit 111 and has a phase shift of 270° from the phase of the emitted light Tx.

[0062] Depth information can be derived from a pixel signal representing phase information of 0°, a pixel signal representing phase information of 90°, a pixel signal representing phase information of 180°, and a pixel signal representing phase information of 270° as follows. ０ , a pixel signal representing phase information of 90° is S ９０ , a pixel signal representing 180° phase information is S １８０ , and a pixel signal representing the phase information of 270° is S ２７０ Then, the phase shift θ of the reflected light Rx relative to the emitted light Tx is expressed by the following equation 1.

[0063]

[0064] Therefore, the speed of light is c, and the phase modulation frequency of the emitted light Tx is f ｍｏｄ Then, the depth D to the subject U can be expressed by the following equation 2.

[0065]

[0066] 11, the charge distribution transistor CSGa is provided between the photodiode PD that photoelectrically converts the reflected light Rx and the memory unit MEMa, and the charge distribution transistor CSGb is provided between the photodiode PD that photoelectrically converts the reflected light Rx and the memory unit MEMb. The charge distribution transistors CSGa and CSGb distribute and transfer the signal charge accumulated in the photodiode PD to the memory units MEMa and MEMb based on the voltage applied to their gates.

[0067] An overflow transistor OFG is further provided between the photodiode PD and the power supply line. When a voltage is applied to the gate of the overflow transistor OFG, the overflow transistor OFG discharges the charge accumulated in the photodiode PD to the power supply line. The overflow transistor OFG can function as an electronic shutter by resetting the potential of the photodiode PD to the power supply potential.

[0068] The read transistor TRGa is provided between the memory unit MEMa and the floating diffusion unit FD. When a voltage is applied to the gate of the read transistor TRGa, the read transistor TRGa reads out the signal charge accumulated in the memory unit MEMa to the floating diffusion unit FD. The read transistor TRGb is provided between the memory unit MEMb and the floating diffusion unit FD. When a voltage is applied to the gate of the read transistor TRGb, the read out the signal charge accumulated in the memory unit MEMb to the floating diffusion unit FD.

[0069] A reset transistor RST is further provided between the floating diffusion portion FD and the power supply line. When a voltage is applied to the gate of the reset transistor RST, the potential of the floating diffusion portion FD is reset to the power supply potential.

[0070] The amplifier transistor AMP has a gate connected to the floating diffusion portion FD and a source connected to the drain of the selection transistor SEL. The source of the selection transistor SEL is connected to a vertical signal line VSL. A pixel signal corresponding to the amount of signal charge read out to the floating diffusion portion FD is output to the vertical signal line VSL via the amplifier transistor AMP and the selection transistor SEL.

[0071] According to the above configuration, the two-dimensional pixel array 1120 can read out the signal charges distributed to the memory units MEMa and MEMb in the first pixel 1121 to the floating diffusion unit FD and output them as pixel signals. Therefore, the two-dimensional pixel array 1120 can acquire a pixel signal representing phase information of 0° and a pixel signal representing phase information of 180°. Similarly, the two-dimensional pixel array 1120 can acquire a pixel signal representing phase information of 90° and a pixel signal representing phase information of 270° for the second pixel 1122.

[0072] 12 , in response to the emission of phase-modulated emission light Tx from the light emitting unit 111, the two-dimensional pixel array 1120 simultaneously receives reflected light Rx at the first pixel 1121 and the second pixel 1122 and stores the reflected light Rx in the memory units MEMa, MEMb, MEMc, and MEMd. After that, the signal charges stored in the memory units MEMa, MEMb, MEMc, and MEMd are sequentially read out, whereby the two-dimensional pixel array 1120 can acquire pixel signals representing phase information of 0°, pixel signals representing phase information of 90°, pixel signals representing phase information of 180°, and pixel signals representing phase information of 270°, respectively.

[0073] (2.3. Modification) Furthermore, a modification of the two-dimensional pixel array 1120 of the light receiving unit 112 will be described with reference to Fig. 13. Fig. 13 is a schematic plan view showing the configuration of a two-dimensional pixel array 1120a according to the modification.

[0074] 13 , a two-dimensional pixel array 1120 according to the modified example is configured by arranging 4TAP pixels 1123 in row and column directions. The 4TAP pixels 1123 are pixels that acquire phase information of 0°, 90°, 180°, and 270° by receiving reflected light Rx.

[0075] Specifically, four charge distribution transistors CSGa, CSGb, CGSc, and CGSd are provided in the 4TAP pixel 1123. By applying voltages of different phases to the gates of the charge distribution transistors CSGa, CSGb, CGSc, and CGSd, the signal charges generated in the 4TAP pixel 1123 by receiving reflected light Rx can be distributed to the memory units MEMa, MEMb, MEMc, and MEMd, respectively.

[0076] For example, the 4TAP pixel 1123 may apply a voltage with a phase of 0° to the charge sorting transistor CSGa, a voltage with a phase of 180° to the charge sorting transistor CSGb, a voltage with a phase of 90° to the charge sorting transistor CSGc, and a voltage with a phase of 270° to the charge sorting transistor CSGd. In this manner, the 4TAP pixel 1123 can store signal charges representing phase information of 0° in the memory unit MEMa, signal charges representing phase information of 180° in the memory unit MEMb, signal charges representing phase information of 90° in the memory unit MEMc, and signal charges representing phase information of 270° in the memory unit MEMd.

[0077] The two-dimensional pixel array 1120a according to the modified example can acquire information for deriving the depth to the subject U using one 4TAP pixel 1123. Therefore, the two-dimensional pixel array 1120a according to the modified example can measure the depth to the subject U with higher resolution without miniaturizing the pixels.

[0078] 3. Hardware Configuration The hardware configuration of the information processing device 100 according to this embodiment will be described with reference to Fig. 14. Fig. 14 is a block diagram showing an example of the hardware configuration of the information processing device 100 according to this embodiment.

[0079] The functions of the information processing device 100 according to this embodiment may be realized by cooperation between software and hardware described below. The functions of the recognition unit 120, the calculation unit 130, and the execution unit 140 may be executed by, for example, the CPU 901. The functions of the communication unit 160 may be executed by, for example, the communication device 911. The functions of the display unit 150 may be executed by, for example, the output device 907.

[0080] As shown in FIG. 14, the information processing device 100 includes a CPU (Central Processing Unit) 901 , a ROM (Read Only Memory) 902 , and a RAM (Random Access Memory) 903 .

[0081] The information processing device 100 may further include a host bus 904a, a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a drive 909, a connection port 910, a communication device 911, or a sensor 916. The information processing device 100 may have a processing circuit such as a DSP (Digital Signal Processor) or an ASIC (Application Specific Integrated Circuit) instead of or together with the CPU 901.

[0082] The CPU 901 functions as an arithmetic processing device or a control device, and controls operations within the information processing device 100 in accordance with various programs recorded in the ROM 902, the RAM 903, the storage device 908, or a removable recording medium attached to the drive 909. The ROM 902 stores programs used by the CPU 901, calculation parameters, etc. The RAM 903 temporarily stores programs used in the execution of the CPU 901, and parameters used during the execution of the programs.

[0083] The CPU 901, ROM 902, and RAM 903 are interconnected by a host bus 904a capable of high-speed data transmission. The host bus 904a is connected to an external bus 904b, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 904. The external bus 904b is connected to various components via an interface 905.

[0084] The input device 906 is a device that accepts input from a user, such as a mouse, keyboard, touch panel, button, switch, or lever. The input device 906 may also be a microphone that detects the user's voice. The input device 906 may also be, for example, a remote control device that uses infrared rays or other radio waves, or may be an externally connected device that supports operation of the information processing device 100.

[0085] The input device 906 further includes an input control circuit that outputs an input signal generated based on information input by the user to the CPU 901. By operating the input device 906, the user can input various data or instruct the information processing device 100 to perform processing operations.

[0086] The output device 907 is a device that can visually or audibly present information acquired or generated by the information processing device 100 to a user. The output device 907 may be, for example, a display device such as an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), an OLED (Organic Light Emitting Diode) display, a hologram, or a projector, a sound output device such as a speaker or headphones, or a printing device such as a printer. The output device 907 can output information acquired by processing by the information processing device 100 as video such as text or an image, or sound such as voice or audio.

[0087] The storage device 908 is a data storage device configured as an example of a storage unit of the information processing device 100. The storage device 908 may be configured, for example, by a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 908 can store programs executed by the CPU 901, various data, various data acquired from the outside, and the like.

[0088] The drive 909 is a device for reading or writing data from or to a removable recording medium such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and is built into or externally attached to the information processing device 100. For example, the drive 909 can read information recorded on an attached removable recording medium and output the information to the RAM 903. The drive 909 can also write data to an attached removable recording medium.

[0089] The connection port 910 is a port for directly connecting an external device to the information processing device 100. The connection port 910 may be, for example, a Universal Serial Bus (USB) port, an IEEE 1394 port, or a Small Computer System Interface (SCSI) port. The connection port 910 may also be an RS-232C port, an optical audio terminal, or a High-Definition Multimedia Interface (HDMI) (registered trademark) port. By connecting the connection port 910 to an external device, various types of data can be transmitted and received between the information processing device 100 and the external device.

[0090] The communication device 911 is, for example, a communication interface configured with a communication device for connecting to the network 920. The communication device 911 may be, for example, a communication card for a wired or wireless local area network (LAN), Wi-Fi (registered trademark), Bluetooth (registered trademark), or Wireless USB (WUSB). The communication device 911 may also be a router for optical communication, a router for an asymmetric digital subscriber line (ADSL), or a modem for various types of communication.

[0091] The communication device 911 can transmit and receive signals, for example, via the Internet or other communication devices using a predetermined protocol such as TCP / IP. The network 920 connected to the communication device 911 is a wired or wireless network, and may be, for example, an Internet communication network, a home LAN, an infrared communication network, a radio wave communication network, or a satellite communication network.

[0092] It is also possible to create a program that causes hardware such as the CPU 901, ROM 902, and RAM 903 built into a computer to perform functions equivalent to those of the information processing device 100. It is also possible to provide a computer-readable recording medium on which the program is recorded.

[0093] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0094] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0095] Note that the following configurations also fall within the technical scope of the present disclosure. (1) An information processing device comprising: a recognition unit that recognizes a gesture using a detection target made by a subject based on a depth map of the subject; a calculation unit that calculates a speed or acceleration of the detection target making the gesture based on a temporal change in the depth map; and an execution unit that executes an action corresponding to the gesture using an intensity expression based on the calculated speed or acceleration. (2) The information processing device according to (1), wherein the calculation unit calculates the speed or acceleration of the detection target based on a temporal change in the depth of the detection target. (3) The information processing device according to (1) or (2), wherein the recognition unit recognizes a plurality of consecutive gestures using the detection target, and the execution unit executes the action corresponding to a permutation or combination of the plurality of consecutive gestures. (4) The information processing device according to any one of (1) to (3), wherein the subject is a user, and the user interacts with other users using the actions via a network. (5) The information processing device according to (4), wherein the detection target is the user's body. (6) The information processing device according to any one of (1) to (5), further comprising a distance measurement sensor unit that acquires the depth map of the subject. (7) The information processing device according to (6), wherein the distance measurement sensor unit includes: a light emitting unit that emits emission light having a wavelength longer than visible light to the subject; a light receiving unit that receives reflected light of the emission light from the subject; and a distance measuring unit that derives a depth to the subject based on a time of flight from when the emission light is emitted to the subject to when the reflected light is received. (8) The information processing device according to (7), wherein the light emitting unit emits phase-modulated emission light, and the distance measuring unit derives the depth based on a phase shift of the reflected light due to the time of flight. (9) The information processing device according to (7) or (8), wherein the light receiving unit receives the reflected light with a two-dimensional pixel array. (10) The information processing device according to (9), wherein the two-dimensional pixel array acquires 0° phase information, 90° phase information, 180° phase information, and 270° phase information in one light receiving operation.(11) The information processing device according to (10), wherein the two-dimensional pixel array includes a first pixel that acquires the 0° phase information and the 180° phase information, and a second pixel that acquires the 90° phase information and the 270° phase information. (12) The information processing device according to (11), wherein the first pixel acquires the 0° phase information and the 180° phase information by distributing signal charges generated by receiving the reflected light to two charge accumulation units at phase timings of 0° and 180°, and the second pixel acquires the 90° phase information and the 270° phase information by distributing signal charges generated by receiving the reflected light to two charge accumulation units at phase timings of 90° and 270°. (13) The information processing device according to (10), wherein the two-dimensional pixel array includes a 4TAP pixel that acquires the 0° phase information, the 90° phase information, the 180° phase information, and the 270° phase information. (14) The information processing device according to (13), wherein the 4TAP pixel acquires the 0° phase information, the 90° phase information, the 180° phase information, and the 270° phase information by distributing signal charges generated by receiving the reflected light to four charge accumulation units at phase timings of 0°, 90°, 180°, and 270°. (15) An information processing method by a computer, comprising: recognizing a gesture using a detection target by the subject based on a depth map of the subject; calculating a velocity or acceleration of the detection target performing the gesture based on a time change of the depth map; and performing an action corresponding to the gesture using an intensity expression based on the calculated velocity or acceleration. (16) A program for causing a computer to function as: a recognition unit that recognizes a gesture made by a subject using a detection target based on a depth map of the subject; a calculation unit that calculates a speed or acceleration of the detection target making the gesture based on a time change in the depth map; and an execution unit that executes an action corresponding to the gesture using an intensity expression based on the calculated speed or acceleration.

[0096] REFERENCE SIGNS LIST 1 Information processing system 100 Information processing device 110 Distance measurement sensor unit 110a ToF sensor 111 Light emission unit 112 Light receiving unit 113 Distance measurement unit 120 Recognition unit 130 Calculation unit 140 Execution unit 150 Display unit 160 Communication unit 200 Information processing server 920 Network U Subject

Claims

1. An information processing device comprising: a recognition unit that recognizes a gesture made by a subject using a detection target based on a depth map of the subject; a calculation unit that calculates a speed or acceleration of the detection target performing the gesture based on changes over time in the depth map; and an execution unit that executes an action corresponding to the gesture using an intensity expression based on the calculated speed or acceleration.

2. The information processing device according to claim 1, wherein the calculation unit calculates the velocity or acceleration of the detection target based on an amount of change over time in the depth of the detection target.

3. The information processing device according to claim 1, wherein the recognition unit recognizes a plurality of consecutive gestures using the detection target, and the execution unit executes the action corresponding to a permutation or combination of the plurality of consecutive gestures.

4. The information processing device according to claim 1, wherein the subject is a user, and the user interacts with other users through the action via a network.

5. The information processing device according to claim 4, wherein the detection target is the user's body.

6. The information processing device according to claim 1, further comprising a distance measurement sensor unit that acquires the depth map of the subject.

7. The information processing device of claim 6, wherein the distance measurement sensor unit includes: a light emitting unit that emits light having a longer wavelength than visible light toward the subject; a light receiving unit that receives reflected light of the emitted light from the subject; and a distance measurement unit that derives the depth to the subject based on the flight time from when the emitted light is emitted to the subject to when the reflected light is received.

8. The information processing device according to claim 7, wherein the light emitting unit emits phase-modulated emitted light, and the distance measuring unit derives the depth based on a phase shift of the reflected light due to the time of flight.

9. The information processing device according to claim 7, wherein the light receiving section receives the reflected light with a two-dimensional pixel array.

10. The information processing device according to claim 9, wherein the two-dimensional pixel array obtains 0° phase information, 90° phase information, 180° phase information, and 270° phase information in one light receiving operation.

11. The information processing device of claim 10, wherein the two-dimensional pixel array includes a first pixel that acquires the 0° phase information and the 180° phase information, and a second pixel that acquires the 90° phase information and the 270° phase information.

12. The information processing device described in claim 11, wherein the first pixel acquires the 0° phase information and the 180° phase information by distributing the signal charge generated by receiving the reflected light to two charge storage sections at phase timings of 0° and 180°, and the second pixel acquires the 90° phase information and the 270° phase information by distributing the signal charge generated by receiving the reflected light to two charge storage sections at phase timings of 90° and 270°.

13. The information processing device of claim 10, wherein the two-dimensional pixel array includes 4TAP pixels that obtain the 0° phase information, the 90° phase information, the 180° phase information, and the 270° phase information.

14. The information processing device of claim 13, wherein the 4TAP pixel acquires the 0° phase information, the 90° phase information, the 180° phase information, and the 270° phase information by distributing the signal charge generated by receiving the reflected light to four charge storage sections at phase timings of 0°, 90°, 180°, and 270°.

15. An information processing method by a computer, comprising: recognizing a gesture made by a subject using a detection target based on a depth map of the subject; calculating a speed or acceleration of the detection target performing the gesture based on changes over time in the depth map; and executing an action corresponding to the gesture with a strength expression based on the calculated speed or acceleration.

16. A program for causing a computer to function as: a recognition unit that recognizes a gesture made by a subject using a detection target based on a depth map of the subject; a calculation unit that calculates a speed or acceleration of the detection target making the gesture based on changes over time in the depth map; and an execution unit that executes an action corresponding to the gesture using an intensity expression based on the calculated speed or acceleration.

Images