Electronic device, program, and control method

Abstract

Provided is an electronic device comprising: an acquisition unit that acquires imaging data including a hand of an operator of the electronic device as a subject; an estimation unit that estimates skeleton data including a plurality of points corresponding to the hand of the operator included in the imaging data; and a recognition unit that recognizes a gesture by the operator to operate the electronic device based on the distance between two gesture recognition points among the plurality of points included in the skeleton data and a threshold value corresponding to the distance between the imaging unit that captured the imaging data and the hand of the operator.

Description

[Technical field]

[0001] The present invention relates to an electronic device, a program, and a control method. [Background technology]

[0002] Patent Document 1 describes a technique for calculating the position of a specific part of an operator's hand based on skeleton data of the hand of an electronic device operator, and determining a cursor position corresponding to that position. [Prior art document] [Patent documents] [Patent Document 1] Patent No. 7213396 Summary of the Invention [Means for solving the problem]

[0003] According to an embodiment of the present invention, an electronic device is provided. The electronic device may include an acquisition unit that acquires imaging data including a hand of an operator of the electronic device as a subject. The electronic device may include an estimation unit that estimates skeleton data including a plurality of points corresponding to the hand of the operator included in the imaging data. The electronic device may include a recognition unit that recognizes a gesture by the operator to operate the electronic device based on a distance between two gesture recognition points among the plurality of points included in the skeleton data and a threshold value corresponding to a distance between the imaging unit that captured the imaging data and the hand of the operator. In the electronic device, the recognition unit may recognize a gesture by the operator to operate the electronic device based on the threshold value that is larger as the distance between the imaging unit and the hand of the operator is shorter. In any of the electronic devices, the recognition unit may recognize a gesture by the operator based on a distance between a first gesture recognition point corresponding to the tip of the thumb of the hand and a second gesture recognition point corresponding to the tip of the index finger of the hand, which are included in the skeleton data, and the threshold value. The recognition unit may determine that the operator has performed a pinch gesture when the distance between the first gesture recognition point and the second gesture recognition point becomes shorter than the threshold. After determining that the operator has performed the pinch gesture, the recognition unit may determine whether the operator has performed a click gesture, a swipe gesture, or a drag-and-drop gesture based on a time until the pinch gesture is released and an amount of movement of the operator's hand while the pinch gesture is being performed. In any of the electronic devices, the recognition unit may recognize a gesture by the operator based on a distance between the first gesture recognition point and the second gesture recognition point, the threshold, and an angle before and after movement of the second gesture recognition point based on an auxiliary point corresponding to the base of the index finger of the hand, which is included in the skeleton data.In any of the electronic devices, the recognition unit may recognize a gesture by the operator to operate the electronic device, based on the distance between the two gesture recognition points and a threshold value corresponding to the distance between the imaging unit that captured the imaging data and the operator's hand, for a plurality of frames after thinning out the plurality of frames included in the imaging data at a thinning rate that is greater the shorter the distance between the imaging unit and the operator's hand.

[0004] According to an embodiment of the present invention, there is provided an electronic device. The electronic device may include an acquisition unit that acquires imaging data including a hand of an operator of the electronic device as a subject. The electronic device may include an estimation unit that estimates skeleton data including a plurality of points corresponding to the hand of the operator included in the imaging data. The electronic device may include a recognition unit that recognizes a gesture by the operator based on a distance between a first gesture recognition point corresponding to the tip of the thumb of the hand and a second gesture recognition point corresponding to the tip of the index finger of the hand, among the plurality of points included in the skeleton data, and an angle before and after movement of the second gesture recognition point based on an auxiliary point corresponding to the base of the index finger of the hand.

[0005] According to one embodiment of the present invention, there is provided a program for causing a computer to function as the electronic device.

[0006] According to one embodiment of the present invention, there is provided a control method executed by an electronic device. The control method may include an acquisition step of acquiring imaging data including a hand of an operator of the electronic device as a subject. The control method may include an estimation step of estimating skeleton data including a plurality of points corresponding to the hand of the operator included in the imaging data. The control method may include a recognition step of recognizing a gesture by the operator to operate the electronic device based on a distance between two gesture recognition points among the plurality of points included in the skeleton data and a threshold value corresponding to a distance between an imaging unit that captured the imaging data and the hand of the operator.

[0007] According to one embodiment of the present invention, there is provided a control method executed by an electronic device. The control method may include an acquisition step of acquiring imaging data including a hand of an operator of the electronic device as a subject. The control method may include an estimation step of estimating skeleton data including a plurality of points corresponding to the hand of the operator included in the imaging data. The control method may include a recognition step of recognizing a gesture by the operator based on a distance between a first gesture recognition point corresponding to the tip of the thumb of the hand and a second gesture recognition point corresponding to the tip of the index finger of the hand, among the plurality of points included in the skeleton data, and an angle before and after movement of the second gesture recognition point based on an auxiliary point corresponding to the base of the index finger of the hand.

[0008] The above summary of the invention does not list all of the necessary features of the present invention. Also, subcombinations of these features may also be inventions. [Brief description of the drawings]

[0009] [Figure 1] An example of an electronic device 100 is shown diagrammatically. [Diagram 2] 2 illustrates an example of a functional configuration of the electronic device 100. [Diagram 3] 2 is an explanatory diagram for explaining an example of a plurality of points included in skeleton data 30 of a hand 20 of an operator 40. FIG. [Figure 4] 13 is an explanatory diagram for explaining a method for determining a cursor position by a determination unit 128. FIG. [Diagram 5] 1 is an explanatory diagram for explaining at least three noise removal points selected from a plurality of points included in skeleton data 30 of a hand 20. FIG. [Figure 6] 10 is an explanatory diagram for explaining gesture recognition by a recognition unit 130. FIG. [Figure 7]10 is an explanatory diagram for explaining gesture recognition by a recognition unit 130. FIG. [Figure 8] 13 is an explanatory diagram for explaining how a pinch recognition threshold is changed in response to the distance between the imaging unit 102 and the hand 20 of the operator 40. FIG. [Figure 9] 13 is an explanatory diagram for explaining another example of gesture recognition by the recognition unit 130. FIG. [Figure 10] 11 is an explanatory diagram for explaining a noise reduction process in gesture recognition by a recognition unit 130. FIG. [Figure 11] An example of a hardware configuration of a computer 1200 that functions as the electronic device 100 is shown in schematic form. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0011] In the conventional method, when the position of a specific part of the hand of an operator of an electronic device is calculated and the cursor position is determined to correspond to the calculated position, a large movement is required for operation because only the difference in the moving distance of one point is calculated. The electronic device 100 according to the present embodiment determines the cursor position based on a plurality of cursor operation points among a plurality of points included in the skeleton data, thereby making it possible to operate the cursor even with a small movement. For example, it makes it possible to operate the cursor in a large manner even with a small movement, and even a user with relatively poor physical movement can operate the cursor efficiently.

[0012] 1 illustrates an example of an electronic device 100. The electronic device 100 includes an imaging unit 102 capable of imaging an operator 40 of the electronic device 100. The electronic device 100 may be, but is not limited to, a smartphone, a tablet terminal, a PC (Personal Computer), a digital home appliance, a wearable terminal, or the like.

[0013] The imaging unit 102 may be a camera or a sensor built into the electronic device 100. The imaging unit 102 may be an external camera or a sensor. The imaging unit 102 provides imaging data to the electronic device 100. The imaging data may be video data. The imaging data may be still image data captured continuously. The imaging unit 102 may have a function of measuring the distance to an imaging target. The imaging unit 102 may include, for example, a depth camera. The imaging unit 102 may be a so-called stereo camera.

[0014] The electronic device 100 may infer skeleton data including multiple points corresponding to the hand 20 of the operator 40 contained in the imaging data acquired from the imaging unit 102, and may determine a cursor position for operating the electronic device 100 or recognize a gesture by the operator 40 based on some of the multiple points contained in the skeleton data.

[0015] 2 illustrates an example of a functional configuration of the electronic device 100. The electronic device 100 includes an imaging unit 102, a display unit 104, a memory 106, a storage unit 108, and a control unit 120. The control unit 120 includes a setting unit 122, an acquisition unit 124, an estimation unit 126, a determination unit 128, a recognition unit 130, and an operation unit 132.

[0016] The display unit 104 is a display arranged on the front side of the electronic device 100. The display unit 104 may be a touch panel display. The display unit 104 displays icons for operating the electronic device 100 and the like.

[0017] The memory 106 may be configured by a memory of a microcomputer integrated with the control unit 120. The memory 106 may also be a RAM (Random Access Memory) or the like configured by an independent semiconductor device connected to the control unit 120. The memory 106 may temporarily store various programs executed by the control unit 120 and various data referenced by these programs.

[0018] The storage unit 108 may be configured by a writable semiconductor device memory, such as a RAM or a flash memory, built into the electronic device 100. The storage unit 108 may be configured by an external memory connected to the electronic device 100.

[0019] The setting unit 122 performs various settings. The setting unit 122 may perform various settings according to instructions from the operator 40.

[0020] The acquisition unit 124 acquires imaging data from the imaging unit 102. The acquisition unit 124 may acquire imaging data including the hand of the operator 40 as a subject. The setting unit 122 may recognize the hand of the operator 40 by analyzing the imaging data acquired by the acquisition unit 124, and set a reference position of the hand.

[0021] The estimation unit 126 estimates skeleton data corresponding to the hand of the operator 40 included in the imaging data acquired by the acquisition unit 124. The skeleton data may represent the shape of a volumetric object by a set of line segments that form the skeleton of the object. For example, the skeleton data may represent each part of the object by line segments that indicate the axis of the part or line segments that indicate the frame of the part. The skeleton of the object represented by the skeleton data may be different from the actual skeleton of the object. For example, the skeleton data of the hand does not necessarily need to be aligned with the bones of the hand, but may include at least line segments that indicate the position of each finger and the bending of each finger. The skeleton data may also be a collection of points called a skeleton mesh, which is a sample of several representative points of the skeleton.

[0022] The skeleton data includes a plurality of points. Each of the plurality of points corresponds, for example, to the base, joint, tip, etc. of each part. For example, the skeleton data of a hand includes a plurality of points that correspond to the base of the wrist, the base, joint, and tip of the thumb, the base, joint, and tip of the index finger, the base, joint, and tip of the middle finger, the base, joint, and tip of the ring finger, and the base, joint, and tip of the little finger.

[0023] The algorithm used by the estimation unit 126 to estimate the skeleton data is not particularly limited. As an example, the estimation unit 126 may estimate the skeleton data corresponding to the operator's hand using a learning model that has been machine-trained using a set of a large number of hand image data and skeleton data of the hand as training data.

[0024] As a specific example, the estimation unit 126 first extracts an area including a hand from the imaging data. The algorithm for extracting an area including a hand from the imaging data is not particularly limited, and a known algorithm may be used. As an example, the estimation unit 126 may extract an area including a hand from the imaging data by detecting a clenched fist or a palm from the imaging data. In this embodiment, the palm refers to a part of the hand other than the fingers. As an example, when the operator 40 does not clench his / her hand, for example, when the hand of the operator 40 is open, the estimation unit 126 may detect a palm, and when the operator 40 clenches his / her hand, the estimation unit 126 may detect a clenched fist. The estimation unit 126 may extract an area including the hand of the operator 40 based on the position and range of the detected clenched fist or palm. The estimation unit 126 may estimate skeleton data corresponding to the hand from the area including the hand from the imaging data. As an example, the estimation unit 126 may estimate skeleton data corresponding to the hand using a learning model as described above.

[0025] The determination unit 128 determines a cursor position for operating the electronic device 100. The determination unit 128 may determine a cursor position for operating the electronic device 100 based on a plurality of cursor operation points among a plurality of points included in the skeleton data estimated by the estimation unit 126. Which points among the plurality of points included in the skeleton data are to be used as cursor operation points may be set in advance by the setting unit 122. The setting unit 122 sets, for example, two points among the plurality of points included in the skeleton data as cursor operation points. The setting unit 122 may set three or more points among the plurality of points included in the skeleton data as cursor operation points.

[0026] The recognition unit 130 recognizes a gesture by the operator 40 for operating the electronic device 100. The recognition unit 130 may recognize a gesture by the operator 40 based on a distance between two gesture recognition points among a plurality of points included in the skeleton data estimated by the estimation unit 126 and a threshold value corresponding to a distance between the imaging unit 102 that captured the imaging data and the hand of the operator 40. The recognition unit 130 may recognize a gesture by the operator 40 based on a threshold value that is larger as the distance between the imaging unit 102 and the hand of the operator 40 becomes shorter.

[0027] The operation unit 132 operates the electronic device 100 based on the cursor position determined by the determination unit 128 and the gesture by the operator 40 recognized by the recognition unit 130. For example, when the recognition unit 130 recognizes a gesture to select an icon, the operation unit 132 selects the icon corresponding to the cursor position and executes an application corresponding to the icon.

[0028] FIG. 3 is an explanatory diagram for explaining an example of a plurality of points included in the skeleton data 30 of the hand 20 of the operator 40. The skeleton data 30 illustrated in FIG. 3 includes 21 points, namely, point 301, point 302, point 303, point 304, point 305, point 306, point 307, point 308, point 309, point 310, point 311, point 312, point 313, point 314, point 315, point 316, point 317, point 318, point 319, point 320, and point 321. The number of points included in the skeleton data 30 does not have to be 21. The number of points may be less than 21 or more than 21. The plurality of points may include the tips of the fingers, the first joints of the fingers, the second joints of the fingers, and the points of the palm. The points of the palm may include the points of the bases of the fingers. The positions of the plurality of points may be different from those in FIG. 3. For example, the multiple points may include points that are not finger tips or knuckles.

[0029] The setting unit 122 may set a plurality of cursor operation points from a plurality of points included in the skeleton data 30. For example, the setting unit 122 sets a plurality of points selected by the operator 40 from a plurality of points included in the skeleton data 30 as a plurality of cursor operation points. The plurality of cursor operation points may be set arbitrarily. The plurality of cursor operation points may be selected from the same finger. The plurality of cursor operation points may be selected from different fingers. The plurality of cursor operation points may be selected from the palm.

[0030] The setting unit 122 sets, for example, two cursor operation points. As an example, the setting unit 122 sets, as the cursor operation points, a point 302 at the base of the thumb and a point 304 at the tip of the thumb among the multiple points included in the skeleton data 30. By making it possible to arbitrarily select multiple cursor operation points from the multiple points included in the skeleton data 30, for example, even a person who has lost a part of a hand in a traffic accident or the like can select a point corresponding to the remaining part of the hand as a cursor operation point and use these points to determine the position of a cursor for operating the electronic device 100.

[0031] 4 is an explanatory diagram for explaining a method of determining a cursor position by the determination unit 128. Here, a case will be described as an example in which the operator 40 can operate the cursor position by moving the hand 20 with the hand 20 open. The hand 20 shown in dotted line is the hand before the movement, and the hand 20 shown in solid line is the hand after the movement. Here, a case will be described as an example in which a point 302 at the base of the thumb and a point 304 at the tip of the thumb are set as cursor operation points.

[0032] Based on the multiple cursor operation points, the determination unit 128 determines the position of the cursor 51 on the display unit 104 of the electronic device 100. In the example shown in FIG. 4, the determination unit 128 determines the position of the cursor 51 based on a point 302 and a point 304.

[0033] The determination unit 128 may determine the position of the cursor 51 based on a spherical space defined by a plurality of cursor operation points. In the example shown in Fig. 4, the determination unit 128 defines a spherical space 510 based on two points, point 302 and point 304. For example, the determination unit 128 defines a spherical space 510 with point 302 as the center and a length determined based on the distance between point 302 and point 304 as the radius.

[0034] The determination unit 128 may determine the position of the cursor 51 based on an expanded spherical space 511 obtained by expanding the spherical space 510. The determination unit 128 defines the expanded spherical space 511 by expanding the radius of the spherical space 510 in accordance with the distance between the imaging unit 102 and the hand 20 of the operator, for example.

[0035] The determination unit 128 may determine the position of the cursor 51 based on the amount of movement specified by reflecting the angular change of the vector formed by the multiple cursor operation points in the spherical space 510 in the expanded spherical space 511. The determination unit 128 may move the cursor 51 by the amount of movement specified by reflecting the angular change of the vector formed by the multiple cursor operation points in the spherical space 510 in the expanded spherical space 511.

[0036] In the example shown in FIG. 4, the determination unit 128 moves the cursor 51 by the movement amount 52 determined by reflecting in the expanded spherical space 511 the angle 330 between the vector formed by the points 302 and 304 on the hand 20 before the movement and the vector formed by the points 302 and 304 on the hand 20 after the movement.

[0037] The determination unit 128 may move the cursor 51 vertically or horizontally depending on the rotation direction of the hand 20. For example, when the point 304 moves up or down with respect to the point 302 as a reference, the determination unit 128 moves the cursor 51 vertically by the amount of movement 52 specified by reflecting the angle change in the vertical direction in the enlarged spherical space 511. For example, when the point 304 moves forward or backward with respect to the point 302 as a reference, the determination unit 128 moves the cursor 51 horizontally by the amount of movement 52 specified by reflecting the angle change in the forward or backward direction in the enlarged spherical space 511. This makes it possible to provide the operator 40 with an operating environment in which the cursor 51 can be moved up or down by tilting the hand 20 so that the point 304 moves up or down with respect to the point 302 as a reference, and the cursor 51 can be moved horizontally by tilting the hand 20 so that the point 304 moves forward or backward with respect to the point 302 as a reference. The relationship between the rotation direction of the hand 20 and the movement direction of the cursor 51 is not limited to this. The relationship between the rotation direction of the hand 20 and the movement direction of the cursor 51 may be set by the setting unit 122.

[0038] Since the amount of movement corresponding to angle 330 in spherical space 510 is not very large, when using this amount of movement to determine the position of cursor 51, it is necessary to move hand 20 a lot in order to move cursor 51 a lot. In contrast, by using expanded spherical space 511, the amount of movement can be increased, making it possible to move cursor 51 a lot with a small movement of hand 20, and improving the operability of moving cursor 51.

[0039] Even when the operator 40 does not intend to do so, the arm, hand 20, and fingers of the operator 40 always sway up and down, left and right, and back and forth to a certain degree. Due to the hand 20 swaying in this way, the multiple points included in the skeleton data 30 also sway. Hereinafter, this swaying will be called noise. Noise in the cursor operation points causes meaningless vibration and flickering of the cursor position, thereby degrading the user experience of the operator 40.

[0040] The electronic device 100 according to the present embodiment may further include a function for reducing such noise. The electronic device 100 sets at least three noise removal points from a plurality of points included in the skeleton data 30 of the hand 20, and reduces noise by using the at least three noise removal points.

[0041] FIG. 5 is an explanatory diagram for explaining at least three noise removal points selected from a plurality of points included in skeleton data 30 of hand 20. The at least three noise removal points may be set by setting unit 122. For example, setting unit 122 sets a plurality of points selected by operator 40 from a plurality of points included in skeleton data 30 as a plurality of noise removal points. The at least three noise removal points may be arbitrarily selected from a plurality of points included in skeleton data 30. In the example shown in FIG. 5, point 304, point 305, and point 309 are set as noise removal points.

[0042] The determination unit 128 may determine the position of the cursor 51 based on a plurality of cursor operation points identified based on an area (sometimes referred to as a noise removal area 600) constituted by at least three noise removal points included in the skeleton data 30, among a plurality of cursor operation points extracted from consecutive frames included in the imaging data acquired by the acquisition unit 124. For example, after determining the noise removal area 600, the determination unit 128 determines the center of gravity of at least three noise removal points for each of the consecutive frames included in the imaging data, and when the center of gravity is included in the noise removal area 600, determines the position of the cursor 51 based on the plurality of cursor operation points.

[0043] For example, the determination unit 128 determines the position of the cursor 51 based on the remaining cursor operation points obtained by removing, as noise, cursor operation points that do not correspond to the noise removal region 600 from the multiple cursor operation points extracted from consecutive frames included in the imaging data. For example, after determining the noise removal region 600, the determination unit 128 determines the center of gravity of at least three noise removal points for each of the consecutive frames included in the imaging data, and removes, as noise, the multiple cursor operation points of the frame when the center of gravity is not included in the noise removal region 600. The determination unit 128 may determine the position of the cursor 51 based on the remaining cursor operation points of the multiple frames excluding the multiple cursor operation points removed as noise.

[0044] With these features, when the hand 20 of the operator 40 shakes unintentionally, it is possible to prevent the cursor 51 from moving according to the multiple cursor operation points at the time of shaking. This makes it possible to reduce the possibility that the cursor 51 moves against the intention of the operator 40, and improve the user experience of the operator 40.

[0045] The at least three noise removal points may be at least one selected from the cursor operation points, or at least one selected from the palm points. This allows the noise removal area 600 corresponding to the cursor position to be set, and effectively separates the coordinate information to be removed as noise from the coordinate information to be used as a signal for cursor operation. For example, when the multiple cursor operation points are two points, the point 304 at the tip of the thumb and the point 302 at the base of the thumb, the at least three noise removal points may be three points, the point 304 at the tip of the thumb, the point 305 at the base of the index finger, and the point 309 at the base of the middle finger, and the noise removal area 600 may be configured based on the three points. However, this is not limited to this. For example, the at least three noise removal points may not include any multiple cursor operation points at all, or may be more than three points.

[0046] 6 is an explanatory diagram for explaining gesture recognition by the recognition unit 130. The recognition unit 130 may recognize a gesture by the operator 40 by using two gesture recognition points among a plurality of points included in the skeleton data 30. Here, a case will be described in which points 304 and 308 are set as gesture recognition points, and the recognition unit 130 recognizes a pinch gesture by the operator 40. Point 304 may be an example of a first gesture recognition point, and point 308 may be an example of a second gesture recognition point.

[0047] The two gesture recognition points may be set by the setting unit 122. For example, the setting unit 122 sets two points selected by the operator 40 from a plurality of points included in the skeleton data 30 as the two gesture recognition points. The two gesture recognition points may be set arbitrarily. The two gesture recognition points may be the points at the tips of two different fingers. However, this is not limited thereto. For example, the two gesture recognition points may include points that are not the tips of fingers.

[0048] 6, the recognition unit 130 may determine that the operator 40 has made a pinch gesture when the distance between the point 304 and the point 308 changes from a state longer than a predetermined threshold (which may be referred to as a pinch recognition threshold) to a state shorter than the predetermined threshold. After determining that the operator 40 has made a pinch gesture, the recognition unit 130 may determine that the operator 40 has released the pinch gesture when the distance between the point 304 and the point 308 changes from a state shorter than the pinch recognition threshold to a state longer than the pinch recognition threshold.

[0049] The recognition unit 130 may recognize a gesture of the operator 40 by using a pinch recognition threshold corresponding to the distance between the imaging unit 102 and the hand 20. The recognition unit 130 may recognize a gesture by the operator 40 by using a pinch recognition threshold that is larger as the distance between the imaging unit 102 and the hand 20 becomes shorter.

[0050] In the example shown in Fig. 6, the recognition unit 130 may recognize a gesture by the operator 40 based on the distance between the point 308 and the point 304 and a pinch recognition threshold corresponding to the distance between the image capture unit 102 and the hand 20. The recognition unit 130 may continuously compare the distance between the point 308 and the point 304 with the pinch recognition threshold corresponding to the distance between the image capture unit 102 and the hand 20, and determine that the operator 40 has made a pinch gesture when the distance becomes shorter than the pinch recognition threshold. After determining that the operator 40 has made a pinch gesture, the recognition unit 130 may continuously compare the distance between the point 308 and the point 304 with the pinch recognition threshold corresponding to the distance between the image capture unit 102 and the hand 20, and determine that the operator 40 has released the pinch gesture when the distance becomes longer than the pinch recognition threshold.

[0051] The shorter the distance between the imaging unit 102 and the hand 20, the larger the size of the hand 20 in the imaging data, and the larger the amount of movement of the fingers when the operator 40 makes a pinch gesture. Conversely, the longer the distance between the imaging unit 102 and the hand 20, the smaller the size of the hand 20 in the imaging data, and the smaller the amount of movement of the fingers when the operator 40 makes a pinch gesture. Therefore, if the same pinch recognition threshold is used even if the distance between the imaging unit 102 and the hand 20 is different, it is difficult to accurately recognize the pinch gesture. On the other hand, the recognition unit 130 can improve the accuracy of recognizing the pinch gesture by using a pinch recognition threshold corresponding to the distance between the point 308 and the point 304 and the distance between the imaging unit 102 and the hand 20.

[0052] The recognition unit 130 may use a pinch recognition threshold value that is larger as the size of the hand 20 in the captured image data increases, thereby using a pinch recognition threshold value that corresponds to the distance between the image capture unit 102 and the hand 20. The correspondence relationship between the size of the hand 20 in the captured image data and the pinch recognition threshold value may be set in advance by the setting unit 122.

[0053] When the imaging unit 102 has a function of measuring the distance to an imaging target, the recognition unit 130 may use a pinch recognition threshold corresponding to the distance between the imaging unit 102 and the hand 20 measured by the imaging unit 102. The correspondence relationship between the distance between the imaging unit 102 and the hand 20 and the pinch recognition threshold may be set in advance by the setting unit 122.

[0054] The recognition unit 130 may recognize a click gesture by the operator 40. The recognition unit 130 may recognize a swipe gesture by the operator 40. The recognition unit 130 may recognize a drag-and-drop gesture by the operator 40.

[0055] After determining that the operator 40 has made a pinch gesture, the recognition unit 130 may determine whether the operator 40 has made a click gesture, a swipe gesture, or a drag-and-drop gesture based on the time until the pinch gesture is released and the amount of movement of the operator 40's hand while the operator 40 is making the pinch gesture.

[0056] For example, the recognition unit 130 determines that the operator 40 has performed a click gesture when the pinch gesture is released before a predetermined first time has elapsed after the operator 40 has determined that the operator 40 has performed a pinch gesture. For example, the recognition unit 130 determines that the operator 40 has performed a swipe gesture when the first time has elapsed after the operator 40 has determined that the operator 40 has performed a pinch gesture and the time until the hand 20 in the pinch gesture state starts to move does not exceed a predetermined second time, and determines that the operator 40 has performed a drag-and-drop gesture when the second time has elapsed. The recognition unit 130 may determine that the drag has started when the hand 20 starts to move when the first time has elapsed after the operator 40 has determined that the operator 40 has performed a pinch gesture and the time until the hand 20 in the pinch gesture state starts to move exceeds the predetermined second time, and may determine that a drop has been performed when the hand 20 moves and the pinch gesture is released.

[0057] With such a configuration, for example, when the operator 40 puts his / her thumb and index finger together and immediately releases them, it can be determined that the operator 40 has performed a click, when the hand 20 starts to move after the thumb and index finger have been put together for a while, it can be determined that the operator 40 has performed a swipe, and when the hand 20 starts to move after the thumb and index finger have been put together for a while longer, it can be determined that the operator 40 has performed a drag-and-drop. This allows the operator 40 to perform intuitive operations.

[0058] The recognition unit 130 may use different pinch recognition thresholds when determining that a pinch gesture has been made and when determining that a pinch gesture has been released. For example, the recognition unit 130 may determine that a pinch gesture has been released using a pinch recognition threshold value that is greater than the pinch recognition threshold value used when determining that a pinch gesture has been made. This makes it easier to recognize that the pinch gesture is being maintained, even if, for example, the tip of the thumb and the tip of the index finger are slightly separated against the will of the operator 40 during a swipe gesture or a drag-and-drop gesture. Therefore, it is possible to reduce the possibility that the swipe gesture or the drag-and-drop gesture is released against the will of the operator 40, and improve the user experience of the operator 40.

[0059] 7 is an explanatory diagram for explaining gesture recognition by the recognition unit 130. The recognition unit 130 recognizes that a pinch gesture is being made by the operator 40, based on the imaging data continuously acquired by the acquisition unit 124. Here, a state in which the recognition unit 130 specifies an area in the imaging data in which the hand 20 is shown, for the imaging data acquired by the acquisition unit 124, will be described as a start state.

[0060] In step (step may be abbreviated as S) 101, the recognition unit 130 aligns the hand 20 with the center of the coordinates. Specifically, for example, the recognition unit 130 aligns the hand 20 with the center of the coordinates in a RoI (Region of Interest). In S102, the recognition unit 130 rotates the hand 20 to adjust the orientation of the hand 20. Here, the orientation of the hand 20 is adjusted so that the index finger points upward. In S103, the recognition unit 130 reduces noise at the fingertip points. Here, the fingertip points are point 304 at the tip of the thumb and point 308 at the tip of the index finger. Here, the noise reduction is reduction of noise in the X coordinate and noise in the Y coordinate. A specific method of noise reduction will be described later. A series of processes from S101 to S103 may be described as preprocessing. The preprocessing may be performed continuously on the imaging data continuously acquired by the acquisition unit 124. Here, the following description will be continued on the assumption that the recognition unit 130 recognizes that a pinch gesture has been made after the preprocessing.

[0061] In S104, the recognition unit 130 judges whether or not a predetermined first time has passed after the pinch gesture has been performed. If it is judged not to have passed the first time, the process proceeds to S105, where it is judged that a "click" has started, and the process proceeds to S112. If it is judged to have passed the first time, the process proceeds to S106. In S112, the recognition unit 130 judges whether or not the pinch gesture has been released. If it is judged to have been released, it is judged that the "click" has ended, it is judged to be a "click" gesture, and the judgment ends. If it is judged not to have been released, the process proceeds to S110, where it is judged that a "drag and drop" has started.

[0062] In S106, the recognition unit 130 judges whether or not the time until the hand 20 in the pinch gesture state starts to move exceeds a predetermined second time. If it is judged not to have exceeded the second time, the process proceeds to S108, where it is judged that a "swipe" has started, and the process proceeds to S112. If it is judged to have exceeded the second time, the process proceeds to S110, where it is judged that a "drag and drop" has started. In S112, the recognition unit 130 judges whether or not the pinch gesture has been released. If it is judged to have been released, it is judged that the "swipe" has ended, it is judged to be a "swipe" gesture, and the judgment is terminated. If it is judged not to have been released, the process proceeds to S110, where it is judged that a "drag and drop" has started.

[0063] If the recognition unit 130 determines in S110 that "drag and drop" has started, the process proceeds to S112. In S112, the recognition unit 130 determines whether the pinch gesture has been released. If it determines that the pinch gesture has been released, the recognition unit 130 determines that the "drag and drop" has ended, determines that the gesture is a "drag and drop", and ends the determination. If it determines that the gesture has not been released, the process proceeds again to S110 and determines that the "drag and drop" state is continuing. Thereafter, the recognition unit 130 repeats the process of determining that the "drag and drop" state is continuing until it determines that the pinch gesture has been released in S112.

[0064] FIG. 8 is an explanatory diagram for explaining changing the pinch recognition threshold value according to the distance between the imaging unit 102 and the hand 20 of the operator 40. When the distance between the imaging unit 102 and the hand 20 of the operator 40 is short, the hand 20 appears large in the image included in the imaging data, as shown in the upper part of FIG. 8. Therefore, the recognition unit 130 may use a pinch recognition threshold value with a larger value as the distance between the imaging unit 102 and the hand 20 of the operator 40 is shorter. As a specific example, when the size of the part of the image in which the hand 20 appears is X×Y, the value of the pinch recognition threshold value S1 may be Y / 8. When the distance between the imaging unit 102 and the hand 20 of the operator 40 is long, the hand 20 appears small in the image included in the imaging data, as shown in the lower part of FIG. 8. Therefore, the recognition unit 130 may use a pinch recognition threshold value with a smaller value as the distance between the imaging unit 102 and the hand 20 of the operator 40 is longer. For example, if the size of the part of the image in which the hand 20 is shown is X / 4×Y / 4, the value of the pinch recognition threshold S2 may be S1 / 4, i.e., Y / 32. The example shown in FIG. 8 is merely an example, and the size of the part in which the hand 20 is shown and the corresponding pinch recognition threshold value are not limited to these. The pinch recognition threshold may be dynamically set according to the ratio of the size of the hand part. For example, if the hand is far away and the size of the recognizable part of the hand is 25% (1 / 4) smaller than the size when the hand is close, the pinch recognition threshold is dynamically set according to the ratio of the reduced size of the hand part.

[0065] Fig. 9 is an explanatory diagram for explaining another example of gesture recognition by the recognition unit 130. The recognition unit 130 may recognize a gesture by the operator 40 by using an auxiliary point in addition to two gesture recognition points among a plurality of points included in the skeleton data 30. In Fig. 9, an example will be described in which points 308 and 304 are used as two gesture recognition points and point 305 corresponding to the base of the index finger is used as the auxiliary point.

[0066] The recognition unit 130 may recognize a gesture by the operator 40 based on the distance between the point 308 and the point 304 and the angle 700 before and after the movement of the point 304 with respect to the point 305 .

[0067] For example, the recognition unit 130 determines that the operator 40 has made a pinch gesture when the distance between the points 304 and 308 changes from longer to shorter than the pinch recognition threshold and the angle 700 is larger than a predetermined angle. Even if the distance between the points 304 and 308 changes from longer to shorter than the pinch recognition threshold, the recognition unit 130 does not need to determine that the operator 40 has made a pinch gesture when the angle 700 is smaller than a predetermined angle.

[0068] For example, after determining that the operator 40 has made a pinch gesture, if the distance between the points 304 and 308 changes from shorter to longer than the pinch recognition threshold and the angle 700 is larger than a predetermined angle, the recognition unit 130 determines that the operator 40 has released the pinch gesture. Even if the distance between the points 304 and 308 changes from shorter to longer than the pinch recognition threshold, if the angle 700 is smaller than a predetermined angle, the recognition unit 130 does not need to determine that the operator 40 has released the pinch gesture.

[0069] When a gesture is recognized using only two gesture recognition points, noise at the two gesture recognition points may cause erroneous recognition of the gesture. For example, when the recognition unit 130 recognizes a pinch gesture, even though the distance between the two gesture recognition points is originally longer than the pinch recognition threshold, noise may cause the distance to be shorter than the pinch recognition threshold, resulting in erroneous recognition that a pinch gesture has been made. Conversely, even though the distance between the two gesture recognition points is originally shorter than the pinch recognition threshold, noise may cause the distance to be longer than the pinch recognition threshold, resulting in erroneous recognition that a pinch gesture has not been made. In response to this, the recognition unit 130 can reduce the possibility of such erroneous recognition by performing recognition using an auxiliary point.

[0070] FIG. 10 is an explanatory diagram for explaining noise reduction processing in gesture recognition by the recognition unit 130. As shown in FIG.

[0071] The recognition unit 130 may recognize a gesture by the operator 40 based on the distance between the two gesture recognition points and a threshold value corresponding to the distance between the imaging unit 102 and the hand 20 of the operator 40, for multiple frames after thinning out the multiple frames included in the imaging data at a thinning rate that is greater as the distance between the imaging unit 102 and the hand of the operator 40 is shorter. In this way, thinning out one or more frames from multiple consecutive frames included in the imaging data may be referred to as filtering. The larger the filter value, the greater the thinning rate per unit number of frames. Filtering is performed and noise is numerically calculated based on the filter value, thereby reducing noise.

[0072] Using Fig. 10, a description will be given of changing the filter value when recognizing a pinch gesture in accordance with the distance between the imaging unit 102 and the hand 20 of the operator 40. In reality, each of a plurality of points is affected by noise corresponding to the shaking of the hand 20, but for simplicity in Fig. 10, only the noise at point 304 at the tip of the thumb will be described.

[0073] When the distance between the imaging unit 102 and the hand 20 of the operator 40 is short, the hand 20 appears large in the image included in the imaging data, as shown in the upper part of Fig. 10. Therefore, the movement amount of the gesture recognition point according to the change in the shape of the hand 20 is large, and the noise is also large. When the size of the part where the hand 20 is captured is X x Y, the point 304 may move between the solid line and the dashed line, and for example, the movement amount of the point 304 may be Y / 5. Noise may occur between the solid line and the dashed line, and for example, the noise value N1 may be Y / 5.

[0074] Since the movement amount of the gesture recognition point is large, the coordinate change of the gesture recognition point between adjacent frames on the time axis is large. Therefore, even if the filter value is increased and frames are thinned out at a larger thinning rate, the change in the shape of the hand can be detected. The filter value may be, for example, F1. For example, the noise processing may be a process of dividing the noise value N1 by the filter value. For example, the noise value N2 after the noise processing may be N1 / F1, that is, a value obtained by dividing Y / 5 by the filter value F1. This reduces noise, thereby reducing erroneous recognition of gestures and improving the user experience.

[0075] When the distance between the imaging unit 102 and the hand 20 of the operator 40 is long, the hand 20 appears small in the image included in the imaging data, as shown in the lower part of Fig. 10. Therefore, the movement amount of the gesture recognition point according to the change in the shape of the hand 20 is small, and the noise is also small. When the size of the part where the hand 20 is captured is X / 4 x Y / 4, the point 304 may move between the solid line and the dashed line, and for example, the movement amount of the point 304 may be Y / 20. Noise may occur between the solid line and the dashed line, and for example, the noise value N3 may be Y / 20.

[0076] Since the movement amount of the gesture recognition point is small, the coordinate change of the gesture recognition point between adjacent frames on the time axis is small. Therefore, it is difficult to detect the change in the shape of the hand unless the filter value is made small and the frames are thinned out at a smaller thinning rate. The filter value may be, for example, F2. For example, the noise processing may be performed by dividing the noise value N3 by the filter value. For example, the noise value N4 after the noise processing may be N3 / F2, that is, a value obtained by dividing Y / 20 by the filter value F2. The filter value may be dynamically set according to the ratio of the size of the hand part. For example, when the hand is far away and the size of the recognizable hand part is reduced to 25% (1 / 4) of the size when the hand is close, the filter value is dynamically set according to the ratio of the reduced size of the hand part.

[0077] In this way, by changing the filter value for noise processing according to the distance between the imaging unit 102 and the hand 20 of the operator 40, it becomes possible to perform appropriate noise reduction processing according to the movement amount of the gesture recognition point and the noise level. This reduces erroneous recognition regardless of the distance between the imaging unit 102 and the hand 20 of the operator 40, and improves the user experience.

[0078] 11 is a schematic diagram showing an example of a hardware configuration of a computer 1200 functioning as the electronic device 100. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of an apparatus according to the present embodiment, or cause the computer 1200 to execute operations or one or more "parts" associated with the apparatus according to the present embodiment, and / or cause the computer 1200 to execute a process or steps of the process according to the present embodiment. Such a program can be executed by the CPU 1212 to cause the computer 1200 to execute specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0079] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphic controller 1216, which are connected to each other by a host controller 1210. The computer 1200 also includes a GPU 1250. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive may be a DVD-ROM drive, a DVD-RAM drive, etc. The storage device 1224 may be a hard disk drive, a solid state drive, etc. The computer 1200 also includes a legacy input / output unit such as a ROM 1230 and a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0080] The CPU 1212 operates according to a program stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphic controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into itself, and causes the image data to be displayed on the display device 1218.

[0081] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0082] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or a program that depends on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, and the like.

[0083] The programs are provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, the RAM 1214, or the ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by implementing operations or processing of information according to the use of the computer 1200.

[0084] For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 may execute a communication program loaded in the RAM 1214 and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in the RAM 1214, the storage device 1224, a DVD-ROM, or a recording medium such as an IC card, and transmits the read transmission data to a network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.

[0085] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0086] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0087] The above-described programs or software modules may be stored in a computer-readable storage medium on the computer 1200 or in the vicinity of the computer 1200. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

[0088] The blocks in the flowcharts and block diagrams in the present embodiment may represent stages of a process in which an operation is performed or "parts" of an apparatus responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as, for example, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like, including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.

[0089] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.

[0090] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages ​​such as the “C” programming language or similar programming languages.

[0091] The computer-readable instructions may be provided to a general-purpose computer, a special-purpose computer, or a processor or programmable circuit of another programmable data processing device, such as a computer, locally or via a wide-area network (WAN) such as a local area network (LAN), the Internet, etc., so that the processor or programmable circuit of the programmable data processing device executes the computer-readable instructions to generate means for performing the operations specified in the flowchart or block diagram. Here, the computer may be a PC (personal computer), a tablet computer, a smartphone, a workstation, a server computer, a general-purpose computer, a special-purpose computer, etc., and may be a computer system in which multiple computers are connected. Such a computer system in which multiple computers are connected is also called a distributed computing system, and is a computer in the broad sense. In a distributed computing system, multiple computers collectively execute a program by each executing a part of the program and transferring data during program execution between the computers as necessary.

[0092] Examples of processors include computer processors, central processing units (CPUs), processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc. A computer may have one processor or multiple processors. In a multiprocessor system with multiple processors, each processor executes a part of a program, and the multiple processors collectively execute a program by passing data during program execution between the processors as necessary. For example, in executing multitasks, each of the multiple processors may execute a part of each task in small chunks by switching tasks for each time slice. In this case, which part of a program each processor executes changes dynamically. Which part of a program each of the multiple processors executes may be statically determined by programming that takes the multiprocessor into consideration.

[0093] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the description of the claims that such modifications and improvements can also be included in the technical scope of the present invention.

[0094] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and that the process may be performed in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, it does not mean that the process must be performed in this order. [Explanation of symbols]

[0095] 20 hand, 30 skeleton data, 40 operator, 51 cursor, 52 movement amount, 100 electronic device, 102 imaging unit, 104 display unit, 106 memory, 108 storage unit, 120 control unit, 122 setting unit, 124 acquisition unit, 126 estimation unit, 128 decision unit, 130 recognition unit, 132 operation unit, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321 point, 330 angle, 510 spherical space, 511 enlarged spherical space, 600 noise removal area, 700 angle, 1200 computer, 1210 Host controller, 1212, CPU, 1214, RAM, 1216, graphics controller, 1218, display device, 1220, input / output controller, 1222, communication interface, 1224, storage device, 1230, ROM, 1240, input / output chip, 1250, GPU

Claims

1. An electronic device, an acquisition unit that acquires imaging data including a hand of an operator of the electronic device as a subject; an estimation unit that estimates skeleton data including a plurality of points corresponding to the operator's hand included in the imaging data; a recognition unit that recognizes a gesture made by the operator to operate the electronic device, based on a distance between two gesture recognition points among the plurality of points included in the skeleton data and a threshold value corresponding to a distance between an imaging unit that captures the imaging data and a hand of the operator; Equipped with The recognition unit recognizes a gesture by the operator to operate the electronic device based on the distance between the two gesture recognition points and a threshold value corresponding to the distance between the imaging unit that captured the imaging data and the operator's hand, for a plurality of frames after thinning out the plurality of frames included in the imaging data at a thinning rate that is greater the shorter the distance between the imaging unit and the operator's hand.

2. The electronic device according to claim 1 , wherein the recognition unit recognizes a gesture made by the operator to operate the electronic device based on the threshold value, the threshold value being larger as the distance between the imaging unit and the hand of the operator becomes shorter.

3. 2. The electronic device according to claim 1, wherein the recognition unit recognizes a gesture by the operator based on a distance between a first gesture recognition point corresponding to the tip of a thumb of the hand and a second gesture recognition point corresponding to the tip of an index finger of the hand, which are included in the skeleton data, and the threshold value.

4. 4 . The electronic device according to claim 3 , wherein the recognition unit determines that the operator has performed a pinch gesture when a distance between the first gesture recognition point and the second gesture recognition point becomes shorter than the threshold value.

5. 5. The electronic device according to claim 4, wherein the recognition unit, after determining that the operator has performed the pinch gesture, determines whether the operator has performed a click gesture, a swipe gesture, or a drag-and-drop gesture based on a time until the pinch gesture is released and an amount of movement of the operator's hand while the pinch gesture is being performed.

6. 6. The electronic device according to claim 3, wherein the recognition unit recognizes a gesture by the operator based on a distance between the first gesture recognition point and the second gesture recognition point, the threshold value, and an angle before and after movement of the second gesture recognition point based on an auxiliary point corresponding to a base of an index finger of the hand, which is included in the skeleton data.

7. An electronic device, an acquisition unit that acquires imaging data including a hand of an operator of the electronic device as a subject; an estimation unit that estimates skeleton data including a plurality of points corresponding to the operator's hand included in the imaging data; a recognition unit that recognizes a gesture made by the operator based on a distance between a first gesture recognition point corresponding to the tip of the thumb of the hand and a second gesture recognition point corresponding to the tip of the index finger of the hand, among the plurality of points included in the skeleton data, and an angle before and after movement of the second gesture recognition point with respect to an auxiliary point corresponding to the base of the index finger of the hand; Equipped with the recognition unit recognizes a gesture by the operator to operate the electronic device based on the distance between the first gesture recognition point and the second gesture recognition point and the angle before and after movement of the second gesture recognition point relative to the auxiliary point, for a plurality of frames after thinning out the plurality of frames included in the imaging data at a thinning rate that is greater the shorter the distance between the imaging unit that captured the imaging data and the operator's hand.

8. A program for causing a computer to function as the electronic device according to any one of claims 1 to 5.

9. A control method executed by an electronic device, comprising: acquiring image data including a hand of an operator of the electronic device as a subject; an estimation step of estimating skeleton data including a plurality of points corresponding to the operator's hand included in the imaging data; a recognition step of recognizing a gesture made by the operator to operate the electronic device based on a distance between two gesture recognition points among the plurality of points included in the skeleton data and a threshold value corresponding to a distance between an imaging unit that captures the imaging data and a hand of the operator; Equipped with The recognition step is a step of recognizing a gesture by the operator to operate the electronic device, based on the distance between the two gesture recognition points and a threshold value corresponding to the distance between the imaging unit that captured the imaging data and the operator's hand, for a plurality of frames after thinning out the plurality of frames included in the imaging data at a thinning rate that is greater the shorter the distance between the imaging unit and the operator's hand.

10. A control method executed by an electronic device, comprising: acquiring image data including a hand of an operator of the electronic device as a subject; an estimation step of estimating skeleton data including a plurality of points corresponding to the operator's hand included in the imaging data; a recognition step of recognizing a gesture made by the operator based on a distance between a first gesture recognition point corresponding to the tip of the thumb of the hand and a second gesture recognition point corresponding to the tip of the index finger of the hand, among the plurality of points included in the skeleton data, and an angle before and after movement of the second gesture recognition point with respect to an auxiliary point corresponding to the base of the index finger of the hand; Equipped with The recognition step is a step of recognizing a gesture by the operator to operate the electronic device, based on the distance between the first gesture recognition point and the second gesture recognition point and the angle before and after movement of the second gesture recognition point relative to the auxiliary point, for a plurality of frames after thinning out a plurality of frames included in the imaging data at a thinning rate that is greater the shorter the distance between an imaging unit that captured the imaging data and the operator's hand.

Images