Electronic apparatus, program, and control method

Abstract

SOLUTION: An electronic apparatus comprises: an acquisition unit that acquires imaging data including a hand, as a subject, of an operator of the electronic apparatus; an estimation unit that estimates skeleton data including a plurality of points corresponding to the hand of the operator included in the imaging data; and a recognition unit that recognizes a gesture of the operator for operating the electronic apparatus on the basis of the distance between two gesture recognition points of the plurality of points included in the skeleton data, and a threshold corresponding to the distance between an imaging unit, which has imaged the imaging data, and the hand of the operator.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an electronic device, a program, and a control method. [Background technology]

[0002] Patent Document 1 describes a technique for calculating the position of a specific part of an operator's hand based on skeleton data of the operator's hand of an electronic device, and determining a cursor position that corresponds to that position. [Prior art document] [Patent documents] [Patent Document 1] Patent No. 7213396 Summary of the Invention [Means for solving the problem]

[0003] According to one embodiment of the present invention, there is provided an electronic device. The electronic device may include an acquisition unit that acquires imaging data including a hand of an operator of the electronic device as a subject. The electronic device may include an estimation unit that estimates skeleton data including a plurality of points corresponding to the operator's hand included in the imaging data. The electronic device may include a recognition unit that recognizes a gesture by the operator to operate the electronic device based on a distance between two gesture recognition points among the plurality of points included in the skeleton data and a threshold corresponding to the distance between the imaging unit that captured the imaging data and the operator's hand. In the electronic device, the recognition unit may recognize a gesture by the operator to operate the electronic device based on the threshold, which is larger as the distance between the imaging unit and the operator's hand is shorter. In any of the electronic devices, the recognition unit may recognize a gesture by the operator based on a distance between a first gesture recognition point corresponding to the tip of the thumb and a second gesture recognition point corresponding to the tip of the index finger included in the skeleton data and the threshold. The recognition unit may determine that the operator has performed a pinch gesture when the distance between the first gesture recognition point and the second gesture recognition point becomes shorter than the threshold. After determining that the operator has performed the pinch gesture, the recognition unit may determine whether the operator has performed a click gesture, a swipe gesture, or a drag-and-drop gesture based on a time until the pinch gesture is released and an amount of movement of the operator's hand while making the pinch gesture. In any of the electronic devices, the recognition unit may recognize the gesture performed by the operator based on the distance between the first gesture recognition point and the second gesture recognition point, the threshold, and an angle before and after movement of the second gesture recognition point relative to an auxiliary point corresponding to the base of the index finger of the hand, which is included in the skeleton data.In any of the electronic devices, the recognition unit may recognize a gesture by the operator to operate the electronic device based on the distance between the two gesture recognition points and a threshold value corresponding to the distance between the imaging unit that captured the imaging data and the operator's hand, for a plurality of frames after thinning out the plurality of frames included in the imaging data at a thinning rate that increases as the distance between the imaging unit and the operator's hand decreases.

[0004] According to one embodiment of the present invention, there is provided an electronic device. The electronic device may include an acquisition unit that acquires imaging data including a hand of an operator of the electronic device as a subject. The electronic device may include an estimation unit that estimates skeleton data including a plurality of points corresponding to the operator's hand included in the imaging data. The electronic device may include a recognition unit that recognizes a gesture by the operator based on a distance between a first gesture recognition point corresponding to the tip of the thumb of the hand and a second gesture recognition point corresponding to the tip of the index finger of the hand, among the plurality of points included in the skeleton data, and an angle before and after movement of the second gesture recognition point relative to an auxiliary point corresponding to the base of the index finger of the hand.

[0005] According to one embodiment of the present invention, there is provided a program for causing a computer to function as the electronic device.

[0006] According to one embodiment of the present invention, there is provided a control method executed by an electronic device. The control method may include an acquisition step of acquiring imaging data including a hand of an operator of the electronic device as a subject. The control method may include an estimation step of estimating skeleton data including a plurality of points corresponding to the operator's hand included in the imaging data. The control method may include a recognition step of recognizing a gesture by the operator to operate the electronic device based on a distance between two gesture recognition points among the plurality of points included in the skeleton data and a threshold corresponding to the distance between an imaging unit that captured the imaging data and the operator's hand.

[0007] According to one embodiment of the present invention, there is provided a control method executed by an electronic device. The control method may include an acquisition step of acquiring imaging data including a hand of an operator of the electronic device as a subject. The control method may include an estimation step of estimating skeleton data including a plurality of points corresponding to the operator's hand included in the imaging data. The control method may include a recognition step of recognizing a gesture by the operator based on a distance between a first gesture recognition point corresponding to the tip of the thumb of the hand and a second gesture recognition point corresponding to the tip of the index finger of the hand, among the plurality of points included in the skeleton data, and an angle before and after movement of the second gesture recognition point relative to an auxiliary point corresponding to the base of the index finger of the hand.

[0008] The above summary of the invention does not list all of the necessary features of the present invention, and subcombinations of these features may also constitute inventions. [Brief explanation of the drawings]

[0009] [Figure 1] 1 shows a schematic diagram of an example of an electronic device 100. [Figure 2] 1 shows an example of a functional configuration of the electronic device 100. [Figure 3] 10 is an explanatory diagram for explaining an example of a plurality of points included in skeleton data 30 of a hand 20 of an operator 40. FIG. [Figure 4] 10 is an explanatory diagram for explaining a method for determining the cursor position by the determination unit 128. FIG. [Figure 5] FIG. 10 is an explanatory diagram for explaining at least three noise removal points selected from a plurality of points included in skeleton data 30 of a hand 20. [Figure 6] 10 is an explanatory diagram for explaining gesture recognition by a recognition unit 130. FIG. [Figure 7]10 is an explanatory diagram for explaining gesture recognition by a recognition unit 130. FIG. [Figure 8] 10 is an explanatory diagram for explaining that the pinch recognition threshold is changed in accordance with the distance between the image capturing unit 102 and the hand 20 of the operator 40. FIG. [Figure 9] 10 is an explanatory diagram for explaining another example of gesture recognition by the recognition unit 130. FIG. [Figure 10] 10 is an explanatory diagram for explaining noise reduction processing in gesture recognition by a recognition unit 130. FIG. [Figure 11] 1 shows an example of a hardware configuration of a computer 1200 that functions as the electronic device 100. DETAILED DESCRIPTION OF THE INVENTION

[0010] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0011] Conventionally, when the position of a specific part of the hand of an electronic device operator is calculated and a cursor position is determined to correspond to that position, large movements are required for operation because only the difference in the movement distance of one point is calculated. Electronic device 100 according to this embodiment determines the cursor position based on multiple cursor operation points among multiple points included in skeleton data, thereby enabling cursor operation with small movements. For example, large cursor operation can be achieved with small movements, allowing even users with relatively limited mobility to efficiently operate the cursor.

[0012] 1 schematically illustrates an example of an electronic device 100. The electronic device 100 includes an imaging unit 102 capable of capturing an image of an operator 40 of the electronic device 100. The electronic device 100 may be, but is not limited to, a smartphone, a tablet terminal, a PC (Personal Computer), a digital home appliance, a wearable terminal, or the like.

[0013] The imaging unit 102 may be a camera or a sensor built into the electronic device 100. The imaging unit 102 may also be an external camera or a sensor. The imaging unit 102 provides imaging data to the electronic device 100. The imaging data may be video data. The imaging data may also be continuously captured still image data. The imaging unit 102 may have a function to measure the distance to an imaging target. The imaging unit 102 may include, for example, a depth camera. The imaging unit 102 may also be a so-called stereo camera.

[0014] The electronic device 100 may infer skeleton data including multiple points corresponding to the hand 20 of the operator 40 contained in the imaging data acquired from the imaging unit 102, and may determine a cursor position for operating the electronic device 100 or recognize a gesture by the operator 40 based on some of the multiple points contained in the skeleton data.

[0015] 2 schematically illustrates an example of the functional configuration of the electronic device 100. The electronic device 100 includes an imaging unit 102, a display unit 104, a memory 106, a storage unit 108, and a control unit 120. The control unit 120 includes a setting unit 122, an acquisition unit 124, an estimation unit 126, a determination unit 128, a recognition unit 130, and an operation unit 132.

[0016] The display unit 104 is a display arranged on the front side of the electronic device 100. The display unit 104 may be a touch panel display. The display unit 104 displays icons for operating the electronic device 100 and the like.

[0017] The memory 106 may be configured by a memory of a microcomputer integrated with the control unit 120. Alternatively, the memory 106 may be a RAM (Random Access Memory) or the like configured by an independent semiconductor device connected to the control unit 120. The memory 106 may temporarily store various programs executed by the control unit 120 and various data referenced by these programs.

[0018] The storage unit 108 may be configured by a writable semiconductor device memory, such as a RAM or a flash memory, built into the electronic device 100. The storage unit 108 may also be configured by an external memory connected to the electronic device 100.

[0019] The setting unit 122 performs various settings. The setting unit 122 may perform various settings in accordance with instructions from the operator 40.

[0020] The acquisition unit 124 acquires imaging data from the imaging unit 102. The acquisition unit 124 may acquire imaging data that includes the hand of the operator 40 as a subject. The setting unit 122 may recognize the hand of the operator 40 by analyzing the imaging data acquired by the acquisition unit 124, and set a reference position of the hand.

[0021] The estimation unit 126 estimates skeleton data corresponding to the hand of the operator 40 contained in the imaging data acquired by the acquisition unit 124. The skeleton data may represent the shape of a volumetric object using a set of line segments that form the skeleton of the object. For example, the skeleton data may represent each part of the object using line segments that indicate the axis of the part or line segments that indicate the frame of the part. The skeleton of the object represented by the skeleton data may differ from the actual skeleton of the object. For example, the skeleton data of the hand does not necessarily need to follow the bones of the hand, but may include line segments that indicate at least the position of each finger and the bending direction of each finger. Alternatively, the skeleton data may be a collection of points called a skeleton mesh, which is a sample of several representative points of the skeleton.

[0022] The skeleton data includes a plurality of points. Each of the plurality of points corresponds to, for example, the base, joint, tip, etc. of each part. For example, skeleton data of a hand includes a plurality of points that correspond to the base of the wrist, the base, joint, and tip of the thumb, the base, joint, and tip of the index finger, the base, joint, and tip of the middle finger, the base, joint, and tip of the ring finger, and the base, joint, and tip of the little finger.

[0023] The algorithm used by the estimation unit 126 to estimate skeleton data is not particularly limited, but as an example, the estimation unit 126 may estimate skeleton data corresponding to the operator's hand using a learning model that has been machine-learned using pairs of image data of a large number of hands and skeleton data of the hands as training data.

[0024] As a specific example, the estimation unit 126 first extracts a region including a hand from the imaging data. The algorithm for extracting a region including a hand from the imaging data is not particularly limited, and a known algorithm may be used. As an example, the estimation unit 126 may extract a region including a hand from the imaging data by detecting a clenched fist or a palm in the imaging data. Note that in this embodiment, the palm refers to the part of the hand other than the fingers. As an example, the estimation unit 126 may detect a palm when the operator 40 is not clenching his / her hand, for example, when the operator 40's hand is open, and may detect a fist when the operator 40 is clenching his / her hand. The estimation unit 126 may extract a region including the operator 40's hand based on the position and range of the detected clenched fist or palm. The estimation unit 126 may estimate skeleton data corresponding to the hand from the region including the hand from the imaging data. As an example, the estimation unit 126 may estimate skeleton data corresponding to the hand using the learning model described above.

[0025] The determination unit 128 determines a cursor position for operating the electronic device 100. The determination unit 128 may determine a cursor position for operating the electronic device 100 based on a plurality of cursor operation points among a plurality of points included in the skeleton data estimated by the estimation unit 126. Which points among the plurality of points included in the skeleton data are to be used as cursor operation points may be set in advance by the setting unit 122. The setting unit 122 sets, for example, two points among the plurality of points included in the skeleton data as cursor operation points. The setting unit 122 may also set three or more points among the plurality of points included in the skeleton data as cursor operation points.

[0026] The recognition unit 130 recognizes a gesture made by the operator 40 to operate the electronic device 100. The recognition unit 130 may recognize a gesture made by the operator 40 based on the distance between two gesture recognition points among the multiple points included in the skeleton data estimated by the estimation unit 126 and a threshold value corresponding to the distance between the imaging unit 102 that captured the imaging data and the hand of the operator 40. The recognition unit 130 may recognize a gesture made by the operator 40 based on a threshold value that increases as the distance between the imaging unit 102 and the hand of the operator 40 decreases.

[0027] The operation unit 132 operates the electronic device 100 based on the cursor position determined by the determination unit 128 and the gesture by the operator 40 recognized by the recognition unit 130. For example, when the recognition unit 130 recognizes a gesture to select an icon, the operation unit 132 selects the icon corresponding to the cursor position and executes the application corresponding to the icon.

[0028] FIG. 3 is an explanatory diagram illustrating an example of multiple points included in skeleton data 30 of the hand 20 of the operator 40. The skeleton data 30 illustrated in FIG. 3 includes 21 points: point 301, point 302, point 303, point 304, point 305, point 306, point 307, point 308, point 309, point 310, point 311, point 312, point 313, point 314, point 315, point 316, point 317, point 318, point 319, point 320, and point 321. The number of points included in the skeleton data 30 does not have to be 21. The number of points may be less than 21 or more than 21. The multiple points may include points at the tips of the fingers, the first joints of the fingers, the second joints of the fingers, and points on the palm. The points on the palm may include points at the bases of the fingers. The positions of the multiple points may differ from those shown in FIG. 3. For example, the multiple points may include points that are not finger tips or knuckles.

[0029] The setting unit 122 may set multiple cursor operation points from multiple points included in the skeleton data 30. For example, the setting unit 122 sets multiple points selected by the operator 40 from multiple points included in the skeleton data 30 as multiple cursor operation points. The multiple cursor operation points may be set arbitrarily. The multiple cursor operation points may be selected from the same finger. The multiple cursor operation points may be selected from different fingers. The multiple cursor operation points may be selected from the palm.

[0030] The setting unit 122 sets, for example, two cursor operation points. As an example, the setting unit 122 sets, as the cursor operation points, point 302 at the base of the thumb and point 304 at the tip of the thumb from among the multiple points included in the skeleton data 30. By allowing multiple cursor operation points to be arbitrarily selected from the multiple points included in the skeleton data 30, for example, even a person who has lost part of a hand in a traffic accident or the like can select points corresponding to the remaining part of the hand as cursor operation points and use these points to determine the position of a cursor for operating the electronic device 100.

[0031] 4 is an explanatory diagram for explaining a method for determining the cursor position by the determination unit 128. Here, an example will be described in which the operator 40 can operate the cursor position by moving the hand 20 with the hand 20 open. The hand 20 shown by the dotted line is the hand before movement, and the hand 20 shown by the solid line is the hand after movement. Here, an example will be described in which a point 302 at the base of the thumb and a point 304 at the tip of the thumb are set as cursor operation points.

[0032] Based on the multiple cursor operation points, the determination unit 128 determines the position of the cursor 51 on the display unit 104 of the electronic device 100. In the example shown in FIG. 4, the determination unit 128 determines the position of the cursor 51 based on the point 302 and the point 304.

[0033] The determination unit 128 may determine the position of the cursor 51 based on a spherical space defined by a plurality of cursor operation points. In the example shown in Fig. 4, the determination unit 128 defines a spherical space 510 based on two points, point 302 and point 304. For example, the determination unit 128 defines the spherical space 510 with point 302 as the center and a length determined based on the distance between point 302 and point 304 as the radius.

[0034] The determination unit 128 may determine the position of the cursor 51 based on an expanded spherical space 511 obtained by expanding the spherical space 510. The determination unit 128 defines the expanded spherical space 511 by expanding the radius of the spherical space 510 in accordance with the distance between the imaging unit 102 and the operator's hand 20, for example.

[0035] The determination unit 128 may determine the position of the cursor 51 based on the amount of movement identified by reflecting the angular change of the vector formed by the multiple cursor operation points in the spherical space 510 in the expanded spherical space 511. The determination unit 128 may move the cursor 51 by the amount of movement identified by reflecting the angular change of the vector formed by the multiple cursor operation points in the spherical space 510 in the expanded spherical space 511.

[0036] In the example shown in Figure 4, the determination unit 128 moves the cursor 51 by the movement amount 52 determined by reflecting in the expanded spherical space 511 the angle 330 between the vector formed by the points 302 and 304 on the hand 20 before the movement and the vector formed by the points 302 and 304 on the hand 20 after the movement.

[0037] The determination unit 128 may move the cursor 51 vertically or horizontally depending on the rotation direction of the hand 20. For example, when the point 304 moves up or down relative to the point 302, the determination unit 128 moves the cursor 51 vertically by the amount of movement 52 determined by reflecting the angle change in the up or down direction in the expanded spherical space 511. For example, when the point 304 moves forward or backward relative to the point 302, the determination unit 128 moves the cursor 51 horizontally by the amount of movement 52 determined by reflecting the angle change in the forward or backward direction in the expanded spherical space 511. This makes it possible to provide the operator 40 with an operation environment in which the cursor 51 can be moved up or down by tilting the hand 20 so that the point 304 moves up or down relative to the point 302, and the cursor 51 can be moved horizontally by tilting the hand 20 so that the point 304 moves forward or backward relative to the point 302. The relationship between the rotation direction of the hand 20 and the movement direction of the cursor 51 is not limited to this. The relationship between the rotation direction of the hand 20 and the movement direction of the cursor 51 may be set by the setting unit 122.

[0038] The amount of movement corresponding to angle 330 in spherical space 510 is not very large, so when using this amount of movement to determine the position of cursor 51, it is necessary to move hand 20 a lot to move cursor 51 a lot. In contrast, by using expanded spherical space 511, the amount of movement can be increased, making it possible to move cursor 51 a lot with a small movement of hand 20, thereby improving the operability of moving cursor 51.

[0039] Even when the operator 40 does not intend to do so, the arm, hand 20, and fingers of the operator 40 always sway to a certain degree up and down, left and right, and back and forth. Due to this swaying of the hand 20, the multiple points included in the skeleton data 30 also sway. Hereinafter, this swaying will be referred to as noise. Noise in the cursor operation points causes meaningless vibrations and flickering of the cursor position, thereby degrading the user experience of the operator 40.

[0040] The electronic device 100 according to this embodiment may further include a function for reducing such noise. The electronic device 100 sets at least three noise reduction points from a plurality of points included in the skeleton data 30 of the hand 20, and reduces noise using the at least three noise reduction points.

[0041] FIG. 5 is an explanatory diagram illustrating at least three noise removal points selected from a plurality of points included in skeleton data 30 of hand 20. The at least three noise removal points may be set by setting unit 122. For example, setting unit 122 sets a plurality of points selected by operator 40 from a plurality of points included in skeleton data 30 as a plurality of noise removal points. The at least three noise removal points may be arbitrarily selected from a plurality of points included in skeleton data 30. In the example shown in FIG. 5, point 304, point 305, and point 309 are set as noise removal points.

[0042] The determination unit 128 may determine the position of the cursor 51 based on a plurality of cursor operation points identified based on an area (sometimes referred to as a noise removal area 600) constituted by at least three noise removal points included in the skeleton data 30, out of a plurality of cursor operation points extracted from consecutive frames included in the imaging data acquired by the acquisition unit 124. For example, after determining the noise removal area 600, the determination unit 128 determines the center of gravity of the at least three noise removal points for each of the consecutive frames included in the imaging data, and if the center of gravity is included in the noise removal area 600, determines the position of the cursor 51 based on the plurality of cursor operation points.

[0043] For example, the determination unit 128 determines the position of the cursor 51 based on the remaining cursor operation points obtained by removing, as noise, cursor operation points that do not correspond to the noise removal region 600 from the plurality of cursor operation points extracted from consecutive frames included in the imaging data. For example, after determining the noise removal region 600, the determination unit 128 determines the center of gravity of at least three noise removal points for each of the consecutive frames included in the imaging data, and if the center of gravity is not included in the noise removal region 600, removes the plurality of cursor operation points of that frame as noise. The determination unit 128 may determine the position of the cursor 51 based on the remaining cursor operation points of the plurality of frames excluding the plurality of cursor operation points removed as noise.

[0044] As a result, when the hand 20 of the operator 40 shakes unintentionally, the cursor 51 is prevented from moving in accordance with the multiple cursor operation points at the time of shaking. This reduces the possibility that the cursor 51 will move against the intention of the operator 40, and improves the user experience of the operator 40.

[0045] The at least three noise removal points may be at least one selected from the cursor operation points or at least one selected from the palm points. This allows the noise removal region 600 to be set corresponding to the cursor position, effectively separating coordinate information to be removed as noise from coordinate information to be used as a signal for cursor operation. For example, when the multiple cursor operation points are two points, the point 304 at the tip of the thumb and the point 302 at the base of the thumb, the at least three noise removal points may be three points, the point 304 at the tip of the thumb, the point 305 at the base of the index finger, and the point 309 at the base of the middle finger, and the noise removal region 600 may be configured based on these three points. However, this is not limited to this. For example, the at least three noise removal points may not include any cursor operation points, or may include more than three points.

[0046] 6 is an explanatory diagram for describing gesture recognition by the recognition unit 130. The recognition unit 130 may recognize a gesture by the operator 40 using two gesture recognition points out of a plurality of points included in the skeleton data 30. Here, a case will be described in which points 304 and 308 are set as gesture recognition points, and the recognition unit 130 recognizes a pinch gesture by the operator 40. Point 304 may be an example of a first gesture recognition point, and point 308 may be an example of a second gesture recognition point.

[0047] The two gesture recognition points may be set by the setting unit 122. For example, the setting unit 122 sets two points selected by the operator 40 from a plurality of points included in the skeleton data 30 as the two gesture recognition points. The two gesture recognition points may be set arbitrarily. The two gesture recognition points may be points at the tips of two different fingers. However, this is not limiting. For example, the two gesture recognition points may include points that are not the tips of fingers.

[0048] 6, the recognition unit 130 may determine that the operator 40 has made a pinch gesture when the distance between the point 304 and the point 308 changes from a state longer than a predetermined threshold (sometimes referred to as a pinch recognition threshold) to a state shorter than the predetermined threshold. After determining that the operator 40 has made a pinch gesture, the recognition unit 130 may determine that the operator 40 has released the pinch gesture when the distance between the point 304 and the point 308 changes from a state shorter than the pinch recognition threshold to a state longer than the predetermined threshold.

[0049] The recognition unit 130 may recognize a gesture by the operator 40 using a pinch recognition threshold corresponding to the distance between the image capture unit 102 and the hand 20. The recognition unit 130 may recognize a gesture by the operator 40 using a pinch recognition threshold that is larger as the distance between the image capture unit 102 and the hand 20 becomes shorter.

[0050] 6 , the recognition unit 130 may recognize a gesture by the operator 40 based on the distance between the point 308 and the point 304 and a pinch recognition threshold corresponding to the distance between the image capture unit 102 and the hand 20. The recognition unit 130 may continuously compare the distance between the point 308 and the point 304 with the pinch recognition threshold corresponding to the distance between the image capture unit 102 and the hand 20, and determine that the operator 40 has made a pinch gesture when the distance becomes shorter than the pinch recognition threshold. After determining that the operator 40 has made a pinch gesture, the recognition unit 130 may continuously compare the distance between the point 308 and the point 304 with the pinch recognition threshold corresponding to the distance between the image capture unit 102 and the hand 20, and determine that the operator 40 has released the pinch gesture when the distance becomes longer than the pinch recognition threshold.

[0051] The shorter the distance between the imaging unit 102 and the hand 20, the larger the size of the hand 20 in the imaging data, and the larger the amount of finger movement when the operator 40 makes a pinch gesture. Conversely, the longer the distance between the imaging unit 102 and the hand 20, the smaller the size of the hand 20 in the imaging data, and the smaller the amount of finger movement when the operator 40 makes a pinch gesture. Therefore, if the same pinch recognition threshold is used despite different distances between the imaging unit 102 and the hand 20, it becomes difficult to accurately recognize the pinch gesture. In contrast, the recognition unit 130 can improve the accuracy of pinch gesture recognition by using a pinch recognition threshold that corresponds to the distance between point 308 and point 304 and the distance between the imaging unit 102 and the hand 20.

[0052] The recognition unit 130 may use a pinch recognition threshold value that increases as the size of the hand 20 in the imaging data increases, thereby using a pinch recognition threshold value that corresponds to the distance between the imaging unit 102 and the hand 20. The correspondence relationship between the size of the hand 20 in the imaging data and the pinch recognition threshold value may be set in advance by the setting unit 122.

[0053] If the imaging unit 102 has a function of measuring the distance to the imaging target, the recognition unit 130 may use a pinch recognition threshold corresponding to the distance between the imaging unit 102 and the hand 20 measured by the imaging unit 102. The correspondence relationship between the distance between the imaging unit 102 and the hand 20 and the pinch recognition threshold may be set in advance by the setting unit 122.

[0054] The recognition unit 130 may recognize a click gesture by the operator 40. The recognition unit 130 may recognize a swipe gesture by the operator 40. The recognition unit 130 may recognize a drag-and-drop gesture by the operator 40.

[0055] After determining that the operator 40 has made a pinch gesture, the recognition unit 130 may determine whether the operator 40 has made a click gesture, a swipe gesture, or a drag-and-drop gesture based on the time until the pinch gesture is released and the amount of movement of the operator 40's hand while the operator 40 is making the pinch gesture.

[0056] For example, the recognition unit 130 determines that the operator 40 has performed a click gesture if the pinch gesture is released before a predetermined first time has elapsed after determining that the operator 40 has performed a pinch gesture. For example, the recognition unit 130 determines that the operator 40 has performed a swipe gesture if the first time has elapsed after determining that the operator 40 has performed a pinch gesture and the time until the hand 20 in the pinch gesture state starts to move does not exceed a predetermined second time, and determines that the operator 40 has performed a drag-and-drop gesture if the second time has elapsed. The recognition unit 130 may determine that the drag has started when the hand 20 starts to move if the first time has elapsed after determining that the operator 40 has performed a pinch gesture and the time until the hand 20 in the pinch gesture state starts to move exceeds the predetermined second time, and may determine that a drop has been performed when the hand 20 has moved and the pinch gesture has been released.

[0057] With this configuration, for example, when the operator 40 puts his / her thumb and index finger together and then immediately releases them, it can be determined that the operator 40 has performed a click, when the hand 20 starts to move after the thumb and index finger have been put together for a while, it can be determined that the operator 40 has performed a swipe, and when the hand 20 starts to move after the thumb and index finger have been put together for a little longer, it can be determined that the operator 40 has performed a drag-and-drop. This enables intuitive operation by the operator 40.

[0058] The recognition unit 130 may use different pinch recognition thresholds when determining that a pinch gesture has been performed and when determining that a pinch gesture has been released. For example, the recognition unit 130 may determine that a pinch gesture has been released using a pinch recognition threshold that is greater than the pinch recognition threshold used when determining that a pinch gesture has been performed. This makes it easier to recognize that the pinch gesture is being maintained, even if, for example, the tip of the thumb and the tip of the index finger are slightly separated against the operator 40's intention during a swipe gesture or a drag-and-drop gesture. This reduces the possibility that the swipe gesture or the drag-and-drop gesture will be released against the operator 40's intention, thereby improving the user experience of the operator 40.

[0059] 7 is an explanatory diagram for explaining gesture recognition by the recognition unit 130. The recognition unit 130 recognizes that a pinch gesture is being made by the operator 40 based on the imaging data continuously acquired by the acquisition unit 124. Here, a state in which the recognition unit 130 has specified an area in the imaging data acquired by the acquisition unit 124 that includes the hand 20 will be described as a start state.

[0060] In step (sometimes abbreviated as S) 101, the recognition unit 130 aligns the hand 20 with the center of the coordinate system. Specifically, for example, the recognition unit 130 aligns the hand 20 with the center of the coordinate system in a Region of Interest (RoI). In S102, the recognition unit 130 rotates the hand 20 to adjust the orientation of the hand 20. Here, the orientation of the hand 20 is adjusted so that the index finger points upward. In S103, the recognition unit 130 reduces noise at the fingertip points. Here, the fingertip points are point 304 at the tip of the thumb and point 308 at the tip of the index finger. Here, the noise reduction refers to the reduction of noise in the X coordinate and noise in the Y coordinate. A specific method of noise reduction will be described later. A series of processes from S101 to S103 may be referred to as preprocessing. The preprocessing may be performed continuously on the imaging data continuously acquired by the acquisition unit 124. Here, the following description will be continued assuming that the recognition unit 130 has recognized that a pinch gesture has been made after the preprocessing.

[0061] In S104, the recognition unit 130 determines whether a predetermined first time has elapsed since the pinch gesture was made. If it is determined that the first time has not elapsed, the process proceeds to S105, where it is determined that a "click" has started, and the process proceeds to S112. If it is determined that the first time has elapsed, the process proceeds to S106. In S112, the recognition unit 130 determines whether the pinch gesture has been released. If it is determined that the pinch gesture has been released, it is determined that the "click" has ended, and the recognition unit 130 determines that the pinch gesture has been released, and the process ends. If it is determined that the pinch gesture has not been released, the process proceeds to S110, where it is determined that a "drag and drop" has started.

[0062] In S106, the recognition unit 130 determines whether the time until the hand 20 in the pinch gesture state starts to move exceeds a predetermined second time. If it determines that it has not exceeded the second time, the process proceeds to S108, where it is determined that a "swipe" has started, and proceeds to S112. If it determines that it has exceeded the second time, the process proceeds to S110, where it is determined that a "drag and drop" has started. In S112, the recognition unit 130 determines whether the pinch gesture has been released. If it determines that it has been released, it determines that the "swipe" has ended, determines that it is a "swipe" gesture, and ends the determination. If it determines that it has not been released, the process proceeds to S110, where it is determined that a "drag and drop" has started.

[0063] If the recognition unit 130 determines in S110 that a "drag and drop" has started, the process proceeds to S112. In S112, the recognition unit 130 determines whether the pinch gesture has been released. If it determines that the pinch gesture has been released, the recognition unit 130 determines that the "drag and drop" has ended, determines that the gesture is a "drag and drop," and ends the determination. If it determines that the pinch gesture has not been released, the process proceeds again to S110, and determines that the "drag and drop" state is continuing. Thereafter, the recognition unit 130 repeats the process of determining that the "drag and drop" state is continuing until it determines that the pinch gesture has been released in S112.

[0064] FIG. 8 is an explanatory diagram illustrating changing the pinch recognition threshold in accordance with the distance between the imaging unit 102 and the hand 20 of the operator 40. When the distance between the imaging unit 102 and the hand 20 of the operator 40 is short, the hand 20 appears large in the image included in the imaging data, as shown in the upper part of FIG. 8. Therefore, the recognition unit 130 may use a larger pinch recognition threshold as the distance between the imaging unit 102 and the hand 20 of the operator 40 becomes shorter. As a specific example, if the size of the portion of the image in which the hand 20 appears is X×Y, the value of the pinch recognition threshold S1 may be Y / 8. When the distance between the imaging unit 102 and the hand 20 of the operator 40 is long, the hand 20 appears small in the image included in the imaging data, as shown in the lower part of FIG. 8. Therefore, the recognition unit 130 may use a smaller pinch recognition threshold as the distance between the imaging unit 102 and the hand 20 of the operator 40 becomes longer. For example, if the size of the portion of the image in which the hand 20 is shown is X / 4 × Y / 4, the value of the pinch recognition threshold S2 may be S1 / 4, i.e., Y / 32. The size of the portion in which the hand 20 is shown and the corresponding pinch recognition threshold value are not limited to those shown in FIG. 8 . The pinch recognition threshold may be dynamically set according to the ratio of the size of the hand portion. For example, if the hand is far away and the size of the recognizable hand portion is 25% (1 / 4) smaller than the size when the hand is close, the pinch recognition threshold is dynamically set according to the ratio of the reduced size of the hand portion.

[0065] 9 is an explanatory diagram for describing another example of gesture recognition by the recognition unit 130. The recognition unit 130 may recognize a gesture by the operator 40 by using an auxiliary point in addition to two gesture recognition points among the multiple points included in the skeleton data 30. In FIG. 9, an example will be described in which points 308 and 304 are used as the two gesture recognition points and point 305 corresponding to the base of the index finger is used as the auxiliary point.

[0066] The recognition unit 130 may recognize the gesture made by the operator 40 based on the distance between the point 308 and the point 304 and the angle 700 before and after the movement of the point 304 with respect to the point 305 .

[0067] For example, the recognition unit 130 determines that the operator 40 has made a pinch gesture when the distance between the point 304 and the point 308 changes from longer than the pinch recognition threshold to shorter than the pinch recognition threshold and the angle 700 is larger than a predetermined angle. Even if the distance between the point 304 and the point 308 changes from longer than the pinch recognition threshold to shorter than the pinch recognition threshold, the recognition unit 130 does not need to determine that the operator 40 has made a pinch gesture when the angle 700 is smaller than the predetermined angle.

[0068] For example, after determining that the operator 40 has made a pinch gesture, the recognition unit 130 determines that the operator 40 has released the pinch gesture if the distance between the points 304 and 308 changes from being shorter than the pinch recognition threshold to being longer and the angle 700 is larger than a predetermined angle. Even if the distance between the points 304 and 308 changes from being shorter than the pinch recognition threshold to being longer, the recognition unit 130 does not need to determine that the operator 40 has released the pinch gesture if the angle 700 is smaller than the predetermined angle.

[0069] When attempting to recognize a gesture using only two gesture recognition points, noise at the two gesture recognition points may cause erroneous gesture recognition. For example, when the recognition unit 130 recognizes a pinch gesture, even though the distance between the two gesture recognition points is actually longer than the pinch recognition threshold, noise may cause the distance to be shorter than the pinch recognition threshold, resulting in the recognition unit 130 erroneously recognizing that a pinch gesture has been made. Conversely, even though the distance between the two gesture recognition points is actually shorter than the pinch recognition threshold, noise may cause the distance to be longer than the pinch recognition threshold, resulting in the recognition unit 130 erroneously recognizing that a pinch gesture has not been made. In contrast, by having the recognition unit 130 perform recognition using an auxiliary point as well, the possibility of such erroneous recognition can be reduced.

[0070] FIG. 10 is an explanatory diagram for explaining noise reduction processing in gesture recognition by the recognition unit 130. In FIG.

[0071] The recognition unit 130 may recognize a gesture by the operator 40 based on the distance between the two gesture recognition points and a threshold value corresponding to the distance between the imaging unit 102 and the hand 20 of the operator 40, from the multiple frames included in the imaging data, with the thinning rate increasing as the distance between the imaging unit 102 and the hand 20 of the operator 40 decreases. Thus, thinning one or more frames from multiple consecutive frames included in the imaging data may be referred to as filtering. The larger the filter value, the higher the thinning rate per unit frame. By performing filtering and performing a numerical calculation on the noise based on the filter value, noise can be reduced.

[0072] 10, a description will be given of changing the filter value when recognizing a pinch gesture in accordance with the distance between the imaging unit 102 and the hand 20 of the operator 40. In reality, each of a plurality of points is affected by noise corresponding to the shaking of the hand 20, but for simplicity, only the noise at point 304 at the tip of the thumb will be described in FIG.

[0073] When the distance between the imaging unit 102 and the hand 20 of the operator 40 is short, the hand 20 appears large in the image included in the imaging data, as shown in the upper part of Fig. 10. Therefore, the movement amount of the gesture recognition point in response to changes in the shape of the hand 20 is large, and the noise is also large. When the size of the part where the hand 20 is captured is X x Y, the point 304 may move between the solid line and the dashed line, and for example, the movement amount of the point 304 may be Y / 5. Noise may occur between the solid line and the dashed line, and for example, the noise value N1 may be Y / 5.

[0074] Since the movement of the gesture recognition point is large, the coordinate change of the gesture recognition point between adjacent frames on the time axis is large. Therefore, even if the filter value is increased and frames are thinned out at a higher thinning rate, changes in the shape of the hand can be detected. The filter value may be, for example, F1. For example, noise processing may be performed by dividing the noise value N1 by the filter value. For example, the noise value N2 after noise processing may be N1 / F1, that is, the value obtained by dividing Y / 5 by the filter value F1. This reduces noise, thereby reducing erroneous gesture recognition and improving the user experience.

[0075] When the distance between the imaging unit 102 and the hand 20 of the operator 40 is long, the hand 20 appears small in the image included in the imaging data, as shown in the lower part of FIG. 10. Therefore, the amount of movement of the gesture recognition point in response to changes in the shape of the hand 20 is small, and noise is also small. When the size of the portion where the hand 20 is captured is X / 4 × Y / 4, the point 304 may move between the solid line and the dashed line; for example, the amount of movement of the point 304 may be Y / 20. Noise may occur between the solid line and the dashed line; for example, the noise value N3 may be Y / 20.

[0076] Because the movement of the gesture recognition points is small, the coordinate change of the gesture recognition points between adjacent frames on the time axis is small. Therefore, it is difficult to detect changes in the hand shape unless the filter value is reduced and frames are thinned at a smaller thinning rate. The filter value may be, for example, F2. For example, noise processing may be performed by dividing the noise value N3 by the filter value. For example, the noise value N4 after noise processing may be N3 / F2, i.e., Y / 20 divided by the filter value F2. The filter value may be dynamically set according to the size ratio of the hand part. For example, if the hand is far away and the size of the recognizable hand part is 25% (1 / 4) smaller than the size when the hand is close, the filter value is dynamically set according to the size ratio of the reduced hand part.

[0077] In this way, by changing the filter value for noise processing according to the distance between the image capturing unit 102 and the hand 20 of the operator 40, it becomes possible to perform appropriate noise reduction processing according to the amount of movement of the gesture recognition point and the magnitude of the noise. This reduces erroneous recognition regardless of the distance between the image capturing unit 102 and the hand 20 of the operator 40, and improves the user experience.

[0078] 11 schematically illustrates an example of the hardware configuration of a computer 1200 that functions as the electronic device 100. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of an apparatus according to the present embodiment, or can cause the computer 1200 to perform operations associated with the apparatus according to the present embodiment or one or more "parts" thereof, and / or can cause the computer 1200 to perform a process according to the present embodiment or steps of the process. Such a program can be executed by the CPU 1212 to cause the computer 1200 to perform specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

[0079] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are interconnected by a host controller 1210. The computer 1200 also includes a GPU 1250. The computer 1200 also includes input / output units such as a communications interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid-state drive, or the like. The computer 1200 also includes a ROM 1230 and legacy input / output units such as a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0080] The CPU 1212 operates according to programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into the graphics controller itself, and causes the image data to be displayed on the display device 1218.

[0081] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0082] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or programs that depend on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, etc.

[0083] The programs are provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, RAM 1214, or ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and causes cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be configured by implementing operations or processing of information in accordance with the use of the computer 1200.

[0084] For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 may execute a communication program loaded into the RAM 1214 and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in the RAM 1214, the storage device 1224, a DVD-ROM, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.

[0085] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. to be read into the RAM 1214, and may perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0086] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and may undergo information processing. The CPU 1212 may perform various types of processing on data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequences of the programs, and write the results back to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. on the recording medium. For example, if multiple entries, each having an attribute value of a first attribute associated with an attribute value of a second attribute, are stored on the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0087] The above-described programs or software modules may be stored in a computer-readable storage medium on or near the computer 1200. A recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can also be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

[0088] The blocks in the flowcharts and block diagrams in the present embodiments may represent stages of a process in which an operation is performed or "parts" of an apparatus responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, including integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as field programmable gate arrays (FPGAs) and programmable logic arrays (PLAs), including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.

[0089] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by an appropriate device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture, including instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray disc, memory stick, integrated circuit card, etc.

[0090] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages ​​such as the “C” programming language or similar programming languages.

[0091] Computer-readable instructions may be provided to a general-purpose computer, a special-purpose computer, or a processor or programmable circuit of another programmable data processing device, such as a computer, locally or via a local area network (LAN) or a wide area network (WAN) such as the Internet, so that the processor or programmable circuit of the programmable data processing device executes the computer-readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Here, the computer may be a personal computer (PC), a tablet computer, a smartphone, a workstation, a server computer, a general-purpose computer, a special-purpose computer, or the like, or may be a computer system in which multiple computers are connected. Such a computer system in which multiple computers are connected is also called a distributed computing system, and is a broad definition of computers. In a distributed computing system, multiple computers collectively execute a program by each executing a portion of the program and passing data between computers as needed during program execution.

[0092] Examples of processors include a computer processor, a central processing unit (CPU), a processing unit, a microprocessor, a digital signal processor, a controller, a microcontroller, etc. A computer may have one or more processors. In a multiprocessor system with multiple processors, each processor executes a portion of a program and passes data between processors as needed during program execution, allowing the multiple processors to collectively execute a program. For example, in multitasking, each of the multiple processors may execute a portion of each task in small chunks by switching tasks at time slice intervals. In this case, which portion of a program each processor executes changes dynamically. Which portion of a program each of the multiple processors executes may also be statically determined by multiprocessor-aware programming.

[0093] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention.

[0094] It should be noted that the execution order of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and that the processes can be performed in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, this does not mean that the processes must be performed in this order. [Explanation of symbols]

[0095] 20 hand, 30 skeleton data, 40 operator, 51 cursor, 52 movement amount, 100 electronic device, 102 imaging unit, 104 display unit, 106 memory, 108 storage unit, 120 control unit, 122 setting unit, 124 acquisition unit, 126 estimation unit, 128 determination unit, 130 recognition unit, 132 operation unit, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321 point, 330 angle, 510 spherical space, 511 expanded spherical space, 600 noise removal area, 700 angle, 1200 computer, 1210 Host controller, 1212 CPU, 1214 RAM, 1216 graphics controller, 1218 display device, 1220 input / output controller, 1222 communication interface, 1224 storage device, 1230 ROM, 1240 input / output chip, 1250 GPU

Claims

1. An electronic device, an acquisition unit that acquires imaging data including a hand of an operator of the electronic device as a subject; an estimation unit that estimates skeleton data including a plurality of points corresponding to the operator's hand included in the imaging data; a recognition unit that recognizes a gesture made by the operator to operate the electronic device based on a distance between two gesture recognition points among the plurality of points included in the skeleton data and a threshold value corresponding to a distance between an imaging unit that captured the imaging data and a hand of the operator; An electronic device comprising:

2. The electronic device according to claim 1 , wherein the recognition unit recognizes a gesture made by the operator to operate the electronic device based on the threshold value, which is larger as the distance between the imaging unit and the operator's hand becomes shorter.

3. 2. The electronic device according to claim 1, wherein the recognition unit recognizes the gesture by the operator based on a distance between a first gesture recognition point corresponding to a tip of a thumb of the hand and a second gesture recognition point corresponding to a tip of an index finger of the hand, the distance being included in the skeleton data, and the threshold value.

4. 4. The electronic device according to claim 3, wherein the recognition unit determines that the operator has performed a pinch gesture when a distance between the first gesture recognition point and the second gesture recognition point becomes shorter than the threshold value.

5. 5. The electronic device according to claim 4, wherein the recognition unit determines whether the operator has performed a click gesture, a swipe gesture, or a drag-and-drop gesture based on a time period from when the pinch gesture is released to when the pinch gesture is released and an amount of movement of the operator's hand while the pinch gesture is being performed.

6. 6. The electronic device according to claim 3, wherein the recognition unit recognizes the gesture by the operator based on a distance between the first gesture recognition point and the second gesture recognition point, the threshold value, and an angle before and after movement of the second gesture recognition point relative to an auxiliary point corresponding to a base of an index finger of the hand, the auxiliary point being included in the skeleton data.

7. 6. The electronic device according to claim 1, wherein the recognition unit recognizes a gesture by the operator to operate the electronic device based on a distance between the two gesture recognition points and a threshold value corresponding to a distance between the imaging unit that captured the imaging data and the operator's hand, for a plurality of frames obtained by thinning out the plurality of frames included in the imaging data at a thinning rate that increases as the distance between the imaging unit and the operator's hand decreases.

8. An electronic device, an acquisition unit that acquires imaging data including a hand of an operator of the electronic device as a subject; an estimation unit that estimates skeleton data including a plurality of points corresponding to the operator's hand included in the imaging data; a recognition unit that recognizes a gesture made by the operator based on a distance between a first gesture recognition point corresponding to the tip of the thumb of the hand and a second gesture recognition point corresponding to the tip of the index finger of the hand, among the plurality of points included in the skeleton data, and an angle before and after movement of the second gesture recognition point with reference to an auxiliary point corresponding to the base of the index finger of the hand; An electronic device comprising:

9. A program for causing a computer to function as the electronic device according to any one of claims 1 to 5.

10. A control method executed by an electronic device, comprising: an acquisition step of acquiring imaging data including a hand of an operator of the electronic device as a subject; an estimation step of estimating skeleton data including a plurality of points corresponding to the operator's hand included in the imaging data; a recognition step of recognizing a gesture made by the operator to operate the electronic device based on a distance between two gesture recognition points among the plurality of points included in the skeleton data and a threshold value corresponding to a distance between an imaging unit that captured the imaging data and a hand of the operator; A control method comprising:

11. A control method executed by an electronic device, comprising: an acquisition step of acquiring imaging data including a hand of an operator of the electronic device as a subject; an estimation step of estimating skeleton data including a plurality of points corresponding to the operator's hand included in the imaging data; a recognition step of recognizing a gesture made by the operator based on a distance between a first gesture recognition point corresponding to the tip of the thumb of the hand and a second gesture recognition point corresponding to the tip of the index finger of the hand, among the plurality of points included in the skeleton data, and an angle before and after movement of the second gesture recognition point with reference to an auxiliary point corresponding to the base of the index finger of the hand; A control method comprising:

Images