Air gesture interaction method and device and electronic equipment

By calculating the target coordinates and features of the hand bone point with the wrist joint as the coordinate origin, and identifying and responding to the user's air gestures, the gesture trajectory offset problem is solved, and the response speed and user experience of electronic devices are improved.

CN118915902APending Publication Date: 2024-11-08BEIJING CHEHEJIA AUTOMOBILE TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202310511378.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-08
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The user's air-to-air gestures are counteracted due to the relative movement of the torso, arms, and wrist joints, and the electronic devices cannot respond in time, affecting the user experience.

Method used

By using the wrist joint as the coordinate origin, the target coordinates of the hand bone point are calculated, the gesture characteristics are identified, and the operation guidance icon is displayed when the setting conditions are met, and the corresponding operations are performed to achieve accurate identification and response to air-distance gestures.

Benefits of technology

Effectively identifying users' air-distance gestures reduces the impact of relative movement of body parts and improves the response speed and user experience of electronic devices to air-distance gestures.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118915902A_ABST
    Figure CN118915902A_ABST
Patent Text Reader

Abstract

The invention provides an air gesture interaction method and device and electronic equipment, and the method comprises the steps: according to a target coordinate of a hand skeleton point with a wrist joint as a coordinate origin, calculating at least one first gesture feature of the hand skeleton point of a first gesture image when the at least one first gesture feature of the hand skeleton point of the first gesture image meets a first set condition of a first air gesture; and displaying an operation guide icon corresponding to the first air gesture when the hand skeleton point in the first gesture image meets a first set condition of the first air gesture, and executing a target operation corresponding to the second air gesture when the hand skeleton point in the second gesture image meets a second set condition of the second air gesture indicated by the operation guide icon, so that the target operation is executed when the current air gesture is recognized. According to the target coordinate of the hand skeleton point with the wrist joint as the coordinate origin, the gesture category to which the current air gesture belongs is judged, the influence of relative movement of some parts of the body is avoided, the air gesture of the user can be effectively recognized, the electronic equipment can respond to the air gesture of the user in time, and the user experience is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to an air gesture interaction method, device and electronic device. Background Art

[0002] At present, with the continuous development of electronic technology, users' use of electronic devices (such as mobile phones) is no longer limited to operating by touching the touch screen. Some electronic devices can recognize the gestures performed by users in three-dimensional space (i.e., air gestures) through devices such as cameras or ultrasonic sensors, thereby realizing air-based operations on some functions of the electronic devices.

[0003] However, some of the user's air gestures (such as waving), due to the influence of the relative movement of some parts of the body, do not necessarily produce a movement trajectory in the preset direction. For example, the user's torso, upper arms, and wrist joints may have movements in the opposite direction to the hands, which combined to offset the movement trajectory of the hands, resulting in the electronic device being unable to respond to air gestures in a timely manner, affecting the user experience. Summary of the invention

[0004] The present disclosure aims to solve one of the technical problems in the above-mentioned technologies at least to some extent.

[0005] To this end, the present disclosure proposes an interactive method, device and electronic device for air gestures, which can calculate at least one first gesture feature of the hand skeleton points in a first gesture image according to the target coordinates of the hand skeleton points with the wrist joint as the coordinate origin, and display an operation guide icon corresponding to the first air gesture when at least one gesture feature meets the first setting condition of the first air gesture. Furthermore, when the hand skeleton points in the second gesture image are calculated to meet the second setting condition of the second air gesture indicated by the operation guide icon, the target operation corresponding to the second air gesture is executed. This achieves that when identifying the current air gesture, the gesture category to which the current air gesture belongs can be judged according to the target coordinates of the hand skeleton points with the wrist joint as the coordinate origin. This is not affected by the relative movement of some parts of the body, and the user's air gesture can be effectively identified, so that the electronic device can respond to the user's air gesture in a timely manner, thereby improving the user experience.

[0006] The first aspect of the present disclosure proposes an interactive method for air gestures, including: acquiring a first gesture image collected at a first moment, and identifying the first gesture image to obtain target coordinates of hand skeleton points of the first gesture image under a target gesture category, wherein the target coordinates are coordinates in a three-dimensional coordinate system established with the wrist joint corresponding to the hand skeleton point as the coordinate origin; based on the target coordinates of the hand skeleton point, calculating at least one first gesture feature of the hand skeleton point of the first gesture image; when the at least one first gesture feature meets the first set condition of the first air gesture, displaying an operation guide icon corresponding to the first air gesture; acquiring a second gesture image collected at a second moment, and calculating at least one second gesture feature of the hand skeleton point in the second gesture image; wherein the second moment is located after the first moment; when the at least one second gesture feature meets the second set condition of the second air gesture indicated by the operation guide icon, executing the target operation corresponding to the second air gesture.

[0007] The second aspect of the present disclosure proposes an interactive device for air gestures, including: an acquisition module, used to acquire a first gesture image collected at a first moment, and perform skeleton point recognition on the first gesture image to obtain target coordinates of hand skeleton points of the first gesture image under a target gesture category, wherein the target coordinates are coordinates in a three-dimensional coordinate system established with the wrist joint corresponding to the hand skeleton point as the coordinate origin; a first calculation module, used to calculate at least one first gesture feature of the hand skeleton point of the first gesture image based on the target coordinates of the hand skeleton point; a first control module, used to display an operation guide icon corresponding to the first air gesture when the at least one first gesture feature meets the first set condition of the first air gesture; a second calculation module, used to acquire a second gesture image collected at a second moment, and calculate at least one second gesture feature of the hand skeleton point in the second gesture image; wherein the second moment is located after the first moment; and a second control module, used to execute the target operation corresponding to the second air gesture when the at least one second gesture feature meets the second set condition of the second air gesture indicated by the operation guide icon.

[0008] The third aspect embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the air gesture interaction method described in the first aspect embodiment of the present disclosure.

[0009] The fourth aspect embodiment of the present disclosure proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the interactive method of air gestures described in the first aspect embodiment of the present disclosure is implemented.

[0010] The fifth aspect embodiment of the present disclosure proposes a computer program product, including computer instructions, on which a computer program is stored. When the computer instructions are executed by a processor, the interactive method of air gestures described in the first aspect embodiment of the present disclosure is implemented.

[0011] The sixth aspect embodiment of the present disclosure proposes a vehicle, comprising the electronic device described in the third aspect embodiment of the present disclosure.

[0012] The technical solution disclosed in the present invention obtains a first gesture image collected by an electronic device at a first moment, and recognizes the first gesture image to obtain the target coordinates of the hand skeleton point of the first gesture image under the target gesture category, wherein the target coordinates are coordinates in a three-dimensional coordinate system established with the wrist joint corresponding to the hand skeleton point as the coordinate origin; based on the target coordinates of the hand skeleton point, at least one first gesture feature of the hand skeleton point of the first gesture image is calculated; when the at least one first gesture feature meets the first setting condition of the first air gesture, an operation guide icon corresponding to the first air gesture is displayed; a second gesture image collected at a second moment is obtained, and at least one second gesture feature of the hand skeleton point in the second gesture image is calculated; wherein the second moment is located after the first moment; when the at least one second gesture feature meets the second setting condition of the second air gesture indicated by the operation guide icon, the target operation corresponding to the second air gesture is performed, thereby realizing that when recognizing the current air gesture, the gesture category to which the current air gesture belongs is judged according to the target coordinates of the hand skeleton point with the wrist joint as the coordinate origin, which is not affected by the relative movement of some parts of the body, and the user's air gesture can be effectively recognized, so that the electronic device can respond to the user's air gesture in time, thereby improving the user experience.

[0013] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0015] Figure 1 A schematic diagram of hand movement trajectory offset in the related art;

[0016] Figure 2A schematic diagram of a flow chart of an air gesture interaction method provided according to an embodiment of the present disclosure;

[0017] Figure 3 A schematic diagram of a flow chart of an air gesture interaction method provided according to an embodiment of the present disclosure;

[0018] Figure 4 A schematic diagram of hand skeleton points in a three-dimensional coordinate system provided according to an embodiment of the present disclosure;

[0019] Figure 5 A schematic diagram of hand skeleton points in a three-dimensional coordinate system provided according to an embodiment of the present disclosure;

[0020] Figure 6 A schematic diagram of the angle between a target vector and a first set plane and a second set plane in a three-dimensional coordinate system according to an embodiment of the present disclosure;

[0021] Figure 7 A schematic diagram of a flow chart of an air gesture interaction method provided according to an embodiment of the present disclosure;

[0022] Figure 8 A schematic diagram of a second setting condition of an operation guide icon corresponding to a first air gesture and a second air gesture indicated by the operation guide icon according to an embodiment of the present disclosure;

[0023] Fig. 9 A schematic diagram of a flow chart of an air gesture interaction method provided according to an embodiment of the present disclosure;

[0024] Fig.10 A schematic diagram of a flow chart of an air gesture interaction method provided according to an embodiment of the present disclosure;

[0025] Fig.11 A schematic diagram of a flow chart of an air gesture interaction device provided according to an embodiment of the present disclosure;

[0026] Fig.12 The present invention is a structural block diagram of an electronic device provided according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] Embodiments of the present disclosure are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0028] In the related technology, when the electronic device recognizes a user's air gesture, the electronic device displays a first display interface of a first application; the electronic device detects through a camera that the user inputs the first air gesture; in response to the first air gesture, the electronic device moves an indication icon in the first display interface along a movement trajectory of the first air gesture in a preset direction; in response to the first air gesture, the electronic device displays a second display interface of the first application.

[0029] Taking browsing the mobile e-book APP as an example, the mobile phone recognizes the air gestures performed by the user through the image captured by the camera. If it is detected that the user performs a first air gesture of waving to the left, the mobile phone may respond to the first air gesture to perform the first operation of "flip to the previous page" in the e-book APP. If it is detected that the user performs a second air gesture of waving to the right, the mobile phone may respond to the second air gesture to perform the second operation of "flip to the next page" in the e-book APP. In this solution, when the user intends to turn the page, the first air gesture of waving left (or right) does not necessarily produce a movement trajectory in the preset direction. For example, the user's torso, upper arm, and wrist joints may have opposite movements, which combine to offset the movement trajectory of the hand. Figure 1 As shown, the electronic device cannot respond to air gestures, affecting the user experience.

[0030] In view of the above problems, the present disclosure provides an air gesture interaction method, device and electronic device.

[0031] The following describes the air gesture interaction method, device and electronic device of the embodiments of the present disclosure with reference to the accompanying drawings.

[0032] Figure 2 The figure is a flow chart of an air gesture interaction method provided according to an embodiment of the present disclosure. It should be noted that the air gesture interaction method of the embodiment of the present disclosure can be applied to the air gesture interaction device of the embodiment of the present disclosure, and the device can be an electronic device.

[0033] like Figure 2 As shown, the air gesture interaction method may include the following steps:

[0034] Step 201 , obtaining a first gesture image collected at a first moment, and recognizing the first gesture image to obtain target coordinates of hand skeleton points of the first gesture image under a target gesture category.

[0035] The target coordinates are coordinates in a three-dimensional coordinate system established with the wrist joint corresponding to the hand bone point as the coordinate origin.

[0036] In the disclosed embodiment, a first gesture image captured by an electronic device at a first moment through a camera or an ultrasonic sensor or other device may be obtained, and the first gesture image may be recognized to obtain a target gesture category to which the gesture in the first gesture image belongs, and target coordinates of hand skeleton points under the target gesture category. The target gesture category may be a left hand gesture or a right hand gesture, and the hand skeleton points may be 21 hand key points under a left hand gesture or a right hand gesture.

[0037] As an example, the target coordinates of the hand skeleton points of the first gesture image under the target gesture category may be obtained based on the trained skeleton point recognition model.

[0038] As another example, a gesture category algorithm may be used to determine the target gesture category in the first gesture image. Furthermore, after determining the target gesture category of the first gesture image, the coordinates of the target bone points in the first gesture image may be specified, and the coordinates of other bone points may be determined based on the set positional relationship between the other bone points and the target bone points. Thus, the target coordinates of the hand bone points of the first gesture image under the target gesture category may be obtained.

[0039] In addition, it should be noted that the target coordinates of the hand bone points of the first gesture image under the target gesture category are three-dimensional coordinates in a three-dimensional coordinate system established with the wrist joint corresponding to the hand bone point as the coordinate origin. It should be noted that when the target gesture category is a left-hand gesture, the wrist joint is the left wrist joint, and when the target gesture category is a right-hand gesture, the wrist joint is the right wrist joint.

[0040] Step 202: Calculate at least one first gesture feature of the hand skeleton points in the first gesture image based on the target coordinates of the hand skeleton points.

[0041] In an embodiment of the present disclosure, based on the target coordinates of the hand skeleton points, at least one first gesture feature of the hand skeleton points of the first gesture image can be calculated, wherein the at least one first gesture feature may include: a first angle between a target vector between the coordinate origin of the three-dimensional coordinate system and the first specified skeleton point and a first set plane in the three-dimensional coordinate system, a second angle between the target vector and a second set plane in the three-dimensional coordinate system, and at least one target ratio between the coordinate value of the first specified skeleton point and a target length, wherein the target length is the length between the coordinate origin and the second set skeleton point.

[0042] Step 203: When at least one first gesture feature satisfies a first setting condition of a first air gesture, an operation guide icon corresponding to the first air gesture is displayed.

[0043] In an embodiment of the present disclosure, the air gesture to which the current gesture belongs can be determined based on at least one first gesture feature, and the electronic device can display an operation guide icon corresponding to the current gesture.

[0044] As an example, when at least one first gesture feature satisfies the first set condition of the first air gesture, the electronic device may display an operation guide icon corresponding to the first air gesture.

[0045] Among them, the first air gesture may include: an upward swipe activation gesture, a downward swipe activation gesture, a left swipe activation gesture, a right swipe activation gesture or a back swipe activation gesture. The operation guide icon corresponding to the upward swipe activation gesture is a page scroll up icon, the operation guide icon corresponding to the downward swipe activation gesture is a page scroll down icon, the operation guide icon corresponding to the left swipe activation gesture is a page scroll left icon, the page scroll right icon corresponding to the right swipe activation gesture, and the operation guide icon corresponding to the back swipe activation gesture is a back to previous page icon.

[0046] Step 204: Acquire a second gesture image collected at a second moment, and calculate at least one second gesture feature of a hand skeleton point in the second gesture image.

[0047] Among them, the second moment is located after the first moment.

[0048] In order to improve the accuracy of gesture recognition, after the electronic device displays the operation guide icon corresponding to the first air gesture, the second gesture image collected by the electronic device at the second moment can continue to be obtained, wherein the second gesture image can be an image collected when the user responds to the operation guide icon and makes an air gesture, and then, in order to accurately identify the air gesture in the second gesture image, at least one second gesture feature of the hand bone point in the second gesture image can be calculated. The second moment is located after the first moment, for example, the second moment can be an adjacent moment after the first moment, and the second moment and the first moment can also be non-adjacent moments.

[0049] Step 205: When at least one second gesture feature satisfies a second set condition of the second air gesture indicated by the operation guide icon, execute a target operation corresponding to the second air gesture.

[0050] Furthermore, when at least one second gesture feature satisfies the second set condition of the second air gesture associated with the operation guide icon, the gesture in the second gesture image can be determined as the second air gesture associated with the operation guide icon, and the electronic device can execute the target operation corresponding to the second air gesture.

[0051] For example, the operation guidance icon is a page scroll up icon, and the second air gesture indicated by the operation guidance icon is a page scroll up gesture, and the electronic device can execute the page scroll up.

[0052] In summary, by acquiring a first gesture image captured at a first moment and identifying the first gesture image, the target coordinates of the hand skeleton points of the first gesture image under the target gesture category are obtained, wherein the target coordinates are coordinates in a three-dimensional coordinate system established with the wrist joint corresponding to the hand skeleton point as the coordinate origin; based on the target coordinates of the hand skeleton point, at least one first gesture feature of the hand skeleton point of the first gesture image is calculated; when at least one first gesture feature meets the first setting condition of the first air gesture, the operation guide icon corresponding to the first air gesture is displayed; the second gesture image captured at a second moment is acquired, and at least one second gesture feature of the hand skeleton point in the second gesture image is calculated; wherein the second moment is located after the first moment; when at least one second gesture feature meets the second setting condition of the second air gesture indicated by the operation guide icon, the target operation corresponding to the second air gesture is executed, thereby, when identifying the current air gesture, according to the target coordinates of the hand skeleton point with the wrist joint as the coordinate origin, the gesture category to which the current air gesture belongs is judged, which is not affected by the relative movement of some parts of the body, and can effectively identify the user's air gesture, so that the electronic device can respond to the user's air gesture in time, thereby improving the user experience.

[0053] In order to clearly illustrate how at least one first gesture feature of the hand skeleton points of the first gesture image is calculated based on the target coordinates of the hand skeleton points in the above embodiment, the present disclosure proposes another interactive method of air gestures.

[0054] Figure 3 FIG. 1 is a flow chart of an air gesture interaction method provided according to an embodiment of the present disclosure. Figure 3 As shown, the air gesture interaction method may include the following steps:

[0055] Step 301 , obtaining a first gesture image collected at a first moment, and recognizing the first gesture image to obtain target coordinates of hand skeleton points of the first gesture image under a target gesture category.

[0056] The target coordinates are coordinates in a three-dimensional coordinate system established with the wrist joint corresponding to the hand bone point as the coordinate origin.

[0057] Step 302: connect the origin of the three-dimensional coordinate system and the first designated bone point in the hand bone points to determine a target vector between the origin of the three-dimensional coordinate system and the first designated bone point.

[0058] As an example, a line is connected between the coordinate origin of the three-dimensional coordinate system and the first specified bone point among the hand bone points to obtain a target vector, wherein the direction of the target vector can be from the coordinate origin to the first specified bone point, or the direction of the target vector is from the first specified bone point to the coordinate origin.

[0059] It should be noted that if Figure 4 and Figure 5 As shown, the first specified bone point can be Figure 4 or Figure 5 Among joints No. 7, No. 8 or No. 9, the present disclosure uses joint No. 7 as the first designated bone point for exemplary description.

[0060] Step 303 : Project the target vector onto the first setting plane and the second setting plane in the three-dimensional coordinate system respectively to determine a first angle between the target vector and the first setting plane and a second angle between the target vector and the second setting plane.

[0061] The first setting plane is perpendicular to the second setting plane.

[0062] In the embodiments of the present disclosure, Figure 6 As shown, the first set plane can be the XOZ plane, and the second set plane can be the YOZ plane. By projecting the target vector onto the XOZ plane, a first angle between the target vector and the XOZ plane can be obtained. By projecting the target vector onto the YOZ plane, a second angle between the target vector and the YOZ plane can be obtained.

[0063] For example, taking the first specified bone point as Figure 4 or Figure 5 Taking joint No. 7 in as an example, the vector of joints 0 to 7 is V07=[x7–x0,y7-y0,z7-z0], and the projection of V07 on the XOZ plane is V07xz=[x7-x0,0,z7-z0]; V07·V07xz=|V07|*|V07xz|*cos(θ1), θ1 is the angle between V07 and V07xz, that is, the angle between V07 and the XOZ plane; θ1=arccos(V07·V07xz / (|V07|*|V07xz|)).

[0064] Similarly, the projection of V07 on the YOZ plane is V07yz = [0, y7-y0, z7-z0]; V07·V07yz = |V07|*|V07yz|*cos(θ2), θ2 is the angle between V07 and V07yz, that is, the angle between V07 and the YOZ plane; θ2 = arccos(V07·V07yz / (|V07|*|V07yz|)).

[0065] It should be noted that the first specified bone point can be Figure 4 or Figure 5 For joint No. 7, joint No. 8 or joint No. 9 in the figure, in order to further improve the accuracy of the first angle and the second angle, as an example, the angles between the lines between the coordinate origin and the joint No. 7, joint No. 8 and joint No. 9 and the first set plane can be calculated respectively, and then the average of the angle between the line between the coordinate origin and the joint No. 7 and the first set plane, the angle between the line between the coordinate origin and the joint No. 8 and the first set plane, and the angle between the line between the coordinate origin and the joint No. 9 and the first set plane can be determined, and the average is used as the first angle.

[0066] Similarly, the angles between the lines connecting the coordinate origin and joint No. 7, joint No. 8 and joint No. 9 and the second set plane can be calculated respectively, and then the average of the angle between the line connecting the coordinate origin and joint No. 7 and the second set plane, the angle between the line connecting the coordinate origin and joint No. 8 and the second set plane, and the angle between the line connecting the coordinate origin and joint No. 9 and the second set plane can be determined, and the average can be used as the second angle.

[0067] For example, the angle between the joint vectors 0 to 7 and the XOZ plane is θ 1-07 =arccos(V07·V07xz / (|V07|*|V07xz|)); Angle θ between joint vectors 0 to 8 and XOZ plane 1-08 =arccos(V08·V08xz / (|V08|*|V08xz|)); Angle θ between joint vectors 0 to 9 and the XOZ plane 1-09 =arcos(V09·V09xz / (|V09|*|V09xz|)); θ1=(θ 1-07 +θ 1-08 +θ 1-09 ) / 3.

[0068] The angle θ between the joint vectors 0 to 7 and the YOZ plane 2-07 =arccos(V07·V07yz / (|V07|*|V07yz|)); Angle θ between joint vectors 0 to 8 and YOZ plane 2-08 =arccos(V08·V08yz / (|V08|*|V08yz|)); Angle θ between joint vectors 0 to 9 and YOZ plane 2-09 =arccos(V09·V09yz / (|V09|*|V09yz|)); θ2=(θ 2-07 +θ 2-08 +θ 2-09 ) / 3.

[0069] Among them, V07xz represents the projection of V07 on the XOZ plane, V08xz represents the projection of V08 on the XOZ plane, V09xz represents the projection of V09 on the XOZ plane, V07yz represents the projection of V07 on the YOZ plane, V08yz represents the projection of V08 on the YOZ plane, V09yz represents the projection of V09 on the YOZ plane, V08 represents the vector between joints 0 to 8, and V09 represents the vector between joints 0 to 9.

[0070] Step 304, obtaining the sub-coordinates of the first designated bone point in each coordinate axis of the three-dimensional coordinate system, and obtaining the target length between the coordinate origin and the second designated bone point in the hand bone point.

[0071] In the embodiments of the present disclosure, Figure 4 or Figure 5 As shown, the first specified bone point is joint 7, and the second specified bone point is Figure 4 or Figure 5 For joint No. 2 in the hand, obtain the sub-coordinates of joint No. 7 on each coordinate axis in the three-dimensional coordinate system from the target coordinates of the hand bone points, obtain the sub-coordinates of joint No. 2 on each coordinate axis in the three-dimensional coordinate system from the target coordinates of the hand bone points, and calculate the target length between the coordinate origin and joint No. 2 according to the sub-coordinates of joint No. 2 on each coordinate axis in the three-dimensional coordinate system.

[0072] It should be noted that when the first specified bone point is joint 8, the second specified bone point corresponds to Figure 4 or Figure 5 In the case of joint 3, when the first specified bone point can be joint 9, the second specified bone point can be Figure 4 or Figure 5 Joint No. 4 in the.

[0073] Step 305 : Compare the sub-coordinates of each coordinate axis with the target length to obtain at least one target ratio between the target coordinates and the target length.

[0074] Furthermore, by comparing the sub-coordinates of each coordinate axis with the target length, at least one target ratio between the sub-coordinates of each coordinate axis and the target length can be obtained.

[0075] The first specified bone point is Figure 4 or Figure 5 Take joint No. 7 and the second specified bone point as joint No. 2 as an example. If the coordinates of joint No. 7 are (x7, y7, z7), and the coordinates of joint No. 2 are (x2, y2, z2), the target length d is the distance from joint No. 0 to joint No. 2: d = (x2^2+y2^2+z2^2)^0.5, the target ratio of the X-axis is x' = x7 / d, the target ratio of the Y-axis is y' = y7 / d, and the target ratio of the Z-axis is z' = z7 / d.

[0076] In order to further improve the accuracy of the target ratio, the average of the ratio of the X-axis coordinate of joint No. 7 to the target length, the ratio of the X-axis coordinate of joint No. 8 to the target length, and the ratio of the X-axis coordinate of joint No. 9 to the target length can be calculated to obtain the target ratio x'=(x7 / d+x8 / d+x9 / d) / 3 of the X-axis; similarly, the target ratio y'=(y7 / d+y8 / d+y9 / d) / 3 of the Y-axis, and the target ratio z'=(z7 / d+z8 / d+z9 / d) / 3 of the Z-axis.

[0077] Step 306: Use the first angle, the second angle, and at least one target ratio as at least one first gesture feature.

[0078] Furthermore, the first angle, the second angle and at least one target ratio are used as at least one first gesture feature of the hand skeleton point of the first gesture image.

[0079] Step 307: When at least one first gesture feature satisfies the first set condition of the first air gesture, control the electronic device to display an operation guide icon corresponding to the first air gesture.

[0080] In the embodiment of the present disclosure, based on the first setting condition of the first air gesture satisfied by at least one first gesture feature, the operation guide icon corresponding to the gesture in the first gesture image can be distinguished.

[0081] As an example, when the first angle is greater than 70 degrees and less than 80 degrees, and the target ratio of the Y axis is less than -1.5, it can be determined that at least one first gesture feature satisfies the first set condition corresponding to the swipe-up activation gesture, and the electronic device can be controlled to display a page scroll up icon;

[0082] As another example, when the first angle is greater than 70 degrees and the target ratio of the Y axis is greater than 1.5, it can be determined that at least one first gesture feature satisfies the first set condition corresponding to the slide down activation gesture, and the electronic device can be controlled to display a page scroll down icon;

[0083] As another example, when the second angle is greater than 70 degrees and the target ratio of the X-axis is greater than 1.5, it can be determined that at least one first gesture feature satisfies the first set condition corresponding to the left-swipe activation gesture, and the electronic device can be controlled to scroll the icon to the left on the display page;

[0084] As another example, when the second angle is greater than 70 degrees and the target ratio of the X-axis is less than -1.5, it can be determined that at least one first gesture feature satisfies the first set condition corresponding to the right-swipe activation gesture, and the electronic device can be controlled to display a page scrolling icon to the right;

[0085] As another example, when the first angle is greater than 80 degrees, the target ratio of the Y axis is less than -1.5, and the target ratio of the Z axis is less than 0.2, it can be determined that at least one first gesture feature satisfies the fallback activation, and the electronic device can be controlled to display the fallback to the previous page icon. Wherein, when at least one first gesture feature does not meet the first set condition of the first air gesture, the gesture image collected by the electronic device continues to be acquired until the third gesture image is acquired; wherein, at least one first gesture feature in the third gesture image meets the first set condition of the first air gesture.

[0086] Step 308: Acquire a second gesture image collected at a second moment, and calculate at least one second gesture feature of a hand skeleton point in the second gesture image.

[0087] Among them, the second moment is located after the first moment.

[0088] Step 309: When at least one second gesture feature satisfies a second set condition of the second air gesture indicated by the operation guide icon, execute a target operation corresponding to the second air gesture.

[0089] In summary, a line is connected between the coordinate origin of the three-dimensional coordinate system and the first specified bone point in the hand bone points to determine the target vector between the coordinate origin of the three-dimensional coordinate system and the first specified bone point; the target vector is projected onto the first set plane and the second set plane in the three-dimensional coordinate system respectively to determine the first angle between the target vector and the first set plane and the second angle between the target vector and the second set plane, wherein the first set plane is perpendicular to the second set plane; the sub-coordinates of the first specified bone point in each coordinate axis of the three-dimensional coordinate system are obtained, and the target length between the coordinate origin and the second specified bone point in the hand bone points is obtained; the sub-coordinates of each coordinate axis are compared with the target length to obtain at least one target ratio between the sub-coordinates of each coordinate axis and the target length; the first angle, the second angle and the at least one target ratio are used as at least one first gesture feature, thereby effectively determining at least one first gesture feature of the hand bone point of the first gesture image, and according to the first setting condition of the first air gesture satisfied by the at least one first gesture feature, the operation guide icon corresponding to the gesture in the first gesture image can be effectively distinguished.

[0090] In order to clearly illustrate how to calculate at least one second gesture feature of the hand skeleton points of the second gesture image in the above embodiment, the present disclosure proposes another interactive method of air gestures.

[0091] Figure 7 FIG. 1 is a flow chart of an air gesture interaction method provided according to an embodiment of the present disclosure. Figure 7 As shown, the air gesture interaction method may include the following steps:

[0092] Step 701: Acquire a first gesture image collected at a first moment, and recognize the first gesture image to obtain target coordinates of hand skeleton points of the first gesture image under a target gesture category.

[0093] The target coordinates are coordinates in a three-dimensional coordinate system established with the wrist joint corresponding to the hand bone point as the coordinate origin.

[0094] Step 702: Calculate at least one first gesture feature of the hand skeleton points in the first gesture image based on the target coordinates of the hand skeleton points.

[0095] Step 703: When at least one first gesture feature satisfies a first set condition of a first air gesture, control the electronic device to display an operation guide icon corresponding to the first air gesture.

[0096] Step 704: Acquire a second gesture image captured at a second moment, and acquire a third gesture image captured at a third moment.

[0097] The third moment is the previous collection moment adjacent to the second moment.

[0098] In an embodiment of the present disclosure, a second gesture image captured by the electronic device at a second moment through a device such as a camera or an ultrasonic sensor, and a third gesture image captured at a third moment can be obtained, wherein the third moment can be the previous adjacent moment of the second moment.

[0099] Step 705, determine the time difference between the second moment and the third moment, and determine the movement distance of the first specified skeleton point at the second moment relative to the third moment based on the difference between the coordinates of the first specified skeleton point at the second moment and the coordinates of the first specified skeleton point at the third moment.

[0100] The moving distance includes the moving distance in the direction of each coordinate axis.

[0101] In the disclosed embodiment, by identifying the second gesture image, the coordinates of the hand skeleton points in the second gesture image under the target gesture category can be obtained, and by identifying the third gesture image, the coordinates of the hand skeleton points in the third gesture image under the target gesture category can be obtained. Furthermore, from the coordinates of the hand skeleton points in the second gesture image under the target gesture category, the coordinates of the first designated skeleton point at the second moment can be obtained. Similarly, from the coordinates of the hand skeleton points in the third gesture image under the target gesture category, the coordinates of the first designated skeleton point at the third moment can be obtained. By comparing the coordinates of the first designated skeleton point at the second moment with the coordinates of the first designated skeleton point at the third moment, the movement distance of the first designated skeleton point at the second moment relative to the third moment can be determined, wherein the movement distance includes the movement distance in the direction of each coordinate axis.

[0102] Step 706: Determine a first moving speed in each coordinate axis direction based on the ratio between the moving distance in each coordinate axis direction and the time difference.

[0103] For example, taking the first specified bone point as Figure 4 or Figure 5 Taking the joint No. 7 in the figure as an example, the time difference between the second moment and the third moment is Δt=1 / FPS, where FPS represents the current frame rate, the moment t represents the second moment, and the moment t-1 represents the third moment. The coordinates of the first specified bone point at the moment t are (x7 t ,y7 t , z7 t ), the coordinates of the first specified bone point at time t-1 are (x7 t-1 ,y7 t-1 , z7 t-1 ), Δx=x7 t -x7 t-1 ; Δy=y7 t -y7 t-1 ; Δz=z7 t -z7 t-1 , V x =Δx / Δt=(x7 t -x7 t-1 ) / (1 / FPS)=(x7 t -x7 t-1 )*FPS;

[0104] V y =(y7 t -y7 t-1 )*FPS;V z =(z7 t -z7 t-1 )*FPS.

[0105] In order to further improve the accuracy of the first moving speed in each coordinate axis direction, as an example, the speeds of joints 7, 8, and 9 on each coordinate axis can be calculated respectively, and the speeds of joints 7, 8, and 9 on each coordinate axis are averaged to obtain the first moving speed in each coordinate axis direction, which can be specifically expressed as the following formula:

[0106] V x =(x7 t -x7 t-1 +x8 t -x8 t-1 +x9 t -x9 t-1 )*FPS / 3;

[0107] V y=(y7 t -y7 t-1 +y8 t -y8 t-1 +y9 t -y9 t-1 )*FPS / 3;

[0108] V z =(z7 t -z7 t-1 +z8 t -z8 t-1 +z9 t -z9 t-1 )*FPS / 3.

[0109] It should be noted that since the coordinates of the bone points in the three-dimensional coordinate system may contain a lot of noise,

[0110] Therefore, in order to improve the stability of the second gesture feature, a filtering algorithm may be used to filter the moving distance in each coordinate axis direction, and based on the ratio between the moving distance in each coordinate axis direction after filtering and the time difference, the first moving speed in each coordinate axis direction is determined. It should be noted that the filtering algorithm may include: mean filtering, median filtering, exponential filtering, Holt double exponential filtering, etc.

[0111] Step 707, fusing the first moving speeds in the directions of the coordinate axes to obtain a fused speed.

[0112] Then, based on the first moving speed in each coordinate axis direction, the fusion speed can be obtained, for example, the fusion speed

[0113] As another possible implementation method, a fourth gesture image captured at a third moment is acquired; wherein the third moment is an adjacent previous acquisition moment of the second moment; the coordinates and movement speed of the first designated bone point in the fourth gesture image at the third moment are combined to obtain a first motion state, and the first motion state is used to predict the second motion state of the first designated bone point at the second moment; based on a set initial covariance matrix, the predicted noise of the first designated bone point in the second gesture image at the second moment is predicted; the predicted noise of the first designated bone point at the second moment and the set observation noise are fused (for example, Kalman fusion) to obtain a fusion weight; the second motion state and the third motion state are weightedly summed using the fusion weight to obtain a fused motion state of the first designated bone point at the second moment; wherein the third motion state includes the coordinates of the first designated bone point at the second moment and the first movement speed in the direction of each coordinate axis; from the fused motion state, the second movement speed of the first designated bone point in the direction of each coordinate axis is obtained; the second movement speed in the direction of each coordinate axis is fused to obtain a fused speed. Among them, based on the set initial covariance matrix, the process of predicting the predicted noise of the first specified bone point in the second gesture image at the second moment is: based on the set initial covariance matrix, perform at least one iteration to obtain the first covariance matrix at the third moment; obtain the state transfer matrix of the second motion state and the second covariance matrix of the state transfer matrix; multiply the state transfer matrix, the first covariance matrix and the transposed matrix of the state transfer matrix to obtain the target product; add the target product and the second covariance matrix to obtain the predicted noise at the second moment.

[0114] That is to say, based on the position P at time t (second time) t and speed V t , and P at time t-1 t-1 and V t-1 The relationship P t =P t-1 +V t-1 *Δt+1 / 2a t-1 Δt 2 , the second motion state can be obtained Specifically, it can be expressed as

[0115]

[0116] in, The first motion state X t-1 , B is the control matrix;

[0117] is the state transfer matrix F;

[0118] In the embodiment of the present disclosure, the covariance matrix (prediction noise) of the state transfer matrix of the second motion state is predefined as Q, and the coefficient of the Q matrix can be 0.1.

[0119] In addition, since the coordinates of the first specified bone point are three-dimensional coordinates, as well as the speed on each coordinate axis, the observation matrix can be pre-set as

[0120] As an example, the covariance matrix R of the observation noise of the second motion state may be predefined. Since the actual noise is large, the matrix coefficient of R may be 2. Therefore, The initial value of the first motion state can be Initial covariance matrix

[0121] In the embodiment of the present disclosure, taking time t as the second time and time t-1 as the third time as an example, Xp t =FX t-1 ; Prediction noise Pp at the second moment t =FP t-1 F T +Q, covariance matrix P t =(IK t H)Pp t - , where Q is the covariance matrix of the state transfer matrix of the second motion state (the second covariance matrix), FP t-1 F T As the target product, at least one (e.g., t-1) iteration can be performed based on the initial covariance matrix to obtain the first covariance matrix at the third moment, where the initial covariance matrix can be preset according to historical experience, I is the unit matrix, and the observation noise R at time t is t =HP t H T , fusion weight K t =Pp t - H T (HPp t - H T +R t );

[0122] Fusion motion state Xk=Xp+K t (X t -HXp); Among them, in this fusion state, the sum of the observation noise and the prediction noise is the smallest. When time t is the second moment, v xk , v yk and v zkThe second moving speed of the first specified bone point in each coordinate axis direction, the fusion speed

[0123] Step 708: When at least one second gesture feature satisfies a second set condition of the second air gesture indicated by the operation guide icon, execute a target operation corresponding to the second air gesture.

[0124] Furthermore, if Figure 8 As shown, when the electronic device displays an operation guide icon corresponding to the first air gesture, it is determined whether at least one second gesture feature satisfies the second set condition of the second air gesture indicated by the operation guide icon.

[0125] As an example, when the electronic device displays a page scroll up icon, the speed of the first designated bone point in the second gesture image in the Y-axis direction is greater than 0.8m / s, and the fusion speed is greater than 1m / s, controlling the electronic device to perform a page scroll up operation.

[0126] As another example, when the electronic device displays a page scroll down icon, the speed of the first designated bone point in the second gesture image in the Y-axis direction is less than -0.8m / s, and the fusion speed is greater than 1m / s, controlling the electronic device to perform a page scroll down operation.

[0127] As another example, when the electronic device displays a page scroll left icon, the speed of the first specified bone point in the second gesture image in the X-axis direction is less than -0.8m / s, and the fusion speed is greater than 1m / s, controlling the electronic device to perform a page scroll left operation.

[0128] As another example, when the electronic device displays a page scroll right icon, the speed of the first designated bone point in the second gesture image in the X-axis direction is 0.8 m / s, and the fusion speed is greater than 1 m / s, controlling the electronic device to perform a page scrolling operation.

[0129] As another example, when the electronic device displays the icon for going back to the previous page, the speed of the first designated bone point in the second gesture image in the Y-axis direction is greater than 0.6 m / s, the speed in the Z-axis direction is greater than 0.6 m / s, and the fusion speed is greater than 0.8 m / s, and the electronic device is controlled to perform the operation of going back to the previous page.

[0130] In addition, it should be noted that in order to improve the user experience, when at least one second gesture feature does not meet the second set condition of the second air gesture associated with the operation guide icon, the display time of the operation guide icon corresponding to the first air gesture is determined; when the display time is greater than or equal to the set time threshold, the electronic device is controlled to cancel the display of the operation guide icon.

[0131] In summary, by acquiring a third gesture image captured at a third moment; wherein the third moment is the previous capture moment adjacent to the second moment; determining the time difference between the second moment and the third moment, and according to the difference between the coordinates of the first designated skeletal point in the second gesture image at the second moment and the coordinates of the first designated skeletal point in the third gesture image at the third moment, determining the movement distance of the first designated skeletal point at the second moment relative to the third moment; wherein the movement distance includes the movement distance in the direction of each coordinate axis; based on the ratio between the movement distance in the direction of each coordinate axis and the time difference, determining the first movement speed in the direction of each coordinate axis; fusing the first movement speed in the direction of each coordinate axis to obtain the fusion speed, thereby, based on the second gesture image captured at the second moment and the third gesture image captured at the third moment, at least one second gesture feature can be effectively determined.

[0132] In order to clearly illustrate how the above embodiment recognizes the first gesture image to obtain the target coordinates of the hand bone points of the first gesture image under the target gesture category, the present disclosure proposes another interactive method of air gestures.

[0133] Fig. 9 FIG. 1 is a flow chart of an air gesture interaction method provided according to an embodiment of the present disclosure. Fig. 9 As shown, the air gesture interaction method may include the following steps:

[0134] Step 901, obtaining a trained skeleton point recognition model.

[0135] As an example, a sample gesture image is obtained, and the hand in the sample gesture image is labeled with a category to obtain a labeled gesture category, and the coordinates of the sample hand skeleton points in the sample gesture image are labeled to obtain the labeled coordinates of the skeleton points, wherein the labeled coordinates are coordinates with the sample wrist joint in the sample hand skeleton points as the coordinate origin; the sample gesture image is input into an initial gesture recognition model to obtain the predicted gesture category and the predicted coordinates of the skeleton points corresponding to the sample gesture image; a first sub-loss value is generated based on a first difference between the labeled gesture category and the predicted gesture category, and a second sub-loss value is generated based on a second difference between the labeled coordinates and the predicted coordinates of the skeleton points; a loss function value is generated based on the first sub-loss value and the second sub-loss value; and the initial gesture recognition model is trained based on the loss function value.

[0136] Among them, the first sub-loss value and the first difference are positively correlated, that is, the smaller the first difference is, the smaller the first sub-loss value is, and conversely, the larger the first difference is, the larger the first sub-loss value is. Similarly, the second sub-loss value and the second difference are positively correlated. Therefore, in the present disclosure, the skeleton point recognition model can be trained according to the loss function value to minimize the loss function value.

[0137] It should be noted that the above only uses the termination condition of the skeleton point recognition model training as an example of minimizing the loss function value. In actual application, other termination conditions can also be set. For example, the termination condition can also be that the number of training times reaches a set threshold, the training time is greater than a set threshold, etc. The present disclosure does not impose any restrictions on this.

[0138] In addition, it should be noted that in the embodiments of the present disclosure, the sample gesture images may be collected online, for example, images containing gestures may be collected online through web crawler technology as sample gesture images, or the sample gesture images may be images containing gestures collected offline, or the sample gesture images may be artificially synthesized images, etc. The embodiments of the present disclosure do not impose any restrictions on this.

[0139] Step 902, obtaining a first gesture image collected at a first moment, and using a trained skeleton point recognition model to recognize the first gesture image to obtain target coordinates of hand skeleton points in the first gesture image under a target gesture category.

[0140] The target coordinates are coordinates in a three-dimensional coordinate system established with the wrist joint in the hand skeleton point as the coordinate origin.

[0141] In an embodiment of the present disclosure, the first gesture image is input into a trained skeleton point recognition model, and the target coordinates of the hand skeleton points under the target gesture category output by the trained skeleton point recognition model are obtained.

[0142] Step 903: Calculate at least one first gesture feature of the hand skeleton points in the first gesture image based on the target coordinates of the hand skeleton points.

[0143] Step 904: When at least one first gesture feature satisfies a first set condition of a first air gesture, an operation guide icon corresponding to the first air gesture is displayed.

[0144] Step 905: Acquire a second gesture image collected at a second moment, and calculate at least one second gesture feature of a hand skeleton point in the second gesture image.

[0145] Among them, the second moment is located after the first moment.

[0146] Step 906: When at least one second gesture feature satisfies a second set condition of the second air gesture indicated by the operation guide icon, execute a target operation corresponding to the second air gesture.

[0147] In summary, by using the trained skeleton point recognition model, the target coordinates of the hand skeleton points of the first gesture image under the target gesture category can be accurately obtained.

[0148] In any embodiment of the present disclosure, if Fig.10 As shown, the air gesture interaction method of the embodiment of the present disclosure can also be implemented based on the following steps:

[0149] 1. Apply the pre-trained neural network (skeleton point recognition model) to extract the skeleton points in the 3D space from the currently collected gesture image. The coordinate origin of the 3D space is the wrist joint;

[0150] 2. Calculate the data features of the 3D skeleton points (at least one first gesture feature), including the ratio of the coordinates of each coordinate axis of the fingertip to the specified length, the angle between the line connecting the fingertip and the wrist joint and the XOZ plane (first angle), and the angle between the line connecting the fingertip and the wrist joint and the YOZ plane (second angle);

[0151] 3. The data features of the 3D skeleton points meet the conditions of the first air gesture, and the operation guide icon corresponding to the first air gesture is displayed;

[0152] 4. Calculate the data features of the 3D skeleton points (at least one second gesture feature), including: the absolute value of the movement speed of the fingertip, the speed in the X-axis direction, the speed in the Y-axis direction, and the speed in the Z-axis direction;

[0153] 5. When the absolute value of the fingertip's moving speed, the speed in the X-axis direction, the speed in the Y-axis direction, and the speed in the Z-axis direction meet the conditions corresponding to the second air gesture, the corresponding operation is performed.

[0154] In order to implement the above-mentioned embodiments, the present disclosure proposes an interactive device for air gestures.

[0155] Fig.11 A schematic diagram of the structure of an air gesture interaction device provided according to an embodiment of the present disclosure.

[0156] like Fig.11 As shown, the air gesture interaction device 1100 includes: an acquisition module 1110, a first calculation module 1120, a first control module 1130, a second calculation module 1140 and a second control module 1150.

[0157] Among them, the acquisition module 1110 is used to acquire the first gesture image collected at the first moment, and perform skeleton point recognition on the first gesture image to obtain the target coordinates of the hand skeleton points of the first gesture image under the target gesture category, wherein the target coordinates are coordinates in a three-dimensional coordinate system established with the wrist joint in the hand skeleton points as the coordinate origin; the first calculation module 1120 is used to calculate at least one first gesture feature of the hand skeleton points of the first gesture image based on the target coordinates of the hand skeleton points; the first control module 1130 is used to display the operation guide icon corresponding to the first air gesture when at least one first gesture feature meets the first setting condition of the first air gesture; the second calculation module 1140 is used to acquire the second gesture image collected at the second moment, and calculate at least one second gesture feature of the hand skeleton points in the second gesture image; wherein the second moment is located after the first moment; the second control module 1150 is used to execute the target operation corresponding to the second air gesture when at least one second gesture feature meets the second setting condition of the second air gesture indicated by the operation guide icon.

[0158] As a possible implementation method of the embodiment of the present disclosure, the first calculation module 1120 is specifically used to: connect the coordinate origin of the three-dimensional coordinate system and the first specified bone point in the hand bone points to determine the target vector between the coordinate origin of the three-dimensional coordinate system and the first specified bone point; project the target vector onto the first set plane and the second set plane in the three-dimensional coordinate system respectively to determine the first angle between the target vector and the first set plane and the second angle between the target vector and the second set plane, wherein the first set plane is perpendicular to the second set plane; obtain the sub-coordinates of the first specified bone point in each coordinate axis of the three-dimensional coordinate system, and obtain the target length between the coordinate origin and the second specified bone point in the hand bone points; compare the sub-coordinates of each coordinate axis with the target length to obtain at least one target ratio between the sub-coordinates of each coordinate axis and the target length; use the first angle, the second angle and at least one target ratio as at least one first gesture feature.

[0159] As a possible implementation method of the embodiment of the present disclosure, the second calculation module 1140 is specifically used to: obtain a third gesture image collected at a third moment; wherein the third moment is the previous collection moment adjacent to the second moment; determine the time difference between the second moment and the third moment, and determine the movement distance of the first specified skeletal point at the second moment relative to the third moment based on the difference between the coordinates of the first specified skeletal point in the second gesture image at the second moment and the coordinates of the first specified skeletal point in the third gesture image at the third moment; wherein the movement distance includes the movement distance in each coordinate axis direction; based on the ratio between the movement distance in each coordinate axis direction and the time difference, determine the first movement speed in each coordinate axis direction; fuse the first movement speed in each coordinate axis direction to obtain a fused speed.

[0160] As a possible implementation of the embodiment of the present disclosure, the second calculation module 1140 is also used to: obtain a fourth gesture image collected at a third moment; wherein the third moment is an adjacent previous collection moment of the second moment; combine the coordinates and movement speed of the first designated bone point in the fourth gesture image at the third moment to obtain a first motion state, and predict the second motion state of the first designated bone point at the second moment based on the first motion state; based on the set initial covariance matrix, predict the predicted noise of the first designated bone point in the second gesture image at the second moment; perform Kalman fusion on the predicted noise of the first designated bone point at the second moment and the set observation noise to obtain a fusion weight; use the fusion weight to perform weighted summation on the second motion state and the third motion state to obtain a fused motion state of the first designated bone point at the second moment; wherein the third motion state includes the coordinates of the first designated bone point at the second moment and the first movement speed in each coordinate axis direction; from the fused motion state, obtain the second movement speed of the first designated bone point in each coordinate axis direction; fuse the second movement speed in each coordinate axis direction to obtain a fused speed.

[0161] As a possible implementation method of an embodiment of the present disclosure, the second calculation module 1140 is also used to: perform at least one iteration based on the set initial covariance matrix to obtain a first covariance matrix at a third moment; obtain a state transfer matrix of the second motion state and a second covariance matrix of the state transfer matrix; multiply the state transfer matrix, the first covariance matrix and the transposed matrix of the state transfer matrix to obtain a target product; add the target product to the second covariance matrix to obtain the predicted noise at the second moment.

[0162] As a possible implementation method of an embodiment of the present disclosure, the acquisition module 1110 is also used to: when at least one first gesture feature does not satisfy the first set condition of the first air gesture, continue to acquire gesture images captured after the first moment until a third gesture image is acquired; wherein at least one first gesture feature in the third gesture image satisfies the first set condition of the first air gesture.

[0163] As a possible implementation method of the embodiment of the present disclosure, the air gesture interaction device 1100 also includes: a display module.

[0164] Among them, the display module is used to determine the display time of the operation guide icon corresponding to the first air gesture when at least one second gesture feature does not meet the second set condition of the second air gesture associated with the operation guide icon; the first control module 1130 is also used to cancel the display of the operation guide icon when the display time is greater than or equal to the set time threshold.

[0165] As a possible implementation method of the embodiment of the present disclosure, the acquisition module 1110 is specifically used to: obtain a trained skeleton point recognition model; use the trained skeleton point recognition model to recognize the first gesture image to obtain the target coordinates of the hand skeleton points of the first gesture image under the target gesture category.

[0166] As a possible implementation method of the embodiment of the present disclosure, the acquisition module 1110 is also used to: acquire a sample gesture image, and perform category labeling on the hand in the sample gesture image to obtain a labeled gesture category, and label the coordinates of the sample hand skeleton points in the sample gesture image to obtain the labeled coordinates of the skeleton points, wherein the labeled coordinates are coordinates with the sample wrist joint in the sample hand skeleton points as the coordinate origin; input the sample gesture image into the initial skeleton point recognition model to obtain the predicted gesture category and the predicted skeleton point coordinates corresponding to the sample gesture image; generate a first sub-loss value based on a first difference between the labeled gesture category and the predicted gesture category, and generate a second sub-loss value based on a second difference between the labeled coordinates and the predicted coordinates of the skeleton points; generate a loss function value based on the first sub-loss value and the second sub-loss value; and train the initial skeleton point recognition model based on the loss function value.

[0167] The interactive device for air gestures of the disclosed embodiment obtains a first gesture image collected at a first moment, and recognizes the first gesture image to obtain target coordinates of hand skeleton points of the first gesture image under a target gesture category, wherein the target coordinates are coordinates in a three-dimensional coordinate system established with the wrist joint corresponding to the hand skeleton point as the coordinate origin; based on the target coordinates of the hand skeleton point, at least one first gesture feature of the hand skeleton point of the first gesture image is calculated; when at least one first gesture feature meets the first set condition of the first air gesture, an operation guide icon corresponding to the first air gesture is displayed; a second gesture image collected at a second moment is obtained, and at least one second gesture feature of the hand skeleton point in the second gesture image is calculated; wherein the second moment is located after the first moment; when at least one second gesture feature meets the second set condition of the second air gesture indicated by the operation guide icon, a target operation corresponding to the second air gesture is performed, thereby realizing that when recognizing the current air gesture, the gesture category to which the current air gesture belongs is judged according to the target coordinates of the hand skeleton point with the wrist joint as the coordinate origin, which is not affected by the relative movement of some parts of the body, and can effectively recognize the user's air gesture, so that the electronic device can respond to the user's air gesture in a timely manner, thereby improving the user experience.

[0168] It should be noted that the above explanation of the method embodiment is also applicable to the device of this embodiment, and will not be repeated here.

[0169] In order to implement the above embodiments, the present disclosure also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the air gesture interaction method described in the above embodiments is implemented.

[0170] In order to implement the above embodiments, the present disclosure also proposes a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the interactive method of air gestures described in the above embodiments is implemented.

[0171] In order to implement the above embodiments, the present disclosure also proposes a computer program product, on which a computer program is stored. When the computer program is executed by a processor, the interactive method of air gestures described in the above embodiments is implemented.

[0172] In order to implement the above embodiments, the present disclosure proposes a vehicle, which may include the electronic device described in the above embodiments.

[0173] Fig.12 The present invention is a structural block diagram of an electronic device provided according to an embodiment of the present disclosure. Fig.12 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0174] like Fig.12 As shown, the electronic device 1200 includes a processor 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a memory 1206 to a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the electronic device 1200 are also stored. The processor 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0175] The following components are connected to the I / O interface 1205: a memory 1206 including a hard disk, etc.; and a communication part 1207 including a network interface card such as a LAN (Local Area Network) card, a modem, etc., and the communication part 1207 performs communication processing via a network such as the Internet; a drive 1208 is also connected to the I / O interface 1205 as needed.

[0176] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1207. When the computer program is executed by the processor 1201, the above-mentioned functions defined in the method of the present disclosure are executed.

[0177] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory 1206 including instructions, and the instructions can be executed by the processor 1201 of the electronic device 1200 to complete the above method. Optionally, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0178] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0179] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0180] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present disclosure belong.

[0181] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0182] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0183] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0184] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0185] The storage medium mentioned above may be a read-only memory, a disk or an optical disk, etc. Although the embodiments of the present disclosure have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present disclosure. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present disclosure.

Claims

1. A method for air gesture interaction, characterized in that: include: Acquire a first gesture image collected at a first moment, and recognize the first gesture image to obtain target coordinates of a hand skeleton point of the first gesture image under a target gesture category, wherein the target coordinates are coordinates in a three-dimensional coordinate system established with a wrist joint corresponding to the hand skeleton point as a coordinate origin; Calculating at least one first gesture feature of the hand skeleton point of the first gesture image based on the target coordinates of the hand skeleton point; When the at least one first gesture feature satisfies a first setting condition of the first air gesture, displaying an operation guide icon corresponding to the first air gesture; Acquire a second gesture image collected at a second moment, and calculate at least one second gesture feature of the hand skeleton point in the second gesture image; wherein the second moment is after the first moment; When the at least one second gesture feature satisfies a second set condition of the second air gesture indicated by the operation guide icon, a target operation corresponding to the second air gesture is performed.

2. The method according to claim 1, characterized in that The calculating, based on the target coordinates of the hand skeleton points, at least one first gesture feature of the hand skeleton points of the first gesture image comprises: Connecting a coordinate origin of the three-dimensional coordinate system and a first designated bone point among the hand bone points to determine a target vector between the coordinate origin of the three-dimensional coordinate system and the first designated bone point; Respectively projecting the target vector onto a first setting plane and a second setting plane in the three-dimensional coordinate system to determine a first angle between the target vector and the first setting plane and a second angle between the target vector and the second setting plane, wherein the first setting plane is perpendicular to the second setting plane; Obtaining the sub-coordinates of the first designated bone point in each coordinate axis of the three-dimensional coordinate system, and obtaining the target length between the coordinate origin and the second designated bone point in the hand bone point; Comparing the sub-coordinates of each coordinate axis with the target length to obtain at least one target ratio between the sub-coordinates of each coordinate axis and the target length; The first angle, the second angle and the at least one target ratio are used as the at least one first gesture feature.

3. The method according to claim 1, characterized in that The at least one second gesture feature includes: a first moving speed and a fusion speed in each coordinate axis direction of the three-dimensional coordinate system, and the at least one second gesture feature of the hand skeleton point calculated in the second gesture image includes: Acquire a third gesture image collected at a third moment; wherein the third moment is a previous collection moment adjacent to the second moment; determining a time difference between the second moment and the third moment, and determining a movement distance of the first designated skeletal point at the second moment relative to the third moment according to a difference between the coordinates of the first designated skeletal point in the second gesture image at the second moment and the coordinates of the first designated skeletal point in the third gesture image at the third moment; wherein the movement distance includes movement distances in the directions of each coordinate axis; Determine a first moving speed in each of the coordinate axis directions based on a ratio between the moving distance in each of the coordinate axis directions and the time difference; The first moving speeds in the directions of the coordinate axes are fused to obtain the fused speed.

4. The method according to claim 3, characterized in that The at least one second gesture feature includes: a first moving speed and a fusion speed in the directions of each coordinate axis of the three-dimensional coordinate system, and the at least one second gesture feature of the hand skeleton point calculated in the second gesture image includes: Acquire a fourth gesture image collected at a third moment; wherein the third moment is a previous collection moment adjacent to the second moment; combining the coordinates and the movement speed of the first designated skeleton point in the fourth gesture image at the third moment to obtain a first movement state, and using the first movement state to predict a second movement state of the first designated skeleton point at the second moment; Based on the set initial covariance matrix, predicting the predicted noise of the first designated skeleton point in the second gesture image at the second moment; fusing the predicted noise of the first designated skeleton point at the second moment with the set observed noise to obtain a fusion weight; The second motion state and the third motion state are weightedly summed by using the fusion weight to obtain the fusion motion state of the first designated skeleton point at the second moment; wherein the third motion state includes the coordinates of the first designated skeleton point at the second moment and the first moving speed in the direction of each coordinate axis; The second moving speed of the first designated skeleton point in the direction of each of the coordinate axes is obtained from the fused motion state, and the second moving speeds in the directions of each of the coordinate axes are fused to obtain the fused speed.

5. The method according to claim 4, characterized in that The predicting the predicted noise of the first designated skeleton point in the second gesture image at the second moment based on the set initial covariance matrix includes: Based on the set initial covariance matrix, perform at least one iteration to obtain the first covariance matrix at the third moment; Acquire a state transfer matrix of the second motion state and a second covariance matrix of the state transfer matrix; Multiplying the state transfer matrix, the first covariance matrix, and the transposed matrix of the state transfer matrix to obtain a target product; The target product is added to the second covariance matrix to obtain the predicted noise at the second moment.

6. The method according to claim 1, characterized in that The method further comprises: When the at least one first gesture feature does not satisfy the first set condition of the first air gesture, continue to acquire gesture images collected after the first moment until a third gesture image is acquired; Among them, at least one first gesture feature in the third gesture image satisfies the first setting condition of the first air gesture.

7. The method according to claim 1, characterized in that The method further comprises: When the at least one second gesture feature does not satisfy the second set condition of the second air gesture associated with the operation guide icon, determining a display time for displaying the operation guide icon corresponding to the first air gesture; When the display time is greater than or equal to a set time threshold, the operation guide icon is cancelled.

8. The method according to claim 1, characterized in that The step of recognizing the first gesture image to obtain target coordinates of hand skeleton points of the first gesture image under a target gesture category includes: Obtain a trained skeleton point recognition model; The first gesture image is recognized using a trained skeleton point recognition model to obtain target coordinates of the hand skeleton points of the first gesture image under the target gesture category.

9. The method according to claim 8, characterized in that The training method of the skeleton point recognition model includes: Acquire a sample gesture image, and perform category annotation on the hand in the sample gesture image to obtain annotated gesture categories, and annotate the coordinates of the sample hand skeleton points in the sample gesture image to obtain annotated coordinates of the skeleton points, wherein the annotated coordinates are coordinates with the sample wrist joint in the sample hand skeleton points as the coordinate origin; Inputting the sample gesture image into an initial skeleton point recognition model to obtain a predicted gesture category and skeleton point predicted coordinates corresponding to the sample gesture image; Generate a first sub-loss value according to a first difference between the labeled gesture category and the predicted gesture category, and generate a second sub-loss value according to a second difference between the labeled coordinates and the predicted coordinates of the skeleton point; Generate a loss function value according to the first sub-loss value and the second sub-loss value; The initial skeleton point recognition model is trained according to the loss function value.

10. An interactive device for air gestures, characterized in that: include: an acquisition module, configured to acquire a first gesture image acquired at a first moment, and perform skeleton point recognition on the first gesture image to obtain target coordinates of hand skeleton points of the first gesture image under a target gesture category, wherein the target coordinates are coordinates in a three-dimensional coordinate system established with a wrist joint corresponding to the hand skeleton point as a coordinate origin; A first calculation module, configured to calculate at least one first gesture feature of the hand skeleton point of the first gesture image based on the target coordinates of the hand skeleton point; A first control module, configured to display an operation guide icon corresponding to the first air gesture when the at least one first gesture feature satisfies a first setting condition of the first air gesture; A second calculation module is used to obtain a second gesture image collected by the electronic device at a second moment, and calculate at least one second gesture feature of the hand skeleton point in the second gesture image; wherein the second moment is after the first moment; The second control module is used to execute the target operation corresponding to the second air gesture when the at least one second gesture feature meets the second set condition of the second air gesture indicated by the operation guide icon.

11. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the air gesture interaction method described in any one of claims 1-9.

12. A computer-readable storage medium, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the air gesture interaction method as described in any one of claims 1-9.

13. A vehicle, characterized in that: Comprising the electronic device as claimed in claim 11.

Citation Information

Cited By

  • Aircraft control method and system based on gesture interaction

    CN120723080A