Gesture action detection method, apparatus, device, and storage medium

Abstract

The application relates to a gesture action detection method, device and equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring video stream data containing a to-be-detected gesture; detecting the to-be-detected gesture in the video stream data, wherein the detection process of an i-th to-be-detected gesture is as follows: extracting a three-dimensional characteristic value of the i-th to-be-detected gesture in the video stream data; obtaining an action category to which the i-th to-be-detected gesture belongs and a similarity belonging to the action category according to the three-dimensional characteristic value of the i-th to-be-detected gesture; and obtaining a detection result of whether the i-th to-be-detected gesture meets an action standard according to the action category of the i-th to-be-detected gesture, a standard category of an i-th standard gesture pre-stored and the similarity. The application is used for solving the problem that the existing skin color threshold recognition mode and the conventional static gesture recognition method cannot accurately judge the gesture category and affect the judgment of the gesture action standard.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a gesture action detection method and device, equipment and a storage medium. BACKGROUND

[0002] In actual industrial operation, life and human-computer interaction process, a series of operations are often performed by hands, and it is important to confirm whether the gesture operation action is standard. For example, in human-computer interaction, the standard of gesture action needs to be determined to deliver more accurate instruction information to the machine; in industrial operation, the standard of hand action operation needs to be detected in the process of packing and sealing articles to prevent the product quality from being affected by improper gesture operation; in fitness, teaching and game, etc., it is necessary to determine whether the teaching gesture action is standard, and in hand washing, it is necessary to detect whether the washing method of the hand is standard to determine whether the hand is washed in place.

[0003] In the existing gesture recognition technology, the recognition and judgment of gestures generally depend on the extraction of skin color. Due to the large difference in human skin color, and when some objects close to the skin color of the hand are encountered, only through the existing skin color threshold recognition method, the gesture is classified and recognized, which is easy to cause judgment error and affect the judgment of the standard of gesture action. In the process of gesture judgment, there is overlapping and shielding, and the gesture category cannot be accurately judged by the conventional static gesture recognition method, thereby affecting the judgment of the standard of gesture action. SUMMARY

[0004] The present application provides a gesture action detection method, device, equipment and storage medium to solve the problem that the existing skin color threshold recognition method and the conventional static gesture recognition method cannot accurately judge the gesture category and affect the judgment of the standard of gesture action.

[0005] In a first aspect, the embodiments of the present application provide a gesture action detection method, comprising:

[0006] Obtaining video stream data containing a to-be-detected gesture;

[0007] Optionally, the to-be-detected gesture in the video stream data is detected, and the detection process of the i-th to-be-detected gesture is as follows:

[0008] Extracting a three-dimensional feature value of the i-th to-be-detected gesture in the video stream data;

[0009] According to the three-dimensional feature value of the i-th to-be-detected gesture, obtaining an action category to which the i-th to-be-detected gesture belongs and a similarity belonging to the action category;

[0010] According to the action category of the i-th to-be-detected gesture, the standard category of the i-th standard gesture pre-stored, and the similarity, a detection result of whether the i-th to-be-detected gesture conforms to an action standard is obtained.

[0011] Optionally, the extracting the three-dimensional feature value of the i-th to-be-detected gesture in the video stream data comprises:

[0012] The video stream data is processed frame by frame to obtain frame images in time sequence;

[0013] The three-dimensional feature value of each frame image is extracted, the three-dimensional feature value being a three-dimensional feature of depth information meeting a skin color threshold range, the skin color threshold being used to distinguish the to-be-detected gesture from other parts except the to-be-detected gesture;

[0014] According to the three-dimensional feature value of each frame image, a starting frame image of the i-th to-be-detected gesture is obtained by identifying the frame image matching the starting image of the i-th standard gesture pre-stored, and an ending frame image of the i-th to-be-detected gesture is obtained by identifying the frame image matching the ending image of the i-th standard gesture pre-stored;

[0015] According to the three-dimensional feature value of all frame images between the starting frame image of the i-th to-be-detected gesture and the ending frame image of the i-th to-be-detected gesture, the three-dimensional feature value of the i-th to-be-detected gesture is obtained.

[0016] Optionally, the obtaining the action category of the i-th to-be-detected gesture and the similarity belonging to the action category according to the three-dimensional feature value of the i-th to-be-detected gesture comprises:

[0017] The three-dimensional feature value of the i-th to-be-detected gesture is input to a pre-trained gesture recognition model;

[0018] The similarity between the three-dimensional feature value of the i-th to-be-detected gesture and the three-dimensional feature value of each standard gesture is calculated through the gesture recognition model, a standard action category of a standard gesture corresponding to a maximum value in the similarity is taken as the action category of the i-th to-be-detected gesture, and the maximum value in the similarity is taken as the similarity belonging to the action category.

[0019] Optionally, the obtaining the detection result of whether the i-th to-be-detected gesture conforms to the action standard according to the action category of the i-th to-be-detected gesture, the standard category of the i-th standard gesture pre-stored, and the similarity comprises:

[0020] It is judged whether the action category of the i-th to-be-detected gesture is consistent with the standard category of the i-th standard gesture pre-stored;

[0021] if not consistent, obtaining a detection result that the i-th gesture to be tested does not conform to the action standard of the i-th standard gesture;

[0022] if consistent, judging whether the similarity meets a preset similarity threshold, if yes, obtaining a detection result that the i-th gesture to be tested conforms to the action standard of the i-th standard gesture, if not, obtaining a detection result that the i-th gesture to be tested does not conform to the action standard of the i-th standard gesture.

[0023] Optionally, after obtaining the detection result that the i-th gesture to be tested conforms to the action standard of the i-th standard gesture, the method further comprises:

[0024] inputting the three-dimensional feature value of the i-th gesture to be tested into the gesture recognition model.

[0025] Optionally, the three-dimensional feature value comprises a palm key point feature value, a finger joint key point feature value and a wrist key point feature value.

[0026] The method of recognizing the three-dimensional feature value of the i-th gesture to be tested to obtain an action category of the i-th gesture to be tested and a similarity belonging to the action category comprises:

[0027] calculating the finger joint key point feature value to obtain a finger bending angle;

[0028] calculating the wrist key point feature value to obtain a wrist rotation angle;

[0029] recognizing the palm key point feature value, the finger bending angle and the wrist rotation angle to obtain the action category of the i-th gesture to be tested and the similarity belonging to the action category.

[0030] Optionally, after obtaining the detection result that the i-th gesture to be tested conforms to the action standard, the method further comprises:

[0031] calculating a total number of the gestures to be tested that meet the detection result of conforming to the action standard;

[0032] comparing the total number of the gestures to be tested that meet the detection result of conforming to the action standard with a total number of all the standard gestures pre-stored, if the total number of all the gestures to be tested that meet the detection result is less than the total number of all the standard gestures pre-stored, judging that all the standard gestures pre-stored are not completed;

[0033] if the total number of the gestures to be tested that meet the detection result of conforming to the action standard is equal to the total number of the standard gestures pre-stored, judging that all the standard gestures pre-stored are completed.

[0034] In a second aspect, an embodiment of the present application provides a gesture action detection device, comprising:

[0035] an acquisition module configured to acquire video stream data containing a to-be-detected gesture;

[0036] a detection module configured to detect the to-be-detected gesture in the video stream data, wherein a detection process of an i-th to-be-detected gesture is as follows: three-dimensional feature values of the i-th to-be-detected gesture in the video stream data are extracted; the three-dimensional feature values of the i-th to-be-detected gesture are identified to obtain an action category of the i-th to-be-detected gesture and a similarity belonging to the action category; and a detection result of whether the i-th to-be-detected gesture conforms to an action standard is obtained according to the action category of the i-th to-be-detected gesture, a standard category of an i-th standard gesture pre-stored, and the similarity.

[0037] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus;

[0038] the memory is configured to store a computer program;

[0039] the processor is configured to execute the program stored in the memory to realize the gesture action detection method in the first aspect.

[0040] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the gesture action detection method in the first aspect.

[0041] The above technical solution provided by the embodiments of the present application has the following advantages compared with the prior art: the method provided by the embodiments of the present application detects three-dimensional feature values of each to-be-detected gesture in video stream data, performs multi-dimensional detection on the to-be-detected gesture, realizes accurate recognition of the contour of the to-be-detected gesture even in the case of overlapping and occlusion of both hands, thereby accurately recognizing the action category of the to-be-detected gesture, and further realizes the purpose of accurately judging whether the to-be-detected gesture conforms to the action standard according to the action category of the to-be-detected gesture and the similarity belonging to the action category. BRIEF DESCRIPTION OF DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0043] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0044] Figure 1 A flowchart of a gesture action detection method provided for the first embodiment of the present application;

[0045] Figure 2 A flowchart of a detection process of the ith gesture to be detected provided for the second embodiment of the present application;

[0046] Figure 3 A flowchart of extracting three-dimensional feature values of the ith gesture to be detected in the video stream data provided for the second embodiment of the present application;

[0047] Figure 4 A structural diagram of a gesture action detection device provided for the embodiments of the present application;

[0048] Figure 5 A structural diagram of an electronic device provided for the embodiments of the present application. DETAILED DESCRIPTION

[0049] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0050] A gesture action detection method is provided in the embodiments of the present application, which is used to detect whether a gesture action in video stream data meets an action standard. The method can be applied in the fields of human-computer interaction, practical teaching, public video monitoring, daily life and industry, etc. The field of human-computer interaction can include human-computer interaction games, intelligent robots, etc. The field of practical teaching can include experimental operation, sign language teaching, etc. The field of public video monitoring can include detection of behaviorally abnormal personnel in airports, public activity places, etc. The field of daily life can include detection of behaviorally abnormal personnel in swimming and bodybuilding, detection of hand washing posture, etc. The field of industry can include logistics packaging, production of products requiring manual operation, etc.

[0051] The method can be directly integrated in the form of a software module in an electronic device, which can be directly a device used for shooting the video stream, or other electronic devices in communication connection with the device used for shooting the video stream, which can be any form of terminal or server.

[0052] Embodiment one

[0053] Referring to Figure 1 The gesture action detection method of the embodiment one of the present application comprises:

[0054] S101, acquiring video stream data containing a to-be-detected gesture.

[0055] In some embodiments, the video stream data is collected by a camera, and the camera is arranged in a spatial region of the hand to be detected, so as to detect the action of the hand in real time and obtain the video stream data containing the to-be-detected gesture.

[0056] The camera can be a depth camera or a camera connected with a depth sensor.

[0057] The to-be-detected gesture in the video stream data is detected to determine whether the to-be-detected gesture meets the requirements of a standard gesture. The to-be-detected gestures in the video stream data are detected one by one in the time sequence of the video stream data, and the detection process of the i-th to-be-detected gesture is as follows:

[0058] S102, extracting a three-dimensional feature value of the i-th to-be-detected gesture in the video stream data.

[0059] The feature points of the to-be-detected gesture are selected from the video stream data, a coordinate system is established according to the shooting direction of the camera, the three-dimensional coordinates of the feature points in the coordinate system are acquired, and the three-dimensional feature value of the to-be-detected gesture is obtained.

[0060] The coordinate system can be a camera coordinate system, with the camera as the origin, the x-axis to the right, the z-axis to the front (towards the direction of the camera), and the y-axis upward (above the camera itself), or a three-dimensional coordinate system can be established with other reference objects as the origin.

[0061] The selected feature points are key points that can represent the features of the hand, which can include the contour feature points of each finger, the contour feature points of the wrist, and the contour feature points of the palm. The selected key points are at least three points on the inner or outer edge of the contour, or can be continuous points along the inner or outer edge of the contour.

[0062] S103, obtaining the action category of the i-th to-be-detected gesture and the similarity belonging to the action category according to the three-dimensional feature value of the i-th to-be-detected gesture.

[0063] S104 , obtaining a detection result of whether the i-th gesture to be measured meets the action standard according to the action category of the i-th gesture to be measured, the standard category of the pre-stored i-th standard gesture, and the similarity.

[0064] In some embodiments, a detection result of whether the gesture to be measured meets the action standard is obtained by determining whether the action category of the i-th gesture to be measured is consistent with the standard category of the pre-stored i-th standard gesture, and whether the similarity reaches a preset threshold range.

[0065] If it is determined that the action category of the i-th gesture to be measured is inconsistent with the standard category of the pre-stored i-th standard gesture, the detection result obtained is that the action category of the i-th gesture to be measured does not meet the standard category of the i-th standard gesture.

[0066] If the action category of the i-th gesture to be measured is determined to be consistent with the standard category of the pre-stored i-th standard gesture, and the similarity meets the preset similarity threshold, the detection result obtained is that the i-th gesture to be measured meets the action standard of the i-th standard gesture.

[0067] If the action category of the i-th gesture to be measured is determined to be consistent with the standard category of the pre-stored i-th standard gesture, and the similarity does not meet the preset similarity threshold, the detection result obtained is that the i-th gesture to be measured does not meet the action standard of the i-th standard gesture.

[0068] The embodiment of the present application performs multi-dimensional detection of the gesture to be measured by detecting the three-dimensional feature values ​​of each gesture to be measured in the video stream data, thereby achieving accurate identification of the outline of the gesture to be measured even when the hands are overlapped and occluded, thereby accurately identifying the action category of the gesture to be measured, and judging whether the gesture to be measured meets the action standard based on the action category of the gesture to be measured and the similarity between the action category and the action category, thereby achieving the purpose of accurately judging whether the gesture to be measured meets the action standard.

[0069] Example 2

[0070] In the second embodiment of the present application, the method of the present application is specifically introduced by detecting the hand washing posture. Figure 2 As shown, the gesture action detection method of the embodiment of the present application introduces the technical features of the gesture recognition model, and the detection process of the i-th gesture to be detected is described in detail. The detection process steps specifically include:

[0071] Before detecting the i-th gesture to be measured by the gesture recognition model, the video stream data is preprocessed, wherein the preprocessing method includes frame processing, depth information processing and format conversion.

[0072] S201, extract the three-dimensional feature value of the i-th to-be-detected gesture in the video stream data, and the three-dimensional feature value includes a palm key point feature value, a finger joint key point feature value, and a wrist key point feature value.

[0073] In some embodiments, the process of extracting the three-dimensional feature value of the i-th to-be-detected gesture in the video stream data refers to Figure 3 as shown, and includes:

[0074] S301, frame processing of the video stream data is performed to obtain a frame image sorted in time sequence.

[0075] In some embodiments, the video stream data can be framed by a long short term memory network (LSTM), and each frame corresponds to an image. For example, if the video frame rate is 25, there are 25 frames per second, that is, 25 images. The continuity of the previous frame and the next frame of the video stream is maintained according to the time sequence information to obtain a video image sequence containing time sequence information.

[0076] The long short term memory network is a neural network algorithm for processing and predicting time series with long delay or interval. The framing method of the video stream data is not limited to this network, and other neural network models can also be used for framing the video stream data, as long as the continuity of the previous frame and the next frame of the framed video stream data can be maintained.

[0077] Each framed image after the above preprocessing is input into a gesture recognition model. The gesture recognition model is a neural network based on deep learning. The gesture recognition model is trained according to a standard gesture video template data set.

[0078] S302, extract the three-dimensional feature value of each framed image, and the three-dimensional feature value is a three-dimensional feature value of depth information meeting a skin color threshold range, and the skin color threshold is used to distinguish the to-be-detected gesture from other parts except the to-be-detected gesture.

[0079] From each framed image, the depth information meeting the skin color threshold range is identified to distinguish the to-be-detected gesture from other parts except the to-be-detected gesture. The distinguishing method can be to remove the part of each framed image except the to-be-detected gesture to obtain a framed image containing the to-be-detected gesture, so as to determine the position of the to-be-detected gesture in space and realize tracking of the to-be-detected gesture. Meanwhile, the influence of objects with similar skin color can be avoided through the skin color threshold and multi-dimensional depth information.

[0080] After obtaining the framed image of the depth information meeting the skin color threshold range, the framed image of the depth information meeting the skin color threshold range is format-converted to meet the input format of the gesture recognition model, and the format-converted framed image is input into the gesture recognition model.

[0081] The three-dimensional feature value of the gesture to be detected is obtained by calculating the coordinates of the three-dimensional feature value in the camera coordinate system from the three-dimensional feature value in the depth information of each sub-frame image.

[0082] In some embodiments, the three-dimensional feature value can include a palm key point feature value, a finger joint key point feature value, and a wrist key point feature value, and the key point is at least three points on the inner and outer edges of the contour. The finger joint is each joint of the finger, generally three joints of the finger.

[0083] S303, according to the three-dimensional feature value of each sub-frame image, identify the sub-frame image matching the starting image of the pre-stored i-th standard gesture, obtain the start frame image of the i-th gesture to be detected, and identify the sub-frame image matching the end image of the pre-stored i-th standard gesture, obtain the end frame image of the i-th gesture to be detected.

[0084] For example, the video template data set of the standard gesture includes seven action categories of washing hands and other action categories, and the seven action categories are "in", "out", "clamping", "arch", "big", "standing", and "wrist" respectively. The first standard gesture is "in", the first image of "in" is the feature of the palm and the palm contact, or the feature of the palm and the finger contact, and the last image is the feature of the palm and the palm separation, or the feature of the palm and the finger separation.

[0085] If the three-dimensional feature value corresponding to the feature of the palm and the palm contact, or the feature of the palm and the finger contact in the first sub-frame image of "in" is identified in the detection of the first gesture to be detected, the sub-frame image corresponding to the three-dimensional feature value is taken as the start frame image of the first gesture to be detected. When the feature corresponding to the last image of the first standard gesture is detected in the sub-frame image of the gesture to be detected, the end frame image is obtained.

[0086] S304, according to the three-dimensional feature value of all sub-frame images between the start frame image of the i-th gesture to be detected and the end frame image of the i-th gesture to be detected, obtain the three-dimensional feature value of the i-th gesture to be detected.

[0087] In some embodiments, the three-dimensional feature value of all sub-frame images between the time corresponding to the start frame image of the i-th gesture to be detected and the time corresponding to the end frame image of the i-th gesture to be detected is calculated by averaging to obtain the averaged three-dimensional feature value.

[0088] The averaging calculation means that the three-dimensional feature value extracted in the gesture recognition model is normalized to a one-dimensional array, and then each one-dimensional array is calculated by matrix to obtain the averaged three-dimensional feature value. In addition to taking the average of the three-dimensional feature values of all sub-frame images, other ways of calculating the three-dimensional feature values can also be used, which are not limited here.

[0089] input the average three-dimensional feature value into a pre-trained gesture recognition model, calculate the similarity between the three-dimensional feature value of the i-th gesture to be tested and the three-dimensional feature value of each standard gesture through the gesture recognition model, and take the standard action category of the standard gesture corresponding to the maximum value in the similarity as the action category of the i-th gesture to be tested, and take the maximum value in the similarity as the similarity belonging to the action category.

[0090] In the action category recognition process of the gesture to be tested, if it is unable to recognize any category belonging to all action categories, the gesture to be tested is judged as other action categories.

[0091] In some embodiments, the step of identifying the three-dimensional feature value of the i-th gesture to be tested to obtain the action category of the i-th gesture to be tested and the similarity belonging to the action category can include:

[0092] S202, calculate the finger joint key point feature value to obtain the finger bending angle, calculate the wrist key point feature value to obtain the wrist rotation angle, identify the palm key point feature value, the finger bending angle and the wrist rotation angle, and obtain the action category of the i-th gesture to be tested and the similarity belonging to the action category.

[0093] S203, obtain the detection result of whether the i-th gesture to be tested meets the action standard, including:

[0094] determine whether the action category of the i-th gesture to be tested is consistent with the standard category of the i-th standard gesture pre-stored.

[0095] If not consistent, obtain the detection result that the action category of the i-th gesture to be tested does not meet the standard category of the i-th standard gesture.

[0096] If consistent, determine whether the similarity meets a preset similarity threshold, if yes, obtain the detection result that the i-th gesture to be tested meets the action standard of the i-th standard gesture, and if not, obtain the detection result that the i-th gesture to be tested does not meet the action standard of the i-th standard gesture.

[0097] In some embodiments, after obtaining the detection result, further comprising:

[0098] S204, display the extracted three-dimensional feature value of the gesture to be tested and the obtained detection result of whether the gesture to be tested meets the action standard in real time to prompt the user.

[0099] In some embodiments, according to the detection process of the i-th gesture to be tested, each gesture to be tested is detected, and after obtaining the detection result of all gestures to be tested, further comprising:

[0100] counting the total number of the detected gestures meeting the action criterion, comparing the total number of the detected gestures meeting the action criterion with the total number of all the pre-stored standard gestures, and determining that the pre-stored all the standard gestures are not completed if the total number of the detected gestures meeting the action criterion is less than the total number of all the pre-stored standard gestures.

[0101] determining that the pre-stored all the standard gestures are completed if the total number of the detected gestures meeting the action criterion is equal to the total number of all the pre-stored standard gestures.

[0102] In some embodiments, after determining that the pre-stored all the standard gestures are not completed or determining that the pre-stored all the standard gestures are completed, the determination result that the pre-stored all the standard gestures are not completed or the detection result that the pre-stored all the standard gestures are completed is displayed to prompt the user.

[0103] In some embodiments, after obtaining the detection result that the ith detected gesture meets the action criterion of the ith standard gesture, the method further comprises:

[0104] S205, inputting the three-dimensional feature value of the ith detected gesture meeting the action criterion into the gesture recognition model. The loss function between the three-dimensional feature value of the detected gesture meeting the action criterion and the three-dimensional feature value of the standard gesture is calculated, so that the gesture recognition model is continuously learned, and the stability and classification accuracy of the model are enhanced.

[0105] Based on the same concept, the embodiments of the present application provide a gesture action detection device. The specific implementation of the device can be referred to the description of the method embodiments, and the repeated parts will not be described here. As shown in the Figure 4 The device mainly comprises:

[0106] The acquisition module 401 is configured to acquire video stream data containing a detected gesture.

[0107] The detection module 402 is configured to detect the detected gesture in the video stream data. The detection process of the ith detected gesture is as follows: extracting the three-dimensional feature value of the ith detected gesture in the video stream data; identifying the three-dimensional feature value of the ith detected gesture to obtain the action category of the ith detected gesture and the similarity belonging to the action category; and obtaining the detection result that the ith detected gesture meets the action criterion according to the action category of the ith detected gesture, the standard category of the pre-stored ith standard gesture and the similarity.

[0108] The embodiment of the present application acquires video stream data containing a to-be-tested gesture through an acquisition module, detects three-dimensional characteristic values of each to-be-tested gesture in the video stream data through a detection module, performs multi-dimensional detection on the to-be-tested gesture, realizes accurate recognition of the contour of the to-be-tested gesture even in the case of overlapping and shielding of the two hands, thereby accurately recognizing the action category of the to-be-tested gesture, and judging whether the to-be-tested gesture conforms to the action standard according to the action category of the to-be-tested gesture and the similarity belonging to the action category, and further realizing the purpose of accurately judging whether the to-be-tested gesture conforms to the action standard.

[0109] Based on the same concept, the embodiment of the present application further provides an electronic device, as shown in the Figure 5 The electronic device mainly includes a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502 and the memory 503 complete mutual communication through the communication bus 504. Wherein the memory 503 stores a program which can be executed by the processor 501, and the processor 501 executes the program stored in the memory 503 to realize the gesture action detection method steps described in the above embodiment.

[0110] The communication bus 504 mentioned in the above electronic device can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 504 can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 5 In the above electronic device, only one thick line is used to represent the communication bus 504, but it does not mean that there is only one bus or only one type of bus.

[0111] The communication interface 502 is used for communication between the above electronic device and other devices.

[0112] The memory 503 can include a random access memory (RAM) and can also include a non-volatile memory, for example, at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor 501.

[0113] The processor 501 described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc., and can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0114] In yet another embodiment of the present application, a computer readable storage medium is also provided, which stores a computer program. When the computer program is run on a computer, the computer is caused to perform the gesture action detection method steps described in the above embodiments.

[0115] In the above embodiments, the implementation can be achieved by software, hardware, firmware or any combination thereof, entirely or partially. When implemented by software, the implementation can be in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed by a computer, the computer instructions generate the processes or functions described in the embodiments of the present application, entirely or partially. The computer can be a general purpose computer, a special purpose computer, a computer network or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions are transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available medium can be a magnetic medium (such as floppy disk, hard disk, magnetic tape, etc.), an optical medium (such as DVD) or a semiconductor medium (such as solid state disk), etc.

[0116] It has to be noted that, in the present document, relational terms are intended only to convey a possible relationship between elements or

[0117] The foregoing is considered as illustrative only of the principles of the application. Numerous modifications and changes will readily occur to those skilled in the art, which modifications and changes are to be understood as intended to be encompassed by the general scope of the application. Accordingly, the application is not to be limited to the above described ones; rather, the above described ones are to serve as a basis for the claims as they follow.

Claims

1. A gesture action detection method, characterized by, The method comprises the following steps: acquiring video stream data containing a to-be-detected gesture, and dividing the video stream data into frames by using a long short-term memory network; detecting the to-be-detected gesture in the video stream data, wherein the detection process of the ith to-be-detected gesture is as follows: extracting three-dimensional feature values of the ith to-be-detected gesture in the video stream data, comprising: selecting feature points of the to-be-detected gesture from the video stream data, establishing a coordinate system according to the shooting direction of the camera, and acquiring three-dimensional coordinates of the feature points in the coordinate system to obtain the three-dimensional feature values of the to-be-detected gesture, wherein the feature points comprise contour feature points of each finger, contour feature points of the wrist and contour feature points of the palm, the feature points are at least three points on the inner edge or the outer edge of the contour, or are continuous points along the inner edge or the outer edge of the contour; obtaining an action category to which the ith to-be-detected gesture belongs and a similarity belonging to the action category according to the three-dimensional feature values of the ith to-be-detected gesture; obtaining a detection result of whether the ith to-be-detected gesture meets an action standard according to the action category of the ith to-be-detected gesture, a standard category of the ith standard gesture pre-stored and the similarity; after obtaining the detection result that the ith to-be-detected gesture meets the action standard of the ith standard gesture, further comprising: inputting the three-dimensional feature values of the ith to-be-detected gesture meeting the action standard into a gesture recognition model to calculate a loss function between the three-dimensional feature values of the to-be-detected gesture meeting the action standard and the three-dimensional feature values of the standard gesture, and optimizing the gesture recognition model by using the loss function. The extraction of the three-dimensional feature values of the ith to-be-detected gesture in the video stream data comprises:

2. The gesture motion detection method according to claim 1, characterized by, processing the video stream data by frame to obtain frame images sorted in time sequence; extracting three-dimensional feature values of each frame image, the three-dimensional feature values being three-dimensional feature values of depth information meeting a skin color threshold range, the skin color threshold being used to distinguish the to-be-detected gesture from other parts except the to-be-detected gesture; according to the three-dimensional feature values of each frame image, identifying the frame image matching a pre-stored starting image of the ith standard gesture to obtain a starting frame image of the ith to-be-detected gesture, and identifying the frame image matching a pre-stored ending image of the ith standard gesture to obtain an ending frame image of the ith to-be-detected gesture; obtaining the three-dimensional feature values of the ith to-be-detected gesture according to the three-dimensional feature values of all frame images between the starting frame image of the ith to-be-detected gesture and the ending frame image of the ith to-be-detected gesture. The obtaining of the action category of the ith to-be-detected gesture and the similarity belonging to the action category according to the three-dimensional feature values of the ith to-be-detected gesture comprises: inputting the three-dimensional feature values of the ith to-be-detected gesture into a pre-trained gesture recognition model; 3. The gesture motion detection method of claim 1, wherein, calculating, by using the gesture recognition model, the similarity between the three-dimensional feature values of the ith to-be-detected gesture and the three-dimensional feature values of each standard gesture, taking the standard action category of the standard gesture corresponding to the maximum value in the similarity as the action category of the ith to-be-detected gesture, and taking the maximum value in the similarity as the similarity belonging to the action category. ​ ​ 4. The gesture motion detection method according to claim 3, characterized by, The detection result of whether the i-th to-be-detected gesture meets the action standard is obtained according to the action category of the i-th to-be-detected gesture, the standard category of the pre-stored i-th standard gesture and the similarity, and the detection result includes: It is judged whether the action category of the i-th to-be-detected gesture is consistent with the standard category of the pre-stored i-th standard gesture; If not, a detection result that the action category of the i-th to-be-detected gesture does not meet the standard category of the i-th standard gesture is obtained; If consistent, it is judged whether the similarity meets a preset similarity threshold, if yes, a detection result that the i-th to-be-detected gesture meets the action standard of the i-th standard gesture is obtained, and if not, a detection result that the i-th to-be-detected gesture does not meet the action standard of the i-th standard gesture is obtained.

5. The gesture action detection method of claim 4, wherein, After the detection result that the i-th to-be-detected gesture meets the action standard of the i-th standard gesture is obtained, the following steps are further included: The three-dimensional feature value of the i-th to-be-detected gesture is input into the gesture recognition model.

6. The gesture motion detection method of claim 1, wherein, The three-dimensional feature value includes a palm key point feature value, a finger joint key point feature value and a wrist key point feature value; The action category of the i-th to-be-detected gesture and the similarity belonging to the action category are obtained according to the three-dimensional feature value of the i-th to-be-detected gesture, and the obtaining includes: The finger joint key point feature value is calculated to obtain a finger bending angle; The wrist key point feature value is calculated to obtain a wrist rotation angle; According to the palm key point feature value, the finger bending angle and the wrist rotation angle, the action category of the i-th to-be-detected gesture and the similarity belonging to the action category are obtained.

7. The gesture motion detection method of claim 1, wherein, After the detection result of whether the i-th to-be-detected gesture meets the action standard is obtained, the following steps are further included: The total number of the to-be-detected gestures meeting the action standard in the detection result is calculated; The total number of the to-be-detected gestures meeting the action standard in the detection result is compared with the total number of all pre-stored standard gestures, if the total number of all to-be-detected gestures in the detection result is less than the total number of all pre-stored standard gestures, it is judged that all pre-stored standard gestures have not been completed; If the total number of the to-be-detected gestures meeting the action standard in the detection result is equal to the total number of the pre-stored standard gestures, it is judged that all pre-stored standard gestures have been completed.

8. An apparatus, comprising: It includes: The acquisition module is used to acquire video stream data containing a to-be-detected gesture, and frame the video stream data through a long short-term memory network; The detection module is used to detect the to-be-detected gesture in the video stream data, The detection process of the ith gesture to be detected is as follows: three-dimensional characteristic values of the ith gesture to be detected in the video stream data are extracted, including: selecting feature points of the gesture to be detected from the video stream data, establishing a coordinate system according to the shooting direction of the camera, obtaining three-dimensional coordinates of the feature points in the coordinate system, and obtaining three-dimensional characteristic values of the gesture to be detected, wherein the feature points include contour feature points of each finger, contour feature points of a wrist and contour feature points of a palm, the feature points are at least three points on an inner edge or an outer edge of a contour or are continuous points along the inner edge or the outer edge of the contour; three-dimensional characteristic values of the ith gesture to be detected are identified, to obtain an action category of the ith gesture to be detected and a similarity belonging to the action category; and a detection result of whether the ith gesture to be detected meets an action standard is obtained according to the action category of the ith gesture to be detected, a standard category of an ith standard gesture pre-stored and the similarity; The optimization module is configured to input the three-dimensional characteristic values of the ith gesture to be detected meeting the action standard into a gesture recognition model, to calculate a loss function between the three-dimensional characteristic values of the gesture to be detected meeting the action standard and three-dimensional characteristic values of a standard gesture, and to optimize the gesture recognition model by using the loss function.

9. An electronic device, comprising: The gesture action detection method comprises the following steps: The processor, the communication interface, the memory and the communication bus are in communication with each other through the communication bus; The memory is configured to store a computer program; The processor is configured to execute the program stored in the memory, to implement the gesture action detection method according to any one of claims 1-7.

10. A computer readable storage medium storing a computer program, characterized in that, The computer program, when executed by the processor, implements the gesture action detection method according to any one of claims 1-7.

Images