Method, device and apparatus for determining trigger gesture

Abstract

The embodiments of the present application relate to the field of human-computer interaction technology, and disclose a method, device, and apparatus for determining a trigger gesture, obtaining N video frames that are sequentially continuous in a video stream, performing target component detection processing on each video frame, obtaining a bounding box of at least one target component, and then performing gesture recognition based on the bounding box of at least one target component and the video frame to determine the gesture corresponding to the video frame. The gestures corresponding to the N video frames constitute a gesture sequence, and finally, the trigger gesture is determined based on the gesture sequence. That is, the target component (human head, human face, or human head plus shoulders) is detected first, and then gesture detection and recognition is performed in a local area near the target component (around the bounding box). Compared with gesture detection and recognition of the entire image, the computing power can be reduced. In addition, the feature granularity of the hand component is relatively large in the local area near the target component, so that the features of the hand component are obvious, which can promote the improvement of the accuracy of gesture detection in each video frame.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of human-computer interaction technology, and in particular, to a method, device, and apparatus for determining a trigger gesture. Background Art

[0002] With the continuous development and popularization of intelligent, electronic and interconnected products, many increasingly intelligent human-computer interaction methods have emerged to meet people's pursuit of personalization and fashion. For example, the touch screen of a smartphone is a human-computer interaction system realized by touch. There are also some products that are controlled by voice interaction, such as smart speakers. As long as the user inputs relevant commands by voice, the product can perform related operations according to the voice input commands. In some products, such as smart cameras, game consoles or drones, human-computer interaction can also be achieved through gestures. Different gestures correspond to different commands. When the product obtains a gesture, it executes the command corresponding to the gesture.

[0003] However, gestures are usually small and flexible in images or video frames, making them difficult to track. With limited computing power, a gesture recognition model is used to detect and identify gestures in the entire image to determine the trigger gesture used to trigger the product to execute the corresponding command. Due to the compression of the recognition model, the hand area accounts for a very small proportion of the entire image, making the gesture recognition model unreliable and unable to meet the accuracy and speed requirements of device interaction, which is prone to false triggering. Summary of the invention

[0004] The main technical problem solved by the embodiments of the present application is to provide a method, device and equipment for determining a trigger gesture, which can improve the accuracy of the trigger gesture and effectively reduce false triggers while keeping the computing power as low as possible, that is, to achieve the robustness and stability of the gesture trigger.

[0005] To solve the above technical problems, in a first aspect, an embodiment of the present application provides a method for determining a trigger gesture, including:

[0006] Obtain N video frames that are consecutive in time sequence in the video stream, where N is an integer greater than or equal to 1;

[0007] Performing target component detection processing on the video frame to obtain a bounding box of at least one target component;

[0008] Perform gesture recognition according to the bounding box of at least one target component and the video frame to determine the gesture corresponding to the video frame, where the gestures corresponding to the N video frames constitute a gesture sequence;

[0009] According to the gesture sequence, the trigger gesture is determined.

[0010] In some embodiments, the aforementioned performing gesture recognition based on the bounding box of at least one target component and the video frame to determine the gesture corresponding to the video frame includes:

[0011] Performing outward expansion processing on the bounding box of at least one target component respectively to obtain at least one outward expanded bounding box;

[0012] Performing hand part detection processing on an area in the video frame that is within at least one outward-expanded bounding box to obtain a hand part detection result;

[0013] Determine the target hand part according to the hand part detection result;

[0014] Perform gesture classification processing on the target hand part to obtain the gesture corresponding to the video frame.

[0015] In some embodiments, the above-mentioned determining the target hand part according to the hand part detection result includes:

[0016] If the hand part detection result includes a hand part, the hand part is determined to be a target hand part;

[0017] If the hand part detection result includes multiple hand parts, the score of each hand part is calculated according to the confidence of each hand part, the distance between the target parts corresponding to the hand part and the pixel area of ​​the hand part, and the hand part with the highest score is determined as the target hand part.

[0018] In some embodiments, the aforementioned calculation of the score of each hand component according to the confidence of each hand component, the distance between the target component corresponding to the hand component, and the pixel area of ​​the hand component includes:

[0019] The score of each hand component is calculated using the following formula;

[0020]

[0021] Among them, Score i is the score of the i-th hand component, i is the label of the hand component, Conf i is the confidence of the i-th hand part, Dist i is the distance between the i-th hand part and the target part, Area i is the pixel area of ​​the ith hand part.

[0022] In some embodiments, the aforementioned determining the triggering gesture includes:

[0023] The number of gestures corresponding to each gesture category in the gesture sequence is counted, and if a first gesture category with the largest number of gestures meets a preset condition, the gesture belonging to the first gesture category is used as a trigger gesture.

[0024] In some embodiments, the aforementioned preset conditions include that the percentage of gestures belonging to the first gesture category in the total number of gestures in the gesture sequence is greater than or equal to a percentage threshold, or the number of gestures belonging to the first gesture category is greater than or equal to a quantity threshold.

[0025] In some embodiments, the method further includes: acquiring a first target component corresponding to the trigger gesture;

[0026] The gesture corresponding to the first target component is tracked and detected in the video stream to determine a new trigger gesture.

[0027] In order to solve the above technical problems, in a second aspect, an interactive method is provided in an embodiment of the present application, including:

[0028] Determine the trigger gesture using the method for determining the trigger gesture in the first aspect above;

[0029] Control the target device to execute the operation instruction corresponding to the trigger gesture.

[0030] To solve the above technical problems, in a third aspect, an embodiment of the present application provides a device for determining a trigger gesture, including:

[0031] An acquisition module, used for acquiring N video frames that are sequentially continuous in the video stream, where N is an integer greater than or equal to 1;

[0032] A detection module, used to perform target component detection processing on the video frame to obtain a boundary box of at least one target component;

[0033] A recognition module, configured to perform gesture recognition based on a bounding box of at least one target component and a video frame, and determine a gesture corresponding to the video frame, wherein the gestures corresponding to the N video frames constitute a gesture sequence;

[0034] The determination module is used to determine the trigger gesture according to the gesture sequence.

[0035] In order to solve the above technical problems, in a fourth aspect, an electronic device is provided in an embodiment of the present application, characterized in that it includes:

[0036] at least one processor, and

[0037] a memory communicatively coupled to at least one processor, wherein:

[0038] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of the first aspect.

[0039] To solve the above technical problems, in a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method of the first aspect.

[0040] Beneficial effects of the embodiments of the present application: Different from the prior art, the method for determining the trigger gesture provided by the embodiments of the present application first obtains N video frames that are sequentially continuous in the video stream, performs target component detection processing on each video frame, obtains the bounding box of at least one target component, then performs gesture recognition based on the bounding box of at least one target component and the video frame, determines the gesture corresponding to the video frame, and the gestures corresponding to the N video frames constitute a gesture sequence, and finally, determines the trigger gesture based on the gesture sequence. In this method, the target component (head, face, or head and shoulder) is detected first, and then gesture detection and recognition is performed in the local area near the target component (around the bounding box). Compared with the gesture detection and recognition of the whole picture, the computing power can be reduced. In addition, the feature granularity of the hand component is relatively large in the local area near the target component, so that the feature of the hand component is obvious, which can promote the improvement of the accuracy of gesture detection in each video frame. In addition, the trigger gesture is obtained by considering the gestures in the N video frames that are sequentially continuous. Compared with the trigger gesture determined by the gesture of a single frame, the accuracy of the trigger gesture can be improved and false triggering can be avoided to a certain extent. For example, if a user accidentally makes a gesture and the gesture suddenly appears less times in the gesture sequence, it will not be recognized as a valid trigger gesture, so the gesture will not be responded to, reducing the occurrence of false triggers. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0042] Figure 1 This is a schematic diagram of an application scenario in which human-computer interaction is achieved through gestures in some embodiments of the present application;

[0043] Figure 2 A flowchart of a method for determining a trigger gesture in some embodiments of the present application;

[0044] Figure 3 A schematic diagram of gestures in some embodiments of the present application;

[0045] Figure 4 A schematic diagram of a bounding box on a video frame in some embodiments of the present application;

[0046] Figure 5A schematic diagram of a bounding box expanding into an outward-expanding bounding box in some embodiments of the present application;

[0047] Figure 6 This is a schematic diagram of a user's posture in front of an electronic device in some embodiments of the present application;

[0048] Figure 7 A flowchart of a method for determining a trigger gesture in some embodiments of the present application;

[0049] Figure 8 A schematic diagram of the structure of an apparatus for determining a trigger gesture in some embodiments of the present application;

[0050] Fig. 9 This is a schematic diagram of the structure of an electronic device in some embodiments of the present application. DETAILED DESCRIPTION

[0051] The present application is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements can also be made without departing from the concept of the present application. These all belong to the protection scope of the present application.

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] It should be noted that, if there is no conflict, the various features in the embodiments of the present application can be combined with each other, all within the scope of protection of the present application. In addition, although the functional module division is performed in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a sequence different from the module division in the device or the flow chart. In addition, the words "first", "second", "third", etc. used herein do not limit the data and execution order, but only distinguish the same items or similar items with basically the same functions and effects.

[0054] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in this specification and in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used in this specification includes any and all combinations of one or more of the related listed items.

[0055] In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0056] See also Figure 1 , Figure 1 The schematic diagram of an application scenario for realizing human-computer interaction through gestures is shown, and the application scenario includes an electronic device and at least one user. The electronic device 10 includes at least one camera 11, and at least one user is within the field of view of the camera 11. The camera 11 captures an image or a video stream. When the user wants to control the electronic device 10 to enable a certain function, a certain gesture can be made.

[0057] The electronic device 10 may be a terminal device with computing and processing capabilities, such as a television equipped with a camera, an electronic photo frame, a game box, an unmanned aerial vehicle, a smart car or a smart camera.

[0058] Controlling the electronic device 10 may be controlling a functional component in the electronic device 10, and the functional component may be a hardware or software module. In one example, the electronic device 10 may include but is not limited to a smart camera, and controlling the smart camera may include but is not limited to controlling one or more functional components such as a lens focus control module, a scene mode switching module, or a flash control module provided in the smart camera.

[0059] Specifically, the electronic device 10 is taken as a smart camera for example. For example, there are three users in front of the smart camera, and the smart camera obtains a video stream, and each video frame in the video stream includes the three users. The user can control the smart camera to execute corresponding instructions through gestures to achieve human-computer interaction to meet the user's shooting needs. For example, through different gestures, the smart camera can be controlled to start or cancel focus, change scene mode, start or close gesture tracking, etc.

[0060] For example, when user A makes an "open palm" gesture, the smart camera recognizes the gesture through the video stream and executes an instruction reflecting "start tracking user A's head component". When user A makes a "like" gesture, the smart camera recognizes the gesture through the video stream and executes an instruction reflecting "start focusing". Therefore, users do not need to manually adjust the smart camera. They can use gestures to perform intelligent control in front of the smart camera to obtain satisfactory photos or videos. It is understandable that when there is only one user in front of the smart camera, the user can control the smart camera to take satisfactory photos or videos for himself through gesture control, without the need for an additional photographer. It is simple and convenient and suitable for use in multiple scenarios.

[0061] Recognizing gestures in video streams is a key factor in gesture control. If the recognized gestures are inaccurate, it is easy to cause the electronic device to perform actions that do not correspond to the gestures, that is, to cause false triggering. In the technical solutions known to the inventors of the present application, a gesture recognition model is usually used to perform full-image gesture detection and recognition to determine the trigger gesture used to trigger the electronic device to execute the corresponding command. However, since gestures are usually small in images or video frames, their postures are flexible, and they are not easy to track, the gesture recognition model is not reliable and cannot meet the accuracy and speed requirements of device interaction, and is prone to false triggering. In addition, full-image gesture detection and recognition requires a lot of computing power and has high requirements on the hardware performance of electronic devices.

[0062] In view of this, some embodiments of the present application provide a method for determining a trigger gesture. First, N video frames that are sequentially continuous in a video stream are obtained, and target component detection processing is performed on each video frame to obtain a bounding box of at least one target component. Then, gesture recognition is performed based on the bounding box of at least one target component and the video frame to determine the gesture corresponding to the video frame. The gestures corresponding to the N video frames constitute a gesture sequence. Finally, the trigger gesture is determined based on the gesture sequence. In this method, the target component (head, face, or head and shoulder) is detected first, and then gesture detection and recognition is performed in a local area near the target component (around the bounding box). Compared with the gesture detection and recognition of the whole picture, the computing power can be reduced. In addition, the feature granularity of the hand component is relatively large in the local area near the target component, so that the feature of the hand component is obvious, which can promote the improvement of the accuracy of gesture detection in each video frame. In addition, the trigger gesture is obtained by considering the gestures in the N video frames that are sequentially continuous. Compared with the trigger gesture determined by the gesture of a single frame, the accuracy of the trigger gesture can be improved and false triggering can be avoided to a certain extent. For example, if a user accidentally makes a gesture or a gesture is misclassified in a certain frame, and the gesture suddenly appears less times in the gesture sequence, it will not be recognized as a valid trigger gesture, so that the electronic device will not respond to the gesture, reducing the occurrence of false triggers.

[0063] The following is a detailed description of the method for determining the trigger gesture in some embodiments of the present application. Figure 2 The method S100 includes but is not limited to the following steps:

[0064] S10: Obtain N video frames that are consecutive in time sequence in the video stream, where N is an integer greater than or equal to 1.

[0065] The video stream can be acquired by the camera in the above application scenario embodiment. It can be understood that the video stream includes multiple temporally continuous video frames. The gesture detection and recognition process is performed using N temporally continuous video frames in the video stream. These N video frames include the user's gestures.

[0066] It is understandable that there may be one or more users in front of the camera, so the video frame may include the gesture of one user or multiple users. In order to implement gesture control, it is necessary to determine a valid trigger gesture based on the N video frames to control the operation of the functional components in the electronic device.

[0067] It is understood that the gesture may be a static gesture, such as Figure 3 Some static gestures are shown, such as a palm gesture, an OK gesture, a thumbs-up gesture, an index finger gesture, or a fist gesture, etc. Here, the gestures are merely exemplified, and it is understood that any form and action made by a user through a hand part can be called a gesture, and in this application, no limitation is imposed on gestures.

[0068] S20: Perform target component detection processing on the video frame to obtain a boundary box of at least one target component.

[0069] Since the hand part (i.e., the hand) is small relative to the human body and is usually small in the video frame, it is not easy to track and identify. In order to more quickly and effectively detect and identify the gestures made by the hand part, for each of the N video frames, target part detection processing is first performed to obtain the bounding box of at least one target part.

[0070] The target part is a part of a human body, such as the head, face, or head plus shoulders. These target parts have obvious features and are easy to detect and identify. By identifying the target part, the pixel range where the hand part may appear can be initially locked.

[0071] The target component detection processing in this step can be performed by a pre-trained target component detection neural network. By inputting N video frames into the trained neural network and performing target detection, the position information of the target component can be obtained. The position information of the target component is represented by a bounding box.

[0072] Taking the target part as the head as an example, N video frames are input into the trained head detection neural network. For each video frame, the trained head detection neural network will output the position of each head on the video frame (i.e., the bounding box). Figure 4 As shown, Figure 4 The detection result on a video frame is shown. In the video frame, the head of each person is within the bounding box. The bounding box is a rectangular box that can be represented by parameters (x, y, w, h), where (x, y) represents the center of the bounding box, w represents the width of the bounding box, and h represents the height of the bounding box. It can be understood that these parameters can be determined with reference to the image coordinate system of the video frame. The bounding box reflects the position of the target component.

[0073] It can be understood that if there is one user in the video frame, one bounding box of the target component will be detected, and if there are multiple users in the video frame, multiple bounding boxes of the target components will be detected. Figure 4 The example in the figure is that three users are included in the video frame.

[0074] S30: Perform gesture recognition according to the boundary box of at least one target component and the video frame to determine the gesture corresponding to the video frame, and the gestures corresponding to the N video frames constitute a gesture sequence.

[0075] Based on the human body structure, it can be known that the hand part is near the target part, for example, below or above the target part, etc. Therefore, searching and identifying the hand part near the target part in the video frame, and then identifying the gesture, can effectively reduce the detection and recognition of invalid areas. Here, the invalid area can be understood as the area where the hand part is unlikely to appear, such as the background area far away from the target part, or the body trunk area far away from the target part.

[0076] Specifically, in a video frame, gesture recognition is performed in an area around a boundary box of each target component, so that the gesture corresponding to the video frame can be determined.

[0077] It can be understood that the N video frames are all subjected to gesture recognition in the above manner, thereby obtaining N gestures, and these N gestures constitute a gesture sequence according to the time sequence of the video frames.

[0078] For example, if N is 20, that is, 20 consecutive video frames are used for gesture recognition, and a gesture sequence with a length of 20 is obtained. The gesture sequence can be [OK gesture, palm gesture, ..., palm gesture, fist gesture], that is, sorted in time sequence, the first one is the OK gesture, the middle 18 are palm gestures, and the last one is the fist gesture.

[0079] In some embodiments, the above step S30 specifically includes:

[0080] S31: Performing outward expansion processing on the bounding box of at least one target component respectively to obtain at least one outward expanded bounding box.

[0081] In this embodiment, the bounding box of each target component is expanded to obtain a corresponding expanded bounding box. It can be understood that the pixel area of ​​the video frame framed by the expanded bounding box is the pixel area where the hand component will appear with the highest probability.

[0082] It is understandable that the outward expansion bounding box is obtained by outward expansion based on the bounding box. The specific outward expansion processing method can be set by those skilled in the art. For example, the four boundaries of the bounding box are proportionally enlarged and expanded with the center (x, y) of the bounding box as the center. In some embodiments, the outward expansion is performed in all directions with the center (x, y) of the bounding box as the center. The outward expansion bounding box obtained after the outward expansion can be circular or trapezoidal, etc., and there is no specific limitation on the shape of the outward expansion bounding box.

[0083] In some embodiments, the above step S31 specifically includes:

[0084] S311: Obtain an expansion coefficient, where the expansion coefficient includes a top side expansion amount, a bottom side expansion amount, a left side expansion amount, and a right side expansion amount, wherein the top side expansion amount is smaller than the bottom side expansion amount.

[0085] S312: Perform outward expansion adjustment on four sides of at least one bounding box according to the outward expansion coefficients to obtain at least one outward expanded bounding box.

[0086] It is understandable that the top expansion amount, bottom expansion amount, left expansion amount and right expansion amount in the expansion system can be pixel distances determined based on the video frame resolution and the bounding box. For example, in the expansion coefficient, the top expansion amount top=0.5, the bottom expansion amount down=2, the left expansion amount left=2, and the right expansion amount right=2.

[0087] Thus, for each bounding box in the video frame, the four sides of the bounding box are expanded according to the expansion coefficients to obtain an expanded bounding box. Specifically, Figure 5 As shown, the top edge of the bounding box is expanded according to the top edge expansion amount top, the bottom edge of the bounding box is expanded according to the bottom edge expansion amount down, the left side of the bounding box is expanded according to the left expansion amount left, and the right side of the bounding box is expanded according to the right expansion amount right, to obtain the corresponding expanded bounding box.

[0088] It is worth noting that the top expansion is smaller than the bottom expansion. Because the user's hand will not exceed the head too much, setting the top expansion smaller than the bottom expansion can effectively reduce the range of hand detection and prevent the hand from being absent or not completely in the expansion boundary box, that is, it can balance the hand detection speed and accuracy.

[0089] S32: Perform hand part detection processing on an area in the video frame that is within at least one outward-expanded boundary box to obtain a hand part detection result.

[0090] After obtaining each outward expansion bounding box, the hand part detection process is performed on the area in each outward expansion bounding box in the video frame to obtain a hand part detection result. Here, the hand part detection process is to detect which pixels in each outward expansion bounding box on the video frame are hand parts, that is, to detect and locate each hand part, and the hand part detection result includes the pixel position of each hand part in the video frame.

[0091] It can be understood that the hand part detection processing in this step can be performed by a pre-trained hand part detection neural network. The local images obtained by intercepting the video frames of each outward expansion boundary box are input into the trained hand part detection neural network for target detection, and the pixel position information of the hand part can be obtained.

[0092] S33: Determine the target hand part according to the hand part detection result.

[0093] It is understandable that the above hand part detection result may include one hand part, for example, there is only one user performing a single-hand gesture operation in front of the smart camera. In some scenarios, the hand part detection result may also include multiple hand parts, for example, there are multiple users in front of the smart camera, one of whom is performing a gesture operation (that is, the hand part of the user can be understood as a valid hand part), and the hand parts of other users may be detected (that is, the hand parts of other users can be understood as invalid hand parts).

[0094] Therefore, it is necessary to determine a valid hand part from the hand part detection results as a target hand part. The gesture of the target hand part is the gesture corresponding to the video frame.

[0095] In some embodiments, the target hand part can be determined from the features of each hand part in the hand part detection results. Considering that users who deliberately make gestures to control electronic devices generally place the hand part in front of the body, the effective hand part is closer to the camera, and the camera can collect more clearly, so that the effective hand part is closer to the camera and the pixel area accounts for a larger proportion. Therefore, in some embodiments, the hand part with the largest pixel area proportion can be used as the target hand part. Alternatively, in some embodiments, based on the fact that the effective hand part is clearer, the hand part with the highest confidence can be used as the target hand part. It can be understood that the confidence of the hand part reflects the degree to which the hand part is a real "hand".

[0096] In some embodiments, the above step S33 specifically includes:

[0097] S331: If the hand part detection result includes a hand part, determine that the hand part is a target hand part.

[0098] It is understandable that in a scenario where there is only one user performing a single-hand gesture operation in front of the smart camera, only one hand part can be detected, and thus, the hand part is determined to be the target hand part.

[0099] S332: If the hand part detection result includes multiple hand parts, the score of each hand part is calculated according to the confidence of each hand part, the distance between the target part corresponding to the hand part and the pixel area of ​​the hand part, and the hand part with the highest score is determined as the target hand part.

[0100] In the case where the hand part detection result includes multiple hand parts, for each hand part, the hand part's score is calculated by considering the hand part's confidence, the distance between the hand part and the target part corresponding to the hand part, and the pixel area of ​​the hand part. That is, the multiple hand parts in the video frame are scored based on the above three factors: confidence, distance, and pixel area. The hand part with the highest score is taken as the target hand part.

[0101] It is understandable that users who deliberately make gestures to control electronic devices generally place the hand part in front of the body, at about the same height as the shoulder, so that the effective hand part is closer to the camera and the camera can collect data more clearly. As a result, the effective hand part is closer to the camera, the pixel area accounts for a larger proportion, the confidence is higher, and the distance between the target part corresponding to the hand part is smaller.

[0102] like Figure 6 As shown, Figure 6 A situation is shown in which user A, user B and user C take photos, record videos or live broadcast in front of a smart camera. The video frame includes user A, user B and user C. User A extends his hand parallelly to the smart camera to make an "OK gesture" in the hope of adjusting the focal length. User B raises his hand as far as possible above his head to make a "V-shaped gesture" in order to pose. User C crosses his arms to pose, exposing only part of his hand. Therefore, when detecting hand parts, the hand parts of user A and user B are clearly exposed and have a high confidence level, while the hand part of user C is not completely exposed and has a low confidence level. The hand part of user A is close to the smart camera and has a large pixel area, while the hand parts of user B and user C are relatively far away from the smart camera and have a small pixel area. The hand parts of user A and user C are at a small distance from their respective head parts, while the hand part of user B is at a large distance from his head part. If the score is calculated from a single factor, it cannot fully reflect the real situation, and the real target hand part cannot be accurately determined.

[0103] In this embodiment, the score is calculated based on three factors: the confidence of the hand part, the distance between it and the corresponding target part, and the pixel area. The score is positively correlated with the confidence and the pixel area, and negatively correlated with the distance between the hand part and the corresponding target part. This enables the score to reflect the actual situation and makes the target hand part more accurate.

[0104] In some embodiments, the above step S332 specifically includes:

[0105] The score of each hand component is calculated using the following formula;

[0106]

[0107] Among them, Score i is the score of the i-th hand component, i is the label of the hand component, Conf i is the confidence of the i-th hand part, Dist i is the distance between the i-th hand component and the target component, Area i is the pixel area of ​​the ith hand part. Equivalent to the distance ratio, Equivalent to the pixel area ratio. Alpa is the weight of confidence, beta is the weight of distance ratio, and lambda is the weight of pixel area ratio. By setting the weight, you can set the importance of each item to the overall score.

[0108] In this embodiment, the score of a hand component is the sum of its confidence and pixel area ratio, minus the distance ratio, wherein the pixel area ratio reduces the weight of the pixel area, and the distance ratio reduces the weight of the distance, which can eliminate the special circumstances of the human body and make the score more accurate, which is beneficial to accurately locate the target hand component.

[0109] S34: Perform gesture classification processing on the target hand part to obtain the gesture corresponding to the video frame.

[0110] After the target hand part is acquired, the gesture presented by it is subjected to gesture classification processing, thereby obtaining the gesture corresponding to the video frame.

[0111] It can be understood that the gesture classification processing in this step can be performed by a pre-trained gesture classification neural network. The bounding box of the target hand part is used to capture the image area of ​​the target hand part, and the image area of ​​the target hand part is input into the above-mentioned trained gesture classification neural network for gesture classification. Then, the gesture category of the target hand part can be obtained.

[0112] It is understandable that the gesture types that the gesture classification neural network can detect can be set according to actual needs. For example, the images in the training set for training the gesture classification neural network can include six gesture categories, namely: palm gesture, OK gesture, L-shaped gesture, V-shaped gesture, fist gesture, and other gestures. Thus, the trained gesture classification neural network can identify the gesture categories that the technician needs to identify.

[0113] In this embodiment, when determining the gesture corresponding to the video frame, the bounding box of each target component in the video frame is first expanded to obtain each expanded bounding box, which is the range of the hand component detection processing. The area in the video frame located in each expanded bounding box is subjected to hand component detection processing to obtain the hand component detection result. On the one hand, it can reduce the detection range, which is beneficial to improve the detection speed and reduce the computing power. On the other hand, the hand component has a relatively large feature granularity in the expanded bounding box, which is beneficial to promote the accuracy of the hand component detection result. In addition, according to the hand component detection result, the target hand component is determined, and the target hand component is subjected to gesture classification processing to obtain the gesture corresponding to the video frame. On the one hand, the target hand component is made to be a truly effective hand component, and the gesture corresponding to the video frame is more accurate, thereby making the gesture sequence more accurate. On the other hand, only the target hand component needs to be subjected to gesture classification processing, which can effectively reduce the computing power compared to performing gesture classification processing on each hand component and then determining the gesture corresponding to the video frame.

[0114] S40: Determine a trigger gesture according to the gesture sequence.

[0115] It is understandable that the gesture sequence includes gestures corresponding to N video frames that are continuous in time sequence. After the gesture sequence is acquired, a trigger gesture for triggering the electronic device to execute a corresponding instruction is determined according to the gesture sequence.

[0116] The trigger gesture is obtained by considering the gestures in N video frames that are continuous in time sequence. Compared with determining the trigger gesture by using a single frame gesture, it can improve the accuracy of the trigger gesture and avoid false triggers to a certain extent. For example, if the user accidentally makes a gesture and the gesture suddenly appears less times in the gesture sequence, it will not be identified as a valid trigger gesture, so that the electronic device will not respond to the gesture, reducing the occurrence of false triggers.

[0117] In some embodiments, the above step S40 specifically includes:

[0118] S41: Counting the number of gestures corresponding to each gesture category in the gesture sequence, if a first gesture category with the largest number of gestures meets a preset condition, taking the gesture belonging to the first gesture category as a trigger gesture.

[0119] For example, N is 20, and the statistical results are 18 "palm gestures", 1 "OK gesture", 0 "L-shaped gestures", 0 "V-shaped gestures", and 0 "fist gestures". Among them, the number of "palm gestures" is the largest, which is called the first gesture category. If the first gesture category meets the preset conditions, the gesture belonging to the first gesture category (ie, "palm gesture") will be used as the trigger gesture.

[0120] In some embodiments, the preset condition includes that the percentage of the number of gestures belonging to the first gesture category in the total number of gestures in the gesture sequence is greater than or equal to a percentage threshold. For example, if the percentage of the number of the first gesture category ("palm gesture") in the total number of gestures 20 is 0.9, which is greater than the percentage threshold (e.g., 0.8), then the "palm gesture" is used as the trigger gesture.

[0121] In some embodiments, the preset condition includes that the number of gestures belonging to the first gesture category is greater than or equal to a quantity threshold. For example, if the number of the first gesture category (“palm gesture”) is 18 and is greater than a quantity threshold (e.g., 16), then “palm gesture” is used as a trigger gesture.

[0122] In this embodiment, the target component detection and hand component detection results based on a single frame are prone to missed detection or false detection, and the training samples often cannot cover all user scenarios, resulting in the gesture classification results of a single frame may occasionally be misidentified or misclassified due to inaccurate target component detection results and hand component detection results. In order to prevent false triggering, the above-mentioned false triggering problem can be effectively improved by setting a threshold in combination with statistical gesture sequences.

[0123] In some embodiments, see Figure 7 , the method further comprises:

[0124] S50: Acquire a first target component corresponding to the trigger gesture.

[0125] S60: Tracking and detecting a gesture corresponding to the first target component in the video stream to determine a new triggering gesture.

[0126] After the trigger gesture is determined, the first target component corresponding to the trigger gesture is obtained. For example, if the trigger gesture is made by user A, the target component of user A is the first target component.

[0127] In the subsequent video stream, the gesture corresponding to the first target component is tracked and detected. For example, the gesture of user A is tracked and detected. Specifically, the hand component is detected in the outer expansion boundary box corresponding to user A in the video frame of the video stream, and the gesture of the hand component is recognized. Therefore, the user in front of the electronic device who starts tracking first has priority control. On the one hand, it not only makes gesture control more stable and reduces false triggers caused by multiple gestures, but also effectively reduces computing power by only detecting the gesture corresponding to the first target component in the subsequent video stream.

[0128] It is understandable that the gesture detection of the first target component can also be performed in the manner described in the above embodiment. For example, after the tracking is turned on, the hand component of the first target component in N consecutive video frames is detected, that is, the hand component in the outer expansion boundary box of the first target component is tracked in response, and the hand component is gesture-classified to obtain a gesture sequence after the tracking is turned on. It is understandable that if two hand components of the user appear in the outer expansion boundary box of the first target component, the hand component score can be calculated using the formula in step S332 of the above embodiment, and the hand component with the highest score is used as the hand component for gesture classification and recognition.

[0129] Finally, a new trigger gesture is determined based on the gesture sequence after tracking is enabled. For example, the gesture categories in the gesture sequence after tracking is enabled are counted, and if the proportion of the most numerous gestures is greater than or equal to a percentage threshold or the number of the most numerous gestures is greater than or equal to a quantity threshold, the most numerous gesture is used as a new trigger gesture.

[0130] In this embodiment, after determining the trigger gesture, the first target component corresponding to the trigger gesture is obtained, and in the subsequent video stream, the gesture corresponding to the first target component is tracked and detected, so that the user in front of the electronic device who starts tracking first will have priority control. On the one hand, this not only makes gesture control more stable and reduces false triggers caused by multiple gestures, but also, on the other hand, only detecting the gesture corresponding to the first target component in the subsequent video stream can effectively reduce computing power.

[0131] In summary, the method for determining the trigger gesture provided by the embodiment of the present application is, first, to obtain N video frames that are sequentially continuous in the video stream, perform target component detection processing on each video frame, obtain the bounding box of at least one target component, then, perform gesture recognition based on the bounding box of at least one target component and the video frame, determine the gesture corresponding to the video frame, and the gestures corresponding to the N video frames constitute a gesture sequence, and finally, determine the trigger gesture based on the gesture sequence. In this method, the target component (head, face, or head and shoulder) is detected first, and then gesture detection and recognition is performed in the local area near the target component (around the bounding box). Compared with the gesture detection and recognition of the whole picture, the computing power can be reduced. In addition, the feature granularity of the hand component is relatively large in the local area near the target component, so that the feature of the hand component is obvious, which can promote the improvement of the accuracy of gesture detection in each video frame. In addition, the trigger gesture is obtained by considering the gestures in the N video frames that are sequentially continuous. Compared with the trigger gesture determined by the gesture of a single frame, the accuracy of the trigger gesture can be improved, and false triggering can be avoided to a certain extent. For example, if a user accidentally makes a gesture and the gesture suddenly appears less times in the gesture sequence, it will not be recognized as a valid trigger gesture, so the gesture will not be responded to, reducing the occurrence of false triggers.

[0132] The above describes the method for determining a trigger gesture in an embodiment of the present application. In order to better implement the method of the present application, the device provided in an embodiment of the present application is described below.

[0133] An interactive method provided in an embodiment of the present application includes:

[0134] (1) The trigger gesture is determined using the method for determining the trigger gesture in any of the above embodiments.

[0135] (2) Control the target device to execute the operation instruction corresponding to the trigger gesture.

[0136] When a user wants to control a device to enable a certain function, a certain gesture can be made. The device recognizes the gesture using the method for determining the trigger gesture in any of the above embodiments to detect the gesture and determine the trigger gesture. The device can be called a target device, and controlling the target device can be controlling a functional component in the device, which can be a hardware or software module. In one example, the target device can include but is not limited to a smart camera, and controlling the smart camera can include but is not limited to controlling one or more functional components such as a lens focus control module, a scene mode switching module, or a flash control module set in the smart camera.

[0137] It can be understood that the trigger gesture corresponds to the operation instruction one by one. For example, when user A makes an "open palm" gesture, the smart camera recognizes the trigger gesture through the video stream and executes an operation instruction reflecting "start tracking of user A's head component". When user A makes a "like" gesture, the smart camera recognizes the trigger gesture through the video stream and executes an operation instruction reflecting "start focusing".

[0138] Therefore, the target device is controlled to execute the operation instruction corresponding to the trigger gesture, and the user does not need to manually adjust the target device, and can perform intelligent interactive control through gestures in front of the target device.

[0139] See also Figure 8 , Figure 8 A device for determining a trigger gesture provided in an embodiment of the present application, the device 200 includes:

[0140] The acquisition module 210 is used to acquire N video frames that are consecutive in time sequence in the video stream, where N is an integer greater than or equal to 1.

[0141] The detection module 220 is used to perform target component detection processing on the video frame to obtain a boundary box of at least one target component.

[0142] The recognition module 230 is used to perform gesture recognition based on the boundary box of the at least one target component and the video frame, and determine the gesture corresponding to the video frame, and the gestures corresponding to the N video frames constitute a gesture sequence.

[0143] The determination module 240 is used to determine a trigger gesture according to the gesture sequence.

[0144] The above-mentioned device 200 obtains N video frames that are sequentially continuous in the video stream, performs target component detection processing on each video frame, obtains a bounding box of at least one target component, and then performs gesture recognition based on the bounding box of at least one target component and the video frame to determine the gesture corresponding to the video frame. The gestures corresponding to the N video frames constitute a gesture sequence. Finally, the trigger gesture is determined based on the gesture sequence. In this method, the target component (head, face, or head and shoulder) is detected first, and then gesture detection and recognition are performed in a local area near the target component (around the bounding box). Compared with the gesture detection and recognition of the whole picture, the computing power can be reduced. In addition, the feature granularity of the hand component is relatively large in the local area near the target component, so that the feature of the hand component is obvious, which can promote the improvement of the accuracy of gesture detection in each video frame. In addition, the trigger gesture is obtained by considering the gestures in the N video frames that are sequentially continuous. Compared with the trigger gesture determined by the gesture of a single frame, the accuracy of the trigger gesture can be improved and false triggering can be avoided to a certain extent. For example, if a user accidentally makes a gesture and the gesture suddenly appears less times in the gesture sequence, it will not be recognized as a valid trigger gesture, so the gesture will not be responded to, reducing the occurrence of false triggers.

[0145] See also Fig. 9 , is a hardware structure diagram of an electronic device 10 provided in an embodiment of the present application, specifically, as Fig. 9 As shown, the electronic device 10 includes at least one processor 12 and a memory 13 ( Fig. 9 (a bus connection and a processor are used as an example).

[0146] The processor 12 is used to provide computing and control capabilities to control the electronic device 10 to perform corresponding tasks and control the electronic device 10 to perform any one of the methods for determining a trigger gesture provided in the above embodiments.

[0147] It is understandable that the processor 12 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0148] The memory 13, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the method for determining a trigger gesture in an embodiment of the present invention. The processor 12 can implement any of the methods for determining a trigger gesture provided in the above embodiments by running the non-transitory software programs, instructions and modules stored in the memory 13. Specifically, the memory 13 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 13 may also include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0149] It can be understood that in some embodiments, the electronic device may be a smart device such as a smart camera, a mobile terminal or a drone.

[0150] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the method for determining a trigger gesture as described above.

[0151] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0152] Through the description of the above implementation methods, ordinary technicians in this field can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Ordinary technicians in this field can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Under the concept of the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present application as described above, which are not provided in detail for the sake of simplicity. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for determining a trigger gesture, characterized in that: include: Obtain N video frames that are consecutive in time sequence in the video stream, where N is an integer greater than or equal to 1; Performing target component detection processing on the video frame to obtain a bounding box of at least one target component; Performing hand part detection processing on the surrounding area of ​​the bounding box in the video frame to obtain a hand part detection result; If the hand part detection result includes a hand part, determining the hand part as a target hand part; If the hand component detection result includes multiple hand components, the score of each hand component is calculated using the following formula: Among them, Score i is the score of the i-th hand component, i is the label of the hand component, Conf i is the confidence of the ith hand component, which reflects the degree to which the hand component is a real hand. i is the distance between the i-th hand component and the target component, Area i is the pixel area of ​​the ith hand part, alpha, beta and lambda are weights; Determine the hand component with the highest score as the target hand component; Performing gesture classification processing on the target hand part to obtain a gesture corresponding to the video frame; The gestures corresponding to the N video frames constitute a gesture sequence; A trigger gesture is determined according to the gesture sequence.

2. The method according to claim 1, characterized in that The performing hand part detection processing on the surrounding area of ​​the boundary box in the video frame to obtain the hand part detection result includes: Performing outward expansion processing on the bounding box of the at least one target component respectively to obtain at least one outward expanded bounding box; Perform hand part detection processing on an area in the video frame that is within the at least one outward-expanded boundary box to obtain a hand part detection result.

3. The method according to claim 1, characterized in that The step of determining a trigger gesture according to the gesture sequence includes: The number of gestures corresponding to each gesture category in the gesture sequence is counted, and if a first gesture category with the largest number of gestures meets a preset condition, the gesture belonging to the first gesture category is used as the trigger gesture.

4. The method according to claim 3, characterized in that The preset condition includes that the percentage of the number of gestures belonging to the first gesture category in the total number of gestures in the gesture sequence is greater than or equal to a percentage threshold, or the number of gestures belonging to the first gesture category is greater than or equal to a quantity threshold.

5. The method according to claim 1, characterized in that Also includes: Acquire a first target component corresponding to the trigger gesture; The gesture corresponding to the first target component is tracked and detected in the video stream to determine a new trigger gesture.

6. An interactive method, characterized in that: include: Determine the trigger gesture using the method for determining the trigger gesture as described in any one of claims 1 to 5; The target device is controlled to execute an operation instruction corresponding to the trigger gesture.

7. A device for determining a trigger gesture, characterized in that: include: An acquisition module, used for acquiring N video frames that are sequentially continuous in the video stream, where N is an integer greater than or equal to 1; A detection module, configured to perform target component detection processing on the video frame to obtain a boundary box of at least one target component; The recognition module is used to perform hand part detection processing on the surrounding area of ​​the bounding box in the video frame to obtain a hand part detection result; if the hand part detection result includes one hand part, the hand part is determined to be a target hand part; if the hand part detection result includes multiple hand parts, the score of each hand part is calculated using the following formula: Among them, Score i is the score of the i-th hand component, i is the label of the hand component, Conf i is the confidence of the ith hand component, which reflects the degree to which the hand component is a real hand. i is the distance between the i-th hand component and the target component, Area i is the pixel area of ​​the ith hand component, alpha, beta and lambda are weights; determine the hand component with the highest score as the target hand component; perform gesture classification processing on the target hand component to obtain the gesture corresponding to the video frame; the gestures corresponding to the N video frames constitute a gesture sequence; The determination module is used to determine a trigger gesture according to the gesture sequence.

8. An electronic device, characterized in that: include: at least one processor, and a memory communicatively coupled to the at least one processor, wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method according to any one of claims 1 to 6.

Images