Method, apparatus and electronic device for determining an interaction gesture

Abstract

Embodiments of the present application relate to the technical field of human-computer interaction, and disclose a method and device for determining an interactive gesture and an electronic device. A hand part recognition process is performed on a current video frame to obtain at least one hand part detection frame in the current video frame and a gesture recognition result corresponding to the hand part detection frame. The hand part detection frames corresponding to the current video frame and video frames in a preset historical video frame set are matched to obtain at least one hand part detection frame sequence corresponding to the current video frame. According to the at least one hand part detection frame sequence and the gesture recognition result corresponding to the hand part detection frame in the hand part detection frame sequence, an interactive gesture corresponding to the current video frame is determined. In the above manner, the hand part detection frames belonging to the same position are connected in series, and the gesture statistical result of the same position within a period of time is used to replace the instantaneous gesture. In the case of realizing multi-person interactive control, the accuracy and robustness of the interactive gesture can be improved, and the false triggering can be effectively reduced.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of human-computer interaction, and in particular to a method and device for determining an interactive gesture and an electronic device. BACKGROUND

[0002] With the continuous development and popularization of product intelligence, electrification and interconnection, there are many more and more intelligent human-computer interaction methods to meet people's demand for personalization and fashion. For example, the touch screen of a smart phone is a human-computer interaction system realized by touch. There are also some products controlled by voice interaction, such as smart speakers, and the user only needs to input relevant instructions by voice, and the product can execute relevant operations according to the voice input instructions. In some products, such as smart cameras, game consoles or drones, human-computer interaction can also be realized by gestures, and different gestures correspond to different instructions, and when the product obtains a gesture, it executes the instruction corresponding to the gesture.

[0003] However, the posture of a gesture is variable, and the detection result of a single frame of gesture is extremely dependent on the accuracy of the detection algorithm and is easily affected by external interference, and the accuracy is low and an error response is easily triggered. In addition, from the process of extending the hand to making a specified gesture, the gesture is from an irregular shape to an ideal state, and the irregular shape in this process is easy to be misrecognized, which also leads to low accuracy of gesture recognition and easy triggering of an error response. SUMMARY

[0004] The technical problem solved by the embodiments of the present application is to provide a method and device for determining an interactive gesture and an electronic device, which can improve the accuracy and robustness of interactive gesture recognition and effectively reduce false triggering.

[0005] To solve the above technical problem, in a first aspect, the embodiments of the present application provide a method for determining an interactive gesture, comprising:

[0006] obtaining a current video frame;

[0007] performing hand part recognition processing on the current video frame to obtain at least one hand part detection box in the current video frame and a gesture recognition result corresponding to the hand part detection box;

[0008] matching the hand part detection boxes corresponding to the current video frame and the video frames in a preset historical video frame set to obtain at least one hand part detection box sequence corresponding to the current video frame;

[0009] determining an interactive gesture corresponding to the current video frame according to the at least one hand part detection box sequence and the gesture recognition result corresponding to the hand part detection box in the hand part detection box sequence.

[0010] In some embodiments, the aforementioned hand component recognition processing of the current video frame to obtain at least one hand component detection box in the current video frame and a gesture recognition result corresponding to the hand component detection box includes:

[0011] Perform target component detection processing on the current video frame to obtain the bounding box of at least one target component;

[0012] The bounding box of at least one target component is expanded to obtain at least one expanded bounding box;

[0013] Perform hand part detection processing on the region located within at least one extended bounding box in the current video frame to obtain at least one hand part detection box;

[0014] Perform gesture recognition on hand parts located within at least one hand part detection box in the current video frame to obtain gesture recognition results corresponding to the hand part detection box.

[0015] In some embodiments, the current video frame is the k-th video frame;

[0016] The aforementioned matching of the hand component detection boxes corresponding to the current video frame with those in a preset set of historical video frames yields at least one sequence of hand component detection boxes corresponding to the current video frame, including:

[0017] The k-th video frame and the (k-1)-th video frame are matched with hand component detection boxes according to a preset matching rule. If the first hand component detection box of the k-th video frame and the (k-1)-th video frame are successfully matched, the first hand component detection boxes of the k-th video frame and the (k-1)-th video frame are concatenated into the first hand component detection box sequence. The (k-1)-th video frame and the (k-2)-th video frame are matched with hand component detection boxes according to a preset matching rule. Here, the first hand component detection box is any hand component detection box in the video frame, and the (k-1)-th video frame and the (k-2)-th video frame are video frames in a preset historical video frame set.

[0018] If the first hand component detection box of the k-th video frame fails to match the first hand component detection box of the (k-1)-th video frame, then the k-th video frame and the (k-2)-th video frame will be matched according to the preset matching rules. This process will continue in reverse order until all video frames in the preset historical video frame set have been matched.

[0019] In some embodiments, the aforementioned matching of the hand component detection boxes corresponding to the current video frame with those corresponding to video frames in a preset set of historical video frames to obtain at least one sequence of hand component detection boxes corresponding to the current video frame includes:

[0020] The current video frame is matched with the video frames in the preset historical video frame set according to the preset matching rules to perform hand component detection box matching. If the first hand component detection box of any target video frame in the preset video frame set is successfully matched with the first hand component detection box of the current video frame, the first hand component detection boxes of the target video frame and the current video frame are concatenated into the first hand component detection box sequence. The first hand component detection box is any hand component detection box in the video frame.

[0021] In some embodiments, the aforementioned preset matching rules include:

[0022] If the Euclidean distance between the center coordinates of the first hand component detection boxes of the two video frames being matched is less than or equal to the first threshold, then the first hand component detection boxes of the two video frames are successfully matched.

[0023] In some embodiments, the aforementioned preset matching rules include:

[0024] If the intersection-union ratio (CUI) of the first hand component detection boxes of the two video frames being matched is greater than or equal to the second threshold, then the first hand component detection boxes of the two video frames are successfully matched.

[0025] In some embodiments, the aforementioned preset matching rules include:

[0026] If the absolute distance between the first hand part detection boxes of two video frames that are being matched for hand part detection is less than or equal to the third threshold, then the first hand part detection boxes of the two video frames are successfully matched.

[0027] In some embodiments, determining the interaction gesture corresponding to the current video frame based on at least one hand component detection box sequence and the gesture recognition result corresponding to the hand component detection box in the hand component detection box sequence includes:

[0028] Obtain the valid gestures corresponding to the detection box sequence of each hand component;

[0029] If there is a trigger gesture among the valid gestures, then the trigger gesture is determined to be an interactive gesture;

[0030] If there are multiple trigger gestures among the valid gestures, the trigger gesture that is processed first is determined to be the interactive gesture;

[0031] If no triggering gesture is found among the valid gestures, the process continues to the next video frame after the current video frame.

[0032] In some embodiments, the aforementioned acquisition of valid gestures corresponding to each hand component detection box sequence includes:

[0033] Obtain the first number of first-hand component detection boxes in the first-hand component detection box sequence whose gesture recognition results are the same as the gesture recognition results of the first-hand component detection box in the current video frame. If the proportion of the first number to the number of video frames in the preset historical video frame set is greater than or equal to the first proportion threshold, then determine the gesture recognition result of the first-hand component detection box in the current video frame as the valid gesture corresponding to the first-hand component detection box sequence. Here, the first-hand component detection box sequence is any one of the hand component detection box sequences.

[0034] In some embodiments, the aforementioned hand component detection box sequence includes the hand component detection boxes of video frames that successfully match the hand component detection boxes of the current video frame within a preset time period before the current video frame.

[0035] The aforementioned acquisition of valid gestures corresponding to the detection box sequence of each hand component includes:

[0036] Obtain the second number of first-hand component detection boxes in the first-hand component detection box sequence whose gesture recognition results are the same as those of the first-hand component detection box in the current video frame. If the ratio of the second number to the number of hand component detection boxes in the first-hand component detection box sequence is greater than or equal to the second ratio threshold, then determine the gesture recognition result of the first-hand component detection box in the current video frame as the valid gesture corresponding to the first-hand component detection box sequence. Here, the first-hand component detection box sequence is any one of the hand component detection box sequences.

[0037] To address the aforementioned technical problems, in a second aspect, this application provides an interaction method, including:

[0038] The interactive gestures are determined using the method described in the first aspect.

[0039] Control the target device to execute the operation command corresponding to the interactive gesture.

[0040] To address the aforementioned technical problems, in a third aspect, embodiments of this application provide a device for determining interactive gestures, comprising:

[0041] The acquisition module is used to acquire the current video frame;

[0042] The recognition module is used to perform hand component recognition processing on the current video frame to obtain at least one hand component detection box in the current video frame and the gesture recognition result corresponding to the hand component detection box.

[0043] The matching module is used to match the hand part detection boxes corresponding to the current video frame with the video frames in the preset historical video frame set, so as to obtain at least one hand part detection box sequence corresponding to the current video frame.

[0044] The determination module is used to determine the interactive gesture corresponding to the current video frame based on at least one hand component detection box sequence and the gesture recognition result corresponding to the hand component detection box in the hand component detection box sequence.

[0045] To address the aforementioned technical problems, in a fourth aspect, this application provides an electronic device, comprising:

[0046] At least one processor, and

[0047] A memory communicatively connected to at least one processor, wherein,

[0048] The memory stores instructions that can be executed by at least one processor, such that the instructions are executed by at least one processor to enable the at least one processor to perform the method of the first aspect.

[0049] To address the aforementioned technical problems, in a fifth aspect, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the method of the first aspect.

[0050] The beneficial effects of this application's embodiments are as follows: Unlike existing technologies, the method for determining interactive gestures provided in this application involves performing hand component recognition processing on the current video frame to obtain at least one hand component detection box and a gesture recognition result corresponding to the hand component detection box in the current video frame. Then, the current video frame is matched with the hand component detection boxes corresponding to video frames in a preset historical video frame set to obtain a sequence of at least one hand component detection boxes corresponding to the current video frame. Finally, based on this sequence of at least one hand component detection boxes and the gesture recognition result corresponding to the hand component detection boxes in the sequence, the interactive gesture corresponding to the current video frame is determined. In this method, each hand component detection box sequence obtained through the above hand component detection box matching reflects the position and gesture recognition result of the same hand component (same position) in the current video frame and the preset historical video frame set, respectively. This is equivalent to chaining hand component detection boxes belonging to the same position together, ensuring that the hand component at that position in the present and past belong to the same person. Based on this, the interactive gestures determined by the detection box sequence of each hand component are obtained by considering the gesture recognition results at the same position in the current video frame and historical video frames. Essentially, the statistical results of gesture recognition at the same position over a period of time are used to replace the instantaneous gesture recognition result. This avoids the electronic device triggering an instantaneous gesture and generating an incorrect gesture response when the user changes their gesture or is subject to external interference. Therefore, in the case of multi-person interactive control, the accuracy and robustness of interactive gesture recognition can be improved, and false triggering can be effectively reduced. Attached Figure Description

[0051] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0052] Figure 1 This is a schematic diagram illustrating application scenarios of human-computer interaction achieved through gestures in some embodiments of this application;

[0053] Figure 2 This is a flowchart illustrating the method for determining interactive gestures in some embodiments of this application;

[0054] Figure 3 This is a schematic diagram of gestures in some embodiments of this application;

[0055] Figure 4 This is a schematic diagram of the bounding box on the video frame in some embodiments of this application;

[0056] Figure 5 This is a schematic diagram of the structure of a device for determining interactive gestures in some embodiments of this application;

[0057] Figure 6 This is a schematic diagram of the structure of an electronic device in some embodiments of this application. Detailed Implementation

[0058] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0060] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. In addition, the terms "first," "second," and "third" used herein do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0061] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0062] Furthermore, the technical features involved in the various embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0063] Please see Figure 1 , Figure 1 This diagram illustrates an application scenario for human-computer interaction via gestures. The scenario includes an electronic device and at least one user. The electronic device 10 includes at least one camera 11, and at least one user is within the field of view of the camera 11. The camera 11 captures images or video streams. When the user wants to control the electronic device 10 to activate a specific function, they can make a gesture.

[0064] Among them, electronic device 10 can be a terminal device with computing processing capabilities, such as a video conferencing machine with a camera, a gimbal camera, a television, an electronic photo frame, a game box, an unmanned aerial vehicle, a smart car, or a smart camera.

[0065] The control electronic device 10 can be a device that controls functional components within the electronic device 10, which can be hardware or software modules. In one example, the electronic device 10 can be, but is not limited to, a smart camera. Controlling the smart camera can include, but is not limited to, controlling one or more functional components within the smart camera, such as a lens focus control module, a scene mode switching module, or a flash control module.

[0066] Specifically, taking electronic device 10 as an example of a smart camera, for instance, there are three users in front of the smart camera. The smart camera acquires a video stream, and each video frame in the video stream includes these three users. Users can control the smart camera to execute corresponding commands through gestures, realizing human-computer interaction to meet the user's shooting needs. For example, different gestures can be used to control the smart camera to turn on or off focusing, or different gestures can be used to change the scene mode, turn on or off target tracking, etc.

[0067] For example, when user A makes an "open palm" gesture, the smart camera recognizes the gesture through the video stream and executes the command to "start tracking user A's head." When user B makes a "thumbs up" gesture, the smart camera recognizes the gesture through the video stream and executes the command to "start focusing." Thus, users don't need to manually adjust the smart camera; they can control it intelligently with gestures to get satisfactory photos or videos. In short, when only one user is in front of the smart camera, they can control it to take satisfactory photos or videos without needing an additional photographer. It's simple, convenient, and suitable for various scenarios.

[0068] Recognizing gestures in a video stream is crucial for gesture control. Inaccurate gesture recognition can easily cause electronic devices to perform actions that do not correspond to the gestures, resulting in false triggers. In the technical solutions known to the inventors of this application, gesture detection and recognition are typically performed using a single video frame to determine the interactive gesture used to trigger the electronic device to execute the corresponding command. In other words, the gesture recognition result of a single frame image is used as the final result.

[0069] However, gestures are highly variable, and the accuracy of gesture detection in a single frame is extremely dependent on the accuracy of the detection and is easily affected by external interference, resulting in low accuracy and a tendency to trigger erroneous responses. Furthermore, the process from extending the hand to making the specified gesture involves a transition from an irregular shape to an ideal state. This irregular shape is prone to misidentification, which also leads to low accuracy in gesture recognition and a tendency to trigger erroneous responses.

[0070] In view of this, some embodiments of this application provide a method for determining interactive gestures. The method involves performing hand component recognition processing on the current video frame to obtain at least one hand component detection box and a gesture recognition result corresponding to the hand component detection box in the current video frame. Then, the method matches the hand component detection boxes corresponding to video frames in a preset set of historical video frames to obtain a sequence of at least one hand component detection boxes corresponding to the current video frame. Finally, based on the sequence of at least one hand component detection boxes and the gesture recognition result corresponding to the hand component detection boxes in the sequence, the interactive gesture corresponding to the current video frame is determined. In this method, each hand component detection box sequence obtained through the above hand component detection box matching reflects the position and gesture recognition result of the same hand component (same position) in the current video frame and the preset set of historical video frames, respectively. This is equivalent to concatenating hand component detection boxes belonging to the same position to ensure that the hand component at that position in the present and past belong to the same person. Based on this, the interactive gestures determined by the detection box sequence of each hand component are obtained by considering the gesture recognition results at the same position in the current video frame and historical video frames. Essentially, the statistical results of gesture recognition at the same position over a period of time are used to replace the instantaneous gesture recognition result. This avoids the electronic device triggering an instantaneous gesture and generating an incorrect gesture response when the user changes their gesture or is subject to external interference. Therefore, in the case of multi-person interactive control, the accuracy and robustness of interactive gesture recognition can be improved, and false triggering can be effectively reduced.

[0071] The following provides a detailed description of the methods for determining interactive gestures in some embodiments of this application. Please refer to [link / reference]. Figure 2 The method S100 includes, but is not limited to, the following steps:

[0072] S10: Get the current video frame.

[0073] The camera captures a video stream, which consists of multiple consecutive video frames. The video frame corresponding to the current moment in the video stream is called the current video frame. As time progresses, the current video frame continuously changes.

[0074] It is understandable that there can be one or more users in front of the camera, and therefore, the video frame may include the gestures of one user or multiple users. When the user leaves the camera's field of view, the video frame does not include the user or their gestures.

[0075] Understandably, gestures can be static gestures, such as... Figure 3 Some static gestures are shown, such as the palm gesture, the OK gesture, the thumbs-up gesture, the index finger gesture, or the fist gesture. These gestures are merely illustrative examples; it is understood that any shape or movement made by a user using their hand can be considered a gesture. No limitations are imposed on gestures in this application.

[0076] S20: Perform hand component recognition processing on the current video frame to obtain at least one hand component detection box and the gesture recognition result corresponding to the hand component detection box in the current video frame.

[0077] A pre-trained gesture recognition model can be used to perform convolutional downsampling on the current video frame to locate and classify gestures. The position of the gesture is the hand component detection box, and the category of the gesture is the gesture recognition result corresponding to the hand component detection box. The gesture recognition model can be trained using a convolutional neural network. The application and use of convolutional neural networks are common techniques in the field of algorithms and will not be described in detail here.

[0078] It is understandable that there may be 0, 1, or more hand detection boxes in the current video frame. If 0 hand detection boxes are detected in the current video frame, that is, there is no hand, then the current video frame is skipped and the next video frame is processed.

[0079] In some embodiments, step S20 specifically includes:

[0080] S21: Perform target component detection processing on the current video frame to obtain the bounding box of at least one target component.

[0081] Since the hand component (i.e., the hand) is relatively small compared to the human body and is usually small in video frames, it is not easy to track and recognize. In order to detect and recognize the gestures made by the hand component more quickly and effectively, for the current video frame, target component detection processing is first performed to obtain the bounding box of at least one target component.

[0082] The target component is a part of a human limb, such as the head, face, or head and shoulders. These target components have obvious features and are easy to detect and identify. By identifying the target components, the pixel range in which the hand component may appear can be initially determined.

[0083] The target component detection process in this step can be performed by a pre-trained target component detection neural network. The current video frame is input into the trained target component detection neural network to perform target detection, and the position information of the target component can be obtained. The position information of the target component is represented by a bounding box.

[0084] Taking the head as an example, the current video frame is input into a trained head detection neural network, which then outputs the positions (i.e., bounding boxes) of each head on the current video frame. Figure 4 As shown, Figure 4The display shows the detection results on the current video frame. In this frame, each person's head is within a bounding box, which is a rectangle represented by parameters (x, y, w, h), where (x, y) represents the center of the bounding box, w represents the width, and h represents the height. These parameters can be understood to be determined with reference to the image coordinate system of the current video frame. The bounding box reflects the position of the target component.

[0085] Understandably, if there is one user in the current video frame, the bounding box of one target component will be detected; if there are multiple users in the current video frame, the bounding boxes of multiple target components will be detected. Figure 4 The example shown uses a video frame containing three users.

[0086] S22: Expand the bounding box of at least one target component to obtain at least one expanded bounding box.

[0087] Based on human anatomy, it can be known that the hand component is near the target component, such as below or above the target component. Therefore, by searching for and identifying the hand component near the target component in the video frame, and then recognizing the gesture, the detection and recognition of invalid regions can be effectively reduced. Here, invalid regions can be understood as areas where the hand component is unlikely to appear, such as background areas far away from the target component, or torso areas far away from the target component.

[0088] To obtain the effective region where the hand component is most likely to appear, the bounding boxes of each target component are expanded to obtain corresponding expanded bounding boxes. It can be understood that the pixel region of the video frame enclosed by this expanded bounding box is the pixel region where the hand component is most likely to appear.

[0089] It is understood that the expanded bounding box is obtained by expanding the bounding box. The specific expansion method can be set by those skilled in the art. For example, the four boundaries of the bounding box can be proportionally enlarged and expanded with the center (x, y) of the bounding box as the center. In some embodiments, the expansion can be performed outwards from the center (x, y) of the bounding box, and the resulting expanded bounding box can be circular or trapezoidal, etc., with no specific restrictions on its shape.

[0090] In some embodiments, when the target component is a human body, the bounding box (x, y, w, h) encloses the human body. Since the hand component will not appear below the waist when a person makes a gesture, this information can be used to construct an extended bounding box. The range of the extended bounding box can use the upper half (H / 2) of the bounding box as a reference, and then extend it to the left and right by a certain distance. For example, the width of the extended bounding box is 3W and the height is H / 2, thus constructing an extended bounding box with a height of 0.5H and a width of 3W.

[0091] In some embodiments, when the target component is the head and shoulders, the bounding box (x, y, w, h) encloses the head and shoulders. When making a gesture, the forearm must be raised above the elbow to make the gesture. Based on this characteristic, the entire bounding box can be used as a reference, and the height can be increased by 0.5H upwards and the width increased by W on both sides to construct an expanded bounding box with a height of 1.5H and a width of 3W.

[0092] In some embodiments, when the target component is the head / face region, the bounding box (x, y, w, h) encloses the head / face. Standard gestures will be located on both sides of the face and will not overlap with the face. Therefore, based on the bounding box, an expanded bounding box with a height of 4H and a width of 5H can be constructed by expanding it upward by H, downward by 2H, and left and right by 2W.

[0093] S23: Perform hand part detection processing on the region located within at least one extended bounding box in the current video frame to obtain at least one hand part detection box.

[0094] After obtaining the bounding boxes, hand component detection processing is performed on the regions within each bounding box in the video frame to obtain hand component detection boxes. Here, hand component detection processing detects which pixels in each bounding box on the video frame are hand components, that is, it detects and locates each hand component, and the hand component detection box encloses the pixel position of each hand component in the video frame. The hand component detection box can also be represented by center coordinates and width and height (x, y, w, h), that is, the hand component detection box is the position of the hand component.

[0095] It is understandable that the hand part detection process in this step can be performed by a pre-trained hand part detection neural network. The local images obtained by cropping video frames from each bounding box are input into the trained hand part detection neural network to perform target detection, and the hand part detection box can be obtained.

[0096] S24: Perform gesture recognition on the hand parts located within at least one hand part detection box in the current video frame to obtain the gesture recognition result corresponding to the hand part detection box.

[0097] After obtaining the detection boxes for each hand component in the current video frame, i.e., after obtaining each hand component in the current video frame, gesture recognition is performed on the gestures presented by each hand component, thus generating the gesture recognition result corresponding to each hand component detection box. This gesture recognition result includes the gesture category of the hand component.

[0098] It is understandable that the gesture recognition in this step can be performed by a pre-trained gesture classification neural network. The pixel area within the hand component detection box is input into the pre-trained gesture classification neural network for gesture recognition and classification, and the gesture recognition result corresponding to the hand component detection box can be obtained.

[0099] Understandably, the types of gestures that a gesture classification neural network can detect can be set according to actual needs. For example, the images in the training set for training the gesture classification neural network can include six gesture categories: palm gesture, OK gesture, L-shaped gesture, V-shaped gesture, fist gesture, and other gestures. Thus, a well-trained gesture classification neural network can recognize the gesture categories that the technician needs to identify.

[0100] It is understandable that, since the video stream is acquired in real time, the current video frame is constantly changing as the time sequence progresses. That is, each video frame in the video stream is the current video frame at a certain moment. Therefore, for each video frame in the video stream, the above step S20 is used to perform hand part recognition processing to obtain at least one hand part detection box and the gesture recognition result corresponding to the hand part detection box in each video frame of the video stream.

[0101] S30: Match the hand component detection boxes corresponding to the current video frame with the video frames in the preset historical video frame set to obtain at least one hand component detection box sequence corresponding to the current video frame.

[0102] It's understandable that when multiple users are in front of the camera, multiple hand gestures may be detected in the video stream frames. To avoid the limitations of single-frame detection, multi-frame statistics use the statistical results of consecutive frames as the final interaction gesture. Since multiple hand gestures may exist in each video frame, and generally, within a certain period (a few consecutive frames), the interaction gesture is made by one person. For example, if the first frame contains three trigger gestures (K, L, V) that execute corresponding functions, it's impossible to accurately determine the user's intention in this case, as the three trigger gestures could be due to accidental operation. Therefore, it's necessary to determine the interaction gesture by considering whether these three gestures continue to appear in multiple video frames. Specifically, in subsequent frames 2, 3, 4, 5, 6, 7, 8, and 9, if these three trigger gestures [K, L, V] do not appear consecutively, it indicates an accidental operation. Only the continuously appearing trigger gestures are the interaction gestures the user intends to trigger. In this case, in most situations, only one trigger gesture (e.g., trigger gesture K) will appear consecutively. Multi-frame statistics can then accurately determine the interaction gesture and the user's control intention.

[0103] To capture trigger gestures made by the same user in consecutive frames (a certain period) and avoid interference from gestures by other users, it is necessary to concatenate the gestures of the same person in consecutive frames to form a hand component detection box sequence. Specifically, the hand component detection boxes corresponding to the current video frame are matched with those of video frames in a preset set of historical video frames to obtain at least one hand component detection box sequence corresponding to the current video frame.

[0104] It is understandable that the positions of users remain largely unchanged across multiple consecutive video frames. Consequently, the positions of hand components in the image coordinate system also remain largely unchanged or change only slightly. This means that the offset of the hand component detection box is small across multiple consecutive video frames. Therefore, matching the hand component detection box can refer to matching the positional offset of the hand component detection box. For example, if the positional offset of the hand component detection box in two consecutive video frames is within a certain range, then a match is considered established.

[0105] In some embodiments, the aforementioned current video frame is the k-th video frame, and the aforementioned step S30 specifically includes:

[0106] S31: Match the hand component detection boxes of the k-th video frame and the (k-1)-th video frame according to the preset matching rules. If the first hand component detection box of the k-th video frame and the (k-1)-th video frame are successfully matched, concatenate the first hand component detection boxes of the k-th video frame and the (k-1)-th video frame into the first hand component detection box sequence, and match the hand component detection boxes of the (k-1)-th video frame and the (k-2)-th video frame according to the preset matching rules.

[0107] Among them, the first hand part detection box is any hand part detection box in the video frame, and the (k-1)th and (k-2)th video frames are video frames in the preset historical video frame set.

[0108] In some embodiments, the preset historical video frame set is the n consecutive video frames that are sequentially preceding the current video frame.

[0109] S32: If the first hand part detection box of the k-th video frame fails to match the first hand part detection box of the (k-1)-th video frame, then the k-th video frame and the (k-2)-th video frame are matched according to the preset matching rules. This process is repeated, backtracking the matching until all video frames in the preset historical video frame set are matched.

[0110] The k-th video frame is matched with the (k-1)-th video frame according to a preset matching rule for hand component detection boxes. The (k-1)-th video frame is matched with the (k-2)-th video frame according to the preset matching rule for hand component detection boxes, and so on. If the current video frame successfully matches the first hand component detection box of the previous video frame, the two first hand component detection boxes are concatenated. If the match fails, the previous video frame is skipped, and the current video frame is matched with the video frame two years prior to it for hand component detection boxes, and so on, until a first hand detection box sequence belonging to the same user is finally formed.

[0111] It is understandable that the first-hand part detection box is any hand part detection box in the video frame. Therefore, the first-hand part detection box is only different in name and does not impose any restrictions on the hand part detection box. The above method is used to match any hand part detection box in the video frame.

[0112] In this embodiment, by matching frames one by one, hand component detection boxes belonging to the same location (same user) are linked together to form a hand component detection box sequence, which is beneficial for capturing the trigger gestures made by the same user in consecutive frames (a certain period).

[0113] In some embodiments, the aforementioned step S30 specifically includes:

[0114] S33: Match the current video frame with the video frames in the preset historical video frame set according to the preset matching rules. If the first hand part detection box of any target video frame in the preset video frame set matches the first hand part detection box of the current video frame, then concatenate the first hand part detection boxes of the target video frame and the current video frame into the first hand part detection box sequence. The first hand part detection box is any hand part detection box in the video frame.

[0115] Similarly, the first-hand component detection box is any hand component detection box in the video frame. Therefore, the first-hand component detection box is only different in name and does not impose any restrictions on the hand component detection box. The above method is used to match any hand component detection box in the video frame.

[0116] In this embodiment, by performing single-frame matching between the current video frame and video frames in a preset set of historical video frames, the hand component detection boxes belonging to the same location (same user) are linked together to form a hand component detection box sequence, which is beneficial for capturing the trigger gestures made by the same user in consecutive frames (a certain period).

[0117] In some embodiments, the aforementioned preset matching rule includes: if the Euclidean distance between the center coordinates of the first hand component detection boxes of two video frames that are performing hand component detection box matching is less than or equal to a first threshold, then the first hand component detection boxes of the two video frames are successfully matched.

[0118] For example, the first hand component detection boxes of two video frames used for hand component detection box matching are respectively represented by Box... k (X k Y k W k H k ) and Box k-i (X k-i Y k-i W k-i H k-i ) indicates that Box k The center coordinates are Center k (X k +0.5*W k Y k +0.5*Hk ),the center coordinates of Box k-i is Center k-i (X k-i + 0.5 * W k-i , Y k-i + 0.5 * H k-i ).

[0119] The Euclidean distance between the two center coordinates is distance = sqrt((center k .x - center k-i .x)^2+(center k .y - center k-i .y)^2). The first threshold is thr1. If distance ≤ thr1, it is considered that the two first-hand component detection frames are successfully matched. If distance > thr1, it is considered that the two first-hand component detection frames have no association and the matching fails.

[0120] In this embodiment, by comparing the center coordinates of the two hand component detection frames, it is possible to accurately determine whether the two hand component detection frames are matched.

[0121] In some embodiments, the foregoing preset matching rule includes: if the intersection over union of the first-hand component detection frames of two video frames for which hand component detection frame matching is performed is greater than or equal to the second threshold, then the first-hand component detection frames of these two video frames are successfully matched.

[0122] The intersection of the first-hand component detection frames of two video frames for which hand component detection frame matching is performed is intersection, and the union is union. Then intersection = Area(Box k & Box k-i ), Union = Area(Box k ) + Area(Box k-i ) - intersection. Thus, the intersection over union iou = intersection / union is calculated. The second threshold is thr2. If iou ≥ thr2, it is considered that the two first-hand component detection frames are successfully matched. If iou < thr2, it is considered that the two first-hand component detection frames have no association and the matching fails.

[0123] In this embodiment, by comparing the intersection over union of the two hand component detection frames, it is possible to accurately determine whether the two hand component detection frames are matched.

[0124] In some embodiments, the aforementioned preset matching rule includes: if the absolute distance between the first hand part detection boxes of two video frames that are being matched for hand part detection is less than or equal to a third threshold, then the first hand part detection boxes of the two video frames are successfully matched.

[0125] The absolute distance is calculated as the sum of the Euclidean distances between the four pairs of vertices of the two first-hand component detection boxes multiplied by the ratio of the areas of the two first-hand component detection boxes, called area_ratio. The larger area is used as the ratio of the smaller area to the larger area, ensuring that area_ratio >= 1. Therefore, the absolute distance D = (dist) / (dist) is calculated. tl +dist tr +dist bl +dist br )*area_ratio, where dist tl , dist tr dist bl dist br These are the Euclidean distances between the four pairs of vertices of the two first-hand component detection boxes. The third threshold is thr3. If the absolute distance D ≤ thr3, the two first-hand component detection boxes are considered to have successfully matched. If D > thr3, the two first-hand component detection boxes are considered to have no connection and the match fails.

[0126] In this embodiment, by comparing the absolute distance between the two hand component detection frames, it can be accurately determined whether the two hand component detection frames match.

[0127] S40: Determine the interactive gesture corresponding to the current video frame based on at least one hand component detection box sequence and the gesture recognition result corresponding to the hand component detection box in the hand component detection box sequence.

[0128] After obtaining at least one hand component detection box sequence, based on the gesture recognition results corresponding to the hand component detection boxes in the hand component detection box sequence, the interactive gesture corresponding to the current video frame can be determined according to each hand component detection box sequence and the corresponding gesture recognition result. That is, by tracing back the gesture recognition results in historical video frames, the interactive gesture corresponding to the current video frame can be determined, and it can be determined that the interactive gesture is a stable trigger gesture made by the same user.

[0129] In some embodiments, the aforementioned step S40 specifically includes:

[0130] S41: Obtain the valid gestures corresponding to the detection box sequence of each hand component.

[0131] S42: If there is a trigger gesture among the valid gestures, then the trigger gesture is determined to be an interactive gesture.

[0132] S43: If there are multiple trigger gestures among the valid gestures, then the trigger gesture that is processed first is determined to be the interactive gesture.

[0133] S44: If there is no trigger gesture among the valid gestures, continue processing the next video frame of the current video frame.

[0134] For example, there are three hand detection box sequences. The first hand detection box sequence is [Y, Y, Y, Y, Q, Y, Y, Q, Y, Y, Y], where Y is the gesture the user wants to display and Q is a distracting gesture. The valid gesture corresponding to this hand detection box sequence is Y. The second hand detection box sequence is [K, K, L, K, K, K, K, Q, K, K, K], where K is the gesture the user wants to display and L and Q are distracting gestures. The valid gesture corresponding to this hand detection box sequence is K. The third hand detection box sequence is [L, L, L, K, L, L, L, L, L, L, L], where L is the gesture the user wants to display and K is a distracting gesture. The valid gesture corresponding to this hand detection box sequence is L.

[0135] Understandably, if among gestures Y, K, and L, K belongs to a pre-defined set of trigger gestures (i.e., K is a trigger gesture), and Y and L are not trigger gestures, then trigger gesture K is determined to be an interactive gesture. If both K and L are trigger gestures, the algorithm prioritizes processing trigger gesture L, thus determining trigger gesture L to be an interactive gesture. If none of gestures Y, K, and L belong to a trigger gesture (i.e., no trigger gesture exists), then processing continues to the next video frame.

[0136] In some embodiments, the aforementioned step S41 specifically includes:

[0137] S411: Obtain the first number of first-hand component detection boxes in the first-hand component detection box sequence whose gesture recognition results are the same as the gesture recognition results of the first-hand component detection box in the current video frame. If the ratio of the first number to the number of video frames in the preset historical video frame set is greater than or equal to the first ratio threshold, then determine that the gesture recognition result of the first-hand component detection box in the current video frame is a valid gesture corresponding to the first-hand component detection box sequence.

[0138] Among them, the first hand component detection box sequence is any one of the hand component detection box sequences.

[0139] For example, starting from the current video frame, all first-hand part detection boxes in N frames of a preset historical video frame set are counted. First-hand part detection boxes at the same location are concatenated to obtain S (S<=N) first-hand part detection boxes. These S first-hand part detection boxes are classified, and the number of first-hand part detection boxes with the same gesture category as the first-hand part detection box in the current video frame is recorded as M. When M / N is greater than a first percentage threshold of 0.7, it is considered that more than 70% of the gesture categories in the past N frames are consistent with the gesture category of the first-hand part detection box in the current video frame. The gesture category of the first-hand part detection box in the current video frame can then represent the gesture that the user determined to display in the past N frames.

[0140] Compared to simply counting the most frequent gesture categories in the S first-hand component detection frames, this method counts the percentage of gesture categories in the S first-hand component detection frames that match the gesture categories in the current video frame. This effectively avoids situations where the gestures in the first-hand component detection frames of the current video frame have changed while the previous S first-hand component detection frames contained other gestures, leading to a lag in gesture category statistics.

[0141] In some embodiments, the hand detection frame sequence includes hand detection frames from video frames that successfully match the hand detection frame of the current video frame within a preset time period prior to the current video frame. In this embodiment, the preset historical video frames are video frames within a preset time period prior to the current video frame.

[0142] The aforementioned step S41 specifically includes:

[0143] S412: Obtain the second number of first-hand component detection boxes in the first-hand component detection box sequence whose gesture recognition results are the same as those of the first-hand component detection box in the current video frame. If the ratio of the second number to the number of hand component detection boxes in the first-hand component detection box sequence is greater than or equal to the second ratio threshold, then determine that the gesture recognition result of the first-hand component detection box in the current video frame is a valid gesture corresponding to the first-hand component detection box sequence.

[0144] Among them, the first hand component detection box sequence is any one of the hand component detection box sequences.

[0145] Starting from the current video frame, count all first-hand component detection boxes in the video frames (preset video frame set) within the past preset time t seconds. For example, t can be 1 second. Concatenate the first-hand component detection boxes at the same position to obtain R first-hand component detection boxes. Count the number of video frames in the past t seconds whose gesture category is the same as that of the first-hand component detection box of the current video frame. When P / R is greater than the second percentage threshold of 0.7, it is considered that more than 70% of the gesture categories in the past t seconds are consistent with the gesture category of the first-hand component detection box of the current video frame. The gesture category of the first-hand component detection box of the current video frame can represent the gesture that the user determined to display in the past t seconds.

[0146] In this embodiment, the number R of first-hand component detection boxes that match the first-hand component detection box of the current video frame in the video frames within t seconds is not fixed and is not affected by the time consumption of inter-frame processing and missed detections, making the statistical results more stable and the effective gestures corresponding to the first-hand component detection box sequence more accurate.

[0147] In summary, the method for determining interactive gestures provided in this application involves performing hand component recognition processing on the current video frame to obtain at least one hand component detection box and a gesture recognition result corresponding to the hand component detection box in the current video frame; then, matching the hand component detection boxes corresponding to the current video frame with those of video frames in a preset historical video frame set to obtain a sequence of at least one hand component detection boxes corresponding to the current video frame; finally, determining the interactive gesture corresponding to the current video frame based on the sequence of at least one hand component detection boxes and the gesture recognition result corresponding to the hand component detection boxes in the sequence. In this method, each hand component detection box sequence obtained through the above hand component detection box matching reflects the position and gesture recognition result of the same hand component (same position) in the current video frame and the preset historical video frame set, respectively. This is equivalent to chaining hand component detection boxes belonging to the same position together, ensuring that the hand component at that position in the present and past belong to the same person. Based on this, the interactive gestures determined by the detection box sequence of each hand component are obtained by considering the gesture detection results at the same position in the current video frame and historical video frames. Essentially, the statistical results of gestures at the same position over a period of time are used to replace instantaneous gestures, avoiding erroneous gesture responses triggered by instantaneous gestures when the user changes gestures or external interference. Therefore, in the case of multi-person interactive control, the accuracy and robustness of interactive gestures can be improved, effectively reducing false triggers.

[0148] This application also provides an interaction method, which includes:

[0149] (1) The interactive gesture is determined by the method of determining the interactive gesture as described in any of the above embodiments.

[0150] (2) Control the target device to execute the operation command corresponding to the interactive gesture.

[0151] When a user wants to control a device to enable a certain function, they can make a gesture. The device identifies and determines the gesture using the method for determining interactive gestures in any of the above embodiments. This device can be referred to as the target device, and controlling the target device can be controlling a functional component within the device. This functional component can be a hardware or software module. In one example, the target device may include, but is not limited to, a smart camera. Controlling the smart camera may include, but is not limited to, controlling one or more functional components within the smart camera, such as a lens focus control module, a scene mode switching module, or a flash control module.

[0152] Understandably, interactive gestures correspond one-to-one with operation commands. For example, when user A makes a "palm open" gesture, the smart camera recognizes the gesture and executes the operation command "start tracking user A's head component". When user A makes a "thumbs up" gesture, the smart camera recognizes the gesture and executes the operation command "start focusing".

[0153] Therefore, by controlling the target device to execute the operation command corresponding to the interactive gesture, the user can perform intelligent interactive control in front of the target device without manually adjusting it.

[0154] The methods for determining interactive gestures and interaction methods in the embodiments of this application have been described above. In order to better implement the methods of this application, the apparatus provided in the embodiments of this application will be described next.

[0155] Please see Figure 5 , Figure 5 This application provides an embodiment of a device for determining interactive gestures. The device 200 includes:

[0156] The acquisition module 210 is used to acquire the current video frame.

[0157] The recognition module 220 is used to perform hand part recognition processing on the current video frame to obtain at least one hand part detection box in the current video frame and the gesture recognition result corresponding to the hand part detection box.

[0158] The matching module 230 is used to match the hand part detection boxes corresponding to the current video frame with the video frames in the preset historical video frame set to obtain at least one hand part detection box sequence corresponding to the current video frame.

[0159] The determining module 240 is used to determine the interactive gesture corresponding to the current video frame based on at least one hand component detection box sequence and the gesture recognition result corresponding to the hand component detection box in the hand component detection box sequence.

[0160] The aforementioned device 200 performs hand component recognition processing on the current video frame to obtain at least one hand component detection box and a gesture recognition result corresponding to the hand component detection box in the current video frame. Then, it matches the hand component detection boxes corresponding to video frames in a preset set of historical video frames to obtain a sequence of at least one hand component detection boxes corresponding to the current video frame. Finally, based on the sequence of at least one hand component detection boxes and the gesture recognition result corresponding to the hand component detection boxes in the sequence, it determines the interactive gesture corresponding to the current video frame. In this method, each hand component detection box sequence obtained through the above hand component detection box matching reflects the position and gesture recognition result of the same hand component (same position) in the current video frame and the preset set of historical video frames, respectively. This is equivalent to chaining hand component detection boxes belonging to the same position together, ensuring that the hand component at that position in the present and past belong to the same person. Based on this, the interactive gestures determined by the detection box sequence of each hand component are obtained by considering the gesture detection results at the same position in the current video frame and historical video frames. Essentially, the statistical results of gestures at the same position over a period of time are used to replace instantaneous gestures, avoiding erroneous gesture responses triggered by instantaneous gestures when the user changes gestures or external interference. Therefore, in the case of multi-person interactive control, the accuracy and robustness of interactive gestures can be improved, effectively reducing false triggers.

[0161] Please see Figure 6 This is a hardware structure diagram of an electronic device 10 provided in an embodiment of this application. Specifically, as shown... Figure 6 As shown, the electronic device 10 includes at least one processor 12 and a memory 13 connected in communication. Figure 6 (Taking a bus connection and a single processor as an example).

[0162] The processor 12 is used to provide computing and control capabilities to control the electronic device 10 to perform corresponding tasks and to control the electronic device 10 to perform any of the methods for determining interactive gestures provided in the above embodiments.

[0163] It is understood that processor 12 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0164] The memory 13, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions / modules corresponding to the method for determining interactive gestures or interaction methods in the embodiments of the present invention. The processor 12 can implement any of the methods for determining interactive gestures or interaction methods provided in the above embodiments by running the non-transitory software programs, instructions, and modules stored in the memory 13. Specifically, the memory 13 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 13 may also include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0165] It is understood that, in some embodiments, the electronic device may be a smart device such as a smart camera, a mobile terminal, or a drone.

[0166] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the aforementioned method for determining interactive gestures or interaction methods.

[0167] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0168] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for determining interactive gestures, characterized in that, include: Get the current video frame; Perform hand component recognition processing on the current video frame to obtain at least one hand component detection box in the current video frame and a gesture recognition result corresponding to the hand component detection box; The current video frame is matched with video frames in a preset historical video frame set according to a preset matching rule to perform hand component detection box matching. If the first hand component detection box of any target video frame in the preset historical video frame set is successfully matched with the first hand component detection box of the current video frame, the first hand component detection boxes of the target video frame and the current video frame are concatenated into a first hand component detection box sequence, where the first hand component detection box is any hand component detection box in the video frame. Based on the at least one hand component detection box sequence and the gesture recognition result corresponding to the hand component detection box in the hand component detection box sequence, the interactive gesture corresponding to the current video frame is determined, wherein the hand component detection box sequence is a series of gestures of the same hand component in consecutive video frames.

2. The method according to claim 1, characterized in that, The step of performing hand component recognition processing on the current video frame to obtain at least one hand component detection box in the current video frame and a gesture recognition result corresponding to the hand component detection box includes: The current video frame is subjected to target component detection processing to obtain the bounding box of at least one target component; The bounding boxes of the at least one target component are expanded to obtain at least one expanded bounding box; Hand part detection processing is performed on the region located within the at least one extended bounding box in the current video frame to obtain at least one hand part detection box; Gesture recognition is performed on the hand parts located within the at least one hand part detection box in the current video frame to obtain the gesture recognition result corresponding to the hand part detection box.

3. The method according to claim 1, characterized in that, The current video frame is the k-th video frame; The step of matching the hand component detection boxes corresponding to the current video frame with those corresponding to video frames in a preset set of historical video frames to obtain at least one sequence of hand component detection boxes corresponding to the current video frame includes: The k-th video frame and the (k-1)-th video frame are matched with hand component detection boxes according to a preset matching rule. If the first hand component detection box of the k-th video frame and the (k-1)-th video frame are successfully matched, the first hand component detection boxes of the k-th video frame and the (k-1)-th video frame are concatenated into the first hand component detection box sequence. The (k-1)-th video frame and the (k-2)-th video frame are matched with hand component detection boxes according to the preset matching rule. The first hand component detection box is any hand component detection box in the video frame, and the (k-1)-th video frame and the (k-2)-th video frame are video frames in the preset historical video frame set. If the first hand component detection box of the k-th video frame fails to match the first hand component detection box of the (k-1)-th video frame, then the k-th video frame is matched with the (k-2)-th video frame according to the preset matching rules. This process is repeated, backtracking the matching until all video frames in the preset historical video frame set are matched.

4. The method according to claim 1, characterized in that, The preset matching rules include: If the Euclidean distance between the center coordinates of the first hand component detection boxes of the two video frames being matched is less than or equal to a first threshold, then the first hand component detection boxes of the two video frames are successfully matched.

5. The method according to claim 1, characterized in that, The preset matching rules include: If the intersection-union ratio of the first hand component detection boxes of the two video frames being matched is greater than or equal to the second threshold, then the first hand component detection boxes of the two video frames are successfully matched.

6. The method according to claim 1, characterized in that, The preset matching rules include: If the absolute distance between the first hand component detection boxes of two video frames that are being matched for hand component detection is less than or equal to a third threshold, then the first hand component detection boxes of the two video frames are successfully matched.

7. The method according to claim 1, characterized in that, The step of determining the interactive gesture corresponding to the current video frame based on the at least one hand component detection box sequence and the gesture recognition result corresponding to the hand component detection box in the hand component detection box sequence includes: Obtain the valid gestures corresponding to each of the hand component detection box sequences; If one of the valid gestures is a trigger gesture, then the trigger gesture is determined to be the interaction gesture; If there are multiple trigger gestures among the valid gestures, then the trigger gesture that is processed and acquired first is determined to be the interaction gesture; If no triggering gesture is found among the valid gestures, then the processing continues to the next video frame of the current video frame.

8. The method according to claim 7, characterized in that, The step of obtaining the valid gestures corresponding to each of the hand component detection box sequences includes: Obtain a first number of first hand component detection boxes in the first hand component detection box sequence whose gesture recognition results are the same as those of the first hand component detection box in the current video frame. If the ratio of the first number to the number of video frames in the preset historical video frame set is greater than or equal to a first ratio threshold, then determine that the gesture recognition result of the first hand component detection box in the current video frame is a valid gesture corresponding to the first hand component detection box sequence, wherein the first hand component detection box sequence is any one of the hand component detection box sequences.

9. The method according to claim 7, characterized in that, The hand component detection box sequence includes the hand component detection boxes of video frames that successfully match the hand component detection boxes of the current video frame within a preset time period before the current video frame. The step of obtaining the valid gestures corresponding to each of the hand component detection box sequences includes: Obtain a second number of first hand component detection boxes in the first hand component detection box sequence whose gesture recognition results are the same as those of the first hand component detection box in the current video frame. If the ratio of the second number to the number of hand component detection boxes in the first hand component detection box sequence is greater than or equal to a second ratio threshold, then determine that the gesture recognition result of the first hand component detection box in the current video frame is a valid gesture corresponding to the first hand component detection box sequence, wherein the first hand component detection box sequence is any one of the hand component detection box sequences.

10. An interaction method, characterized in that, include: The interactive gesture is determined using the method for determining interactive gestures as described in any one of claims 1-9; Control the target device to execute the operation command corresponding to the interactive gesture.

11. A device for determining interactive gestures, characterized in that, include: The acquisition module is used to acquire the current video frame; The recognition module is used to perform hand component recognition processing on the current video frame to obtain at least one hand component detection box in the current video frame and a gesture recognition result corresponding to the hand component detection box. The matching module is used to match the current video frame with video frames in a preset historical video frame set according to a preset matching rule to perform hand component detection box matching. If the first hand component detection box of any target video frame in the preset historical video frame set is successfully matched with the first hand component detection box of the current video frame, then the first hand component detection boxes of the target video frame and the current video frame are concatenated into a first hand component detection box sequence, where the first hand component detection box is any hand component detection box in the video frame. The determining module is used to determine the interactive gesture corresponding to the current video frame based on the at least one hand component detection box sequence and the gesture recognition result corresponding to the hand component detection box in the hand component detection box sequence, wherein the hand component detection box sequence is a series of gestures of the same hand component in consecutive video frames.

12. An electronic device, characterized in that, include: At least one processor, and The memory communicatively connected to the at least one processor, wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the method as described in any one of claims 1-9.

Images