Gesture detection method and apparatus, and terminal device and computer-readable storage medium

Abstract

Disclosed in the present application are a gesture detection method and apparatus, and a terminal device and a computer-readable storage medium. The gesture detection method comprises: determining first positions of feature points in a first image of a target entity, wherein the feature points correspond to body key points of the target entity; and on the basis of the first positions, executing a preprocessing operation for gesture detection of the target entity, wherein the preprocessing operation includes a first preprocessing operation and / or a second preprocessing operation, the first preprocessing operation is used for determining a gesture detection region in a second image of the target entity, in order to perform gesture detection on the gesture detection region, and the second preprocessing operation is used for determining whether a gesture of the target entity meets an activation condition, in order to perform gesture detection when it is determined that the gesture of the target entity meets the activation condition.

Description

Gesture detection method, device, terminal equipment and computer readable storage medium

[0001] Cross-reference to Related Applications

[0002] This application claims priority to Chinese Patent Application No. 202411044887.4, filed on July 31, 2024, entitled “Gesture detection method, device, terminal equipment and computer readable storage medium,” the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present application belongs to the technical field of gesture detection, and particularly relates to a gesture detection method, device, terminal equipment and computer readable storage medium. BACKGROUND

[0004] In recent years, gesture detection technology has been increasingly widely applied in various fields such as game entertainment, medical health, automobile driving, and security monitoring. With the continuous development of artificial intelligence technology, the accuracy and reliability of gesture detection technology have gradually increased, bringing many conveniences to people's lives.

[0005] However, the gesture detection scheme in the related art usually inputs the original image of the target object collected without judgment into the model, and the model identifies the valid gesture in the full image range according to the detection mechanism, and outputs the position, size and gesture type of the valid gesture. The algorithmic cost required by such a scheme is huge, and the efficiency of gesture detection is very low. Some schemes attempt to compress the size of the model input image to improve detection efficiency, but the detection accuracy is difficult to meet the requirements, i.e., it is difficult to achieve effective balance between detection accuracy and detection efficiency. SUMMARY

[0006] The embodiments of the present application provide a gesture detection method, device, terminal equipment and computer readable storage medium, which can solve the problem that the detection accuracy and detection efficiency of gesture detection are difficult to achieve effective balance.

[0007] A first aspect of the embodiments of the present application provides a gesture detection method, comprising:

[0008] determining a first position of a feature point in a first image of a target object, wherein the feature point corresponds to a body key point of the target object;

[0009] performing a preprocessing operation of gesture detection of the target object according to the first position;

[0010] The preprocessing operation includes a first preprocessing operation and / or a second preprocessing operation. The first preprocessing operation is used to determine a gesture detection region in the second image of the target object, so as to perform gesture detection on the gesture detection region. The second preprocessing operation is used to determine whether the gesture of the target object meets an activation condition, so as to perform gesture detection when it is determined that the gesture of the target object meets the activation condition.

[0011] A second aspect of the embodiment of the present application provides a gesture detection apparatus, which comprises:

[0012] A determination module is configured to determine a first position of a feature point in the first image of the target object, wherein the feature point corresponds to a body key point of the target object.

[0013] A preprocessing module is configured to perform a preprocessing operation of gesture detection of the target object according to the first position.

[0014] The preprocessing operation includes a first preprocessing operation and / or a second preprocessing operation. The first preprocessing operation is used to determine a gesture detection region in the second image of the target object, so as to perform gesture detection on the gesture detection region. The second preprocessing operation is used to determine whether the gesture of the target object meets an activation condition, so as to perform gesture detection when it is determined that the gesture of the target object meets the activation condition.

[0015] A third aspect of the embodiment of the present application provides a terminal device, which comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The processor executes the computer program to implement the steps of the gesture detection method.

[0016] A fourth aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps of the gesture detection method.

[0017] A fifth aspect of the embodiment of the present application provides a computer program product, which is run on a terminal device, so that the terminal device can implement the steps of the gesture detection method.

[0018] The first aspect of the embodiments of the present application provides a gesture detection method. Before gesture detection, the first position of the feature point corresponding to the body key point in the first image of the target object is determined accurately, and then, the pre-processing operation of gesture detection of the target object is performed according to the first position. In one case, the pre-processing operation can accurately determine the gesture detection region in the second image of the target object, so that subsequent gesture detection can be performed only on the gesture detection region. Therefore, the detection efficiency can be improved on the basis of ensuring good detection accuracy, and the effective balance between detection accuracy and detection efficiency is achieved. In another case, the pre-processing operation can accurately determine whether the gesture of the target object meets the activation condition, and then subsequent gesture detection can be performed on the image region of the target object only when it is determined that the gesture of the target object meets the activation condition. This scheme can filter out unnecessary image regions for detection. On the basis of not reducing the detection accuracy, the computing cost required for gesture detection is significantly reduced, and the detection efficiency is significantly improved. Therefore, the overall gesture detection scheme of the embodiments of the present application can effectively filter out images or image regions that may interfere with gesture detection or are unnecessary for gesture detection by performing a simple and reasonable image processing process before gesture detection. Therefore, the detection accuracy and the detection efficiency can be ensured to be high.

[0019] It can be understood that the beneficial effects of the second aspect to the fifth aspect described above can be referred to the related description in the first aspect described above, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiments or related technical description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] FIG. 1a is a schematic diagram of a gesture detection region positioned according to the method of the present application;

[0022] FIG. 1b is a partial image obtained after cropping according to the gesture detection region of FIG. 1a;

[0023] FIG. 1c is a partial schematic diagram of a first image provided by an embodiment of the present application;

[0024] FIG. 2 is a schematic flowchart of a gesture detection method provided by an embodiment of the present application;

[0025] FIG. 3a is a schematic diagram of a first image provided by an embodiment of the present application;

[0026] FIG. 3b is a schematic diagram of a first image according to another embodiment of the present application;

[0027] FIG. 3c is a schematic diagram of a principle of determining a first size of a target face region according to an embodiment of the present application;

[0028] FIG. 3d is a schematic diagram of a principle of determining a body region of a target object according to an embodiment of the present application;

[0029] FIG. 3e is a schematic diagram of a principle of determining a body region of a target object according to another embodiment of the present application;

[0030] FIG. 4a is a schematic diagram of feature points in a first image according to another embodiment of the present application;

[0031] FIG. 4b is a schematic diagram of a principle of determining a gesture detection region according to an embodiment of the present application;

[0032] FIG. 4c is a schematic diagram of a principle of determining a gesture detection region according to another embodiment of the present application;

[0033] FIG. 4d is a schematic diagram of a principle of determining a gesture detection region according to another embodiment of the present application;

[0034] FIG. 5a and FIG. 5b are schematic diagrams of a gesture detection process in a multi-person mode according to different embodiments of the present application;

[0035] FIG. 5c is a schematic diagram of a gesture detection process in a single-person mode according to an embodiment of the present application;

[0036] FIG. 6 is a schematic diagram of a structure of a gesture detection apparatus according to an embodiment of the present application;

[0037] FIG. 7 is a schematic diagram of a structure of a terminal device according to an embodiment of the present application. DETAILED DESCRIPTION

[0038] In the following description, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. However, persons skilled in the art will understand that the present application can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present application.

[0039] It should be understood that the term "comprising" as used in the specification and in the claims indicates the presence of the recited features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0040] It should also be understood that the term "and / or" as used herein refers to a conjunction, an association, one or more of any combination of associated listed items as well as all possible combinations, and includes these combinations.

[0041] In addition, in the description of the present application and the appended claims, the terms "first", "second", "third" and the like are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0042] In the present application, the reference "one embodiment" or "some embodiments" and the like means that the specific features, structures or characteristics described in connection with the embodiment are included in one or more embodiments of the present application. Therefore, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in further some embodiments" and the like appearing in different places in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "include", "contain", "have" and their variants mean "include but not limited to", unless otherwise specifically emphasized.

[0043] First of all, it should be pointed out that the information collection process (such as face image collection process, fingerprint information collection process, etc.) / feature extraction process involved in the present application is executed with the user's knowledge and permission, and does not belong to the act of interfering with public interests.

[0044] As mentioned earlier, the gesture detection scheme in the related art usually inputs the original image of the target object collected without judgment into the model, and the model identifies the valid gesture in the full image range according to the detection mechanism, and outputs the position, size and gesture type of the valid gesture. It can be understood that in the case of large original image size, the calculation amount consumed by gesture detection is very large, and the detection efficiency is low. Especially in the scene that needs real-time gesture tracking (for example, body game), the user's gesture is often detected several seconds after the user finishes the gesture, the delay time of gesture detection is very long, the user experience is very poor, leading to a straight decline in user satisfaction.

[0045] Some schemes compress the original image and send the compressed image into the model for detection. This scheme has a relatively stringent requirement for the user's action, the position and the number of people in the visual scene. Once the requirement is not met, the gesture cannot be detected, or the wrong gesture is detected. This scheme is prone to cause the problem of excessive compression of the hand region in the original image, which further leads to the difficulty in achieving the expected detection accuracy of the model. That is, this scheme needs to sacrifice the detection accuracy as a price to improve the detection efficiency. Obviously, this scheme cannot effectively balance the detection accuracy and the detection efficiency.

[0046] To solve such problems, embodiments of the present application introduce a local cropping strategy. As shown in FIG. 1a, the hand region (gray box in the figure) can be roughly located from the whole image, and then the original image is cropped according to the hand region. As shown in FIG. 1b, the local image of the cropped hand region is taken as the input image of the model. Both the detection accuracy and the detection efficiency can be improved. It can be understood that the size of the cropped local image is smaller, and the amount of calculation required for model detection is greatly reduced. Moreover, since the image is not excessively compressed, even if the proportion of the hand region in the original image is small, the detection accuracy can still be guaranteed. In this scheme, how to roughly locate the hand region is the key. A feasible scheme is as follows: in the continuous frame tracking process, the position box of the palm is determined according to the gesture detection result of the previous frame, and then the next frame image is cropped according to the position box. The small image after cropping is sent into the model for gesture detection. However, this scheme cannot cope with all scenarios, especially in the initial stage of gesture detection or when the gesture is lost or there is no gesture in the gesture detection. In this scheme, the detection region of the next frame depends heavily on the detection result of the previous frame. If the hand moves quickly or the previous frame does not detect the gesture, it is difficult to determine the accurate detection region of the next frame. Therefore, when the gesture is continuously detected, the frame loss is serious. Or, when there is no gesture at a certain moment, there is no gesture position of the previous frame to rely on, and full image detection still needs to be switched to in order to find a new gesture. Therefore, this scheme also cannot effectively balance the detection accuracy and the detection efficiency. Considering this situation, the current frame can be based on the current frame, and the gesture detection region can be roughly located by detecting the position of the human feature point in the current frame. Then, the image is cropped according to the gesture detection region, and the cropped local image is sent into the gesture detection model for detection. This scheme does not depend on the detection result of the previous frame, and the scope of application of the scheme is more extensive, and the detection accuracy and the detection efficiency are both high.

[0047] In addition, it is found through test analysis that, in the gesture tracking process, the target object does not make an effective gesture at all times, and there are some periods when no control gesture is made, that is, the gesture of the target object is an invalid gesture. Especially in a multi-person control scenario, for example, in a team somatosensory relay game, one of the members is in a competition process, and the other members are in a preparation state. In the related art, in order to ensure detection accuracy, each frame of image is sent into a gesture detection model for detection, which will cause a large amount of unnecessary detection, not only occupying computing resources, but also possibly causing a decrease in detection efficiency. It is found through test analysis that an effective control gesture made by the target object usually needs to raise the hand, that is, only in the state of raising the hand can an effective gesture be made. The gesture made in the state of not raising the hand is usually an invalid gesture, that is, does not meet the gesture activation condition. The gesture made after the hand is raised can be an effective gesture, that is, only after the hand is raised can the gesture activation condition be met. As shown in FIG. 1c, the gesture made by the user with the hand raised is an effective gesture. It can be understood that different effective gestures can correspond to different control instructions. Therefore, in the embodiments of the present application, it can also be determined whether the gesture meets the activation condition, and only when it is determined that the gesture meets the activation condition, gesture detection is performed. In this way, invalid detection can be effectively reduced, and the detection efficiency is further improved on the basis of ensuring the detection accuracy.

[0048] Based on the above inventive concept, the embodiments of the present application provide a gesture detection method, device, terminal equipment, computer readable storage medium and computer program product different from the related art.

[0049] First, the embodiment of the present application provides a gesture detection method. The method can be used in various scenarios where gesture detection is desired. Including but not limited to: gesture detection / recognition in virtual reality (VR) and augmented reality (AR), gesture detection / recognition in smart home devices (for example, a television product with gesture recognition, users can complete control operations such as moving the television screen, selecting confirmation, switching pages, zooming and rotating through gestures such as palm waving and fist grabbing; for example, a 3D body sensing camera integrates a camera and an infrared sensor that can recognize user actions, users can control characters in the game by simply twisting their body parts), gesture control in the automotive industry, gesture detection / recognition in medical device control (for example, doctors can control the operation of medical devices such as surgical instruments and diagnostic equipment through gestures to improve work efficiency and patient experience), gesture detection / recognition in the education industry (for example, teachers can more intuitively teach and demonstrate by using gesture-controlled teaching aids or demonstration devices), and the like. The scheme can better balance detection accuracy and detection efficiency, that is, it can ensure high gesture detection accuracy and gesture detection efficiency. For simplicity, the specific implementation of the gesture detection method of the embodiment of the present application will be described below with the example of gesture control of a smart television.

[0050] As shown in FIG. 2, the gesture detection method provided by the embodiment of the present application includes the following steps S220 and S240.

[0051] Step S220, determining a first position of a feature point in a first image of a target object, wherein the feature point corresponds to a body key point of the target object.

[0052] In the embodiment of the present application, the target object can be one or more objects for which gesture detection is desired. The object can be any suitable type of object capable of making gestures, including but not limited to various age groups of human bodies, animals, and hands of movable robots. Of course, in some special examples, the object can also be an animal capable of making gestures. For simplicity, the following will be described with the example of the object being a human body. The target object in the embodiment of the present application can be one or more people for which gesture detection is desired. For example, in the scenario where a user controls a smart television with gestures, the target object can be a human body in front of the television screen.

[0053] Before step S220, a first image of the target object can be acquired. For example, the first image can be an image of a human body in front of a screen captured by a visual sensor of a smart television in real time. The visual sensor can continuously capture images in a visual range at a preset frequency, and the first image can be an image containing a human body captured at a current time, for example, a current frame. The first image can be a color image, or a grayscale image or a binary image. The first image can be an original image captured by the visual sensor, or an image obtained by performing image preprocessing on the original image. The image preprocessing can include image cropping, image filtering, image brightness adjustment, contrast adjustment, and other image processing operations facilitating key point detection.

[0054] In this step, the first positions of the feature points in the first image can be determined by using various suitable methods. For example, when the target object is a human body, the feature points determined in this step can correspond to human body key points, and therefore various suitable human body key point detection algorithms can be used to implement this step, including but not limited to traditional computer vision processing algorithms (such as edge detection, corner detection, template matching, etc.), key point detection algorithms based on deep learning, etc. Specifically, these algorithms can be used to locate the positions of the feature points corresponding to the human body key points to be detected in the first image. In one specific example, a key point detection model can be pre-trained, and in this step, the first image can be input into the model to obtain the probability that each pixel point belongs to a key point, and then the feature points corresponding to each key point in the first image can be located.

[0055] Specifically, the feature points determined in this step can at least include feature points corresponding to hand key points of the target object. The hand key points can be arbitrarily set according to actual needs, and specifically can be key points that are not easily occluded. For example, key points representing the centers of each wrist, which will be referred to as wrist feature points hereinafter. Of course, the hand key points can also include key points such as palm center points, wrist joint points, fingertip points, palm edge points, etc. In addition, in some examples, the feature points determined in this step can also include feature points corresponding to face key points, upper limb key points, torso key points, lower limb key points, etc. of the target object. For example, as shown in FIG. 4a, the face key points can include a left eye center point, a right eye center point, a nose tip point, a left earlobe point, and a right earlobe point. The upper limb key points can include a left shoulder point, a right shoulder point, a left elbow point, and a right elbow point. The lower limb key points can include a left thigh root point, a right thigh root point, a left knee point, a right knee point, a left ankle point, and a right ankle point. For example, the key point detection model can detect the first positions of 17 feature points corresponding to the 17 key points shown in FIG. 4a in the first image. For example, the coordinates of the pixels corresponding to the 17 key points can be obtained.

[0056] At step S240, a pre-processing operation of gesture detection on the target object is performed according to the first position. The pre-processing operation includes a first pre-processing operation and / or a second pre-processing operation. The first pre-processing operation is used to determine a gesture detection region in the second image of the target object, so as to perform gesture detection on the gesture detection region. The second pre-processing operation is used to determine whether the gesture of the target object meets an activation condition, so as to perform gesture detection when it is determined that the gesture of the target object meets the activation condition.

[0057] In the embodiments of the present application, the pre-processing operation of gesture detection on the target object can be performed according to the first position of the feature point in the first image obtained at step S220 by using various suitable methods. It can be understood that the pre-processing operation is an operation for facilitating gesture detection.

[0058] In one example, the pre-processing operation can only include the first pre-processing operation. Taking a human body in front of a television screen as an example, the first pre-processing operation is used to determine a gesture detection region in the second image of the human body, i.e., to determine a local image region in the second image corresponding to the hand of the human body, so as to facilitate subsequent gesture detection on the local image region. That is, this scheme is to determine a gesture detection region corresponding to the hand of the human body according to the position of the feature point of the human body in the first image. In this way, subsequent gesture detection can be performed only on the gesture detection region, which can avoid interference from other image regions in the image, avoid invalid detection, reduce the amount of detection calculation, and improve the detection efficiency. For example, the second image can be cropped to obtain a hand small image according to the determined gesture detection region, and then only the hand small image is sent into a trained gesture detection model or gesture recognition model, which can accurately and quickly output the position of the palm region, and the gesture recognition model can also output the category of the recognized gesture. In this way, the subsequent control instruction represented by the gesture can be determined according to the category and the corresponding relationship between the category and the instruction, and the corresponding action can be performed according to the control instruction.

[0059] It should be noted that the second image is an image captured at a first time point when the first position of the feature point in the first image is determined. In some cases, the first time point is later than the capturing time point of the first image. In this case, the second image and the first image are different images. In other cases, in the case of a faster processing speed, the time point of performing step S220 is approximately equal to the time point of performing the first preprocessing operation, i.e., real-time detection of each frame of image can be achieved, and the first image and the second image can be the same image. That is, the first image is a current frame (for example, the i-th frame captured after the smart television startup action is recognized) captured in real time at the time point of performing step S220, and the second image is a current frame (for example, the j-th frame captured after the smart television startup action is recognized) captured in real time at the time point of performing the first preprocessing operation. In the real-time detection scenario, i=j, and the first image and the second image are the same image; in the scenario where the processing speed of the hardware is limited and does not meet the real-time detection requirement, i

[0060] In the embodiments of the present application, a plurality of suitable methods can be used to determine the gesture detection region. Exemplarily but not limitatively, the gesture detection region can be determined according to the positions of the feature points corresponding to the determined hand key points. For example, in the example in which the feature points include wrist feature points, the corresponding positions in the second image, such as the second positions, can be determined according to the first positions of each wrist feature point. Then, a larger region can be obtained by expanding a certain size around the second positions according to a preset expansion rule, as the gesture detection region. The preset expansion rule can be set arbitrarily according to actual requirements. For example, the expansion size corresponding to different expansion directions can be determined according to the first positions of a plurality of feature points. For example, the expansion size can be determined according to the distance between two eye feature points or ear feature points of the target object, and the expansion size can also be determined according to the distance between the elbow feature point and the wrist feature point. Exemplarily, the expansion size can be a multiple of the calculated distance. Then, the gesture detection region can be obtained by expanding a corresponding size in different expansion directions around the second positions. For another example, the size of the target face region in the first image, such as the face width, can be obtained by face detection in advance, and then the expansion size corresponding to different expansion directions can be determined according to the face width.

[0061] In another example, the pre-processing operation can only include the second pre-processing operation. Taking an example where the target object is a human body in front of a TV screen, the second pre-processing operation can be an operation of determining whether the gesture of the human body meets the activation condition according to the first positions of the feature points. Further, in a case where it is determined that the activation condition is met, gesture detection can be performed on the image region of the human body; in a case where it is determined that the activation condition is not met, the image region of the human body can be filtered out, i.e., gesture detection is not performed on the image region. It can be easily understood that this scheme can filter out unnecessary image regions for detection, significantly reduce the computing cost required for gesture detection without reducing the detection accuracy, thereby significantly improving the gesture detection efficiency.

[0062] Specifically, various suitable methods can be used to determine whether the gesture of the human body meets the activation condition according to the first positions of the feature points. As described above, only a gesture made in a case where the hand is raised can be a valid gesture, i.e., meets the gesture activation condition; a gesture made in a case where the hand is not raised is usually an invalid gesture, i.e., does not meet the gesture activation condition. Therefore, in this step, various suitable judgment logics can be used to determine whether the corresponding hand is raised according to the first positions of the feature points. Alternatively, the feature points determined in step S220 can include wrist feature points and elbow feature points, and in this step, whether the corresponding hand is raised can be determined according to the relative positional relationship between the wrist feature points and the elbow feature points of the same hand. For example, the vertical coordinates (e.g., with the top-left corner of the image as the origin) of the wrist feature points and the elbow feature points in the image can be compared. Illustratively, the difference between the vertical coordinate of the wrist feature point and the vertical coordinate of the elbow feature point can be calculated, and if the difference is greater than a difference threshold value (which can be set according to actual needs), it can be determined that the hand is not raised, and vice versa. In a specific example, the difference threshold value can be 0. If the vertical coordinate of the wrist feature point is greater than the vertical coordinate of the elbow feature point, it can be indicated that the wrist feature point is below the elbow feature point, i.e., the hand is not raised; on the contrary, if the vertical coordinate of the wrist feature point is less than or equal to the vertical coordinate of the elbow feature point, it can be indicated that the wrist feature point is above the elbow feature point, i.e., the hand is raised. Alternatively, the feature points determined in step S220 can include wrist feature points and chest feature points, and in this step, whether the corresponding hand is raised can be determined according to the relative positional relationship between the wrist feature points and the chest feature points. Similarly, if the vertical coordinate of the wrist feature point is greater than the vertical coordinate of the chest feature point, it can be determined that the hand is not raised; on the contrary, if the vertical coordinate of the wrist feature point is less than or equal to the vertical coordinate of the chest feature point, it can be determined that the hand is raised. Of course, other relative positional relationships between hand feature points and feature points of corresponding body parts can also be used to reasonably determine whether the hand is raised, and further determine whether the gesture of the human body meets the activation condition.

[0063] In yet another example, the pre-processing operation can include a second pre-processing operation and a first pre-processing operation executed in sequence. And the execution of the first pre-processing operation depends on the result of the second pre-processing operation. For example, first, it can be determined according to the first position of the feature point in the first image whether the gesture of the target object (or the gesture of a certain hand of the target object) meets the activation condition. If the activation condition is met, then further determine the gesture detection region for the target object (or a certain hand of the target object) according to the first position of the feature point, and then perform gesture detection on the gesture detection region (for example, crop the image according to the gesture detection region and send it into the gesture detection model for detection). This scheme can further improve the detection efficiency and detection accuracy of the model.

[0064] The above gesture detection method of the embodiments of the present application first accurately determines the first position of the feature point corresponding to the body key point in the first image of the target object before gesture detection, and then performs the pre-processing operation of gesture detection on the target object according to the first position. In one case, since the pre-processing operation can accurately determine the gesture detection region in the second image of the target object, subsequent gesture detection can be performed only on the gesture detection region, so the detection efficiency can be improved on the basis of ensuring good detection accuracy, and the effective balance of detection accuracy and detection efficiency is achieved. In another case, since the pre-processing operation can accurately determine whether the gesture of the target object meets the activation condition, and then subsequent gesture detection can be performed only on the image region of the target object in the case where it is determined that the gesture of the target object meets the activation condition. This scheme can filter out unnecessary image regions for detection, significantly reduce the computing power cost required for gesture detection without reducing the detection accuracy, and significantly improve the detection efficiency, thereby achieving a balance between detection accuracy and detection efficiency. Therefore, the overall gesture detection scheme of the embodiments of the present application can effectively filter out images or image regions that may interfere with gesture detection or are unnecessary for gesture detection by performing a simple and reasonable image processing process before gesture detection, so as to ensure high detection accuracy and detection efficiency.

[0065] In one embodiment, the gesture detection method further includes step S210.

[0066] Step S210, performing face detection on the first image to obtain a target face region corresponding to the face of the target object in the first image.

[0067] The target object can be a target human body, and the face detection can be a human face detection. The target face region can be a target human face region. A trained human face detection model can be used to detect the target human face from the input image and output the position and size of the target human face, the feature points, and the human face similarity score. For example, in step S210, the first image shown in FIG. 3a can be input into any suitable human face detection model (including but not limited to R-CNN series models, YOLO models, SSD models, RetinaFace models, etc.), and the model can output the position of the human face region. For example, the detection information of the human face region (the solid line rectangular position box in the figure) can be obtained. For example, the coordinates of the upper left corner of the rectangular position box in the figure and the width and height of the rectangular position box can be obtained. The rectangular position box can contain the main region of the target human face.

[0068] Step S220 determines the first positions of the feature points in the first image of the target object, including step S221 and step S222.

[0069] Step S221 regionally expands the target face region to obtain a body region corresponding to the body of the target object.

[0070] In step S221, a variety of suitable methods can be used to regionally expand the target face region to obtain a body region corresponding to the body of the target object. The target face region can be regionally expanded according to the position of the target face region and the size of the target face region, so that the expanded body region contains the active region of the human hand. Specifically, it can include the whole body region of the human body or the body region above the legs. Alternatively, the target face region can be expanded in a plurality of expansion directions around the target face region by a corresponding size according to the width of the target face region. Alternatively, the target face region can be expanded in a plurality of expansion directions around the target face region by a corresponding size according to the height of the target face region. The size of the target face region expanded in different expansion directions can be the same or different. Alternatively, the target face region can be expanded in a plurality of expansion directions around the target face region by a corresponding size according to the width and height of the target face region.

[0071] As shown in FIG. 3a, the human face region (the solid line rectangular position box in the figure) detected by the human face detection can be regionally expanded to obtain a body region (the dashed line rectangular position box in the figure). As shown in FIG. 3b, for the case where the first image includes multiple human bodies, a body region can be determined for each human body. That is, the position box of each body region can only include the body region of one human body.

[0072] Step S222 performs body key point detection on the body region to obtain the first positions of the feature points.

[0073] The first image can be cropped according to the determined body region to obtain a local image of the body region, and the local image can be input into the human key point detection model to obtain a heat map of the probability that each pixel in the local image belongs to each feature point. According to the pixel value of each pixel in the heat map, the coordinates of each feature point in the local image can be obtained. For example, the model can output the coordinates p(x p ,y p ) and the confidence score of each feature point in the local image shown in FIG. 4a. Then, the coordinates of each feature point can be converted to obtain the coordinates of each feature point in the first image. Taking the human body in FIG. 3a as an example, the coordinates of the feature points of the human body in FIG. 3b can be obtained by using the following calculation method. First, a confidence threshold th can be set, and only a feature point with a score greater than th is considered to be a reliable feature point, and a feature point with a score less than the threshold is directly discarded. Exemplarily, FIG. 4a can be a local image cropped from an image according to a body region (a virtual rectangular frame region in FIG. 3b) expanded through a face region. According to the position of the body region in the first image and the coordinates of the feature points in the local image, the coordinates of the feature points in the first image can be calculated. When the score is greater than th, assuming that the rectangular position frame of the body region is e_rect(e_x, e_y, e_w, e_h), the coordinates of a feature point in the local image are p(x p ,y p ), and the coordinates of the feature point in the first image are p1(x p’ ,y p’ ), then x p’ =x p +e_x; y p’ =y p +e_y.

[0074] The schemes for determining the positions of the feature points of a human body in a human body image in the related art mainly include the following two kinds: the first kind is to take an original image as an input image, and regard the original image as an image of a unique human body regardless of how many human bodies are in the original image, and then determine the positions of the feature points of each part of the human body recognized by a human key point detection model; and the second kind is to first detect the overall regions of all human bodies in the original image by a target detection model, that is, obtain the position frames of the entire body regions of each human body, and then send the local images in the image regions in each position frame after being cropped into the human key point detection model to obtain the positions of the feature points of each human body. However, in actual applications, both of the two kinds of schemes have many disadvantages and cannot well meet the needs of gesture detection. For the first kind of scheme, when the original image is taken as the input image, due to the limitation of the human key point detection model, only one image can be taken as the image of a human body target, and when there are multiple human bodies in the image, the model will report an error or only infer the feature points of one person, and discard the remaining human body targets. In this way, when other people enter the scene, the correct result cannot be obtained. For the second kind of scheme, when the human body targets in the original image are detected by the target detection model, the model output result is limited by the positions of the human bodies. When the body regions of multiple people intersect with each other or the distances between two people are far, the model cannot detect the position regions of each person, and the human body target detection model has a long inference delay and occupies a large system resource space, which is not conducive to the running of the program.

[0075] According to the above scheme of the embodiment of the present application, the face detection result can be used to quickly expand the body region of each target object from the face region of each target object, and then the key point detection can be performed on the body region of each target object to determine the gesture detection region. For the case that the target object is a human body, the local image of each human body can be locked by face detection, and then the locked human body local image can be cropped into a single small image and sent to the human key point detection model. Compared with the scheme in the related art that can only detect one human body in an original image or can only add a human body detection target model to detect multiple people under limited conditions, the above scheme of the embodiment of the present application can accurately and in real time detect the feature points of multiple human bodies in an image under various scenes. The scheme of the embodiment of the present application has a wider application range and higher detection efficiency and accuracy.

[0076] In the above scheme, the target face region of the target object is obtained by performing face detection on the first image, and then the body region of the body of the target object is quickly and accurately obtained by performing region expansion on the target face region. Furthermore, the first positions of the feature points are obtained by performing body key point detection on the body region. It can be understood that face detection is generally faster than full-body detection because it only focuses on a small region in the image. Once the face is quickly located, the search region can be expanded on this basis, thereby speeding up the overall detection process. The body region can be quickly and accurately obtained by expanding the face region. In addition, by first detecting the face and then only detecting the body key points in the limited region around the face, the amount of image data that needs to be processed can be significantly reduced, thereby reducing the consumption of computing resources. In addition, this scheme can also facilitate the locking of different target objects. Even in the case where different target objects are close to each other, key point detection can be performed on different target objects respectively, and the feature points corresponding to each target object can be accurately obtained. Thus, for each target object, the corresponding gesture detection region can be effectively and accurately locked. The detection efficiency and detection accuracy can be further improved.

[0077] In an embodiment, the step S221 of performing region expansion on the target face region to obtain the body region of the body of the target object comprises: expanding the target face region along the image width direction according to the width of the target face region, and expanding the target face region along the image height direction according to the height of the target face region to obtain the body region.

[0078] Referring to FIG. 3b and FIG. 3c, all the faces in FIG. 3b can be detected by using the face detection model, and a face position frame corresponding to each face can be generated. As shown in FIG. 3b, three face position frames corresponding to three human bodies respectively can be obtained. As shown in FIG. 3c, the width of one face position frame is w (i.e. face width) for example, the height is h (i.e. face height) for example, and the coordinate of the top-left corner point of the face position frame is (x, y) for example. The face position frame can be represented as face_rect(x, y, w, h). The width direction of the image can be referred to as the left-right direction of the image, and the height direction of the image can be referred to as the up-down direction of the image. In one example, a human body position frame (hereinafter referred to as a human body frame) of the body region can be obtained by expanding the basic frame of the face position frame (hereinafter referred to as a face frame) by Ln, Rn, Tn and Bn units in the left-right and up-down directions of the image (Ln, Rn, Tn and Bn can be expansion multiples in different expansion directions). The expansion size in the left-right direction can be Ln times and Rn times of the face width w respectively. Ln and Rn can be set according to actual requirements. They can be the same or different. Similarly, the expansion size in the up-down direction can be Tn times and Bn times of the face height h respectively. Tn and Bn can be set according to actual requirements. They can be the same or different.

[0079] In one specific example, the calculation process of the human body frame according to the face frame is as follows:

[0080] Firstly, it is assumed that the human body frame is (e_x, e_y, e_w, e_h), wherein (e_x, e_y) is the coordinate of the top-left corner point of the human body frame, and e_w and e_h are the width and height of the human body frame respectively. Exemplarily, the coordinate of the top-left corner point of the image can be the origin (0, 0), the y-axis direction can be the image height direction downward, and the x-axis direction can be the image width direction rightward. Considering that the top-left corner point of the human body frame cannot exceed the image range, the coordinate of the top-left corner point of the human body frame can be determined according to the following formula: e_x = max{(x - Ln x w), 0}; e_y = max{(y - Tn x h), 0}.

[0081] That is, if the left boundary of the human body frame after expansion according to the expansion multiple exceeds the left boundary of the image, the left boundary of the image is taken as the left boundary of the human body frame; if the upper boundary of the human body frame after expansion according to the expansion multiple exceeds the upper boundary of the image, the upper boundary of the image is taken as the upper boundary of the human body frame.

[0082] Then, the width and height of the human body frame can be calculated by using various suitable methods. The following method 1 and method 2 are two examples.

[0083] Method 1: Firstly, the width e_w of the human body frame is determined. It can be understood that the width of the human body frame = the actual width of the left boundary of the face frame extending to the left side + the width of the face frame + the actual width of the right boundary of the face frame extending to the right side. And the left actual extension width = x - e_x; suppose the right actual extension width = Rn x w; then e_w = x - e_x + w + Rn x w. In order to avoid the human body frame exceeding the pre-defined maximum right boundary, boundary detection needs to be done. The maximum right boundary of the human body frame can be the right boundary of the first image, or a pre-set position left to the right boundary of the first image. The pre-set position can be a position at a certain distance from the right boundary, which can be set arbitrarily according to actual needs, for example, 1 pixel, 10 pixels, etc. For example, assuming the width of the first image is W, the horizontal coordinate of the maximum right boundary of the human body frame can be less than or equal to W. For example, the horizontal coordinate of the maximum right boundary of the human body frame is equal to W - 1. Wherein, 1 represents 1 pixel. It can be understood that setting the maximum right boundary of the human body frame to be equal to W - 1 can ensure that when processing image data, the index range of the array will not be exceeded, thereby avoiding the problem of calculation overflow. If e_x + e_w ≤ W - 1, it means that extending the right boundary of the face frame by Rn times the face width will not exceed the pre-set maximum right boundary of the human body frame, then e_w = x - e_x + w + Rn x w is true. And if e_x + e_w > W - 1, it means that extending the right boundary of the face frame by Rn times the face width will exceed the pre-set maximum right boundary of the human body frame, then e_w = W - 1 - e_x.

[0084] Next, the height e_h of the human body frame can be determined according to similar principles. It can be understood that the height of the human body frame = the actual height of the upper boundary of the face frame extending upwards + the height of the face frame + the actual height of the lower boundary of the face frame extending downwards. Wherein, the actual height of the upper boundary extending upwards = y - e_y; suppose the actual height of the lower boundary extending downwards = Bn x h; then e_h = y - e_y + h + Bn x h. In order to avoid the human body frame exceeding the pre-defined maximum lower boundary (to prevent the human body frame from going out of bounds), boundary detection needs to be done. Assuming the height of the first image is H, the vertical coordinate of the maximum lower boundary of the human body frame can be less than or equal to H. For example, the maximum lower boundary of the human body frame is equal to H - 1. Then if e_y + e_h ≤ H - 1, e_h = y - e_y + h + Bn x h is true. And if e_y + e_h > H - 1, it means that extending the lower boundary of the face frame by Bn times the face width will exceed the pre-set maximum lower boundary of the human body frame, then e_h = H - 1 - e_y.

[0085] Method 2: Assuming the maximum right boundary of the human body frame is the right boundary of the first image, and the maximum lower boundary of the human body frame is the lower boundary of the first image. The right lower corner point (x a ,y a ) of the face frame. Wherein, x a= x + w, y a = y + h. Then, the bottom right corner point (x b , y b ) of the human body frame can be determined according to the following formula: x b = min{(x a + Rn x w), W}; y b = min{(y a + Bn x h), H}.

[0086] Then, the width and height of the human body frame can be further obtained according to e_w = x b - e_x and e_h = y b - e_y.

[0087] The area expansion multiples Ln, Rn, Tn, Bn corresponding to different expansion directions adopted in the above examples can be set to appropriate values according to actual needs. It can be understood that the human body frame determined in the embodiments of the present application is mainly used to estimate the approximate position of the hand region. According to experiments and experience summaries, considering the limit of the hand movement region, the maximum movement region in the width direction of the human body is the width region reached when the two hands are stretched horizontally, and the maximum movement region in the height direction of the human body is the height region reached when the two hands are raised vertically to be completely lowered.

[0088] In order not to lose the hand image, in one example, as shown in FIG. 3d, the expansion proportions of the human body frame corresponding to the full body by expanding the face frame upward, downward, leftward and rightward respectively can be determined as: 3 (Ln), 3 (Rn), 3 (Tn), 8 (Bn), that is, the face frame is expanded by 3 face widths leftward and rightward, respectively, and is expanded by 3 face heights upward and 8 face heights downward, respectively. While considering that the area expansion multiples in FIG. 3d can maximize the guarantee of intercepting the full body, the image is large, and when the morphological recognition of the human body image is used to determine the palm position in order to make gesture detection, the features of the feet are not needed, that is, only the position of the upper body containing the hands is needed. In another example, as shown in FIG. 3e, in order to improve the image cropping and detection efficiency, a part of the bottom expansion can be removed, and the expansion proportions of the human body frame corresponding to the full body by expanding the face frame upward, downward, leftward and rightward respectively can be determined as: 3 (Ln), 3 (Rn), 3 (Tn), 4 (Bn), that is, the face frame is expanded by 4 face heights downward, so that the human body frame does not contain the calf region. In this scheme, the local image of the upper body region of the human body obtained by cropping according to the human body frame can be subjected to human key point detection, which can improve the detection efficiency and save the calculation amount of key point detection.

[0089] It can be understood that the above scheme has simple execution logic and small amount of calculation. The body region can be obtained in real time and accurately by region expansion on the face region of the target object. The efficiency and accuracy of key point detection can be improved by performing body key point detection on the body region. The accuracy and efficiency of subsequent gesture detection are further improved. This helps to realize real-time and accurate gesture tracking of the target object.

[0090] In addition, with reference to FIG. 3b again, when there are multiple people in the visual range of the visual sensor, that is, there are multiple human bodies in the first image, even if the bodies of different human bodies overlap, according to the face frame detected for each human body, the human body frame is obtained by expansion according to the above region expansion method, and the first image is cropped into multiple small-size input images according to each human body frame. Each small-size input image can then be sent separately into the human body key point detection model for inference to obtain a detection result. In this way, as long as the face frame exists, a local image centered on each human body can be intercepted as an input image for human body key point detection, and feature points for each human body can be obtained quickly and accurately. That is, the body region of each human body in FIG. 3b can be quickly and accurately divided according to the above face detection + region expansion method. The key point detection for each body region can accurately detect the feature points of each human body, and is not easily disturbed by the human body next to it. Therefore, the application range of this method is also wider, and the body region of each target object can be quickly and accurately determined in multiple scenes.

[0091] In addition, since this scheme can efficiently and accurately locate the body region of each target human body in the first image. Therefore, the feature points of each human body can be detected in the same image, and the gesture detection region of each human body can be accurately obtained, that is, accurate positioning can be provided for gesture detection in a multiple-person scene, which is very helpful for relocking when a mobile gesture is lost or for use when a gesture is bound to a specific human body. Therefore, this method not only has high detection accuracy, but also has strong versatility. It can achieve target human body locking and tracking only by using a simple face detection model (without adding a human target detection model), and is easy to deploy and apply.

[0092] In an embodiment, in the case where the current mode is a single-target mode, step S210 performs face detection on the first image to obtain a target face region corresponding to the face of the target object in the first image, including the following steps S211-S213:

[0093] Step S211: performing face detection on the first image to obtain an initial face region corresponding to each object;

[0094] Step S212: performing face recognition on each initial face region to obtain identity information of the object corresponding to each face region;

[0095] Step S213, according to the identity information, screening the target face region from each initial face region.

[0096] It can be understood that in order to meet the diversified control needs of users, the system can set multiple gesture control modes, or set different gesture control modes in different periods of gesture control. One of them is a single-person mode. In the single-person mode, multiple people are allowed to appear in the gesture control scene, but only one person who meets the requirements is locked for gesture detection, and others are not detected. In a specific example, in step S211, the first image can be input into the trained face detection model, for example, input into the face detection model, and the face region of each person in the first image is output. For example, the image of 3 people shown in FIG. 3b is input into the face detection model, and the position frame of the 3 faces in the image can be obtained. The area contained in each position frame can be an initial face region, which can be called an initial face region. Then, the image can be cropped according to the three initial face regions to obtain a local image of each initial face region, that is, three local images are obtained. In step S212, a variety of suitable methods can be used to perform face recognition on the three local images respectively. For example, the local image of the initial face region of each person can be input into the trained face recognition model to obtain the identity information of each person, that is, the face label information of the three people in the image is obtained. In step S212, the target face region (target face region) can be selected from the three initial face regions according to the face label information. For example, the system can pre-set one of the people as the target person (for example, the face of the user who pre-records the control TV is input, and the initial face region with the same face label as the input user face is selected as the target face region in this step). In subsequent step S221, the body region of the target person can be obtained by only expanding the target face region. In this way, gesture detection can be performed on the target person at least for a period of time. In this way, the gesture of the desired detection person can be quickly and accurately locked according to the actual needs, the accuracy of gesture detection is improved, accurate gesture control is realized, and the algorithm cost consumed by such unnecessary detection can be significantly reduced, further improving the detection efficiency.

[0097] In the above scheme, on the basis of accurately detecting the initial face region of each object in the first image, the face region of the target object can be quickly and accurately locked according to the face recognition result through face recognition. It can be understood that, in the example in which the target object is a human body, face recognition is added on the basis of face detection, which can bind the hand region and the face region of the human body, and accurately distinguish which human body the hand region at different positions in the image belongs to. In this way, the target human body can be quickly and accurately given gesture control permission. Subsequently, gesture detection can be performed only on the gesture detection region of the target human body, and the gesture permission of other people is shielded. That is, through face recognition, the gesture detection region can be quickly and accurately bound to the object making the gesture, which can significantly reduce the probability of gesture misoperation, truly filter out interference from the bottom, and improve the accuracy of gesture control. In general, this scheme binds the gesture detection region to the face recognition result, not only achieves the purpose of meeting the user's individual control needs and enriching the gesture application scenarios, but also significantly reduces the probability of gesture misoperation, thereby significantly improving the accuracy of gesture control and reducing the cost of gesture detection, and greatly improving the efficiency of gesture detection.

[0098] In an embodiment, the preprocessing operation includes a second preprocessing operation, and the feature points include wrist feature points corresponding to the wrist parts of the target object and elbow feature points corresponding to the elbow parts of the target object. Step S240 performs the preprocessing operation on the gesture of the target object according to the first position, including step S241.1 and step S241.2. Step S241.1, for each wrist feature point of the target object, determines the palm orientation of the hand corresponding to the wrist feature point according to the relative positional relationship between the wrist feature point and the elbow feature point associated with the wrist feature point. Step S241.2, according to the palm orientation of each hand, determines whether the gesture of the target object meets the activation condition.

[0099] First of all, it needs to be pointed out that the palm orientation in the application embodiment refers to the orientation of the upper limb of the target object. For example, the target object is a human body, when the forearm of the arm of the human body is drooping, the upper limb of the human body is downward, that is, the palm orientation is downward; when the forearm of the arm is raised, the upper limb of the human body is upward, that is, the palm orientation is upward. Therefore, the palm orientation of the hand is associated with the raised state of the hand. The palm upward corresponds to the raised state of the hand, and the palm downward corresponds to the drooping state of the hand (i.e. the hand is not raised).

[0100] For the same target object, the palm orientations of the two hands can be the same or different. In FIG. 4b, the palm orientations of the two hands of the target human body are both downward, in FIG. 4d, the palm orientations of the two hands of the target human body are both upward, and in FIG. 4c, the palm orientation of the left hand of the target human body is upward, and the palm orientation of the right hand is downward.

[0101] As mentioned above, in the gesture tracking process, the target object does not make valid gestures all the time, and there are some time periods when no control gestures are made. Generally, a valid gesture can only be made when the hand is raised. Gestures made when the hand is not raised are usually invalid gestures, which do not meet the gesture activation condition. Only gestures made when the hand is raised can meet the gesture activation condition.

[0102] The feature points of the target object determined in step S220 include the elbow feature points and the wrist feature points. For each human body, the left elbow feature point and the left wrist feature point corresponding to the left hand, and the right elbow feature point and the right wrist feature point corresponding to the right hand can be determined. The elbow feature point and the wrist feature point corresponding to the same hand have a correlation relationship. The labels of different feature points detected by the key point detection model are different, for example, the serial numbers of the left elbow feature point and the right elbow feature point are different. Therefore, the correlation relationship between the elbow feature point and the wrist feature point and the corresponding hand (for example, corresponding to the left hand or the right hand) can be determined according to the label of the feature point. It can be understood that the relative position relationship between the elbow feature point and the wrist feature point corresponding to the same hand can accurately represent the palm orientation of the hand.

[0103] The feature points of the target object obtained in step S220 are, for example, the 17 feature points shown in FIG. 4a. In step S241.1, the coordinates of the wrist feature point and the elbow feature point corresponding to each hand can be searched from the 17 feature points. Then, for each hand, the relative position relationship between the wrist feature point and the elbow feature point can be determined. Then, according to the relative position relationship, the palm orientation of the hand corresponding to the wrist feature point is determined. The relative position relationship can have various representation methods, as long as it can reflect the hand raised state. Exemplarily but not limitatively, the relative position relationship of the two can be directly represented by the difference of the longitudinal coordinates of the two. For example, the longitudinal coordinate difference can be obtained by subtracting the longitudinal coordinate of the elbow feature point from the longitudinal coordinate of the wrist feature point corresponding to the same hand. Then, according to the difference, the palm orientation of the corresponding hand can be determined. For example, a suitable difference threshold value can be set in advance. Then, in the case that the difference is greater than the difference threshold value, it can be determined that the palm orientation of the hand is downward, that is, the corresponding hand is not raised. Further, in step S241.2, it is determined that the gesture of the corresponding hand does not meet the activation condition. Conversely, then, in the case that the difference is less than or equal to the difference threshold value, it can be determined that the palm orientation of the hand is upward, that is, the corresponding hand is raised. Further, in step S241.2, it is determined that the gesture of the corresponding hand meets the activation condition. In this way, before gesture detection, it can be quickly determined that the left hand gesture of the human body in FIG. 4c does not meet the activation condition, while the right hand gesture meets the activation condition.

[0104] In the above scheme, for each wrist feature point of the target object, the palm orientation of the hand corresponding to the wrist feature point can be quickly and accurately determined according to the relative position relationship between the wrist feature point and the elbow feature point associated with the wrist feature point, and then it can be determined in real time and accurately whether the gesture of the target object meets the activation condition before gesture detection. In this way, in the subsequent step, gesture detection / recognition can be performed only on the gesture detection area of the hand that meets the activation condition. This scheme has a small amount of calculation, and therefore can further reduce the cost of gesture detection and improve the efficiency of gesture detection.

[0105] In an embodiment, the pre-processing operation further includes a first pre-processing operation. The step S240 of performing the pre-processing operation on the gesture of the target object according to the first position further includes a step S241.3.

[0106] The step S241.3 determines the gesture detection area in the second image of the target object according to the first position, on the condition that the gesture of the target object meets the activation condition.

[0107] In this scheme, it can be determined whether the gesture of each hand of each target object meets the activation condition, and then the gesture detection area is determined only for the hand that meets the activation condition. This scheme further reduces the amount of calculation, and therefore can further reduce the cost of gesture detection and improve the efficiency of gesture detection.

[0108] In an embodiment, the pre-processing operation includes a first pre-processing operation, and the feature point includes a wrist feature point corresponding to the wrist of the target object. The step S240 of performing the pre-processing operation on the gesture of the target object according to the first position includes a step S242.

[0109] The step S242 determines the gesture detection area according to the second position of the wrist feature point and the first size of the target face region in the first image. The target face region corresponds to the face of the target object, the corresponding position of the second position in the second image falls in the gesture detection area, and the size of the gesture detection area is positively correlated with the first size.

[0110] Referring to FIG. 4b, the target object is a target human body, and the target face region is a target face region. The first size of the target face region can include the width of the target face region and / or the height of the target face region. Of course, other sizes of the target face region can also be included. Hereinafter, the first size is taken as an example of including the width of the target face region. First, the width of the target face region can be determined in a variety of suitable ways.

[0111] In an embodiment, the width of the target face region can be determined according to the face detection result. In this example, the feature points determined in step S220 can not include the facial feature points. For example, the face detection model can output a rectangular position frame for each face region, and the width of the rectangular position frame can be the face width.

[0112] In another embodiment, the first dimension includes the width of the target face region, and the feature points further include eye feature points corresponding to the eyes of the target object and / or ear feature points corresponding to the ears of the target object. The method further includes determining the width of the target face region according to the relative positional relationship between the two eye feature points and / or the relative positional relationship between the two ear feature points.

[0113] For example, the width of the target face region can be determined according to the position of the facial feature points in the determined feature points. Specifically, the width of the target face region can be determined according to the relative positional relationship between the two eye feature points and / or the relative positional relationship between the two ear feature points. For example, the width of the target face region can be determined according to the distance between the two eye feature points. The distance can be the Euclidean distance between the two points, or the distance between the coordinate components in the image width direction. Those skilled in the art can understand the specific implementation of this scheme, and for the sake of brevity, will not be repeated here.

[0114] The above method of determining the width of the target face region according to the relative positional relationship between the facial feature points is more accurate, so that the gesture detection region can be accurately determined, and the accuracy of gesture detection is improved. Moreover, this scheme does not depend on the face detection result, so that the scalability of the scheme is better.

[0115] After determining the first dimension of the target face region, a variety of suitable methods can be used to determine the gesture detection region according to the first dimension and the coordinates of the wrist feature points. For example, but not limited to, a position of the hand wrist feature point can be taken as a reference position, and a corresponding dimension can be expanded in multiple directions around the reference position. The dimension can be equal to a multiple of the first dimension, for example, can be equal to a multiple of the width of the target face region, so that the expanded gesture detection region contains the possible activity region of the hand.

[0116] According to the position of the wrist feature point and the first size of the target face region, the gesture detection region of the hand of the target object can be quickly and accurately determined. In addition, the gesture detection region can be directly obtained by key point detection on the current frame, and thus the gesture detection region is not dependent on the gesture detection result of the previous frame, and can be applied to various gesture detection scenarios. In particular, in a gesture tracking scenario, the gesture detection region can be relatively accurately pre-positioned in real time, and the situation of gesture loss can be greatly avoided. Therefore, the robustness of the scheme is better. In addition, the execution logic of the scheme is relatively simple, and thus the detection efficiency is also relatively high.

[0117] In an embodiment, the step S242 of determining the gesture detection region according to the second position of the wrist feature point and the first size of the target face region in the first image comprises steps S242.1, S242.2 and S242.3.

[0118] In the step S242.1, a third position in the second image is determined according to the second position of the wrist feature point, wherein the position of the third position in the second image is the same as the position of the second position in the first image. In the step S242.2, the third position is respectively expanded in a plurality of expansion directions around the third position by a corresponding second size on the second image to obtain a plurality of expansion boundaries, wherein the second size corresponding to each expansion direction is equal to the first size corresponding to the expansion direction multiplied by a preset multiple corresponding to the expansion direction. In the step S242.3, the gesture detection region is determined according to the enclosed region of the plurality of expansion boundaries.

[0119] Firstly, in the example that the first image and the second image are the same image, the position of the wrist feature point in the second image can be directly located. In the example that the first image and the second image are different images, the third position can be located in the second image at the same coordinate position. Then, the gesture detection region can be obtained by expanding the third position through the expansion boundaries. It can be understood that the method of expanding the gesture detection region from the third position is similar to the method of determining the body frame from the face frame. For the sake of brevity, only one specific example is listed below for illustration.

[0120] For example, referring to FIG. 4b, a region is extended as a gesture region based on the left wrist feature point and the right wrist feature point of the human body in the figure, respectively. Taking the left hand as an example, suppose that the left wrist feature point coordinate is p1(x1, y1), the left elbow feature point coordinate is p2(x2, y2), the left eye feature point is p3(x3, y3), the right eye feature point is p4(x4, y4), and the position frame (for example, called palm frame) of the hand extension region is h_rect(x*, y*, width, height). The width and height are the width and height of the palm frame, respectively. Then, the target face width D can be calculated first. D2= (x3-x4)2+ (y3-y4)2.

[0121] Exemplarily, the plurality of extension directions can include four directions of up, down, left and right of the third position. The second size corresponding to each of the four extension directions is equal to the first size corresponding to the extension direction multiplied by the preset multiple corresponding to the extension direction. The preset multiples corresponding to the four extension directions of up, down, left and right are T_n, B_n, L_n and R_n, respectively. T_n, B_n, L_n and R_n can be set arbitrarily according to actual requirements, and their values can be the same or different. In one example, T_n=2, B_n=L_n=R_n=1. That is, the second size of the third position corresponding to the p1 coordinate in the second image is extended to the left and right by the width of D, upward by 2D, and downward by 1D.

[0122] Specifically, first, the horizontal coordinate x* and the vertical coordinate y* of the top-left corner point of the extended palm frame can be calculated: x * = max{(x1-D), 0}; y * = max{(y1-D), 0}.

[0123] Then, the width of the palm frame is calculated. Suppose that the actual width is extended by 1D on the left and right, and the frame width is 2D, that is, width=2D. To prevent the palm frame from exceeding the boundary, a boundary judgment is needed. Suppose that the width of the first image is W, and the maximum right boundary of the palm frame is equal to W-1. If x1+width≤W-1, then width=2D is true. If x1+width>W-1, then width=W-1-x1.

[0124] Similarly, the height of the palm frame is calculated. Suppose that the height of the palm frame is height=3D. To prevent the palm frame from exceeding the boundary, a boundary judgment is needed. Suppose that the height of the first image is H, and the maximum lower boundary of the palm frame is equal to H-1. If y1+height≤H-1, then height=3D is true. If y1+height> H-1, then height=H-1-y1.

[0125] Thus, the palm frame coordinates and size can be calculated: h_rect(x*, y*, width, height).

[0126] After obtaining the palm frame corresponding to each hand, the region contained in the palm frame can be directly used as the gesture detection region of the hand, or the region contained in the palm frame can be further processed to obtain the palm region of the hand.

[0127] The method of extending the second size in multiple directions around the wrist feature point to determine the gesture detection region has small calculation amount, fully considers the range in which the hand can move, and the obtained gesture detection region is more reasonable and accurate. In particular in the gesture tracking scenario, the gesture can be effectively prevented from being lost. Therefore, the gesture detection efficiency and detection accuracy can be significantly improved.

[0128] In an implementation, the feature points further include an elbow feature point of an elbow of the target object, and the multiple extension directions include a first height direction and a second height direction. The method further includes steps S231 and S232.

[0129] In step S231, for each wrist feature point of the target object, a palm orientation of a hand corresponding to the wrist feature point is determined according to a relative position relationship between the wrist feature point and an elbow feature point associated with the wrist feature point. The implementation of this step is similar to the implementation of the aforementioned step S241.1, and thus is not described herein again for brevity.

[0130] In step S232, a first preset multiple corresponding to the first height direction and a second preset multiple corresponding to the second height direction are determined according to the palm orientation. The first preset multiple is greater than the second preset multiple, the first height direction is the same as the palm orientation, and the second height direction is opposite to the palm orientation. For example, if the palm orientation is upward, the first height direction is an upward extension direction, and the second direction is a downward extension direction, and the multiple of the upward extension is greater than the multiple of the downward extension; if the palm orientation is downward, the first height direction is a downward extension direction, and the second direction is an upward extension direction, and the multiple of the downward extension is greater than the multiple of the upward extension.

[0131] The palm position is determined according to the coordinates of the wrist and the elbow. The palm direction is determined to be upward or downward. The origin of the coordinate system is at the top left corner, and the coordinates increase from left to right and from top to bottom. As described above, if the y coordinate of p2 is less than the y coordinate of p1, i.e., y2 < y1, it is considered that the palm is in the lower part. Conversely, when y2 ≥ y1, the hand is basically in a raised state, and it is considered that the palm is in the upper part.

[0132] The first preset multiple and the second expansion multiple can be set arbitrarily according to actual needs. For example, the first preset multiple is 2, the second expansion multiple is 1, and the width of the target face region is D. As shown in FIG. 4c, when it is determined that the palm of the left hand of the human body in the figure faces downward, the height of the expansion downward from the third position is 2D, and the height of the expansion upward from the third position is D; when it is determined that the palm of the right hand of the human body in the figure faces upward, the height of the expansion upward from the third position is 2D, and the height of the expansion upward and downward from the third position is D. In other words, when the hand is raised, the height of the expansion upward is greater than the height of the expansion downward; and when the hand is not raised, the height of the expansion downward is greater than the height of the expansion upward.

[0133] It can be understood that the palm orientation during the gesture made by the target object can generally accurately reflect the moving direction of the current gesture, and thus the moving direction of the gesture can be roughly predicted according to the palm orientation. In the above scheme, the expansion size in the direction of the palm orientation is set to be relatively large, so that more prediction space (detection region) is reserved for the moving direction of the gesture. In this way, even in the case where the hardware processing speed is limited and the wrist feature point of each frame cannot be determined in real time, the gesture moved in the moving direction of the gesture in the subsequent frame can be captured more completely through the sufficient detection region reserved, so that the gesture detection accuracy can be ensured to be relatively high during the gesture movement.

[0134] In an implementation, step S242.3 of determining the gesture detection region according to the enclosed region of the plurality of expansion boundaries comprises the following steps.

[0135] Step S242.31, for each wrist feature point of the target object, the enclosed region determined for the wrist feature point is taken as a hand region of the hand corresponding to the wrist feature point, to obtain two hand regions;

[0136] Step S242.32, whether the two hand regions overlap is judged.

[0137] Step S242.33, if yes, the two hand regions are merged, and the merged region is taken as the gesture detection region.

[0138] Step S242.34, if no, each hand region is taken as a gesture detection region.

[0139] At this time, a region is expanded as a gesture image cropping region with the feature points of the left and right wrists as the reference points. It is assumed that the region is gesture.rect(x, y, width, height), the face width is expanded by T_N, B_N, L_N, and R_N units respectively upward, downward, leftward, and rightward, and the face width is replaced by the distance between the two ears width obtained above. The coordinates and size calculation are consistent with the face expansion mode above, and are not described herein.

[0140] As shown in FIG. 4d, the palm frame of each target object can be obtained according to the region expansion method in the foregoing examples (two dashed boxes in the figure), and the regions contained in the two palm frames are a hand region. Then, a variety of suitable methods can be used to determine whether the two hand regions overlap.

[0141] For example, the hand region A of the left hand is (x a ,y a ,w a ,h a ), wherein (x a ,y a ) is the top-left point coordinate of the region; the hand region B of the right hand is (x b ,y b ,w b ,h b ). Exemplarily but not limitatively, whether the two hand regions overlap can be determined according to the following conditions: ① x a >x b +w b ② x b >x a +w a ③ y a >y b +h b ④ y b >y a +h a

[0142] Wherein, if the position and size of the two hand regions do not satisfy each of the above four conditions, it is determined that the two hand regions overlap; otherwise, it is determined that the two hand regions do not overlap.

[0143] Exemplarily, if it is determined that the two regions overlap, the merged region C (x c ,y c ,w c ,h c ) can be calculated according to the following method.

[0144] Firstly, the bottom-right point coordinate (x a +w a ,y a +h a ) of the hand region A and the bottom-right point coordinate (x b +w b ,y b +h b ) of the hand region B can be calculated.

[0145] Then, X c =min{xa x b};y c =min{y a ,y b}

[0146] Then, the bottom-right corner point coordinates (x d ,y d ) of the region C can be calculated: x d =max{x a +w a ,x b +w b};y d =max{y a +h a ,y b +h b}。

[0147] Finally, w c =x d -x c ;h c =y d -y c .

[0148] As shown in FIG. 4d, the merged region is the region enclosed by the solid line box shown in the figure. Exemplarily, the merged region can be sent into a gesture detection model as a gesture detection region for detection. In this way, the computing power cost can be saved, and the gesture detection efficiency can be further improved.

[0149] In another implementation, the step S242.3 of determining the gesture detection region according to the enclosed region of the plurality of extended boundaries comprises: for each wrist feature point of the target object, taking the enclosed region determined for the wrist feature point as a gesture detection region, to obtain two gesture detection regions, wherein each gesture detection region corresponds to one hand of the target object.

[0150] The method further comprises:

[0151] The step S251 of performing gesture detection on the two gesture detection regions respectively to determine whether there is an effective region of an effective gesture in the two gesture detection regions and the type of the effective gesture;

[0152] The step S252 of calculating a first distance between the center of the effective region and the center of each gesture detection region to obtain two first distances if yes;

[0153] The step S253 of determining the hand corresponding to the gesture detection region with the smaller first distance as the hand of the effective gesture;

[0154] In step S254, the instruction information corresponding to the valid gesture is determined according to the hand making the valid gesture and the type of the valid gesture.

[0155] In some special scenarios, the instruction information represented by the left and right hands of the target object making the same gesture can be different. For example, the instruction information represented by the left hand making a fist can be different from the instruction information represented by the right hand making a fist. Therefore, the hand making the valid gesture needs to be accurately determined. It can be understood that if the distance between the left hand and the right hand of the target human body is close, and one of the two hands makes a valid gesture while the other does not make a valid gesture, in the example that the second image is different from the first image, the two gesture detection regions determined according to the first image overlap. If the local image cropped after merging the two regions is sent to the model for gesture detection, it is difficult to accurately determine which hand makes the valid gesture, and thus it is difficult to accurately determine the corresponding instruction information.

[0156] In the above scheme of the embodiments of the present application, the two hand regions can not be merged, but the two hand regions can be sent to the gesture detection model as a gesture detection region for gesture detection, and the valid gesture region in each gesture detection region is obtained. The valid gesture region can be the position box of the minimum circumscribed rectangle of the palm making the valid gesture. It can be understood that in the case that the two gesture detection regions overlap, the model can output the label information of the valid gesture region and the gesture type for the two gesture detection regions. Then, the distance between the position center of the valid gesture region and the position center of the two gesture detection regions can be calculated as the first distance. Then, the hand corresponding to the gesture detection region with the smaller first distance can be determined as the hand making the valid gesture. For example, if the distance between the position center of the gesture detection region corresponding to the left hand and the position center of the valid gesture region is 10px, and the distance between the position center of the gesture detection region corresponding to the right hand and the position center of the valid gesture region is 15px, it can be determined that the left hand makes the valid gesture. Further, the specific instruction information of the valid gesture can be accurately determined according to the type of the valid gesture and the hand making the valid gesture. This scheme can ensure higher gesture recognition accuracy.

[0157] As mentioned above, in an ideal case, the hardware processing speed is fast enough, the time of performing step S220 is approximately equal to the time of performing the first preprocessing operation, i.e. real-time detection of each frame of image is achieved, and thus the first image and the second image can be the same image. In a scenario of allowing multiple people to control gestures, e.g. in a multiple people mode, multiple people are allowed to activate gestures at the same time, and during the activation period, people are allowed to continue to perform gesture commands. Specifically, as shown in FIG. 5a, in this mode, all faces can be detected by using a face detection model to obtain multiple face regions. Then, region expansion can be performed on each face region to obtain a corresponding body region. Then, the first image can be cropped according to the body regions to obtain multiple body sub-images. Then, human key point detection can be performed on each body sub-image to obtain a feature point set of each body. The feature points can at least include wrist feature points. Then, according to the positions of the wrist feature points in the first image, hand region expansion can be performed to obtain two hand regions of each body as gesture detection regions. The hand regions are cropped into hand sub-images, which are input into the gesture detection model to obtain effective gesture regions in each hand region of each body according to the detection results. If no effective gesture region is included in the two hand regions of each body, it means that the body does not make an effective gesture; if an effective gesture region is included in the two hand regions of each body, it means that the body makes an effective gesture, and the gesture can be activated and the corresponding instruction can be executed.

[0158] In an embodiment, the second image is an image captured at a first time instant at which the first position is determined, and the first time instant is later than the capturing time instant of the first image. The method further includes step S214. In step S214, an offset of the gesture position of the target object in a first time is determined, where the first time is a duration from the capturing time instant to the first time instant.

[0159] In step S240, a preprocessing operation of gesture detection of the target object is performed according to the first position, including: determining a gesture detection region according to the first position and the offset.

[0160] In the case of limited hardware processing speed, the first time when the first position of the feature point in the first image is obtained through key point detection is different from the first time, that is, there is a certain detection delay. In this case, the second image can be another image collected after the first image. For example, the first image is the i-th frame (i≥1), and the second image can be the (i+1)-th frame. The first time can be the duration of collecting the 1st frame of image. The embodiments of the present application do not limit the calculation method of the offset of the gesture position. In a specific example, the average value of the displacement of the gesture position coordinates detected in several consecutive frames can be used as the offset of the gesture position. The offset of the gesture position of the first frame in the first detection is 0 by default. As shown in FIG. 5b, after each gesture detection, the offset of the gesture position detected in the current detection can be calculated (if it is the first detection, the offset of the gesture position is 0 by default) in the case of non-first detection. Then, after the corresponding hand region is determined for each target human body in the next first image by using the face detection, body region expansion, key point detection, and hand region expansion method in sequence, for example, the corresponding hand region 1 is determined for the face region 1, the hand region 1 can be corrected according to the calculated offset, so as to obtain the corrected hand region 1*. In this way, the determined gesture detection region can be more accurate.

[0161] In the above scheme, considering the case of asynchronization between the image for feature point detection and the image for gesture detection, the offset of the gesture position is calculated to correct the gesture detection region in the second image, so that the corrected gesture detection region is more accurate. Therefore, the gesture detection accuracy can be further improved.

[0162] In some examples, under the premise of ensuring detection accuracy, in order to further improve detection efficiency and save computing resources, the steps of face detection and key point detection can be performed at a certain frequency. For example, once every 20 ms or once every 5 frames. For example, the first image is the i-th frame, i∈1, 6, 11, …, 1+5n; n is a positive integer, and the second image is the (i+1)-th frame. For the case where the current frame is the (i+2)-th frame to the (i+4)-th frame, the gesture detection region of the current frame can be determined according to the effective gesture region obtained through gesture detection of the last frame and the offset of the gesture position.

[0163] In addition, as shown in FIG. 5c, in the single-person mode, the gesture detection region can also be corrected by the gesture position offset. Exemplarily, in the single-person mode, only one person is allowed to activate the gesture at the same time, and during the activation, only the person is allowed to continue to execute the gesture command. In this mode, it can be preset which hand gesture of the person will be detected and the other person will not be detected. The face area of all human bodies in the image is detected by using a face detection model, and then the identity information of all face areas can be identified by using a face recognition model to find out the target face and continuously track. The human body feature points detected in the human body area of the target face are generated to form a hand region, and the picture is cropped according to the region and sent to the gesture detection model for detection; if it is not the first time to detect the face, if the gesture of the person has been detected in the previous frames and the gesture movement amount of the previous frame has been calculated, then the hand region generated according to the human body feature points this time needs to be corrected by adding the movement amount of the gesture position of the previous frame. The above process can lock the gesture of a certain person, and the gesture of other people will not be detected, so as to achieve the gesture detection effect in the single-person mode.

[0164] The embodiment of the present application also provides a gesture detection device for executing the method steps in the above method embodiments. The device can be a virtual appliance in a terminal device, run by a processor of the terminal device, or can be the terminal device itself.

[0165] As shown in FIG. 6, the gesture detection device 600 provided by the embodiment of the present application includes:

[0166] The determining module 610 is configured to determine a first position of a feature point in a first image of a target object, wherein the feature point corresponds to a body key point of the target object.

[0167] The preprocessing module 620 is configured to perform a preprocessing operation of gesture detection of the target object according to the first position; wherein the preprocessing operation includes a first preprocessing operation and / or a second preprocessing operation; the first preprocessing operation is used to determine a gesture detection region in a second image of the target object, so as to perform gesture detection on the gesture detection region; the second preprocessing operation is used to judge whether the gesture of the target object meets an activation condition, so as to perform gesture detection in a case where it is determined that the gesture of the target object meets the activation condition.

[0168] In an implementation, the preprocessing operation includes the first preprocessing operation, the feature point includes a wrist feature point corresponding to a wrist part of the target object, and the preprocessing module 620 includes:

[0169] The region determining sub-module is configured to determine a gesture detection region according to the second position of the wrist feature point and a first size of a target face region in the first image; the target face region corresponds to a face of the target object, the corresponding position of the second position in the second image falls in the gesture detection region, and the size of the gesture detection region is positively correlated with the first size.

[0170] In an implementation, the region determining sub-module comprises:

[0171] The first determining unit is configured to determine a third position in the second image according to the second position of the wrist feature point, wherein the position of the third position in the second image is the same as the position of the second position in the first image.

[0172] The first extending unit is configured to extend the third position to a plurality of extension directions respectively by corresponding second sizes on the second image to obtain a plurality of extension boundaries, wherein the second size corresponding to each extension direction is equal to the first size corresponding to the extension direction multiplied by a preset multiple corresponding to the extension direction.

[0173] The second determining unit is configured to determine the gesture detection region according to an enclosed region of the plurality of extension boundaries.

[0174] In an implementation, the feature point further comprises an elbow feature point corresponding to an elbow part of the target object, the plurality of extension directions comprises a first height direction and a second height direction, and the gesture detection device 600 further comprises:

[0175] The palm orientation determining module is configured to, for each wrist feature point of the target object, determine a palm orientation of a hand corresponding to the wrist feature point according to a relative position relationship between the wrist feature point and an elbow feature point associated with the wrist feature point.

[0176] The multiple determining module is configured to determine a first preset multiple corresponding to the first height direction and a second preset multiple corresponding to the second height direction according to the palm orientation; wherein the first preset multiple is greater than the second preset multiple, the first height direction is the same as the palm orientation, and the second height direction is opposite to the palm orientation.

[0177] In an implementation, the second determining unit is specifically configured to:

[0178] For each wrist feature point of the target object, the enclosed region determined for the wrist feature point is taken as a hand region of a hand corresponding to the wrist feature point to obtain two hand regions.

[0179] It is judged whether the two hand regions overlap.

[0180] If yes, the two hand regions are merged, and the merged region is taken as the gesture detection region.

[0181] If no, each hand region is respectively taken as a gesture detection region.

[0182] In an implementation, the second determining unit is specifically configured to:

[0183] For each wrist feature point of the target object, the enclosing region determined for the wrist feature point is taken as a gesture detection region, and two gesture detection regions are obtained, wherein each gesture detection region corresponds to a hand of the target object.

[0184] The gesture detection apparatus 600 further includes:

[0185] The gesture detection module is configured to perform gesture detection on the two gesture detection regions respectively, and determine whether there is an effective gesture region and a type of the effective gesture in the two gesture detection regions.

[0186] The calculation module is configured to, if yes, calculate first distances between centers of the effective gesture region and centers of the two gesture detection regions, and obtain two first distances.

[0187] The hand determination module is configured to determine the hand corresponding to the gesture detection region with the smaller first distance as the hand of the effective gesture.

[0188] The instruction determination module is configured to determine instruction information corresponding to the effective gesture according to the hand making the effective gesture and the type of the effective gesture.

[0189] In an implementation, the preprocessing operation includes a second preprocessing operation, the feature points include wrist feature points corresponding to wrist parts of the target object and elbow feature points corresponding to elbow parts of the target object, and the preprocessing module 620 includes:

[0190] The third determining unit is configured to, for each wrist feature point of the target object, determine a palm orientation of a hand corresponding to the wrist feature point according to a relative position relationship between the wrist feature point and an elbow feature point associated with the wrist feature point.

[0191] The activation judging unit is configured to judge whether the gesture of the target object meets an activation condition according to the palm orientation of each hand.

[0192] In an implementation, the preprocessing operation further includes a first preprocessing operation, and the preprocessing module 620 further includes:

[0193] The fourth determining unit is configured to, under the condition that the gesture of the target object meets the activation condition, determine gesture detection regions in a second image of the target object according to the first position.

[0194] In an implementation, the first dimension includes a width of the target face region, the feature points further include eye feature points of an eye of the target object and / or ear feature points of an ear of the target object, and the gesture detection apparatus 600 further includes:

[0195] a face dimension determination module configured to determine the width of the target face region according to a relative positional relationship between the two eye feature points and / or a relative positional relationship between the two ear feature points.

[0196] In an implementation, the gesture detection apparatus 600 further includes:

[0197] a face detection module configured to perform face detection on the first image to obtain a target face region of a face of the target object in the first image;

[0198] The determination module 610 includes:

[0199] a second expansion unit configured to expand the target face region to obtain a body region of a body of the target object;

[0200] a key point detection unit configured to perform body key point detection on the body region to obtain the first positions of the feature points.

[0201] In an implementation, the second expansion unit is specifically configured to:

[0202] expand the target face region along an image width direction according to the width of the target face region, and expand the target face region along an image height direction according to a height of the target face region to obtain the body region.

[0203] In an implementation, when the current mode is the single-target mode, the face detection module includes:

[0204] a face detection unit configured to perform face detection on the first image to obtain initial face regions of the respective objects;

[0205] a face recognition unit configured to perform face recognition on each of the initial face regions to obtain identity information of the object corresponding to each of the face regions;

[0206] a screening unit configured to screen the target face region from the initial face regions according to the identity information.

[0207] In an implementation, the second image is an image captured at a first time point at which the first positions are determined, and the first time point is later than a capturing time point of the first image, and the gesture detection apparatus 600 further includes:

[0208] The offset determination module is configured to determine an offset of the gesture position of the target object in a first time, wherein the first time is a duration from the acquisition moment to the first moment.

[0209] The preprocessing module 620 is further configured to:

[0210] According to the first position and the offset, the gesture detection region is determined.

[0211] As shown in FIG. 7, the embodiment of the present application further provides a terminal device 700, which comprises at least one processor 710 (only one processor is shown in FIG. 7), a memory 720, and a computer program 730 stored in the memory 720 and executable on the at least one processor 710, and the processor 710 implements the steps of the gesture detection method described above when executing the computer program 730.

[0212] The terminal device can include but is not limited to a processor and a memory, and FIG. 7 is only an example of the terminal device and does not constitute a limitation on the terminal device, and can include more or fewer components than those shown, or combine certain components, or different components. The processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0213] It should be noted that the information interaction, execution process and the like between the above-described apparatus / module, since based on the same concept as the method embodiments of the present application, the specific functions and the technical effects brought by the same can be referred to the method embodiments part, and will not be described here.

[0214] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. Each functional module in the embodiment can be integrated in one processing module, or each module can be physically present separately, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of software functional module. In addition, the specific name of each functional module is only for convenient distinction, and does not limit the protection scope of the present application. The specific working process of the module in the above system can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0215] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program. When the computer program is executed by a processor, the steps in the gesture detection method can be realized.

[0216] The embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device can realize the steps in the gesture detection method.

[0217] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A gesture detection method, comprising: determining a first position of a feature point in a first image of a target object, wherein the feature point corresponds to a body key point of the target object; performing a pre-processing operation of gesture detection of the target object according to the first position; wherein the pre-processing operation comprises a first pre-processing operation and / or a second pre-processing operation; the first pre-processing operation is used to determine a gesture detection region in a second image of the target object, so as to perform gesture detection on the gesture detection region; the second pre-processing operation is used to determine whether a gesture of the target object meets an activation condition, so as to perform gesture detection when it is determined that the gesture of the target object meets the activation condition.

2. The gesture detection method of claim 1, wherein, The pre-processing operation comprises the first pre-processing operation, the feature point comprises a wrist feature point corresponding to a wrist part of the target object, and the performing of the pre-processing operation of the gesture of the target object according to the first position comprises: determining the gesture detection region according to a second position of the wrist feature point and a first size of a target face region in the first image; wherein the target face region corresponds to a face of the target object, the corresponding position of the second position in the second image falls in the gesture detection region, and the size of the gesture detection region is positively correlated with the first size.

3. The gesture detection method of claim 2, wherein, The determining of the gesture detection region according to the second position of the wrist feature point and the first size of the target face region in the first image comprises: determining a third position in the second image according to the second position of the wrist feature point, wherein the position of the third position in the second image is the same as the position of the second position in the first image; expanding the third position to a plurality of expansion directions respectively to obtain a plurality of expansion boundaries on the second image, wherein the second size corresponding to each expansion direction is equal to the first size corresponding to the expansion direction multiplied by a preset multiple corresponding to the expansion direction; determining the gesture detection region according to a surrounding region of the plurality of expansion boundaries. 4.The gesture detection method of claim 3, wherein the feature point further comprises an elbow feature point corresponding to an elbow part of the target object, the plurality of expansion directions comprises a first height direction and a second height direction, and the method further comprises: for each wrist feature point of the target object, determining a palm orientation of a hand corresponding to the wrist feature point according to a relative position relationship between the wrist feature point and an elbow feature point associated with the wrist feature point; determining a first preset multiple corresponding to the first height direction and a second preset multiple corresponding to the second height direction according to the palm orientation; wherein the first preset multiple is greater than the second preset multiple, the first height direction is the same as the palm orientation, and the second height direction is opposite to the palm orientation.

5. The gesture detection method of claim 3, wherein, The determining of the gesture detection region according to the surrounding region of the plurality of expansion boundaries comprises: For each wrist feature point of the target object, the enclosing region determined for the wrist feature point is taken as a hand region of a hand corresponding to the wrist feature point, to obtain two hand regions; It is judged whether the two hand regions overlap; If yes, the two hand regions are merged, and the merged region is taken as the gesture detection region; If no, each hand region is taken as a gesture detection region.

6. The gesture detection method of claim 3, wherein, The gesture detection region is determined according to the enclosing regions of the plurality of extended boundaries, including: For each wrist feature point of the target object, the enclosing region determined for the wrist feature point is taken as a gesture detection region, to obtain two gesture detection regions, wherein each gesture detection region corresponds to a hand of the target object; The method further includes: Gesture detection is performed on the two gesture detection regions respectively, to determine whether there is an effective region of an effective gesture in the two gesture detection regions and a type of the effective gesture; If yes, a first distance between a center of the effective region and a center of each gesture detection region is calculated, to obtain two first distances; The hand corresponding to the gesture detection region with the smaller first distance is determined as the hand of the effective gesture; According to the hand making the effective gesture and the type of the effective gesture, instruction information corresponding to the effective gesture is determined.

7. The gesture detection method of any one of claims 1 to 6, wherein, The preprocessing operation includes the second preprocessing operation, or further includes the first preprocessing operation, the feature points include wrist feature points corresponding to wrist parts of the target object and elbow feature points corresponding to elbow parts of the target object, and the preprocessing operation on the gesture of the target object is performed according to the first position, including: For each wrist feature point of the target object, a palm orientation of a hand corresponding to the wrist feature point is determined according to a relative position relationship between the wrist feature point and an elbow feature point associated with the wrist feature point, and it is judged whether the gesture of the target object meets an activation condition according to the palm orientation of each hand. And / or For the case where the preprocessing operation includes the first preprocessing operation, a gesture detection region in a second image of the target object is determined according to the first position, on the condition that the gesture of the target object meets the activation condition.

8. The gesture detection method of any one of claims 2 to 6, wherein, The first size includes a width of the target face region, the feature points further include eye feature points corresponding to eye parts of the target object and / or ear feature points corresponding to ear parts of the target object, and the method further includes: A width of the target face region is determined according to a relative position relationship between two eye feature points and / or a relative position relationship between two ear feature points.

9. The gesture detection method according to any one of claims 1 to 6, further comprising: Face detection is performed on the first image, to obtain a target face region in the first image, the target face region corresponding to a face of the target object; The first position of the feature point in the first image of the target object is determined, including: Region extension is performed on the target face region, to obtain a body region corresponding to a body of the target object; perform body key point detection on the body region to obtain a first position of the feature point.

10. The gesture detection method of claim 9, wherein, The region expansion on the target face region to obtain a body region corresponding to a body of the target object comprises: According to the width of the target face region, the target face region is expanded along the image width direction, and according to the height of the target face region, the target face region is expanded along the image height direction to obtain the body region.

11. The gesture detection method of claim 9, wherein, In a case where the current mode is a single-target mode, the face detection on the first image to obtain a target face region corresponding to a face of the target object in the first image comprises: perform face detection on the first image to obtain initial face regions corresponding to each object; perform face recognition on each initial face region to obtain identity information of the object corresponding to each face region respectively; According to the identity information, the target face region is selected from each initial face region.

12. The gesture detection method of any one of claims 1 to 6, wherein, The second image is an image collected at a first time point at which the first position is determined, and the first time point is later than a collection time point of the first image. The method further comprises: determining an offset of a gesture position of the target object within a first time, wherein the first time is a duration from the collection time point to the first time point; The pre-processing operation on the gesture detection of the target object according to the first position comprises: According to the first position and the offset, the gesture detection region is determined.

13. A gesture detection apparatus, comprising: a determination module configured to determine a first position of a feature point in a first image of a target object, wherein the feature point corresponds to a body key point of the target object; a pre-processing module configured to perform a pre-processing operation on gesture detection of the target object according to the first position; The pre-processing operation comprises a first pre-processing operation and / or a second pre-processing operation; the first pre-processing operation is used to determine a gesture detection region in a second image of the target object, so as to perform gesture detection on the gesture detection region; and the second pre-processing operation is used to judge whether a gesture of the target object meets an activation condition, so as to perform gesture detection in a case where it is determined that the gesture of the target object meets the activation condition.

14. A terminal device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the gesture detection method according to any one of claims 1 to 12 when executing the computer program.

15. A computer readable storage medium, the computer readable storage medium storing a computer program, wherein the computer program is executable by a processor to implement the steps of the gesture detection method according to any one of claims 1 to 12.

16. A computer program product, the computer program product being executable on a terminal device to enable the terminal device to implement the steps of the gesture detection method according to any one of claims 1 to 12.

Images