Gesture detection method and device, terminal equipment and computer readable storage medium

By determining the position of feature points before gesture detection and performing preprocessing operations, in response to the problem of low gesture detection efficiency in the prior art, the balance of accuracy and efficiency of gesture detection is achieved, and the computing power cost is reduced.

CN118982867APending Publication Date: 2024-11-19SHENZHEN HONGHE INNOVATION INFORMATION TECH CO LTD

Patent Information

Application Number
CN202411044887.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Existing gesture detection technology has difficulties in achieving a balance between detection accuracy and detection efficiency. It usually requires high computing power costs and low detection efficiency, making it difficult to improve efficiency on the basis of ensuring accuracy.

Method used

By determining the position of the characteristic point of the target object before gesture detection, performing preprocessing operations to determine the gesture detection area or judge the gesture activation conditions, thereby detecting only the necessary areas to reduce invalid detection.

Benefits of technology

It realizes a significant reduction in the computing power cost of gesture detection without reducing detection accuracy, improves detection efficiency, and achieves an effective balance of accuracy and efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982867A_ABST
    Figure CN118982867A_ABST
Patent Text Reader

Abstract

The invention discloses a gesture detection method and device, terminal equipment and a computer readable storage medium, and the method comprises the steps: determining a first position of a feature point in a first image of a target object, wherein the feature point corresponds to a body key point of the target object; executing a preprocessing operation for gesture detection of the target object according to the first position; wherein the pretreatment operation comprises first pretreatment operation and / or second pretreatment operation; the first preprocessing operation is used for determining a gesture detection area in a second image of the target object so as to perform gesture detection on the gesture detection area; the second preprocessing operation is used for judging whether the gesture of the target object meets the activation condition or not, so that gesture detection is carried out under the condition that the gesture of the target object meets the activation condition. According to the scheme, images or image areas which may interfere with gesture detection or do not need gesture detection can be effectively filtered, so that relatively high detection precision and detection efficiency can be ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of gesture detection technology, and in particular, relates to a gesture detection method, apparatus, terminal device and computer-readable storage medium. Background Art

[0002] In recent years, gesture detection technology has been increasingly used in a variety of fields, including gaming, healthcare, car driving, and security monitoring. And with the continuous development of artificial intelligence technology, the accuracy and reliability of gesture detection technology have gradually increased, bringing many conveniences to people's lives.

[0003] However, gesture detection solutions in the prior art usually input the original image of the target object collected into the model without judgment. The model recognizes valid gestures in the entire image according to the detection mechanism and outputs the location, size and gesture type of the valid gesture. The computing power required for this solution is huge, and the efficiency of gesture detection is very low. Some solutions try to compress the size of the model input image to improve the detection efficiency, but the detection accuracy is difficult to meet the requirements, that is, it is difficult to achieve an effective balance between detection accuracy and detection efficiency. Summary of the invention

[0004] The embodiments of the present application provide a gesture detection method, apparatus, terminal device, and computer-readable storage medium, which can solve the problem that it is difficult to achieve an effective balance between detection accuracy and detection efficiency of gesture detection.

[0005] A first aspect of an embodiment of the present application provides a gesture detection method, including:

[0006] Determining a first position of a feature point in a first image of a target object, wherein the feature point corresponds to a body key point of the target object;

[0007] performing a preprocessing operation for gesture detection of the target object according to the first position;

[0008] Among them, the preprocessing operation includes a first preprocessing operation and / or a second preprocessing operation; the first preprocessing operation is used to determine a gesture detection area in the second image of the target object, so as to perform gesture detection on the gesture detection area; the second preprocessing operation is used to determine whether the gesture of the target object meets the activation conditions, so as to perform gesture detection when it is determined that the gesture of the target object meets the activation conditions.

[0009] A second aspect of an embodiment of the present application provides a gesture detection device, including:

[0010] a determination module, configured to determine a first position of a feature point in a first image of a target object, wherein the feature point corresponds to a body key point of the target object;

[0011] a preprocessing module, configured to perform a preprocessing operation for gesture detection of the target object according to the first position;

[0012] Among them, the preprocessing operation includes a first preprocessing operation and / or a second preprocessing operation; the first preprocessing operation is used to determine a gesture detection area in the second image of the target object, so as to perform gesture detection on the gesture detection area; the second preprocessing operation is used to determine whether the gesture of the target object meets the activation conditions, so as to perform gesture detection when it is determined that the gesture of the target object meets the activation conditions.

[0013] A third aspect of an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the steps of the above-mentioned gesture detection method when executing the computer program.

[0014] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned gesture detection method are implemented.

[0015] The first aspect of the embodiment of the present application provides a gesture detection method, before gesture detection, first accurately determines the first position of the feature point corresponding to the key point of the body in the first image of the target object, and then performs a preprocessing operation for gesture detection of the target object according to the first position. In one case, since the preprocessing operation can accurately determine the gesture detection area in the second image of the target object, gesture detection can be performed only for the gesture detection area in the future, so that the detection efficiency can be improved on the basis of ensuring good detection accuracy, and an effective balance between detection accuracy and detection efficiency is achieved. In another case, since the preprocessing operation can accurately determine whether the gesture of the target object meets the activation condition, gesture detection can be performed on the image area of ​​the target object only when it is determined that the gesture of the target object meets the activation condition. This scheme can filter out unnecessary image areas for detection, significantly reduce the computing power cost required for gesture detection without reducing the detection accuracy, and can significantly improve the detection efficiency, thereby also achieving a balance between detection accuracy and detection efficiency. Therefore, the overall gesture detection scheme of the embodiment of the present application can effectively filter out images or image areas that may interfere with gesture detection or unnecessary gesture detection by performing a simple and reasonable image processing process before gesture detection, thereby ensuring high detection accuracy and detection efficiency.

[0016] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1a This is a schematic diagram of a raw image for gesture detection;

[0019] Figure 1b According to the method in the prior art Figure 1a The compressed image;

[0020] Figure 1c It is located according to the method of the invention concept of this application Figure 1a Schematic diagram of the gesture detection area in FIG.

[0021] Figure 1d is in accordance with Figure 1c The local image obtained by cropping the gesture detection area;

[0022] Figure 1e and Figure 1f They are respectively partial schematic diagrams of two first images provided in different embodiments of the present application;

[0023] Figure 2 is a schematic flow chart of a gesture detection method provided by an embodiment of the present application;

[0024] Figure 3a is a schematic diagram of a first image provided by an embodiment of the present application;

[0025] Figure 3b is a schematic diagram of a first image provided by another embodiment of the present application;

[0026] Figure 3c is a schematic diagram of a principle for determining a first size of a target facial area provided by an embodiment of the present application;

[0027] Figure 3d is a schematic diagram of a principle for determining a body region of a target object provided by an embodiment of the present application;

[0028] Figure 3e is a schematic diagram of a principle for determining a body region of a target object provided by another embodiment of the present application;

[0029] Figure 4a is a schematic diagram of feature points in a first image provided by another embodiment of the present application;

[0030] Figure 4b is a schematic diagram of a determination principle of a gesture detection area provided by an embodiment of the present application;

[0031] Figure 4c is a schematic diagram of a determination principle of a gesture detection area provided by another embodiment of the present application;

[0032] Figure 4d is a schematic diagram of a determination principle of a gesture detection area provided by another embodiment of the present application;

[0033] Figure 5a and Figure 5b is a schematic diagram of a gesture detection process in a multi-person mode provided by different embodiments of the present application;

[0034] Figure 5c is a schematic diagram of a gesture detection process in a single-player mode provided by an embodiment of the present application;

[0035] Figure 6 is a structural schematic diagram of a gesture detection device provided by an embodiment of the present application;

[0036] Figure 7 It is a structural diagram of a terminal device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0037] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0038] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0039] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0040] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0041] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0042] First of all, it should be noted that the information collection process (such as face image collection process, fingerprint information collection process, etc.) / feature extraction process involved in this application is performed with the user's knowledge and permission, and does not constitute an act that harms the public interest.

[0043] As mentioned above, the gesture detection scheme in the prior art usually inputs the original image of the target object collected into the model without judgment. The model recognizes valid gestures in the whole image according to the detection mechanism, and outputs the position, size and gesture type of the valid gesture. For example, in a smart home environment, a user controls the smart TV in the living room by making specific gestures facing the TV screen. For another example, a user plays a somatosensory game by making gestures in front of the screen of a smart somatosensory game device. The smart TV or game device is provided with a visual sensor (for example, a camera), which can collect scene images within the visual range in real time, and directly send the collected original image to the gesture detection model for global gesture detection. It can be understood that when the original image size is large, the amount of calculation consumed by gesture detection is very large, and the detection efficiency is low. Especially in scenarios where real-time gesture tracking is required (for example, somatosensory games), the user's gesture is often detected several seconds after the user has made the gesture. The delay time of gesture detection is very long, and the user experience is very poor, resulting in a sharp drop in user satisfaction.

[0044] Some solutions compress the original image and feed the compressed image into the model for detection. This solution has strict requirements on the user's movements, location, and number of people in the visual scene. If these requirements are not met, the gesture will not be detected or the wrong gesture will be detected. For example, when the user's hand is far away from the camera, the hand area accounts for a small proportion of the original image. If the high-resolution original image is compressed into a low-resolution input image for detection, the detection accuracy is likely to be greatly reduced. Figure 1a As shown in Figure 1, the original image has a higher resolution (e.g., 3920px×2160px), and the hand area in the image is clearly visible. Figure 1bAs shown in the figure, the actual model input image resolution (e.g. 600px×600px) obtained after the specified pixel compression is reduced, and the number of pixels in the hand area in the image is also significantly reduced, which is likely to affect the detection effect of the model. In other words, this solution is prone to excessive compression of the hand area in the original image, which in turn makes it difficult for the model detection accuracy to meet expectations. In other words, this solution improves detection efficiency at the expense of detection accuracy. Obviously, this solution cannot effectively balance detection accuracy and detection efficiency.

[0045] In order to solve this type of problem, the embodiment of the present application introduces a local clipping strategy, such as Figure 1c As shown in , the hand region (the gray box in the figure) can be roughly located from the overall image, and then the original image can be cropped according to the hand region. Figure 1d As shown in the figure, the cropped local image of the hand area is used as the input image of the model. It can not only ensure good detection accuracy, but also improve detection efficiency. It can be understood that the size of the cropped local image becomes smaller, and the amount of calculation required for model detection is greatly reduced. In addition, since the image is not over-compressed, even if the hand area accounts for a small proportion of the original image, a high detection accuracy can still be guaranteed. In this scheme, how to roughly locate the hand area is the key. A feasible scheme is as follows: in the process of continuous frame tracking, the position frame of the palm is determined according to the gesture detection result of the previous frame, and then the next frame image is cropped according to the position frame, and the cropped small image is sent to the model for gesture detection. However, this scheme cannot cope with all scenarios, especially in the early stage of gesture detection or when the gesture is lost or there is no gesture during gesture detection, this scheme cannot be applied. In this scheme, the detection area of ​​the next frame is heavily dependent on the detection result of the previous frame. If the hand moves faster or the gesture is not detected in the previous frame, it is difficult to determine the accurate detection area in the next frame. Therefore, when the gesture is detected continuously, the frame loss is more serious. Or, when there is no gesture at all at a certain moment, there is no gesture position in the previous frame to rely on, and it is still necessary to switch to full-image detection to discover new gestures. Therefore, this solution is also difficult to effectively balance detection accuracy and detection efficiency. Considering this situation, we can be based on the current frame, starting from the key features of the human body, and roughly locate the gesture detection area by detecting the position of the human feature points in the current frame. Then, the image is cropped according to the gesture detection area, and the cropped local image is sent to the gesture detection model for detection. This solution does not rely on the detection results of the previous frame, has a wider range of application, and has higher detection accuracy and efficiency.

[0046] In addition, through test analysis, it was found that in the gesture tracking process, the target object does not always make effective gestures, and there are some periods of time when no control gestures are made, that is, the gestures of the target object are invalid gestures. Especially in multi-person control scenarios, for example, in a team somatosensory relay competition game, one member is in the process of competing while the other members are in a preparation state. In the prior art, in order to ensure the detection accuracy, each frame of the image is sent to the gesture detection model for detection, which will result in a large number of unnecessary detections, which not only occupies computing resources, but also may lead to a reduction in detection efficiency. Through test analysis, it was found that the target object usually needs to raise the hand for an effective control gesture, that is, an effective gesture can only be made when the hand is raised. Gestures made without raising the hand are usually invalid gestures, that is, they do not meet the gesture activation conditions. Such as Figure 1e As shown in , gestures with the hand down are usually invalid gestures. Only gestures made after the hand is raised can be valid gestures, that is, only after the hand is raised can the gesture activation conditions be met. Figure 1f As shown, the gesture made by the user raising his hand is a valid gesture. It can be understood that different valid gestures can correspond to different control instructions. Therefore, in the embodiment of the present application, it is also possible to roughly determine whether the gesture meets the activation conditions first, and only perform gesture detection when it is determined that the gesture meets the activation conditions. In this way, invalid detection can be effectively reduced, and detection efficiency can be further improved on the basis of ensuring detection accuracy.

[0047] Based on the above-mentioned inventive concept, the embodiments of the present application provide a gesture detection method, apparatus, terminal device, computer-readable storage medium and computer program product that are different from the prior art.

[0048] First, the embodiment of the present application provides a gesture detection method. The method can be used in various scenarios where gesture detection is desired. Including but not limited to: gesture detection / recognition in virtual reality (VR) and augmented reality (AR), gesture detection / recognition in smart home devices (for example, a TV product with gesture recognition, a user can complete the control of the movement of the TV screen, selection confirmation, page switching, zooming and rotation through gestures such as waving the palm, grabbing with a fist, etc.; for another example, a camera and infrared sensor that can recognize user movements are integrated in a 3D somatosensory camera, and the user can control the character in the game by twisting the body part), in-vehicle gesture control in the automotive industry, gesture detection / recognition in medical equipment control (for example, doctors can use gestures to control the operation of surgical instruments, diagnostic equipment and other medical equipment to improve work efficiency and patient experience), gesture detection / recognition in the education industry (for example, by using gestures to control teaching aids or demonstration equipment, teachers can teach and demonstrate more intuitively), etc. This solution can better balance the detection accuracy and detection efficiency, that is, it can simultaneously ensure high gesture detection accuracy and gesture detection efficiency. For the sake of simplicity, the following text takes the gesture control scenario of a smart TV as an example to explain the specific implementation of the gesture detection method of the embodiment of the present application.

[0049] like Figure 2 As shown, the gesture detection method provided in the embodiment of the present application includes the following steps S220 and S240.

[0050] Step S220: determining a first position of a feature point in the first image of the target object, wherein the feature point corresponds to a key point of the body of the target object.

[0051] In an embodiment of the present application, the target object may be one or more objects for which gesture detection is desired. The object may be any suitable type of object capable of making gestures, including but not limited to humans of various ages, animals, and robots with movable hands. Of course, in some special examples, the object may also be an animal capable of making gestures. For simplicity, the following text will take the human body as an example for explanation. The target object in an embodiment of the present application may be one or more persons for whom gesture detection is desired. For example, in a scenario where a user controls a smart TV using gestures, the target object may be a human body in front of the TV screen.

[0052] Before step S220, a first image of the target object may be acquired first. For example, the first image may be an image of a human body in front of the screen captured in real time by a visual sensor of a smart TV. The visual sensor may continuously capture images within the visual range at a preset frequency, and the first image may be an image containing a human body captured at the current moment, for example, referred to as the current frame hereinafter. The first image may be a color image, or a grayscale image, or a binary image. The first image may be an original image captured by a visual sensor, or an image obtained by performing image preprocessing on the original image. Image preprocessing may include image processing operations such as image cropping, image filtering, image brightness adjustment, and contrast adjustment that facilitate key point detection.

[0053] In this step, a variety of suitable methods can be used to determine the first position of the feature point in the first image. Taking the target object as a human body as an example, the feature points determined in this step can correspond to the key points of the human body, so a variety of suitable human key point detection algorithms can be used to implement this step, including but not limited to traditional computer vision processing algorithms (such as edge detection, corner detection, template matching, etc.), key point detection algorithms based on deep learning, etc. Specifically, these algorithms can be used to locate the position of the feature points corresponding to the human key points expected to be detected in the first image. In a specific example, a key point detection model can be pre-trained. In this step, the first image can be input into the model to obtain the probability that each pixel belongs to a key point, and then the feature points corresponding to each key point in the first image are located.

[0054] Specifically, the feature points determined in this step may at least include feature points corresponding to the target object's hand key points. The hand key points can be arbitrarily set according to actual needs, and specifically can be key points that are not easily blocked. For example, the key points representing the center of each wrist, hereinafter the feature points corresponding to the key points will be referred to as wrist feature points. Of course, hand key points may also include points such as palm points, wrist joint points, fingertip points, palm edge points, and so on. In addition, in some examples, the feature points determined in this step may also include feature points corresponding to the target object's facial key points, upper limb key points, torso key points, lower limb key points, and so on. Exemplarily, as Figure 4a As shown, the facial key points may include the left eye center point, the right eye center point, the nose tip point, the left earlobe point, and the right earlobe point. The upper limb key points may include the left shoulder point, the right shoulder point, the left elbow point, and the right elbow point. The lower limb key points may include the left thigh root point, the right thigh root point, the left knee point, the right knee point, the left ankle point, and the right ankle point. For example, the key point detection model may detect the corresponding Figure 4a The first positions of the 17 feature points of the 17 key points shown in FIG. For example, the coordinates of the pixels corresponding to the 17 key points can be obtained.

[0055] Step S240, performing a preprocessing operation for gesture detection of the target object according to the first position. The preprocessing operation includes a first preprocessing operation and / or a second preprocessing operation. The first preprocessing operation is used to determine a gesture detection area in the second image of the target object, so as to perform gesture detection on the gesture detection area. The second preprocessing operation is used to determine whether the gesture of the target object meets the activation condition, so as to perform gesture detection when it is determined that the gesture of the target object meets the activation condition.

[0056] In the embodiment of the present application, a plurality of suitable methods can be used to perform a preprocessing operation for gesture detection of the target object according to the first position of the feature point in the first image obtained in step S220. It can be understood that the preprocessing operation is an operation that facilitates gesture detection.

[0057] In one example, the preprocessing operation may include only the first preprocessing operation. Taking the target object as a human body in front of a TV screen as an example, the first preprocessing operation is used to determine the gesture detection area in the second image of the human body, that is, to determine the local image area corresponding to the hand of the human body in the second image, so as to facilitate the subsequent gesture detection for the local image area. In other words, this scheme is to determine the gesture detection area corresponding to the hand of the human body according to the position of the human feature point in the first image. In this way, gesture detection can be performed only for the gesture detection area in the future, which can avoid interference from other image areas in the image, avoid invalid detection, reduce the amount of detection calculation, and improve detection efficiency. For example, the second image can be cropped according to the determined gesture detection area to obtain a small hand image, and then only the small hand image is sent to the trained gesture detection model or gesture recognition model. The model can accurately and quickly output the position of the palm area, and the gesture recognition model can also output the category of the recognized gesture. In this way, the control instruction represented by the gesture can be determined according to the category and the correspondence between the category and the instruction, and the corresponding action can be performed according to the control instruction.

[0058] It should be noted that the second image is an image captured at the first moment when the first position of the feature point in the first image is determined. In some cases, the first moment is later than the capture moment of the first image. In this case, the second image and the first image are different images. In other cases, when the processing speed is fast, the moment of executing step S220 is approximately equal to the moment of executing the first preprocessing operation, that is, real-time detection of each frame of the image can be achieved, and the first image and the second image can be the same image. In other words, the first image is the current frame captured in real time at the moment of executing step S220 (for example, the i-th frame captured after the smart TV turns on motion recognition), and the second image is the current frame captured in real time at the moment of executing the first preprocessing operation (for example, the j-th frame captured after the smart TV turns on motion recognition). In the real-time detection scenario, i=j, the first image and the second image are the same image; in the scenario where the processing speed is limited by the hardware and does not meet the real-time detection requirements, i<j, the first image and the second image are two different images.

[0059] In the embodiment of the present application, a variety of suitable methods can be used to determine the gesture detection area. Exemplarily and non-restrictively, the gesture detection area can be determined according to the position of the feature point corresponding to the key point of the hand. For example, in the example where the feature point includes the wrist feature point, the corresponding position in the second image, such as the second position, can be determined according to the first position of each wrist feature point. Then, the second position can be used as a reference, and a preset expansion rule can be used to expand a certain size to the surrounding area to obtain a larger area as the gesture detection area. The preset expansion rule can be arbitrarily set according to actual needs. For example, the expansion size corresponding to different expansion directions can be determined according to the first position of multiple feature points. For example, the expansion size can be determined according to the distance between the two eye feature points or ear feature points of the target object, and the expansion size can also be determined according to the distance between the elbow feature point and the wrist feature point. Exemplarily, the expansion size can be a multiple of the calculated distance. Then, the second position can be used as a reference to expand the corresponding size to different expansion directions around to obtain the gesture detection area. For another example, the size of the target face region in the first image, such as the face width, may be obtained in advance through face detection, and then the expansion sizes corresponding to different expansion directions may be determined according to the face width.

[0060] In another example, the preprocessing operation may include only the second preprocessing operation. Taking the target object as a human body in front of a TV screen as an example, the second preprocessing operation may be an operation of judging whether the gesture of the human body meets the activation condition based on the first position of the feature point. Furthermore, if it is determined that the activation condition is met, gesture detection may be performed on the image area of ​​the human body; if it is determined that the activation condition is not met, the image area of ​​the human body may be filtered out, that is, gesture detection is not performed on the image area. It is easy to understand that this solution can filter out image areas that are unnecessary to detect, and significantly reduce the computing power cost required for gesture detection without reducing the detection accuracy, thereby significantly improving the efficiency of gesture detection.

[0061] Specifically, a variety of suitable methods can be used to determine whether the gesture of the human body meets the activation condition according to the first position of the feature point. As described above, only a gesture made when the hand is raised can be a valid gesture, that is, it meets the gesture activation condition; and a gesture made when the hand is not raised is usually an invalid gesture, that is, it does not meet the gesture activation condition. Therefore, in this step, a variety of suitable judgment logics can be used to determine whether the corresponding hand is raised according to the first position of the feature point. Optionally, the feature points determined in step S220 may include wrist feature points and elbow feature points. In this step, it can be determined whether the corresponding hand is raised according to the relative position relationship between the wrist feature points and the elbow feature points corresponding to the same hand. For example, the size of the ordinates of the wrist feature points and the elbow feature points in the image can be compared (for example, with the upper left corner of the image as the origin). Exemplarily, the difference between the ordinate of the wrist feature point and the ordinate of the elbow feature point can be calculated. If the difference is greater than the difference threshold (which can be set according to actual needs), it can be determined that the hand is not raised, otherwise it is determined that the hand is raised. In a specific example, the difference threshold can be 0. If the ordinate of the wrist feature point is greater than the ordinate of the elbow feature point, it can be indicated that the wrist feature point is below the elbow feature point, that is, the hand is not raised; conversely, if the ordinate of the wrist feature point is less than or equal to the ordinate of the elbow feature point, it can be indicated that the wrist feature point is above the elbow feature point, that is, the hand is raised. Alternatively, the feature points determined in step S220 may include wrist feature points and chest feature points. In this step, it can be determined whether the corresponding hand is raised according to the relative position relationship between the corresponding wrist feature points and the chest feature points. Similarly, if the ordinate of the wrist feature point is greater than the ordinate of the chest feature point, it can be determined that the hand is not raised; conversely, if the ordinate of the wrist feature point is less than or equal to the ordinate of the chest feature point, it can be determined that the hand is raised. Of course, the relative position relationship between other hand feature points and feature points corresponding to other body parts can also be used to reasonably determine whether the hand is raised, and then determine whether the gesture of the human body meets the activation conditions.

[0062] In another example, the preprocessing operation may include a second preprocessing operation and a first preprocessing operation that are performed successively. Moreover, the execution of the first preprocessing operation depends on the result of the second preprocessing operation. For example, it is possible to first determine whether the gesture of the target object meets the activation condition (or whether the gesture of a certain hand of the target object meets the activation condition) based on the first position of the feature point in the first image. If the activation condition is met, the gesture detection area is further determined for the target object (or a certain hand of the target object) based on the first position of the feature point, and then gesture detection is performed for the gesture detection area (for example, the image is cropped according to the gesture detection area and sent to the gesture detection model for detection). This solution can further improve the detection efficiency and detection accuracy of the model.

[0063] As mentioned above, the gesture detection scheme in the prior art usually inputs the collected original image of the target object into the model without judgment. The model recognizes the valid gesture in the whole image according to the detection mechanism, and outputs the position, size and gesture type of the valid gesture. This scheme either has low detection accuracy or huge calculation amount, low detection efficiency, and it is difficult to achieve an effective balance between detection accuracy and detection efficiency. According to the above-mentioned gesture detection method of the embodiment of the present application, before gesture detection, the first position of the feature point corresponding to the key point of the body in the first image of the target object is accurately determined, and then, according to the first position, the preprocessing operation of gesture detection of the target object is performed. In one case, since the preprocessing operation can accurately determine the gesture detection area in the second image of the target object, gesture detection can be performed only for the gesture detection area subsequently, so that the detection efficiency can be improved on the basis of ensuring good detection accuracy, and an effective balance between detection accuracy and detection efficiency is achieved. In another case, since the preprocessing operation can accurately determine whether the gesture of the target object meets the activation condition, gesture detection can be performed on the image area of ​​the target object only when it is determined that the gesture of the target object meets the activation condition. This solution can filter out unnecessary image areas for detection, significantly reduce the computing power cost required for gesture detection without reducing detection accuracy, and significantly improve detection efficiency, thereby achieving a balance between detection accuracy and detection efficiency. Therefore, the overall gesture detection solution of the embodiment of the present application can effectively filter out images or image areas that may interfere with gesture detection or unnecessary gesture detection by performing a simple and reasonable image processing process before gesture detection, thereby ensuring high detection accuracy and detection efficiency.

[0064] In one embodiment, the method further includes step S210.

[0065] Step S210: performing face detection on the first image to obtain a target face region corresponding to the face of the target object in the first image.

[0066] The target object may be a target human body, and the facial detection may be a human face detection. The target facial region may be a target human face region. The trained human face detection model may be used to detect the target human face from the input image, and the position and size of the target human face, feature points, and human face similarity score may be output. For example, in step S210, Figure 3a The first image shown is fed into any suitable face detection model (including but not limited to R-CNN series models, YOLO models, SSD models, RetinaFace models, etc.), and the model can output the position of the face area. For example, the detection information of the face area (the green rectangular position box in the figure) can be obtained. For example, the coordinates of the upper left corner of the rectangular position box in the figure and the width and height of the rectangular position box can be obtained. The rectangular position box can contain the main area of ​​the target face.

[0067] Step S220 determines the first position of the feature point in the first image of the target object, including step S221 and step S222.

[0068] Step S221, performing region expansion on the target facial region to obtain a body region corresponding to the target object's body.

[0069] In step S221, a variety of suitable methods can be used to expand the target facial region to obtain the body region of the body of the corresponding target object. The target facial region can be expanded according to the position of the target facial region and the size of the target facial region, so that the body region obtained after expansion includes the active area of ​​the human hand. Specifically, it can include the whole body area of ​​the human body or the body area above the legs. Optionally, the target facial region can be expanded to a corresponding size in multiple expansion directions around it only according to the width of the target facial region. Alternatively, the target facial region can be expanded to a corresponding size in multiple expansion directions around it only according to the height of the target facial region. The sizes of the target facial region expanded to different expansion directions can be the same or different. Alternatively, the target facial region can be expanded to a corresponding size in multiple expansion directions around it according to the width and height of the target facial region.

[0070] like Figure 3a As shown in FIG. 1 , the face region (the green rectangular position frame in the figure) obtained by face detection can be expanded to obtain the body region (the blue rectangular position frame in the figure). Figure 3b As shown, in the case where the first image includes multiple human bodies, a body region may be determined for each human body, that is, the position frame of each body region may only include the body region of one human body.

[0071] Step S222, performing body key point detection on the body area to obtain the first position of the feature point.

[0072] The first image can be cropped according to the determined body region to obtain a local image of the body region, which can be sent to the human key point detection model to obtain a heat map of the probability that each pixel in the local image belongs to each feature point. The coordinates of the pixels at the location of each feature point in the local image can be obtained based on the pixel value of each pixel in the heat map. For example, the model can output Figure 4a The coordinates p(x) of each feature point in the local image shown in p ,y p ) and confidence score (score). Then, the coordinates of each feature point can be transformed to obtain the coordinates of each feature point in the first image. Figure 4a yes Figure 3a Take a human body in for example, the following calculation method can be used to obtain the characteristic points of the human body. Figure 3b First, the confidence threshold can be set to th. Only feature points with a score greater than th are considered reliable feature points, and feature points with a score less than th are directly discarded. For example, Figure 4a It can be based on the body area expanded from the face area ( Figure 3b The blue rectangular frame area in the image) is a partial image cropped from the image. According to the position of the body area in the first image and the coordinates of the feature point in the partial image, the coordinates of the feature point in the first image can be inferred. When the score is greater than th, assuming that the rectangular position box of the body area is e_rect (e_x, e_y, e_w, e_h), the coordinates of a feature point in the partial image p (x p ,y p ), the coordinates of the feature point in the first image are p1(x p’ ,y p’ ), then x p’ =x p +e_x;y p’ =y p +e_y.

[0073] The solutions for determining the positions of human feature points in human images in the prior art mainly include the following two methods: the first method is to use the original image as the input image, and no matter how many human bodies there are in the original image, the original image is regarded as the image of only one human body, and the positions of the feature points of each part of the human body identified by the human key point detection model are detected. The second method is to first detect the overall area of ​​all human bodies through the target detection model of the original image, that is, to obtain the position frame of the entire body area of ​​each human body, and then send the cropped local image of the image area in each position frame to the human key point detection model to obtain the position of the feature point of each human body. However, in practical applications, both of these solutions have many disadvantages and cannot meet the needs of gesture detection well. For the first solution, when the original image is used as the input image, due to the limitation of the human key point detection model, only one image can be regarded as the image of one human target. When there are multiple human bodies in the image, the model will report an error or only infer the feature points of one person, and discard the remaining human targets. In this way, when there are other people in the scene, it is likely that the correct result cannot be obtained. For the second solution, when the human target in the original image is detected by the target detection model, the model output result is limited by the position of the human body. When the body areas of multiple people intersect with each other or the distance between two people is far, the model cannot detect the location area of ​​each person. In addition, the inference delay of the human target detection model is very long, which occupies a large amount of system resource space and is not conducive to the operation of the program.

[0074] According to the above scheme of the embodiment of the present application, the face detection result can be used to quickly expand the body area of ​​each target object based on the face area of ​​each target object, and then the key point detection can be quickly performed on the body area of ​​each target object and the gesture detection area can be determined. In the case where the target object is a human body, the partial image of each human body can be locked by face detection, and then the locked partial image of the human body can be cropped into a single small image and sent to the human key point detection model. Compared with the prior art that can only detect one human body in the original image, or can only add a human body detection target model to detect multiple people under limited conditions, the above scheme of the embodiment of the present application can detect the feature points of multiple human bodies in an image in real time and accurately in a variety of scenarios. The scheme of the embodiment of the present application has a wider scope of application, and higher detection efficiency and accuracy.

[0075] In the above scheme, the target facial region of the target object is obtained by performing facial detection on the first image, and then the body region of the body corresponding to the target object is quickly and accurately obtained by performing regional expansion on the target facial region. Furthermore, the first position of the feature point is obtained by performing body key point detection on the body region. It can be understood that facial detection is usually faster than whole body detection because it only focuses on a small area in the image. Once the face is quickly located, the search area can be expanded on this basis, thereby accelerating the overall detection process. The body region can be quickly and accurately obtained by expanding the facial region. In addition, by first detecting the face and then performing body key point detection only in a limited area around the face, the amount of image data to be processed can be significantly reduced, thereby reducing the consumption of computing resources. In addition, this scheme can also facilitate locking different target objects. Even if the distances between different target objects are close, key point detection can be performed for different target objects respectively to accurately obtain the feature points corresponding to each target object. Therefore, the corresponding gesture detection area can be effectively and accurately locked for each target object. The detection efficiency and detection accuracy can be further improved.

[0076] In one embodiment, step S221 performs region expansion on the target facial region to obtain a body region of the body corresponding to the target object, including: expanding the target facial region along the image width direction according to the width of the target facial region, and expanding the target facial region along the image height direction according to the height of the target facial region to obtain a body region.

[0077] refer to Figure 3b and Figure 3c , the face detection model can be used to detect Figure 3b All faces in the image can be found, and a face location box corresponding to each face can be generated. Figure 3b As shown in , three face position frames corresponding to three human bodies can be obtained. Figure 3cAs shown, the width of one face position frame is, for example, w (i.e., face width), the height is, for example, h (i.e., face height), and the coordinates of the upper left corner point of the face position frame are, for example, (x, y), then the face position frame can be expressed as face_rect(x, y, w, h). The image width direction can be referred to as the left and right direction of the image, and the image height direction can be referred to as the up and down direction of the image. In an example, the face position frame (hereinafter referred to as the face frame) can be used as the basic frame, and the basic frame of Ln, Rn, Tn, and Bn units can be expanded along the left, right, top, and bottom of the image (Ln, Rn, Tn, and Bn can be expansion multiples in different expansion directions) to obtain the human body position frame (hereinafter referred to as the human body frame) of the body area. The expansion size along the left and right directions can be Ln times and Rn times the face width w, respectively. Ln and Rn can be set according to actual needs. The two can be the same or different. Similarly, the up and down expansion sizes can be Tn times and Bn times the face height h, respectively. Tn and Bn can be set according to actual needs, and the two can be the same or different.

[0078] In a specific example, the calculation process of obtaining the human body frame according to the face frame is as follows:

[0079] First, assume that the human body frame is (e_x, e_y, e_w, e_h), where (e_x, e_y) is the coordinate of the upper left corner of the human body frame, and e_w and e_h are the width and height of the human body frame, respectively. Exemplarily, the coordinate of the upper left corner of the image can be the origin (0,0), the y-axis direction can be downward in the image height direction, and the x-axis direction can be rightward in the image width direction. Considering that the upper left corner of the human body frame cannot exceed the image range, the coordinate of the upper left corner of the human body frame can be determined according to the following formula:

[0080] e_x=max{(x-Ln×w),0};

[0081] e_y=max{(y-Tn×h),0}.

[0082] That is, if the left boundary of the human body frame after expansion according to the expansion multiple exceeds the left boundary of the image, the left boundary of the image is used as the left boundary of the human body frame; if the upper boundary of the human body frame after expansion according to the expansion multiple exceeds the upper boundary of the image, the upper boundary of the image is used as the upper boundary of the human body frame.

[0083] Then, a variety of suitable methods may be used to calculate the width and height of the human body frame, and the following method 1 and method 2 are two examples thereof.

[0084] Method 1: First, determine the width e_w of the human body frame. It can be understood that the width of the human body frame = the actual width of the left boundary of the face frame extended to the left + the width of the face frame + the actual width of the right boundary of the face frame extended to the right. The actual extension width on the left = x-e_x; assuming that the actual extension width on the right = Rn×w; then e_w = x-e_x+w+Rn×w. In order to prevent the human body frame from exceeding the predefined maximum right boundary, boundary detection is required. The maximum right boundary of the human body frame can be the right boundary of the first image, or it can be located at a preset position on the left side of the right boundary of the first image. The preset position can be a position at a certain distance from the right boundary. The certain distance can be arbitrarily set according to actual needs, such as 1 pixel, 10 pixels, etc. For example, assuming that the width of the first image is W, the horizontal coordinate of the maximum right boundary of the human body frame can be less than or equal to W. For example, the horizontal coordinate of the maximum right boundary of the human body frame is equal to W-1. Among them, 1 represents 1 pixel. It can be understood that setting the maximum right boundary of the human body frame equal to W-1 can ensure that the index range of the array will not be exceeded when processing image data, thereby avoiding the problem of calculation overflow. If e_x+e_w≤W-1, it means that expanding the right boundary of the face frame by Rn times the face width will not exceed the preset maximum right boundary of the human body frame, then e_w=x-e_x+w+Rn×w is true. If e_x+e_w>W-1, it means that expanding the right boundary of the face frame by Rn times the face width will exceed the preset maximum right boundary of the human body frame, then e_w=W-1-e_x.

[0085] Next, the height e_h of the human frame can be determined according to a similar principle. It can be understood that the height of the human frame = the actual height of the upper boundary of the face frame extended upward + the height of the face frame + the actual height of the lower boundary of the face frame extended downward. Among them, the actual height of the upper boundary extended upward = y-e_y; assuming that the actual height of the lower boundary extended downward = Bn×h; then e_h = y-e_y+h+Bn×h. In order to prevent the human frame from exceeding the predefined maximum lower boundary (to prevent the human frame from crossing the boundary), boundary detection is required. Assuming that the height of the first image is H, the ordinate of the maximum lower boundary of the human frame can be less than or equal to H. For example, the maximum lower boundary of the human frame is equal to H-1. If e_y+e_h≤H-1, then e_h=y-e_y+h+Bn×h is true. If e_y+e_h>H-1, it means that expanding the lower boundary of the face frame by Bn times the width of the face will exceed the preset maximum lower boundary of the human frame, and at this time e_h=H-1-e_y.

[0086] Method 2: Assume that the maximum right boundary of the human frame is the right boundary of the first image, and the maximum lower boundary of the human frame is the lower boundary of the first image. The lower right corner point of the face frame (x a ,y a ). Among them, x a=x+w,y a =y+h. Then, the lower right corner point (x b ,y b ):

[0087] x b =min{(x a +Rn×w),W};

[0088] y b =min{(y a +Bn×h),H}.

[0089] Then, we can further calculate the value of e_w=x b -e_x and e_h = y b -e_y gets the width and height of the human body box.

[0090] The area expansion multiples Ln, Rn, Tn, and Bn corresponding to different expansion directions used in the above examples can be set to appropriate values ​​according to actual needs. It can be understood that the human body frame determined in the embodiment of the present application is mainly used to estimate the approximate position of the hand area. According to experiments and experience, considering the limit of the human hand activity area, the maximum activity area in the width direction of the human body is the width area reached when the two hands are stretched horizontally, and the maximum activity area in the height direction of the human body is the height area reached when the two hands are raised vertically to fully lowered.

[0091] In order not to lose the hand image, in one example, Figure 3d As shown, it can be determined that the expansion ratios of the face frame to the corresponding body frame are: 3(Ln), 3(Rn), 3(Tn), 8(Bn), that is, the face frame expands to the left and right by 3 face widths, expands to the top by 3 face heights, and expands to the bottom by 8 face heights. Figure 3d Although the area expansion multiple in can maximize the capture of the whole body, the image is too large, and when the shape recognition of the human body image is used to determine the position of the palm for gesture detection, the characteristic shape of the foot is not required, that is, only the position of the upper body including the hand is required. In another example, Figure 3e As shown, in order to improve the efficiency of image cropping and detection, a part of the bottom extension can be removed, and the expansion ratios of the face frame to the corresponding body frame upward, downward, left and right can be determined as: 3 (Ln), 3 (Rn), 3 (Tn), 4 (Bn), that is, 4 face heights are extended downward so that the calf area is not included in the body frame. In this scheme, the local image of the upper body area cropped according to the body frame can be used for human key point detection, which can improve the detection efficiency and save the calculation amount of key point detection.

[0092] It can be understood that the above scheme has simple execution logic and small amount of calculation. By expanding the facial area of ​​the target object, the body area can be obtained in real time and accurately, and then the key point detection of the body area can improve the efficiency and accuracy of the key point detection. And further improve the accuracy and efficiency of subsequent gesture detection. It is helpful to achieve real-time and accurate gesture tracking of the target object.

[0093] In addition, refer again Figure 3b , when there are multiple people within the visual range of the visual sensor, that is, there are multiple human bodies in the first image, even if different human bodies overlap, based on the face frame detected for each human body, the human body frame is expanded according to the above-mentioned area expansion method, and the first image is cropped into multiple small-sized input images according to each human body frame. After that, each small-sized input image can be sent to the human key point detection model for reasoning to obtain the detection result. In this way, as long as the face frame exists, a local image centered on each person's body can be intercepted as the input image for human key point detection, and the feature points for each human body can be quickly and accurately obtained. In other words, the above-mentioned face detection + area expansion method can quickly and accurately divide the human body into multiple small-sized input images. Figure 3b The body area of ​​each human body in the image. The subsequent key point detection for each body area can accurately detect the feature points corresponding to each human body, and is not easily disturbed by the human body next to it. Therefore, the scope of application of this method is also wider, and the body area of ​​each target object can be quickly and accurately determined in a variety of scenarios.

[0094] Moreover, since this scheme can efficiently and accurately locate the body areas of multiple target human bodies in the first image, the body feature points of multiple human bodies can be detected in the same image, and the gesture detection area of ​​each human body can be accurately obtained, which can provide accurate positioning for gesture detection in multi-person scenes, which is very helpful for re-locking when a mobile gesture is lost, or for binding a gesture to a specific human body. Therefore, this method not only has high detection accuracy, but also has strong versatility. It only needs a simple face detection model to achieve locking and tracking of the target human body (without adding a human target detection model), and is easy to deploy and apply.

[0095] In one implementation, when the current mode is the single target mode, step S210 performs face detection on the first image to obtain a target face region corresponding to the face of the target object in the first image, including the following steps:

[0096] Step S211, performing face detection on the first image to obtain initial face regions corresponding to each object;

[0097] Step S212, performing facial recognition on each initial facial region to obtain identity information of the object corresponding to each facial region;

[0098] Step S213, screening out a target facial region from each initial facial region according to the identity information.

[0099] It is understandable that in order to meet the diverse control needs of users, the system can set multiple gesture control modes, or set different gesture control modes at different times of gesture control. One of them is single-person mode. In single-person mode, multiple people are allowed to appear in the gesture control scene, but only one person who meets the requirements is locked for gesture detection, and other people are not subject to gesture detection. In a specific example, in step S211, the first image can be input into a trained face detection model, for example, input into the face detection model, and the face area of ​​each human body in the first image is output. For example, Figure 3b The images of the three human bodies shown are input into the face detection model, and the position frames of the three faces in the image can be obtained. The area contained in each position frame can be an initial face area, which can be called an initial face area, that is, three initial face areas can be obtained. Then, the image can be cropped according to the three initial face areas to obtain a partial image of each initial face area, that is, three partial images. In step S212, a variety of suitable methods can be used to perform face recognition on the three partial images respectively. For example, the partial image of the initial face area of ​​each human body can be input into the trained face recognition model to obtain the identity information of each human body, that is, to obtain the face label information of each of the three people in the figure. In step S212, the target face area (target face area) can be screened out from the three initial face areas according to the face label information. For example, the system pre-sets one of them as the target human body (for example, the face of the user who controls the TV is pre-entered, and the initial face area with the same face label as the entered user can be screened out in this step as the target face area). In the subsequent step S221, the target face area can be expanded to obtain the target body area. In this way, gesture detection can be performed only on the target body for at least a period of time. In this way, the gesture of the desired body can be quickly and accurately locked according to actual needs, the accuracy of gesture detection can be improved, and precise gesture control can be achieved. The computing power cost consumed by such unnecessary detection can also be significantly reduced, further improving the detection efficiency.

[0100] In the above scheme, on the basis of accurately detecting the initial facial area of ​​each object in the first image, facial recognition can be further used to quickly and accurately lock the facial area of ​​the target object according to the facial recognition result. It can be understood that in the example where the target object is a human body, adding facial recognition on the basis of face detection can bind the hand area and the face area of ​​the human body, and accurately distinguish which human body the hand areas at different positions in the image belong to. In this way, the gesture control authority can be quickly and accurately granted to the target human body. Subsequently, gesture detection can be performed only on the gesture detection area of ​​the target human body, while the gesture authority of other people can be shielded. That is, through facial recognition, the gesture detection area can be quickly and accurately bound to the object making the gesture, which can significantly reduce the purpose of gesture misoperation, truly filter interference from the bottom layer, and improve the accuracy of gesture control. In general, this scheme binds the gesture detection area with the facial recognition result, which not only achieves the purpose of meeting the user's personalized control needs and enriching the gesture application scenarios, but also significantly reduces the probability of gesture misoperation, thereby significantly improving the accuracy of gesture control, and also reduces the cost of gesture detection, greatly improving the efficiency of gesture detection.

[0101] In one embodiment, the preprocessing operation includes a second preprocessing operation, and the feature points include wrist feature points corresponding to the wrist of the target object and elbow feature points corresponding to the elbow of the target object. Step S240 performs a preprocessing operation on the gesture of the target object based on the first position, including step S241.1 and step S241.2. Step S241.1, for each wrist feature point of the target object, determine the palm orientation of the hand corresponding to the wrist feature point based on the relative position relationship between the wrist feature point and the elbow feature point associated with the wrist feature point. Step S241.2, determine whether the gesture of the target object meets the activation condition based on the palm orientation of each hand.

[0102] First of all, it should be noted that the palm orientation in the application embodiment refers to the orientation of the upper limbs of the target object. For example, if the target object is a human body, then when the forearm of one of the human arms is drooping, the upper limb is facing downward, that is, the palm orientation is downward; and when the forearm of the arm is raised, the upper limb is facing upward, that is, the palm orientation is upward. Therefore, the palm orientation of the hand is associated with the raised state of the hand. The palm facing upward corresponds to the raised state of the hand, and the palm facing downward corresponds to the drooped state of the hand (that is, the hand is not raised).

[0103] For the same target object, the palm directions of the two hands may be the same or different. Figure 4b The palms of the two hands of the target human body shown in FIG. are both facing downward, and Figure 4d The palms of the two hands of the target human body shown in FIG. are both facing upward. Figure 4cThe palm of the left hand of the target human body shown in FIG. 8 is facing upward, while the palm of the right hand is facing downward.

[0104] As mentioned above, during the gesture tracking process, the target object does not always make valid gestures, and there are periods of time when no control gestures are made. Generally, valid gestures can only be made when the hand is raised. Gestures made without raising the hand are usually invalid gestures and do not meet the gesture activation conditions. Only gestures made when the hand is raised can meet the gesture activation conditions.

[0105] The feature points determined for the target object in step S220 include elbow feature points and wrist feature points. For each human body, the left elbow feature points and left wrist feature points corresponding to the left hand, and the right elbow feature points and right wrist feature points corresponding to the right hand can be determined. There is an association relationship between the elbow feature points and the wrist feature points corresponding to the same hand. The labels of different feature points detected by the key point detection model are different, for example, the serial numbers of the left elbow feature points and the right elbow feature points are different. Therefore, the association relationship between the elbow feature points and the wrist feature points and the corresponding hand (for example, the corresponding left hand or the right hand) can be determined based on the number of the feature points. It can be understood that the relative position relationship between the elbow feature points and the wrist feature points corresponding to the same hand can accurately characterize the palm orientation of the hand.

[0106] The characteristic points of the target object obtained in step S220 are, for example, Figure 4a 17 feature points shown in FIG. 1 , in step S241.1, the coordinates of the wrist feature points and elbow feature points corresponding to each hand can be searched from these 17 feature points. Then, for each hand, the relative position relationship between the wrist feature point and the elbow feature point can be determined. Then, according to the relative position relationship, the palm direction of the hand corresponding to the wrist feature point is determined. The relative position relationship can be represented in a variety of ways, as long as it can reflect the state of hand lifting. Exemplarily and not restrictively, the relative position relationship between the two can be directly represented by the difference between the ordinates of the two. For example, the ordinate of the elbow feature point can be subtracted from the ordinate of the wrist feature point corresponding to the same hand to obtain the ordinate difference. Afterwards, the palm direction of the corresponding hand can be determined according to the difference. For example, a suitable difference threshold can be pre-set. Then, when the difference is greater than the difference threshold, it can be determined that the palm direction of the hand is downward, which means that the corresponding hand is not lifted. Further, in step S241.2, it is determined that the gesture of the corresponding hand does not meet the activation condition. On the contrary, if the difference is less than or equal to the difference threshold, it can be determined that the palm of the hand is facing upward, which means that the corresponding hand is raised. Then, in step S241.2, it is determined that the gesture of the corresponding hand meets the activation condition. According to this method, the activation condition can be quickly determined before the gesture detection. Figure 4cThe left hand gesture of the human body does not meet the activation conditions, while the right hand gesture meets the activation conditions.

[0107] In the above scheme, for each wrist feature point of the target object, according to the relative position relationship between the wrist feature point and the elbow feature point associated with the wrist feature point, the palm orientation of the hand corresponding to the wrist feature point can be quickly and accurately determined, and then whether the gesture of the target object meets the activation conditions can be accurately judged in real time before gesture detection. In this way, in the subsequent steps, gesture detection / recognition can be performed only on the gesture detection area of ​​the hand that meets the activation conditions. This scheme has a small amount of calculation, and therefore, it can further reduce the cost of gesture detection and improve the efficiency of gesture detection.

[0108] In one embodiment, the preprocessing operation further includes a first preprocessing operation. Step S240 performs a preprocessing operation on the gesture of the target object according to the first position, and further includes step S241.3.

[0109] Step S241.3, under the condition that it is determined that the gesture of the target object meets the activation condition, determine the gesture detection area in the second image of the target object according to the first position.

[0110] In this solution, it is possible to first determine whether the gesture of each hand of each target object meets the activation condition, and then determine the gesture detection area only for the hands that meet the activation condition. The amount of calculation in this solution is further reduced, so the cost of gesture detection can be further reduced, and the efficiency of gesture detection can be further improved.

[0111] In one embodiment, the preprocessing operation includes a first preprocessing operation, and the feature point includes a wrist feature point corresponding to the wrist of the target object. Step S240 performs a preprocessing operation on the gesture of the target object according to the first position, including step S242.

[0112] Step S242, determining a gesture detection area according to the second position of the wrist feature point and the first size of the target facial area in the first image, wherein the target facial area corresponds to the face of the target object, the corresponding position of the second position in the second image falls within the gesture detection area, and the size of the gesture detection area is positively correlated with the first size.

[0113] refer to Figure 4b , the target object is a target human body, and the target facial region is a target human face region. The first size of the target human face region may include a width of the target human face region and / or a height of the target human face region. Of course, other sizes of the target human face region may also be included. The following is an example in which the first size includes the width of the target human face region. First, the width of the target human face region may be determined in a variety of suitable ways.

[0114] In one embodiment, the width of the target face region can be determined based on the face detection result. In this example, the feature points determined in step S220 may not include facial feature points. For example, the face detection model may output a rectangular position frame for each face region, and the width of the rectangular position frame may be the face width.

[0115] In another embodiment, the first size includes the width of the target facial area, and the feature points further include eye feature points corresponding to the eyes of the target object and / or ear feature points corresponding to the ears of the target object. The method further includes: determining the width of the target facial area according to the relative position relationship between the two eye feature points and / or the relative position relationship between the two ear feature points.

[0116] Exemplarily, the width of the target facial area can be determined based on the position of the facial feature points in the determined feature points. Specifically, it can be determined based on the relative position relationship between two eye feature points and / or the relative position relationship between two ear feature points. Take determining the width of the target facial area based on the relative position relationship between two eye feature points as an example. Exemplarily, the width of the target facial area is determined based on the distance between the two eye feature points, and the distance can be the Euclidean distance between the two points, or the distance between the coordinate components in the width direction of the image. A person of ordinary skill in the art can understand the specific implementation of this solution, and for the sake of brevity, it will not be repeated here.

[0117] The above method of determining the width of the target facial area based on the relative position relationship between facial feature points is more accurate, so that the gesture detection area can be accurately determined, and the accuracy of gesture detection can be improved. In addition, this solution does not rely on the face detection result, so the solution is more scalable.

[0118] After determining the first size of the target facial region, a variety of suitable methods may be used to determine the gesture detection region based on the first size and the coordinates of the wrist feature point. Exemplarily and not restrictively, the position of the human wrist feature point may be used as a reference position, and the corresponding size may be expanded in multiple directions around the reference position, and the size may be equal to a multiple of the first size, for example, equal to a multiple of the width of the target facial region, so that the expanded gesture detection region includes the possible activity area of ​​the hand.

[0119] In the above scheme, the gesture detection area of ​​the hand of the target object can be quickly and accurately determined according to the position of the wrist feature point and the first size of the target facial area. In addition, since the gesture detection area can be directly obtained by detecting the key points of the current frame, it does not rely on the gesture detection result of the previous frame and can be applied to a variety of gesture detection scenarios. In particular, in the gesture tracking scenario, the gesture detection area can still be pre-positioned relatively real-time and accurately, which can largely avoid the situation of gesture tracking being lost. Therefore, this scheme has better robustness. In addition, the execution logic of this scheme is also relatively simple, so the detection efficiency is also higher.

[0120] In one implementation, step S242 determines the gesture detection area according to the second position of the wrist feature point and the first size of the target facial area in the first image, and includes step S242.1, step S242.2 and step S242.3.

[0121] Step S242.1, based on the second position of the wrist feature point, determine the third position in the second image, wherein the position of the third position in the second image is the same as the position of the second position in the first image. Step S242.2, on the second image, respectively expand the third position in multiple expansion directions around the corresponding second size to obtain multiple expansion boundaries, wherein the second size corresponding to each expansion direction is equal to the first size corresponding to the expansion direction multiplied by the preset multiple corresponding to the expansion direction. Step S242.3, determine the gesture detection area based on the enclosed area of ​​the multiple expansion boundaries.

[0122] First, in an example where the first image and the second image are the same image, the position of the wrist feature point in the second image can be directly located. In an example where the first image and the second image are different images, the third position can be located in the second image at the same coordinate position. Afterwards, the gesture detection area can be obtained by extending the boundary based on the third position. It can be understood that the method of obtaining the gesture detection area by extending the third position is similar to the aforementioned method of determining the body frame based on the face frame. For the sake of brevity, only one specific example is listed below for illustration.

[0123] For example, refer to Figure 4b , respectively, taking the left wrist feature point and the right wrist feature point of the human body in the figure as the reference, expand an area as the corresponding gesture area. Taking the left hand as an example, assuming the coordinates of the left wrist feature point p1(x1,y1), the coordinates of the left elbow feature point p2(x2,y2), the left eye feature point p3(x3,y3), the right eye feature point p4(x4,y4), the position box of the hand extension area (for example, called the palm box) h_rect(x * ,y *,width,height). Where width and height are the width and height of the palm frame respectively. Then we can first calculate the width D of the target face. Where D 2 =(x3-x4) 2 +(y3-y4) 2 .

[0124] Exemplarily, the multiple expansion directions may include the four directions of up, down, left, and right of the third position. The second size corresponding to the four expansion directions, the second size corresponding to each expansion direction is equal to the first size corresponding to the expansion direction multiplied by the preset multiple corresponding to the expansion direction. The preset multiples corresponding to the four expansion directions of up, down, left, and right are T_n, B_n, L_n, and R_n respectively. T_n, B_n, L_n, and R_n can be set arbitrarily according to actual needs, and their values ​​can be the same or different. In one example, T_n=2, B_n=L_n=R_n=1. That is, the second size of the third position corresponding to the p1 coordinate in the second image can be expanded to the left and right to be equal to the width of D, to the height of 2 D upward, and to the height of one D downward.

[0125] Specifically, first, the horizontal coordinate x of the upper left corner of the expanded palm frame can be calculated: * and the ordinate y * :

[0126] x * =max{(x1-D),0};

[0127] y * =max{(y1-D),0}.

[0128] Then, calculate the width of the palm frame, assuming that the left and right sides actually extend by one D, then the frame width is 2D, that is, width = 2D. To prevent the palm frame from crossing the border, a boundary judgment is required. Assume that the width of the first image is W, and the maximum right boundary of the palm frame is equal to W-1. If x1+width≤W-1, then width=2D is true. And if x1+width>W-1, then width=W-1-x1.

[0129] Similarly, calculate the palm frame height height, assuming that the frame height height = 3D. To prevent the palm frame from crossing the border, a boundary judgment is required. Assume that the height of the first image is H, and the maximum lower boundary of the palm frame is equal to H-1. If y1+height≤H-1, then height=3D is true. And if y1+height>H-1, then height=H-1-y1.

[0130] Thus, the palm frame coordinates and size can be calculated: h_rect(x * ,y* , width, height).

[0131] After obtaining the palm frame corresponding to each hand, the area included in the palm frame can be directly used as the gesture detection area of the hand, or the area included in the palm frame can be further processed to obtain the palm area of the hand.

[0132] The method of determining the gesture detection area by expanding the corresponding second size in multiple directions around the position of the wrist feature points has a small amount of calculation, fully considers the possible movement range of the hand, and the obtained gesture detection area is more reasonable and accurate. Especially in the scenario of gesture tracking, it can effectively avoid losing the gesture. Therefore, the gesture detection efficiency and detection accuracy can be significantly improved.

[0133] In one implementation manner, the feature points further include elbow feature points corresponding to the elbows of the target object, and the multiple expansion directions include a first height direction and a second height direction. The method further includes step S231 and step S232.

[0134] Step S231, for each wrist feature point of the target object, determine the palm orientation of the hand corresponding to the wrist feature point according to the relative position relationship between the wrist feature point and the elbow feature point associated with the wrist feature point. The implementation manner of this step is similar to the implementation manner of the前述 step S241.1. For the sake of brevity, it will not be elaborated here.

[0135] Step S232, according to the palm orientation, determine a first preset multiple corresponding to the first height direction and a second preset multiple corresponding to the second height direction. Wherein, the first preset multiple is greater than the second preset multiple, the first height direction is the same as the palm orientation, and the second height direction is opposite to the palm orientation. For example, if the palm orientation is upward, the first height direction is the upward expansion direction, and the second direction is the downward expansion direction, and the expansion multiple upward is greater than the expansion multiple downward; if the palm orientation is downward, the first height direction is the downward expansion direction, and the second direction is the upward expansion direction, and the expansion multiple downward is greater than the expansion multiple upward.

[0136] Judge the palm position. According to the coordinates of the wrist and elbow, it is roughly possible to judge whether the palm is facing upward or downward. The origin of the coordinate system is in the upper left corner, and the coordinates increase from left to right and from top to bottom. As mentioned above, if the y coordinate of p2 is less than the y coordinate of p1, that is, y2 < y1, then in this case, it is considered that the palm is in the lower part. On the contrary, when y2 ≥ y1, the human hand is basically in the raised state, and in this case, it is considered that the palm is in the upper part.

[0137] The first preset multiple and the second expansion multiple can be arbitrarily set according to actual needs. For example, the first preset multiple is 2, the second expansion multiple is 1, and the width of the target face area is D. Then, as Figure 4c As shown, when it is determined that the palm of the left hand of the human body in the figure is facing downward, the height extended downward from the third position is 2D, and the height extended upward from the third position is D; when it is determined that the palm of the right hand of the human body in the figure is facing upward, the height extended upward from the third position is 2D, and the height extended upward and downward from the third position is D. In other words, when the hand is raised, the height extended upward is greater than the height extended downward; and when the hand is not raised, the height extended downward is greater than the height extended upward.

[0138] It can be understood that the direction of the palm of the target object during the process of making a gesture can usually accurately reflect the moving direction of the current gesture. Therefore, based on the direction of the palm, the direction of the gesture movement can be roughly predicted. In the above scheme, the expansion size along the direction of the palm direction is set to be larger, so that more prediction space (detection area) can be reserved for the direction of gesture movement. In this way, even if the hardware processing speed is limited and the wrist feature points of each frame cannot be determined in real time, the gesture after moving along the direction of the gesture movement in the subsequent frames can be accurately captured more completely through the reserved sufficient detection area, so that a high gesture detection accuracy can be always guaranteed during the gesture movement process.

[0139] In one implementation, step S242.3 determines the gesture detection area according to the enclosed area of ​​the multiple extended boundaries, including the following steps:

[0140] Step S242.31, for each wrist feature point of the target object, the enclosed area determined for the wrist feature point is used as the hand area of ​​the hand corresponding to the wrist feature point, and two hand areas are obtained;

[0141] Step S242.32, determining whether the two hand regions overlap;

[0142] Step S242.33, if yes, then merge the two hand regions and use the merged region as the gesture detection region;

[0143] Step S242.34, if not, each hand area is used as a gesture detection area.

[0144] At this time, take the feature points of the left and right wrists as the reference points, expand an area as the gesture image cropping area, assume that the area is gesture.rect(x, y, width, height), expand the face width of T_N, B_N, L_N, R_N units up and down, left and right respectively, and replace the face width with the inter-ear distance width obtained above. The coordinate and size calculation are the same as the face expansion method above, which will not be elaborated here.

[0145] like Figure 4dAs shown, the palm frames of the left and right hands of each target object can be obtained according to the region expansion method in the above example (two red dotted frames in the figure), and the area contained in each of the two palm frames is a hand region. Then, a variety of appropriate methods can be used to determine whether the two hand regions overlap.

[0146] For example, the hand area A of the left hand is (x a ,y a ,w a ,h a ), where (x a ,y a ) is the coordinate of the upper left corner of the area; the hand area B of the right hand is (x b ,y b ,w b ,h b ). By way of example but not limitation, it may be determined whether two hand regions overlap according to the following conditions:

[0147] ①x a >x b +w b ;

[0148] ②x b >x a +w a ;

[0149] ③y a >y b +h b ;

[0150] ④y b >y a +h a ;

[0151] If the positions and sizes of the two hand regions do not satisfy any of the above four conditions, it is determined that the two hand regions overlap; otherwise, it is determined that the two hand regions do not overlap.

[0152] For example, if it is determined that two regions overlap, the merged region C(x c ,y c ,w c ,h c ).

[0153] First, the coordinates of the lower right corner of the hand area A (x a +w a ,y a +h a ), the coordinates of the lower right corner of the hand area B (x b +w b ,yb +h b ).

[0154] Then, we can get: X c =min{x a ,x b};y c =min{y a ,y b}.

[0155] After that, the coordinates of the lower right corner of region C (x d ,y d ):

[0156] x d =max{x a +w a ,x b +w b};y d =max{y a +h a ,y b +h b}.

[0157] Finally, we can get: c =x d -x c ;h c =y d -y c .

[0158] like Figure 4d As shown, the merged area is the area contained in the red solid line frame shown in the figure. Exemplarily, the merged area can be used as a gesture detection area and sent to the gesture detection model for detection. In this way, the computing power cost can be saved and the gesture detection efficiency can be further improved.

[0159] In another embodiment, step S242.3 determines the gesture detection area based on the enclosed area of ​​multiple extended boundaries, including: for each wrist feature point of the target object, the enclosed area determined for the wrist feature point is used as a gesture detection area to obtain two gesture detection areas, where each gesture detection area corresponds to a hand of the target object.

[0160] The method further includes:

[0161] Step S251, performing gesture detection on the two gesture detection areas respectively, and determining whether there is a valid area with a valid gesture and the type of the valid gesture in the two gesture detection areas;

[0162] Step S252, if yes, then calculating the first distances between the center of the effective area and the centers of the two gesture detection areas respectively to obtain two first distances;

[0163] Step S253, determining the hand corresponding to the gesture detection area with the smaller first distance as the hand with the valid gesture;

[0164] Step S254: determining instruction information corresponding to the valid gesture according to the hand making the valid gesture and the type of the valid gesture.

[0165] In some special scenarios, the instruction information represented by the same gesture made by the left and right hands of the target object may be different. For example, the instruction information represented by the left fist and the instruction information represented by the right fist may have different meanings, so it is necessary to accurately determine the hand that made the effective gesture. It can be understood that if the left and right hands of the target body are close to each other, and one of the two hands makes a valid gesture while the other does not, then in the example where the second image is different from the first image, the two gesture detection areas determined based on the first image overlap. If the local image cropped after the two areas are merged is sent to the model for gesture detection, it is impossible to accurately locate which hand made the effective gesture, making it difficult to accurately determine the corresponding instruction information.

[0166] In the above scheme of the embodiment of the present application, the two hand areas may not be merged, but the two hand areas may be sent to the gesture detection model as a gesture detection area for gesture detection, and the valid gesture area in each gesture detection area may be obtained. The valid gesture area may be the position frame of the minimum circumscribed rectangle of the palm that makes a valid gesture. It can be understood that when the two gesture detection areas overlap, the model can output label information of the valid gesture area and the gesture type for both gesture detection areas. Afterwards, the distance between the position center of the valid gesture area and the position center of the two gesture detection areas may be calculated as the first distance. Afterwards, the hand corresponding to the gesture detection area with the smaller first distance may be determined as the hand that makes a valid gesture. For example, if the distance between the position center of the gesture detection area corresponding to the left hand and the position center of the valid gesture area is 10px, and the distance between the position center of the gesture detection area corresponding to the right hand and the position center of the valid gesture area is 15px, it can be determined that the left hand has made the valid gesture. Furthermore, the specific instruction information of the valid gesture can be accurately determined according to the type of the valid gesture and the hand that makes the valid gesture. This scheme can ensure higher gesture recognition accuracy.

[0167] As mentioned above, in an ideal case, the hardware processing speed is fast, and the time of executing step S220 is approximately equal to the time of executing the first preprocessing operation, that is, real-time detection of each frame of image can be achieved, and the first image and the second image can be the same image. In a scenario where multi-person gesture control is allowed, for example, in multi-person mode, multiple people are allowed to activate gestures at the same time, and during the activation period, it is allowed to change people to continue to execute gesture commands. Specifically, Figure 5a As shown, in this mode, all faces can be detected using a face detection model to obtain multiple face regions. Then, each face region can be expanded to obtain a corresponding human region. Then, the first image can be cropped according to these human regions to obtain multiple human thumbnails. Then, human key point detection can be performed for each human thumbnail to obtain a set of feature points for each human body. These feature points can include at least wrist feature points. After that, the hand region can be expanded according to the position of each wrist feature point in the first image to obtain the hand regions of the two hands of each human body as gesture detection regions. The hand regions are cropped into hand thumbnails and sent to the gesture detection model, and the valid gesture regions in each hand region of each human body are obtained according to the detection results. If the two hand regions of each human body do not include a valid gesture region, it means that the human body has not made a valid gesture; if the two hand regions of each human body include a valid gesture region, it means that the human body has made a valid gesture, the gesture can be activated, and the corresponding instruction can be executed.

[0168] In one embodiment, the second image is an image captured at a first moment when the first position is determined, and the first moment is later than the capture moment of the first image. The method further includes step S214. Step S214, determining an offset of the gesture position of the target object within a first time, wherein the first time is a duration from the capture moment to the first moment.

[0169] Step S240 performs a preprocessing operation for gesture detection of the target object according to the first position, including: determining a gesture detection area according to the first position and the offset.

[0170] When the hardware processing speed is limited, the first moment when the first position of the feature point in the first image is obtained through key point detection is different from the first moment, that is, there is a certain detection delay. In this case, the second image may be another image captured after the first image is captured. For example, the first image is the i-th frame (i≥1), and the second image may be the i+1-th frame. Then the first time may be the duration of capturing 1 frame of image. The embodiment of the present application does not limit the method for calculating the offset of the gesture position. In a specific example, the offset of the gesture position can be taken as the displacement average of the gesture position coordinates detected in several consecutive frames. The offset of the gesture position of the first frame in the first detection defaults to 0. As Figure 5b As shown, after each gesture detection, if it is not the first detection, the offset of the gesture position detected at the current time can be calculated first (if it is the first detection, the offset of the gesture position defaults to 0). Then, after the corresponding hand area is determined for each target human body in the next first image by successively using the methods of face detection, body area expansion, key point detection, and hand area expansion, for example, the corresponding hand area 1 is determined for face area 1, and the hand area 1 can be corrected according to the calculated offset, so that the corrected hand area 1*. In this way, the determined gesture detection area can be made more accurate.

[0171] In the above scheme, considering the asynchronism between the image detected by feature points and the image detected by gestures, the gesture detection area in the second image is corrected by calculating the offset of the gesture position, so that the corrected gesture detection area is more accurate. Thus, the gesture detection accuracy can be further improved.

[0172] In some examples, under the premise of ensuring detection accuracy, in order to further improve detection efficiency and save computing resources, the steps of face detection and key point detection can be performed at a certain frequency. For example, it is performed once every 20ms or once every 5 frames. For example, the first image is the i-th frame, i1,6,11,...,1+5n; n is a positive integer, and the second image is the i+1-th frame. Then, for the case where the current frame is the i+2-th frame to the i+4-th frame, the gesture detection area of ​​the current frame can be determined based on the effective gesture area obtained by the gesture detection of the previous frame plus the calculated gesture position offset.

[0173] In addition, if Figure 5c As shown, in single-person mode, the gesture detection area can also be corrected by using the offset of the gesture position. Exemplarily, in single-person mode, only one person is allowed to activate the gesture at the same time, and during the activation period, only the person is allowed to continue to execute the gesture command. In this mode, it can be pre-set whose gesture will be detected and others will not be detected. The face detection model is used to detect the face area of ​​all human bodies in the image, and then the face recognition model can be used to identify the identity information of all face areas, find the target face, and continue to track. The human feature points detected for the human body area of ​​the target face are used to generate the hand area, and the image is cropped according to the area and sent to the gesture detection model for detection; if it is not the first time to detect the face, if the gesture of the person has been detected in the previous few frames, and the gesture movement of the previous frame has been calculated, then the hand area generated according to the human feature points this time needs to be corrected by adding the movement of the gesture position of the previous frame. The above process can lock the gesture of one person, and the gestures of others will not be detected, so as to achieve the gesture detection effect of the single-person mode.

[0174] The embodiment of the present application also provides a gesture detection device, which is used to execute the method steps in the above method embodiment. The device can be a virtual appliance in a terminal device, which is run by a processor of the terminal device, or it can be the terminal device itself.

[0175] like Figure 6 As shown, the gesture detection device 600 provided in the embodiment of the present application includes:

[0176] A determination module 610 is used to determine a first position of a feature point in a first image of a target object, wherein the feature point corresponds to a body key point of the target object;

[0177] The preprocessing module 620 is used to perform a preprocessing operation for gesture detection of the target object based on the first position; wherein the preprocessing operation includes a first preprocessing operation and / or a second preprocessing operation; the first preprocessing operation is used to determine a gesture detection area in the second image of the target object, so as to perform gesture detection on the gesture detection area; the second preprocessing operation is used to determine whether the gesture of the target object meets the activation conditions, so as to perform gesture detection when it is determined that the gesture of the target object meets the activation conditions.

[0178] In one embodiment, the preprocessing operation includes a first preprocessing operation, the feature point includes a wrist feature point corresponding to the wrist of the target object, and the preprocessing module 620 includes:

[0179] The area determination submodule is used to determine the gesture detection area according to the second position of the wrist feature point and the first size of the target facial area in the first image; wherein the target facial area corresponds to the face of the target object, the corresponding position of the second position in the second image falls within the gesture detection area, and the size of the gesture detection area is positively correlated with the first size.

[0180] In one embodiment, the region determination submodule includes:

[0181] a first determining unit, configured to determine a third position in the second image according to the second position of the wrist feature point, wherein a position of the third position in the second image is the same as a position of the second position in the first image;

[0182] A first expansion unit is used to expand the third position in the second image toward multiple surrounding expansion directions by corresponding second sizes to obtain multiple expansion boundaries, wherein the second size corresponding to each expansion direction is equal to the first size corresponding to the expansion direction multiplied by a preset multiple corresponding to the expansion direction;

[0183] The second determining unit is used to determine the gesture detection area according to the enclosed area of ​​the multiple extended boundaries.

[0184] In one embodiment, the feature point further includes an elbow feature point corresponding to the elbow of the target object, the multiple extension directions include a first height direction and a second height direction, and the gesture detection device 600 further includes:

[0185] A palm orientation determination module, for determining, for each wrist feature point of the target object, the palm orientation of the hand corresponding to the wrist feature point according to the relative position relationship between the wrist feature point and the elbow feature point associated with the wrist feature point;

[0186] The multiple determination module is used to determine a first preset multiple corresponding to a first height direction and a second preset multiple corresponding to a second height direction according to the direction of the palm; wherein the first preset multiple is greater than the second preset multiple, the first height direction is the same as the palm direction, and the second height direction is opposite to the palm direction.

[0187] In one implementation, the second determining unit is specifically configured to:

[0188] For each wrist feature point of the target object, the enclosed area determined for the wrist feature point is used as the hand area of ​​the hand corresponding to the wrist feature point, and two hand areas are obtained;

[0189] Determine whether the two hand regions overlap;

[0190] If yes, the two hand regions are merged, and the merged region is used as the gesture detection region;

[0191] If not, each hand area is used as a gesture detection area.

[0192] In one implementation, the second determining unit is specifically configured to:

[0193] For each wrist feature point of the target object, the enclosed area determined for the wrist feature point is used as a gesture detection area to obtain two gesture detection areas, wherein each gesture detection area corresponds to a hand of the target object;

[0194] The gesture detection device 600 further includes:

[0195] A gesture detection module, used to perform gesture detection on the two gesture detection areas respectively, and determine whether there is a valid area with valid gestures and the type of valid gestures in the two gesture detection areas;

[0196] a calculation module, configured to calculate first distances between the center of the effective area and the centers of the two gesture detection areas respectively, to obtain two first distances;

[0197] A hand determination module, used to determine the hand corresponding to the gesture detection area with the smaller first distance as the hand of the valid gesture;

[0198] The instruction determination module is used to determine the instruction information corresponding to the valid gesture according to the hand making the valid gesture and the type of the valid gesture.

[0199] In one embodiment, the preprocessing operation includes a second preprocessing operation, the feature points include wrist feature points corresponding to the wrist of the target object and elbow feature points corresponding to the elbow of the target object, and the preprocessing module 620 includes:

[0200] a third determining unit, configured to determine, for each wrist feature point of the target object, a palm orientation of the hand corresponding to the wrist feature point according to a relative position relationship between the wrist feature point and an elbow feature point associated with the wrist feature point;

[0201] The activation judgment unit is used to judge whether the gesture of the target object meets the activation condition according to the palm direction of each hand.

[0202] In one embodiment, the preprocessing operation further includes a first preprocessing operation, and the preprocessing module 620 further includes:

[0203] The fourth determining unit is configured to determine, under the condition that it is determined that the gesture of the target object meets the activation condition, a gesture detection area in the second image of the target object according to the first position.

[0204] In one embodiment, the first size includes a width of the target facial area, the feature points further include eye feature points corresponding to the eyes of the target object and / or ear feature points corresponding to the ears of the target object, and the gesture detection device 600 further includes:

[0205] The facial size determination module is used to determine the width of the target facial area according to the relative position relationship between the two eye feature points and / or the relative position relationship between the two ear feature points.

[0206] In one implementation, the gesture detection device 600 further includes:

[0207] A face detection module, configured to perform face detection on the first image to obtain a target face region corresponding to the face of a target object in the first image;

[0208] The determination module 610 includes:

[0209] A second expansion unit is used to expand the target facial region to obtain a body region corresponding to the target object;

[0210] The key point detection unit is used to detect the key points of the body area to obtain the first position of the feature point.

[0211] In one implementation, the second expansion unit is specifically configured to:

[0212] According to the width of the target facial region, the target facial region is expanded along the image width direction, and according to the height of the target facial region, the target facial region is expanded along the image height direction to obtain the body region.

[0213] In one implementation, when the current mode is the single target mode, the face detection module includes:

[0214] A face detection unit, configured to perform face detection on the first image to obtain an initial face region corresponding to each object;

[0215] A facial recognition unit, used to perform facial recognition on each initial facial region to obtain identity information of an object corresponding to each facial region;

[0216] The screening unit is used to screen out a target facial region from each initial facial region according to the identity information.

[0217] In one implementation, the second image is an image captured at a first moment when the first position is determined, and the first moment is later than a capture moment of the first image. The gesture detection device 600 further includes:

[0218] An offset determination module, used to determine an offset of a gesture position of a target object within a first time, wherein the first time is a duration from a collection moment to a first moment;

[0219] The pre-processing module 620 is also used for:

[0220] A gesture detection area is determined according to the first position and the offset.

[0221] like Figure 7 As shown, the embodiment of the present application further provides a terminal device 700, including: at least one processor 710 ( Figure 7 Only one processor is shown in the figure), a memory 720, and a computer program 730 stored in the memory 720 and executable on at least one processor 710. When the processor 710 executes the computer program 730, the steps of the above-mentioned gesture detection method are implemented.

[0222] The terminal device may include but is not limited to a processor and a memory. Figure 7It is only an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than shown in the figure, or combine certain components, or different components. The processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0223] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0224] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned function allocation can be completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The functional modules in the embodiment can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. In addition, the specific names of the functional modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0225] The embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned gesture detection method can be implemented.

[0226] An embodiment of the present application provides a computer program product. When the computer program product is executed on a terminal device, the terminal device can implement the steps in the above-mentioned gesture detection method.

[0227] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A gesture detection method, characterized in that: include: Determining a first position of a feature point in a first image of a target object, wherein the feature point corresponds to a body key point of the target object; performing a preprocessing operation for gesture detection of the target object according to the first position; Among them, the preprocessing operation includes a first preprocessing operation and / or a second preprocessing operation; the first preprocessing operation is used to determine a gesture detection area in the second image of the target object, so as to perform gesture detection on the gesture detection area; the second preprocessing operation is used to determine whether the gesture of the target object meets the activation conditions, so as to perform gesture detection when it is determined that the gesture of the target object meets the activation conditions.

2. The gesture detection method according to claim 1, characterized in that: The preprocessing operation includes the first preprocessing operation, the feature point includes a wrist feature point corresponding to a wrist of the target object, and the preprocessing operation of the gesture of the target object is performed according to the first position, including: determining the gesture detection area according to a second position of the wrist feature point and a first size of the target facial area in the first image; The target facial area corresponds to the face of the target object, the corresponding position of the second position in the second image falls within the gesture detection area, and the size of the gesture detection area is positively correlated with the first size.

3. The gesture detection method according to claim 2, characterized in that: The determining the gesture detection area according to the second position of the wrist feature point and the first size of the target facial area in the first image includes: determining a third position in the second image according to the second position of the wrist feature point, wherein a position of the third position in the second image is the same as a position of the second position in the first image; On the second image, the third position is respectively expanded in a plurality of surrounding expansion directions by corresponding second sizes to obtain a plurality of expansion boundaries, wherein the second size corresponding to each expansion direction is equal to the first size corresponding to the expansion direction multiplied by a preset multiple corresponding to the expansion direction; The gesture detection area is determined according to an enclosed area of ​​the multiple extended boundaries.

4. The gesture detection method according to claim 3, characterized in that: The feature points further include an elbow feature point corresponding to the elbow of the target object, the multiple extension directions include a first height direction and a second height direction, and the method further includes: For each wrist feature point of the target object, determining the palm orientation of the hand corresponding to the wrist feature point according to the relative position relationship between the wrist feature point and the elbow feature point associated with the wrist feature point; Determine, according to the palm orientation, a first preset multiple corresponding to the first height direction and a second preset multiple corresponding to the second height direction; The first preset multiple is greater than the second preset multiple, the first height direction is the same as the direction of the palm, and the second height direction is opposite to the direction of the palm.

5. The gesture detection method according to claim 3, characterized in that: The step of determining the gesture detection area according to the enclosed area of ​​the multiple extended boundaries includes: For each wrist feature point of the target object, the enclosed area determined for the wrist feature point is used as the hand area of ​​the hand corresponding to the wrist feature point, to obtain two hand areas; Determining whether the two hand regions overlap; If yes, the two hand regions are merged, and the merged region is used as the gesture detection region; If not, each hand area is used as a gesture detection area.

6. The gesture detection method according to claim 3, characterized in that: The step of determining the gesture detection area according to the enclosed area of ​​the multiple extended boundaries includes: For each wrist feature point of the target object, an enclosed area determined for the wrist feature point is used as a gesture detection area to obtain two gesture detection areas, wherein each gesture detection area corresponds to a hand of the target object; The method further comprises: Performing gesture detection on the two gesture detection areas respectively, determining whether there is a valid area with a valid gesture and the type of the valid gesture in the two gesture detection areas; If yes, then calculating the first distances between the center of the effective area and the centers of the two gesture detection areas respectively to obtain two first distances; Determine the hand corresponding to the gesture detection area with the smaller first distance as the hand of the valid gesture; Instruction information corresponding to the valid gesture is determined according to the hand making the valid gesture and the type of the valid gesture.

7. The gesture detection method according to any one of claims 1 to 6, characterized in that: The preprocessing operation includes the second preprocessing operation, or also includes the first preprocessing operation, the feature points include wrist feature points corresponding to the wrist of the target object and elbow feature points corresponding to the elbow of the target object, and performing the preprocessing operation on the gesture of the target object according to the first position includes: For each wrist feature point of the target object, determine the palm orientation of the hand corresponding to the wrist feature point according to the relative position relationship between the wrist feature point and the elbow feature point associated with the wrist feature point; and determine whether the gesture of the target object meets the activation condition according to the palm orientation of each hand; and / or In the case where the preprocessing operation includes the first preprocessing operation, under the condition that it is determined that the gesture of the target object meets the activation condition, a gesture detection area in the second image of the target object is determined according to the first position.

8. The gesture detection method according to any one of claims 2 to 6, characterized in that: The first size includes a width of the target facial area, the feature points also include eye feature points corresponding to the eyes of the target object and / or ear feature points corresponding to the ears of the target object, and the method further includes: The width of the target facial area is determined according to the relative position relationship between the two eye feature points and / or the relative position relationship between the two ear feature points.

9. The gesture detection method according to any one of claims 1 to 6, characterized in that: The method further comprises: Performing face detection on the first image to obtain a target face region in the first image corresponding to the face of the target object; The determining a first position of a feature point in a first image of the target object comprises: Expanding the target facial region to obtain a body region corresponding to the target object; Perform body key point detection on the body area to obtain the first position of the feature point.

10. The gesture detection method according to claim 9, characterized in that: The step of performing region expansion on the target facial region to obtain a body region corresponding to the body of the target object includes: According to the width of the target facial region, the target facial region is expanded along the image width direction, and according to the height of the target facial region, the target facial region is expanded along the image height direction to obtain the body region.

11. The gesture detection method according to claim 9, characterized in that: When the current mode is the single target mode, performing face detection on the first image to obtain a target face region corresponding to the face of the target object in the first image includes: Performing face detection on the first image to obtain initial face regions corresponding to each object; Performing facial recognition on each initial facial region to obtain identity information of the object corresponding to each facial region; The target facial region is screened out from various initial facial regions according to the identity information.

12. The gesture detection method according to any one of claims 1 to 6, characterized in that: The second image is an image captured at a first moment when the first position is determined, and the first moment is later than a moment when the first image is captured. The method further includes: Determine an offset of a gesture position of the target object within a first time, wherein the first time is a duration from the acquisition moment to the first moment; The performing a preprocessing operation of gesture detection on the target object according to the first position includes: The gesture detection area is determined according to the first position and the offset.

13. A gesture detection device, characterized in that: include: a determination module, configured to determine a first position of a feature point in a first image of a target object, wherein the feature point corresponds to a body key point of the target object; a preprocessing module, configured to perform a preprocessing operation for gesture detection of the target object according to the first position; Among them, the preprocessing operation includes a first preprocessing operation and / or a second preprocessing operation; the first preprocessing operation is used to determine a gesture detection area in the second image of the target object, so as to perform gesture detection on the gesture detection area; the second preprocessing operation is used to determine whether the gesture of the target object meets the activation conditions, so as to perform gesture detection when it is determined that the gesture of the target object meets the activation conditions.

14. A terminal device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the gesture detection method according to any one of claims 1 to 12 when executing the computer program.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the gesture detection method according to any one of claims 1 to 12 are implemented.

Citation Information

Patent Citations

  • System and method for gesture recognition of vehicle

    CN104699238A

  • Gesture recognition method and device and computer readable storage medium

    CN110008818A

  • Control method and control device of mobile robot and mobile robot

    CN114546114A

  • Gesture action recognition method and system and electronic equipment

    CN114863565A

  • Gesture semantic recognition method, electronic equipment, storage medium and program product

    CN116092110A

Cited By

  • Gesture detection method and apparatus, and terminal device and computer-readable storage medium

    WO2026026080A1