Hand detection method and hand key point detection model training method and device

Abstract

The embodiment of the invention provides a hand detection method and a hand key point detection model training method and device, and belongs to the technical field of target detection. The method comprises the following steps: detecting a current frame image through a hand detection model to obtain a current frame hand frame; acquiring a historical frame hand frame; calculating the intersection-to-union ratio between the hand frames; judging whether matching succeeds or not according to the matching threshold value and the intersection-union ratio; and if the matching is successful, inputting the historical frame key point and the current frame image into a hand key point detection model to obtain a heat map, taking a coordinate corresponding to a maximum value of a channel in the heat map as a target key point coordinate, and displaying the target key point coordinate in the current frame image. According to the embodiment of the invention, the historical frame can be used as the second input of the model, and jitter is reduced.

Description

Technical Field

[0001] The present application relates to the field of target detection technology, and in particular to a hand detection method, a hand key point detection model training method and a device. Background Art

[0002] Currently, camera hand key point detection technology has been widely used in many fields, such as gesture recognition, hand posture estimation, hand movement recognition, hand tracking, etc. In the future, camera hand key point detection technology will be applied in more fields, such as industrial automation, etc.

[0003] In the related technology, the existing hand key point detection method has the problem of large hand key point jitter. The hand detection method faces some challenges in dealing with hand jitter, and it is difficult to cope with various complex situations and difficult to be widely used in various scenarios.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to provide a hand detection method and device, an electronic device and a storage medium, aiming to reduce jitter during hand detection.

[0006] To achieve the above objective, an embodiment of the present application provides a hand detection method, which includes the following steps:

[0007] The hand detection model is used to detect the current frame image to obtain the current frame hand frame;

[0008] Acquire a historical frame hand frame, wherein the historical frame hand frame includes historical frame key points;

[0009] Calculating an intersection-over-union ratio between the current frame hand frame and the historical frame hand frame;

[0010] Obtaining a matching threshold, and judging whether the hand frame of the current frame matches the hand frame of the historical frame successfully by using the matching threshold and the intersection-and-union ratio;

[0011] If the match is successful, the historical frame key points and the current frame image are input into the hand key point detection model to obtain a heat map, and the coordinates corresponding to the maximum values ​​of the channels in the heat map are used as the target key point coordinates, and the target key point coordinates are displayed in the current frame image.

[0012] In some embodiments, the method further comprises:

[0013] If the match fails, the current frame hand frame is corrected based on preset rules, the hand area image is cropped from the current frame image according to the current frame hand frame, the hand area image and the empty key point coordinates are input into the hand key point detection model to obtain a heat map, the coordinates corresponding to the maximum values ​​of the channels in the heat map are used as the target key point coordinates, and the target key point coordinates are restored to the current frame image according to the preset rules.

[0014] In some embodiments, the step of modifying the current frame hand frame based on a preset rule and cropping a hand region image from the current frame image according to the current frame hand frame includes:

[0015] If the side lengths of the hand frame of the current frame are inconsistent, the short side of the hand frame of the current frame is symmetrically extended to be consistent with the long side of the hand frame of the current frame;

[0016] Enlarging the current hand frame by a preset multiple;

[0017] Cutting out a hand region image from the current frame image according to the current frame hand frame enlarged by a preset multiple;

[0018] The hand region image is scaled.

[0019] In some embodiments, the historical frame key points and the current frame image are input into the hand key point detection model to obtain a heat map, the coordinates corresponding to the maximum values ​​of the channels in the heat map are used as the target key point coordinates, and the target key point coordinates are displayed in the current frame image, including:

[0020] Correcting the hand frame of the current frame according to the key points of the historical frame, and calculating the affine transformation matrix of the hand frame of the current frame;

[0021] Cropping a hand region image from the current frame image according to the current frame hand frame;

[0022] Mapping the coordinates of the key points of the historical frame to the coordinates of the key points of the historical region image according to the affine transformation matrix;

[0023] Inputting the hand area image and the historical key point coordinates of the hand area image into the hand key point detection model to obtain a heat map;

[0024] The coordinates corresponding to the maximum values ​​of the channels in the heat map are used as the target key point coordinates, the target key point coordinates are restored to the current frame image according to the affine transformation matrix, and the target key point coordinates are displayed in the current frame image.

[0025] In some embodiments, the historical frame key points include historical frame direction key points, and the correcting the current frame hand frame according to the historical frame key points and calculating the affine transformation matrix of the current frame hand frame include:

[0026] If the current frame hand frame does not include all the key points of the historical frames, expanding the current frame hand frame until the current frame hand frame includes all the key points of the historical frames;

[0027] If the side lengths of the hand frame of the current frame are inconsistent, the short side of the hand frame of the current frame is symmetrically extended to be consistent with the long side of the hand frame of the current frame;

[0028] Enlarging the current hand frame by a preset multiple;

[0029] Calculate the rotation angle according to the coordinates of the key points in the historical frame direction;

[0030] The affine transformation matrix of the current frame hand frame is calculated according to the corrected center point of the current frame hand frame, the side length of the current frame hand frame, the rotation angle, and the input size of the hand key point detection model.

[0031] In some embodiments, the obtaining of a matching threshold and judging whether the current frame hand frame successfully matches the historical frame hand frame by using the matching threshold and the intersection-and-union ratio include:

[0032] Determine the best pairing and the best intersection-and-union ratio of the current frame hand frame and the historical frame hand frame according to the matching algorithm and the intersection-and-union ratio;

[0033] Get the matching threshold;

[0034] If the best intersection-over-union ratio is greater than or equal to the matching threshold, it is determined that the hand frame of the current frame matches the hand frame of the historical frame successfully;

[0035] If the best intersection-over-union ratio is less than the matching threshold, it is determined that the current frame hand frame fails to match the historical frame hand frame.

[0036] An aspect of an embodiment of the present application provides a method for training a hand key point detection model, the method comprising the following steps:

[0037] Collect hand images in various scenes, and annotate the hand images with hand frames and several key points;

[0038] Generate a true value heat map according to the key points of the hand image by using a Gaussian function;

[0039] Perturbing the key points of the hand image to obtain a second input;

[0040] Inputting the second input and the hand image into a hand key point detection model for training to obtain a current heat map;

[0041] The loss value between the current heat map and the true value heat map is calculated according to the loss function, and the hand key point detection model is back-propagated according to the loss value until the loss value is less than the preset loss threshold, so as to obtain the trained hand key point detection model, which is used to perform hand detection through the above-mentioned hand detection method.

[0042] In some embodiments, the perturbing the key points of the hand image to obtain the second input includes:

[0043] Dividing the key points of the hand image into a first part of key points and a second part of key points;

[0044] Perturbing the first part of key points according to a preset perturbation multiple, wherein the first part of key points are used for subsequent frame training;

[0045] Setting the coordinates of the second part of key points to zero, wherein the second part of key points are used for initial frame training;

[0046] The first part of key points after disturbance and the second part of key points after zeroing are used as the second input.

[0047] To achieve the above object, another aspect of an embodiment of the present application provides a hand detection device, the device comprising:

[0048] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the hand detection method described above when executing the computer program.

[0049] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the hand detection method described above is implemented.

[0050] The embodiments of the present application include at least the following beneficial effects: the present application provides a hand detection method and device, an electronic device and a storage medium, which detects the current frame image through a hand detection model to obtain a current frame hand frame; obtains the historical frame hand frame to achieve continuous hand tracking without relying on the detection results of a single frame; calculates the intersection-and-union ratio between the current frame hand frame and the historical frame hand frame; obtains a matching threshold, and determines whether the current frame hand frame and the historical frame hand frame are successfully matched through the matching threshold and the intersection-and-union ratio, thereby improving the matching accuracy of the hand frame and enhancing the robustness of hand detection; if the match is successful, the historical frame key points and the current frame image are input into the hand key point detection model to obtain a heat map, and the coordinates corresponding to the maximum values ​​of the channels in the heat map are used as the target key point coordinates, and the target key point coordinates are displayed in the current frame image, and the historical frame key point coordinates are used as the second input of the hand key point detection model, thereby reducing jitter according to the dependency relationship between adjacent frames and improving the accuracy of hand detection.

[0051] The present application also provides a method for training a hand key point detection model, which collects hand images in various scenarios, annotates hand images with hand frames and several key points; generates a true value heat map based on the key points of the hand image through a Gaussian function; perturbs the key points of the hand image to obtain a second input; inputs the second input and the hand image into the hand key point detection model for training to obtain a current heat map, and reduces the jitter during hand detection through the second input, which is beneficial for the model to understand the distribution of key points in the image and the relationship between historical frames and subsequent frames; calculates the loss value of the current heat map and the true value heat map according to the loss function, and performs back propagation on the hand key point detection model according to the loss value until the loss value is less than a preset loss threshold, thereby obtaining a trained hand key point detection model, which is used for hand detection, and can detect the initial frame and subsequent frames, and reduces the jitter according to the dependency relationship between adjacent frames, thereby improving the matching accuracy of the model for the hand frame and the robustness of hand tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flow chart of a hand detection method provided in an embodiment of the present application;

[0053] Figure 2 is a flowchart of the matching failure step of the hand detection method provided in an embodiment of the present application;

[0054] Figure 3 yes Figure 1 Flow chart of step S105 in FIG.

[0055] Figure 4 yes Figure 1 Flow chart of step S104 in FIG.

[0056] Figure 5 is a flow chart of a hand key point detection model training method provided in an embodiment of the present application;

[0057] Figure 6 is a schematic diagram of a marked hand image provided in an embodiment of the present application;

[0058] Figure 7 is an optional flow chart of the hand detection method provided in an embodiment of the present application;

[0059] Figure 8 is a flowchart of hand key point detection in an initial frame provided by an embodiment of the present application;

[0060] Fig. 9 is a subsequent frame hand key point detection flow chart provided in an embodiment of the present application;

[0061] Fig.10 This is a heatmap network diagram provided in an embodiment of the present application;

[0062] Fig.11 is a structural schematic diagram of a hand detection device provided in an embodiment of the present application;

[0063] Fig.12 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0065] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0066] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0068] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0069] 1) Heatmap network, used for tasks such as target detection, pose estimation, image segmentation, etc., to generate heatmaps.

[0070] Currently, camera hand key point detection technology has been widely used in many fields, such as gesture recognition, hand posture estimation, hand movement recognition, hand tracking, etc. In the future, camera hand key point detection technology will be applied in more fields, such as industrial automation, etc.

[0071] In the related technology, the existing hand key point detection method has the problem of large hand key point jitter. The hand detection method faces some challenges in dealing with hand jitter, and it is difficult to cope with various complex situations and difficult to be widely used in various scenarios.

[0072] In summary, the technical problems existing in the relevant technologies need to be improved.

[0073] In view of this, a hand detection method, apparatus, device and medium are provided in the embodiments of the present application. The scheme detects the current frame image through a hand detection model to obtain a current frame hand frame; obtains the historical frame hand frame to realize continuous hand tracking without relying on the detection result of a single frame; calculates the intersection-and-union ratio between the current frame hand frame and the historical frame hand frame; obtains a matching threshold, and judges whether the current frame hand frame and the historical frame hand frame are successfully matched through the matching threshold and the intersection-and-union ratio, thereby improving the matching accuracy of the hand frame and enhancing the robustness of hand detection; if the match is successful, the historical frame key points and the current frame image are input into the hand key point detection model to obtain a heat map, and the coordinates corresponding to the maximum values ​​of the channels in the heat map are used as the target key point coordinates, and the target key point coordinates are displayed in the current frame image. The historical frame key point coordinates are used as the second input of the hand key point detection model, thereby reducing jitter according to the dependency relationship between adjacent frames and improving the accuracy of hand detection.

[0074] The present application also provides a method for training a hand key point detection model, which collects hand images in various scenarios, annotates hand images with hand frames and several key points; generates a true value heat map based on the key points of the hand image through a Gaussian function; perturbs the key points of the hand image to obtain a second input; inputs the second input and the hand image into the hand key point detection model for training to obtain a current heat map, and reduces the jitter during hand detection through the second input, which is beneficial for the model to understand the distribution of key points in the image and the relationship between historical frames and subsequent frames; calculates the loss value of the current heat map and the true value heat map according to the loss function, and performs back propagation on the hand key point detection model according to the loss value until the loss value is less than a preset loss threshold, thereby obtaining a trained hand key point detection model, which is used for hand detection, and can detect the initial frame and subsequent frames, and reduces the jitter according to the dependency relationship between adjacent frames, thereby improving the model's matching accuracy for the hand frame and the robustness of hand tracking.

[0075] The hand detection method and the hand key point detection model training method provided in the embodiment of the present application relate to the field of target detection. The hand detection method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The cloud server, the server can also be a node server in a blockchain network; the software can be an application that implements the hand detection method, etc., but is not limited to the above forms.

[0076] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0077] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0078] Figure 1 is an optional flow chart of the hand detection method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0079] Step S101, detecting the current frame image by using a hand detection model to obtain a current frame hand frame.

[0080] Specifically, the hand box is the bounding box of the hand.

[0081] In some embodiments, the position of the hand is detected from the current frame image by using a trained hand detection model, wherein the hand detection model recognizes the hand frame by using the features obtained by training, and the hand detection model may adopt a convolutional neural network.

[0082] It can be understood that the hand frame of the current frame is represented in the form of a rectangular frame, and the position and size of the frame are predicted by the regression method.

[0083] In this embodiment, the current frame image is detected by a hand detection model to obtain a current frame hand frame, which provides the positioning of the hand in the current frame and provides input for subsequent key point recognition.

[0084] Step S102, obtaining a historical frame hand frame.

[0085] Specifically, the historical frame hand frame includes historical frame key points.

[0086] It can be understood that the historical frame hand frame includes the previous frame hand frame.

[0087] In some embodiments, a pre-stored historical frame hand frame is obtained.

[0088] In this embodiment, obtaining the historical frame hand frame is conducive to achieving continuous hand tracking and helps to solve the problem of hand key point jitter.

[0089] Step S103, calculating the intersection-over-union ratio between the hand frame of the current frame and the hand frame of the historical frame.

[0090] In some embodiments, the intersection-over-union ratio is calculated between the hand frame of the current frame and the hand frame of the previous frame.

[0091] The intersection-union ratio is calculated by dividing the area of ​​the intersection region by the area of ​​the union region.

[0092] It can be understood that by comparing the intersection-over-union ratio between the hand frame of the current frame and the hand frame of the historical frame, their overlap degree can be determined.

[0093] In this embodiment, the intersection-over-union ratio between the hand frame of the current frame and the hand frame of the historical frame is calculated, which is beneficial to the subsequent hand frame matching.

[0094] Step S104, obtaining a matching threshold, and judging whether the hand frame of the current frame matches the hand frame of the historical frame successfully by using the matching threshold and the intersection-and-union ratio.

[0095] In some embodiments, the best pairing and the best intersection-and-union ratio of the current frame hand frame and the historical frame hand frame are determined according to the matching algorithm and the intersection-and-union ratio; and a matching threshold is obtained. The matching algorithm includes but is not limited to greedy matching and Hungarian matching algorithm, the best pairing includes a current frame hand frame and its corresponding historical frame hand frame, and the best intersection-and-union ratio is the intersection-and-union ratio of the best paired hand frames.

[0096] It is understandable that the matching threshold can be dynamically adjusted according to different scenarios and user needs.

[0097] In some embodiments, if the best intersection-over-union ratio is greater than or equal to the matching threshold, it is determined that the hand frame of the current frame matches the hand frame of the historical frame successfully.

[0098] Furthermore, if the best intersection-over-union ratio is less than the matching threshold, it is determined that the hand frame of the current frame fails to match the hand frame of the historical frame.

[0099] In this embodiment, a matching threshold is obtained, and the matching threshold and intersection-over-union ratio are used to determine whether the current frame hand frame matches the historical frame hand frame successfully, thereby helping to accurately determine whether to use the information of the historical frame, thereby improving the matching accuracy of the hand frame and the robustness of hand tracking.

[0100] Step S105, if the match is successful, the historical frame key points and the current frame image are input into the hand key point detection model to obtain a heat map, and the coordinates corresponding to the maximum values ​​of the channels in the heat map are used as the target key point coordinates, and the target key point coordinates are displayed in the current frame image.

[0101] It can be understood that if the match is successful, the current frame can be corrected by the historical frame.

[0102] In some embodiments, the current frame hand frame is corrected according to the historical frame key points, and the affine transformation matrix of the current frame hand frame is calculated; the hand area image is cropped from the current frame image according to the current frame hand frame; the coordinates of the historical frame key points are mapped to the historical key point coordinates of the hand area image according to the affine transformation matrix; the hand area image and the historical key point coordinates of the hand area image are input into the hand key point detection model to obtain a heat map; the coordinates corresponding to the maximum values ​​of the channels in the heat map are used as the target key point coordinates, the target key point coordinates are restored to the current frame image according to the affine transformation matrix, and the target key point coordinates are displayed in the current frame image.

[0103] Optionally, the key points of the historical frames and the current image are input, and the model combines the historical information and the current image to predict the key point positions in the current image.

[0104] Among them, the hand key point detection model outputs a heat map, each heat map channel corresponds to a key point, and each pixel value in the heat map indicates the possibility that the position is the corresponding key point.

[0105] The coordinates of each key point are found through the maximum value of the heat map as the predicted position of the key point in the current frame.

[0106] Finally, the coordinates of these key points are displayed in the current frame image, usually by marking or annotating the location information of these key points.

[0107] Wherein, the current frame hand frame is corrected according to the key points of the historical frame. Optionally, the key points of the historical frame include the key points of the direction of the historical frame.

[0108] Specifically, if the current frame hand frame does not contain all the key points of the historical frames, the current frame hand frame is expanded until the current frame hand frame contains all the key points of the historical frames; if the side lengths of the current frame hand frame are inconsistent, the short side of the current frame hand frame is symmetrically expanded to be consistent with the long side of the current frame hand frame; the current frame hand frame is enlarged by a preset multiple; the rotation angle is calculated according to the coordinates of the key points in the historical frame directions; the affine transformation matrix of the current frame hand frame is calculated according to the corrected center point of the current frame hand frame, the side length of the current frame hand frame, the rotation angle and the input size of the hand key point detection model.

[0109] It can be understood that the affine transformation matrix of the current frame hand frame is calculated according to the correction data of the current frame hand frame, and the hand area image is cropped according to the correction data of the current frame hand frame. The correction data of the current frame hand frame is generated in the correction process of the current frame hand frame.

[0110] Furthermore, the coordinates of the key points of the historical frames are mapped to the coordinates of the historical key points of the hand region image according to the affine transformation matrix.

[0111] In some embodiments, the hand area image and the historical key point coordinates of the hand area image are input into a hand key point detection model to obtain a heat map.

[0112] Furthermore, the coordinates corresponding to the maximum values ​​of the channels in the heat map are used as the coordinates of the target key points, and the coordinates of the target key points are restored to the current frame image according to the affine transformation matrix.

[0113] It can be understood that the pixel in the heat map represents the probability that the position is a key point, and the maximum value of each channel in the heat map is extracted and the corresponding coordinates are found. The coordinates corresponding to the maximum value represent the position of the key point in the hand area image.

[0114] It should be noted that if the matching fails, the current frame hand frame is corrected based on the preset rules, the hand area image is cropped from the current frame image according to the current frame hand frame, the hand area image and the empty key point coordinates are input into the hand key point detection model to obtain a heat map, the coordinates corresponding to the maximum value of the channel in the heat map are used as the target key point coordinates, and the target key point coordinates are restored to the current frame image according to the preset rules.

[0115] In this embodiment, if the match is successful, the historical frame key points and the current frame image are input into the hand key point detection model to obtain a heat map, and the coordinates corresponding to the maximum values ​​of the channels in the heat map are used as the target key point coordinates, and the target key point coordinates are displayed in the current frame image. By fusing the historical frame information, the noise and hand shake problems in a single frame can be overcome, thereby enhancing the accuracy and robustness of hand key point detection.

[0116] In steps S101 to S105 shown in the embodiment of the present application, the current frame image is detected by a hand detection model to obtain a current frame hand frame; the historical frame hand frame is obtained to achieve continuous hand tracking without relying on the detection result of a single frame; the intersection-and-union ratio between the current frame hand frame and the historical frame hand frame is calculated; a matching threshold is obtained, and the matching threshold and the intersection-and-union ratio are used to determine whether the current frame hand frame and the historical frame hand frame are successfully matched, thereby improving the matching accuracy of the hand frame and enhancing the robustness of hand detection; if the match is successful, the historical frame key points and the current frame image are input into the hand key point detection model to obtain a heat map, and the coordinates corresponding to the maximum values ​​of the channels in the heat map are used as the target key point coordinates, and the target key point coordinates are displayed in the current frame image, and the historical frame key point coordinates are used as the second input of the hand key point detection model, thereby reducing jitter according to the dependency relationship between adjacent frames and improving the accuracy of hand detection.

[0117] See also Figure 2 In some embodiments, the hand detection method provided in the embodiment of the present application further includes a matching failure step, and the matching failure step may include but is not limited to steps S201 to S205:

[0118] Step S201, modifying the hand frame of the current frame based on a preset rule.

[0119] In step S201 of some embodiments, if the side lengths of the current frame hand frame are inconsistent, the short side of the current frame hand frame is symmetrically expanded to be consistent with the long side of the current frame hand frame; the current frame hand frame is enlarged by a preset multiple; the hand area image is cropped from the current frame image according to the current frame hand frame; and the hand area image is scaled.

[0120] It can be understood that the preset rules include side length synchronization, hand frame expansion, image cropping, and image scaling.

[0121] In this embodiment, the hand frame of the current frame is corrected based on preset rules to improve the accuracy of the hand area.

[0122] Step S202, cropping a hand area image from the current frame image according to the current frame hand frame.

[0123] In step S202 of some embodiments, a hand region image is cropped from the current frame image according to the corrected current frame hand frame.

[0124] Step S203, input the hand area image and the coordinates of the empty key points into the hand key point detection model to obtain a heat map.

[0125] It can be understood that a matching failure means that no historical frame hand frame matches the current frame hand frame, that is, the current frame does not have a corresponding historical frame.

[0126] In step S203 of some embodiments, since there is no corresponding historical frame, the second input of the hand key point detection model (i.e., the historical key point coordinates of the hand region image) is empty. The hand region image and the empty key point coordinates are input into the hand key point detection model to obtain a heat map.

[0127] Step S204, taking the coordinates corresponding to the maximum value of the channel in the heat map as the coordinates of the target key point.

[0128] In step S204 of some embodiments, the coordinates corresponding to the maximum value points of each channel in the heat map are used as the target key point coordinates, wherein the maximum value points correspond to the predicted positions of the key points in the heat map.

[0129] Step S205, restoring the target key point coordinates to the current frame image according to a preset rule.

[0130] In step S205 of some embodiments, the target key point coordinates are mapped and restored to the current frame image according to a preset rule.

[0131] See also Figure 3 In some embodiments, step S105 may include but is not limited to steps S301 to S305:

[0132] Step S301: If the current frame hand frame does not include all the key points of the historical frames, the current frame hand frame is expanded until the current frame hand frame includes all the key points of the historical frames.

[0133] In step S301 of some embodiments, since the key points of the historical frames need to be mapped to the current image, the hand frame of the current frame needs to include all the key points of the historical frames.

[0134] Step S302: If the side lengths of the hand frame of the current frame are inconsistent, the short side of the hand frame of the current frame is symmetrically extended to be consistent with the long side of the hand frame of the current frame.

[0135] In step S302 of some embodiments, the short side of the hand frame of the current frame is extended to be consistent with the long side of the hand frame of the current frame to achieve side length synchronization.

[0136] Step S303, enlarging the current hand frame by a preset multiple.

[0137] In step S303 of some embodiments, the current hand frame is enlarged by a preset multiple.

[0138] Among them, the preset multiples can be dynamically adjusted according to the application scenario or user needs.

[0139] Step S304, calculating the rotation angle according to the coordinates of the key points in the historical frame direction.

[0140] Specifically, the historical frame key points include historical frame direction key points, and the historical frame direction key points are used to determine and adjust the image direction.

[0141] In step S304 of some embodiments, the palm key point and the middle finger key point are determined as direction key points, and the direction from the palm key point to the middle finger key point is vertically upward as a rule for image rotation.

[0142] It can be understood that the rotation angle is calculated using the coordinates of the key points in the historical frame direction according to the image rotation rule.

[0143] Step S305, calculating the affine transformation matrix of the hand frame of the current frame according to the corrected center point of the hand frame of the current frame, the side length and rotation angle of the hand frame of the current frame, and the input size of the hand key point detection model.

[0144] In step S305 of some embodiments, the affine transformation matrix of the current frame hand frame is calculated according to the corrected center point of the current frame hand frame, the side length of the current frame hand frame, the rotation angle, and the input size of the hand key point detection model. The affine transformation matrix represents the conversion of the hand frame from the original coordinate system to the target coordinate system, and the affine transformation includes the above-mentioned hand frame correction operations such as translation, scaling, and rotation.

[0145] See also Figure 4 In some embodiments, step S104 may include but is not limited to steps S401 to S404:

[0146] Step S401 : determining the best pairing and the best IoU ratio of the hand frame of the current frame and the hand frame of the historical frame according to the matching algorithm and IoU ratio.

[0147] In step S401 of some embodiments, the largest IoU pairing is selected from all IoU ratios between the current frame hand frame and the historical frame hand frame according to a matching algorithm to be determined as the best pairing between the current frame and the historical frame, and the largest IoU ratio is determined as the best IoU ratio.

[0148] It is understandable that when there are multiple current frame hand frames and historical frame hand frames, the matching problem of multiple current frame hand frames and historical frame hand frames can be processed by a data association algorithm, wherein the data association algorithm includes KLT tracking and Kalman filtering algorithms.

[0149] In this embodiment, the best pairing and the best intersection-and-union ratio of the current frame hand frame and the historical frame hand frame are determined according to the matching algorithm and the intersection-and-union ratio, so as to improve the accuracy and efficiency of the matching between the current frame hand frame and the historical frame hand frame.

[0150] Step S402, obtaining a matching threshold.

[0151] It is understandable that the matching threshold may be predetermined or dynamically adjusted according to application scenarios and user needs.

[0152] Step S403: if the best intersection-over-union ratio is greater than or equal to the matching threshold, it is determined that the hand frame of the current frame matches the hand frame of the historical frame successfully.

[0153] In step S403 of some embodiments, if the best intersection-over-union ratio is greater than or equal to the matching threshold, it is determined that the hand frame of the current frame matches the hand frame of the historical frame successfully, and the historical frame is input into the hand key point detection model as the second input data.

[0154] Step S404: if the best intersection-over-union ratio is less than the matching threshold, it is determined that the hand frame of the current frame fails to match the hand frame of the historical frame.

[0155] In step S403 of some embodiments, if the best intersection-over-union ratio is less than the matching threshold, it is determined that the hand frame of the current frame fails to match the hand frame of the historical frame, the current frame does not have a corresponding historical frame, and the current frame is an initial frame.

[0156] Figure 5 is an optional flowchart of the hand key point detection model training method provided in the embodiment of the present application, Figure 5 The method may include but is not limited to steps S501 to S505.

[0157] Step S501, collect hand images in various scenes, and annotate the hand images with hand frames and several key points.

[0158] In step S501 of some embodiments, hand images containing different gestures, different backgrounds, different lighting, and different hand postures are collected.

[0159] Furthermore, the hand image is annotated with the hand frame and the positions of several key points.

[0160] For example, the schematic diagram of the annotated hand image is as follows: Figure 6 shown.

[0161] Furthermore, the annotated hand image is preprocessed. First, the short side of the hand frame is symmetrically expanded to be consistent with the long side, and then the hand area is randomly translated, rotated, and scaled. The translation scale is within 0.5 times the side length, and the scaling range is between 1.0-1.5 times the side length. When rotating, first calculate the angle of the line connecting No. 0 and No. 9, as well as the length of the line segment L0, 9 composed of No. 0 and No. 9, and the length of the line segment L5, 17 composed of No. 5 and No. 17. When L0, 9 is less than 0.7 times of L5, 17, a 360-degree random rotation is performed. Otherwise, the direction of the line connecting No. 0 and No. 9 is turned to the upward [-45, 45] range with a probability of 90%, and a 360-degree random selection is performed with a probability of 10%. Other enhancement methods, such as noise enhancement and color enhancement, are also used. Scale the image to the size of the model input. It can be understood that the key points of No. 0 and No. 9 are direction key points.

[0162] Step S502, generating a true value heat map according to the key points of the hand image by using a Gaussian function.

[0163] In step S502 of some embodiments, after the hand image is preprocessed, a true value heat map is generated according to the key points of the hand image using a Gaussian function.

[0164] In this embodiment, a true value heat map is generated according to the key points of the hand image through a Gaussian function. The method of generating the heat map by Gaussian makes it possible for the model to not be affected by a single pixel during training and to learn the key point positions more stably.

[0165] Step S503: perturb the key points of the hand image to obtain a second input.

[0166] In step S503 of some embodiments, the key points of the hand image are divided into a first part of key points and a second part of key points; the key points of the first part are perturbed according to a preset perturbation multiple, and the key points of the first part are used for subsequent frame training; the coordinates of the key points of the second part are set to zero, and the key points of the second part are used for initial frame training; the perturbed key points of the first part and the zeroed key points of the second part are used as the second input. Among them, the key points of the first part are more than the key points of the second part.

[0167] Exemplarily, the coordinates of the key points are set to all 0 with a probability of 30%. The second part of key points are 30% of the key points of the original hand image.

[0168] In this embodiment, the key points of the hand image are perturbed to obtain the second input, which is not only beneficial to prevent overfitting of the model, but also beneficial to the model reasoning initial frame.

[0169] Step S504: input the second input and the hand image into a hand key point detection model for training to obtain a current heat map.

[0170] In step S504 of some embodiments, a set of coordinates is used as a second input hand key point detection model.

[0171] Among them, the training of the hand key point detection model is divided into two stages, A and B. The A stage contains 5 downsampling modules and 3 upsampling modules. Each module contains several basic layers such as convolution and activation. The second input is concatenated (joined / connected) with the layer after the 5th downsampling through the full connection and reshape layer (reshaping layer / deformation layer), and then restored to the main dimension before concat through the convolution layer. The A stage generates a heat map through convolution after 3 upsampling. One or more B stages can be stacked. Each B stage first concatenates the output of the previous stage and the heat map generated in the previous stage, and then restores it to the main dimension before concat through a convolution. Then, after 3 downsampling and 3 upsampling, another heat map is convolved after the last 3 sampling. After the model training is completed, only the heat map of the last stage is retained as the current heat map when exporting.

[0172] In this embodiment, the second input and the hand image are input into the hand key point detection model for training to obtain the current heat map, which is helpful for the model to understand the distribution of key points in the image and the relationship between historical frames and subsequent frames.

[0173] Step S505, calculate the loss value of the current heat map and the true value heat map according to the loss function, and perform backpropagation on the hand key point detection model according to the loss value until the loss value is less than the preset loss threshold, so as to obtain the trained hand key point detection model.

[0174] In step S505 of some embodiments, the loss value of the current heat map and the true heat map is calculated by the loss function, the gradient is calculated by the loss value, and then the model parameters are updated using the back propagation algorithm until the loss value converges to a preset loss threshold. Specifically, the loss function can use mean square error, mean absolute error, heat map loss, angle loss, etc.

[0175] In this embodiment, the loss value of the current heat map and the true value heat map is calculated according to the loss function, and the hand key point detection model is back-propagated according to the loss value until the loss value is less than the preset loss threshold, so as to obtain the trained hand key point detection model, which is beneficial for the model to locate the hand key points and enhance the generalization ability of the model.

[0176] Figure 7 is an optional flow chart of the hand detection method provided in the embodiment of the present application. Figure 7 The method may include but is not limited to the following steps:

[0177] Step 1: Get the image.

[0178] Wherein, the video stream is processed, and the frame rate of the video stream is above 15 frames. For each frame of the video stream, a frame is first extracted to obtain an image (ie, the current frame image).

[0179] Step 2, detect hands from the image.

[0180] Among them, the hand detection model is used to detect all hand frames, and the IoU (i.e., intersection over union) is calculated between the hand frame of the current frame and all hand frames of the previous frame.

[0181] Step 3: Perform Hungarian matching with the hand in the previous frame based on the IOU.

[0182] Specifically, the history frame adopts the previous frame of the current frame.

[0183] Among them, Hungarian matching is used to obtain the best pairing. Hands with an IoU higher than 0.5 (i.e., the matching threshold) are considered matched, and the others are considered unmatched. For unmatched hands, the initial frame hand key point detection process of step 4 is performed, and for matched hands, the subsequent frame hand key point detection process of step 5 is performed.

[0184] Step 4: If the hand frame of the current frame does not match the hand frame of the previous frame, perform hand key point detection in the initial frame.

[0185] It can be understood that if the hand frame of the current frame does not match the hand frame of the previous frame, the current frame is the initial frame, and the initial frame hand key point detection is performed.

[0186] For example, the initial frame hand key point detection flow chart is as follows: Figure 8 As shown in the figure, the short side of the hand frame is symmetrically expanded to be consistent with the long side, and then the frame is expanded to 1.2 times, this part of the area is cut out on the image and scaled to the input size of the heatmap model, the second input pre_coords is set to a full 0 vector (42 dimensions), and the two inputs are sent to the heatmap model to obtain the heatmap result. The maximum value of each channel heatmap and the corresponding coordinates are selected as the coordinates of the key point, and then the coordinates are restored to the original image according to the translation and scaling information.

[0187] Step 5: If the hand frame of the current frame matches the hand frame of the previous frame, the hand key point detection of the subsequent frame is performed.

[0188] It can be understood that if the hand frame of the current frame matches the hand frame of the previous frame, the subsequent frame is the initial frame, and the hand key point detection of the subsequent frame is performed based on the previous frame and the current frame.

[0189] For example, the subsequent frame hand key point detection flow chart is as follows: Fig. 9As shown. In order to prevent the hand detector from detecting a box that is too small and causing some key points to be unpredictable, the hand box is first corrected according to the key point coordinates of the previous frame. If the box does not frame all the key points of the previous frame, it is expanded to frame them. The short side of the corrected hand box is expanded to be consistent with the long side, and then the hand box is expanded to 1.2 times. The coordinates of key points 0 and 9 in the previous frame are taken to calculate the rotation angle. The goal is that 0 points to 9 and is vertically upward. According to the center point of the corrected hand box, the side length of the enlarged hand box, the rotation angle, and the input size of the hand key point detection model, the affine transformation matrix of the hand box of the current frame is calculated, and then the hand area image is cropped based on the affine transformation. At the same time, the coordinates of the hand key points of the previous frame are mapped to the cropped coordinates according to the affine transformation matrix. The cropped image and the cropped coordinates are input into the hand key point detection model to obtain a heatmap, and the maximum value and corresponding coordinates of each layer of the heatmap are selected as the coordinates of the key point (i.e., the target key point coordinates), and then restored to the original image (i.e., the current frame image) according to the translation, rotation and scaling information.

[0190] In some embodiments, the training phase of the hand keypoint detection model includes the following steps:

[0191] Step 1, data collection.

[0192] Among them, hand images in various scenes are collected, and hand frames and 21 key points are annotated, such as Figure 6 shown.

[0193] Step 2: Image preprocessing.

[0194] Specifically, the short side of the hand frame is first symmetrically expanded to be consistent with the long side, and then the hand area is randomly translated, rotated, and scaled. The translation scale is within 0.5 times the side length, and the scaling range is between 1.0-1.5 times the side length.

[0195] Among them, the distribution of hand key points is Figure 6 For example, when rotating, first calculate the angle of the line connecting 0 and 9, as well as the length of the line segment composed of 0 and 9, L0, 9, and the length of the line segment composed of 5 and 17, L5, 17. When L0, 9 is less than 0.7 times of L5, 17, perform a random rotation within a 360-degree range. Otherwise, with a probability of 90%, turn the direction of the line connecting 0 and 9 to the upward range of [-45, 45], and with a probability of 10%, perform a random selection within a 360-degree range. Other enhancement methods, such as noise enhancement and color enhancement, are also used. The final image is scaled to the size of the model input, and the key points use the Gaussian function to generate the corresponding heatmap.

[0196] Step 3, the perturbed coordinates are used as the second input.

[0197] To simulate the dependency between previous and next frames of the video, the coordinates of the key points of the hand are perturbed to a certain extent as the second input, and the maximum perturbation amplitude does not exceed 0.2 times the side length of the preprocessed image. In order to allow the initial frame model to reason normally, the coordinates of the second input are set to all 0 with a probability of 30%.

[0198] Step 4: Network construction.

[0199] Optionally, the hand key point detection model can use a heatmap network. The heatmap network diagram is as follows Fig.10 shown.

[0200] Among them, compared with the existing heatmap network, the most significant difference of this application is that a set of coordinates is integrated into the network as the second input. Overall, the network is divided into two stages, A and B. Stage A contains 5 downsampling modules and 3 upsampling modules. Each module contains several basic layers such as convolution and activation. The second input is concatenated with the layer after the 5th downsampling through the full connection and reshape layer, and then restored to the main dimension before concatenation through the convolution layer. Stage A generates a heatmap through convolution after 3 upsampling. Stage B can be stacked one or more times. Each stage B first concatenates the output of the previous stage and the heatmap generated in the previous stage, and then restores it to the main dimension before concatenation through a convolution. Then, after 3 downsampling and 3 upsampling, another heatmap is convolved after the last 3 sampling. After the model training is completed, only the heatmap (i.e., heat map) of the last stage is retained when exporting.

[0201] Step 5: Calculate the loss function.

[0202] Among them, the heatmap generated in each stage calculates the L2 loss (mean square error) with the true value heatmap.

[0203] See also Fig.11 The present application also provides a hand detection device that can implement the above hand detection method. The device includes:

[0204] The image detection module 1101 is used to detect the current frame image through a hand detection model to obtain a current frame hand frame;

[0205] A historical frame acquisition module 1102 is used to acquire a historical frame hand frame, where the historical frame hand frame includes historical frame key points;

[0206] A calculation module 1103 is used to calculate the intersection-over-union ratio between the hand frame of the current frame and the hand frame of the historical frame;

[0207] A matching module 1104 is used to obtain a matching threshold, and determine whether the hand frame of the current frame matches the hand frame of the historical frame successfully through the matching threshold and the intersection-and-union ratio;

[0208] The key point detection module 1105 is used to input the historical frame key points and the current frame image into the hand key point detection model if the match is successful, obtain a heat map, use the coordinates corresponding to the maximum value of the channel in the heat map as the target key point coordinates, and display the target key point coordinates in the current frame image.

[0209] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0210] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above hand detection method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0211] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0212] See also Fig.12 , Fig.12 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0213] The processor 1201 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0214] The memory 1202 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1202 may store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1202, and the processor 1201 calls and executes the hand detection method in the embodiments of this application;

[0215] Input / output interface 1203, used to implement information input and output;

[0216] The communication interface 1204 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WI FI, Bluetooth, etc.);

[0217] A bus 1205 that transmits information between various components of the device (e.g., the processor 1201, the memory 1202, the input / output interface 1203, and the communication interface 1204);

[0218] The processor 1201 , the memory 1202 , the input / output interface 1203 and the communication interface 1204 are connected to each other in communication within the device via the bus 1205 .

[0219] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program implements the above-mentioned hand detection method when executed by a processor.

[0220] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0221] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0222] The hand detection method, hand detection device, electronic device and storage medium provided in the embodiments of the present application detect the current frame image through the hand detection model to obtain the current frame hand frame; obtain the historical frame hand frame to realize continuous hand tracking without relying on the detection result of a single frame; calculate the intersection and union ratio between the current frame hand frame and the historical frame hand frame; obtain the matching threshold, and judge whether the current frame hand frame and the historical frame hand frame are successfully matched through the matching threshold and the intersection and union ratio, thereby improving the matching accuracy of the hand frame and enhancing the robustness of hand detection; if the match is successful, the historical frame key points and the current frame image are input into the hand key point detection model to obtain a heat map, and the coordinates corresponding to the maximum values ​​of the channels in the heat map are used as the target key point coordinates, and the target key point coordinates are displayed in the current frame image, and the historical frame key point coordinates are used as the second input of the hand key point detection model, thereby reducing jitter according to the dependency relationship between adjacent frames and improving the accuracy of hand detection.

[0223] The present application also provides a method for training a hand key point detection model, which collects hand images in various scenarios, annotates hand images with hand frames and several key points; generates a true value heat map based on the key points of the hand image through a Gaussian function; perturbs the key points of the hand image to obtain a second input; inputs the second input and the hand image into the hand key point detection model for training to obtain a current heat map, and reduces the jitter during hand detection through the second input, which is beneficial for the model to understand the distribution of key points in the image and the relationship between historical frames and subsequent frames; calculates the loss value of the current heat map and the true value heat map according to the loss function, and performs back propagation on the hand key point detection model according to the loss value until the loss value is less than a preset loss threshold, thereby obtaining a trained hand key point detection model, which is used for hand detection, and can detect the initial frame and subsequent frames, and reduces the jitter according to the dependency relationship between adjacent frames, thereby improving the matching accuracy of the model for the hand frame and the robustness of hand tracking.

[0224] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0225] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0226] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0227] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0228] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0229] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0230] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0231] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0232] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0233] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0234] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A hand detection method, characterized in that: The method comprises the following steps: The hand detection model is used to detect the current frame image to obtain the current frame hand frame; Acquire a historical frame hand frame, wherein the historical frame hand frame includes historical frame key points; Calculating an intersection-over-union ratio between the current frame hand frame and the historical frame hand frame; Obtaining a matching threshold, and judging whether the hand frame of the current frame matches the hand frame of the historical frame successfully by using the matching threshold and the intersection-and-union ratio; If the match is successful, the historical frame key points and the current frame image are input into the hand key point detection model to obtain a heat map, and the coordinates corresponding to the maximum values ​​of the channels in the heat map are used as the target key point coordinates, and the target key point coordinates are displayed in the current frame image.

2. The method according to claim 1, characterized in that The method further comprises: If the match fails, the current frame hand frame is corrected based on preset rules, the hand area image is cropped from the current frame image according to the current frame hand frame, the hand area image and the empty key point coordinates are input into the hand key point detection model to obtain a heat map, the coordinates corresponding to the maximum values ​​of the channels in the heat map are used as the target key point coordinates, and the target key point coordinates are restored to the current frame image according to the preset rules.

3. The method according to claim 2, characterized in that The step of modifying the current frame hand frame based on a preset rule and cropping a hand region image from the current frame image according to the current frame hand frame includes: If the side lengths of the hand frame of the current frame are inconsistent, the short side of the hand frame of the current frame is symmetrically extended to be consistent with the long side of the hand frame of the current frame; Enlarging the current hand frame by a preset multiple; Cutting out a hand region image from the current frame image according to the current frame hand frame enlarged by a preset multiple; The hand region image is scaled.

4. The method according to claim 1, characterized in that: The step of inputting the historical frame key points and the current frame image into the hand key point detection model to obtain a heat map, taking the coordinates corresponding to the maximum values ​​of the channels in the heat map as the target key point coordinates, and displaying the target key point coordinates in the current frame image includes: Correcting the hand frame of the current frame according to the key points of the historical frame, and calculating the affine transformation matrix of the hand frame of the current frame; Cropping a hand region image from the current frame image according to the current frame hand frame; Mapping the coordinates of the key points of the historical frame to the coordinates of the key points of the historical region image according to the affine transformation matrix; Inputting the hand area image and the historical key point coordinates of the hand area image into the hand key point detection model to obtain a heat map; The coordinates corresponding to the maximum values ​​of the channels in the heat map are used as the target key point coordinates, the target key point coordinates are restored to the current frame image according to the affine transformation matrix, and the target key point coordinates are displayed in the current frame image.

5. The method according to claim 1, characterized in that The obtaining of the matching threshold and judging whether the hand frame of the current frame matches the hand frame of the historical frame successfully by using the matching threshold and the intersection-and-union ratio include: Determine the best pairing and the best intersection-and-union ratio of the current frame hand frame and the historical frame hand frame according to the matching algorithm and the intersection-and-union ratio; Get the matching threshold; If the best intersection-over-union ratio is greater than or equal to the matching threshold, it is determined that the hand frame of the current frame matches the hand frame of the historical frame successfully; If the best intersection-over-union ratio is less than the matching threshold, it is determined that the current frame hand frame fails to match the historical frame hand frame.

6. A hand key point detection model training method, characterized in that: The method comprises the following steps: Collect hand images in various scenes, and annotate the hand images with hand frames and several key points; Generate a true value heat map according to the key points of the hand image by using a Gaussian function; Perturbing the key points of the hand image to obtain a second input; Inputting the second input and the hand image into a hand key point detection model for training to obtain a current heat map; The loss value between the current heat map and the true value heat map is calculated according to the loss function, and the hand key point detection model is back-propagated according to the loss value until the loss value is less than a preset loss threshold, so as to obtain the trained hand key point detection model, and the hand key point detection model is used to perform hand detection according to the method described in any one of claims 1-5.

7. The method according to claim 6, characterized in that The perturbing the key points of the hand image to obtain a second input includes: Dividing the key points of the hand image into a first part of key points and a second part of key points; Perturbing the first part of key points according to a preset perturbation multiple, wherein the first part of key points are used for subsequent frame training; Setting the coordinates of the second part of key points to zero, wherein the second part of key points are used for initial frame training; The first part of key points after disturbance and the second part of key points after zeroing are used as the second input.

8. A hand detection device, characterized in that: The device comprises: An image detection module is used to detect the current frame image through a hand detection model to obtain a current frame hand frame; A historical frame acquisition module, used to acquire a historical frame hand frame, wherein the historical frame hand frame includes historical frame key points; A calculation module, used for calculating the intersection-and-union ratio between the current frame hand frame and the historical frame hand frame; A matching module, used for obtaining a matching threshold, and judging whether the hand frame of the current frame matches the hand frame of the historical frame successfully by using the matching threshold and the intersection-and-union ratio; The key point detection module is used to input the historical frame key points and the current frame image into the hand key point detection model if the match is successful, obtain a heat map, use the coordinates corresponding to the maximum value of the channel in the heat map as the target key point coordinates, and display the target key point coordinates in the current frame image.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images