Video-based two-hand identification method and device, equipment, storage medium and program product

Abstract

The invention relates to a video-based two-hand identification method and device, computer equipment, a computer readable storage medium and a computer program product. Comprising the steps of detecting a human body and a hand in frame image data by using a target detection algorithm, and outputting a human body detection frame and a hand detection frame; tracking a human body and a hand in the current video data by using a target tracking algorithm, and outputting a human body tracking frame and a hand tracking frame; fusing the human body detection frame and the human body tracking frame to determine a human body tracking queue; fusing the hand detection frame and the hand tracking frame to determine a hand tracking queue; traversing the human body tracking queue to identify key points, and determining skeleton key points of each human body; and matching the arm key point of each human body with the tracking frame corresponding to each hand to obtain a two-hand recognition result of each human body. By adopting the method, the left hand and the right hand can be distinguished under the condition that the hands are blurred, the two hands can be assigned to specific individuals, and the recognition precision and the recognition efficiency are improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video-based two-hand recognition method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] At present, in the video live broadcast or video conferencing scenarios, people need to perform special effects rendering, gesture remote control and other operations on specific areas of the image. The premise for realizing these functions is to accurately locate the hand position of the target person and identify the left and right hands.

[0003] Most traditional left and right hand recognition technologies are aimed at a single person, and are generally close-up areas (the target detection area accounts for more than 10% of the original detection area). In this scenario, the hand area in the video accounts for a large proportion and there is less external interference. The target is generally detected in the following ways: 1. Combining target detection and classification; 2. End-to-end detection. Among them:

[0004] The first method is to locate the hand through object detection, and then classify the hand area image through the classification model to identify whether the area is the left hand or the right hand. A prerequisite for this method to ensure accuracy is that the resolution of the hand in the image is large, and the algorithm only recognizes the hand area. In some scenes where the hand area is small or the hand is blurred, the algorithm accuracy drops significantly.

[0005] The second method is to jointly train the hand detection task and the left and right hand recognition task. During the training process, a weakly supervised learning method is used to learn some prior information about the left and right hands in the human body. The hand frame coordinates and hand key point information can be directly given, with a faster calculation speed.

[0006] However, the current recognition of left and right hands is essentially the classification of the hand area, and the classification accuracy is limited by the clarity of the hand contour. In some scenes such as the side angle of the hand or the fist posture, the direction of the palm and thumb in the hand area is unclear, and the recognition effect is poor. In addition, in multi-person scenes, the current recognition method does not attribute the left and right hands to specific individuals. Summary of the invention

[0007] Based on this, it is necessary to provide a video-based two-hand recognition method, device, computer equipment, computer-readable storage medium and computer program product to address the above technical problems, which can distinguish between left and right hands when the hands are blurred and attribute the two hands to a specific individual.

[0008] In a first aspect, the present application provides a video-based two-hand recognition method, comprising:

[0009] Extract several frame image data from the current video data, use the target detection algorithm to detect the human body and hand in the frame image data, and output the human body detection frame and the hand detection frame;

[0010] Use the target tracking algorithm to track the human body and hands in the current video data, and output the human body tracking frame and hand tracking frame;

[0011] The human body detection frame and the human body tracking frame are merged to determine a human body tracking queue; the human body tracking queue includes tracking frames corresponding to multiple human bodies;

[0012] The hand detection frame and the hand tracking frame are merged to determine a hand tracking queue; the hand tracking queue includes tracking frames corresponding to multiple hands;

[0013] Traverse the human body tracking queue, identify the key points of the traversed tracking frame, and determine the skeleton key points of each human body; the skeleton key points include the arm key points;

[0014] The key points of each human arm are matched with the tracking frame corresponding to each hand to obtain the recognition results of both hands of each human.

[0015] In one embodiment, a target detection algorithm is used to detect a human body and a hand in frame image data, and a human body detection frame and a hand detection frame are output, including:

[0016] Inputting the frame image data into the human body detection model and the hand detection model respectively;

[0017] Use the human body detection model to perform human body detection and output the human body detection frame;

[0018] Use the hand detection model to detect hands and output the hand detection frame. The human body detection model and the hand detection model use the same network architecture, including: multiple convolutional layers, reshaping layers, fusion layers, and output layers.

[0019] Among them, the reshaping layer is used to reshape the detection box head and category head output by the convolution layer; the fusion layer is used to fuse the results output by the reshaping layer to generate detection box fusion data and category fusion data; the output layer is used to output the detection box result according to the detection box fusion data, and output the classification result according to the category fusion data.

[0020] In one embodiment, the target tracking algorithm includes a bounding box regression network model trained using human body training data and a bounding box regression network model trained using hand training data.

[0021] In one of the embodiments, fusing the human body detection frame and the human body tracking frame to determine the human body tracking queue includes:

[0022] Calculate the intersection-and-union ratio between the human detection frame and the human tracking frame;

[0023] If the intersection-union ratio is greater than the preset threshold, the element corresponding to the human body tracking frame in the human body tracking queue is updated according to the information of the human body detection frame;

[0024] If the intersection-over-union ratio is less than or equal to the preset threshold, a new element is added to the human tracking queue according to the information of the human detection frame.

[0025] In one embodiment, key point recognition is performed on the traversed tracking frame to determine the key points of each human skeleton, including:

[0026] The posture estimation model is used to identify key points of the traversed tracking frame and determine the key points of the skeleton of each human body; the posture estimation model includes a feature extraction module, an encoding module, a post-processing module, a feedforward neural network and a posture estimation network;

[0027] Among them, the feature extraction module is used to extract the human body key point information from the image captured by the traversed tracking frame; the encoding module adopts a multi-head attention mechanism to extract deep feature information from the human body key point information; the post-processing module is used to perform dimensionality reduction, residual connection processing and layer regularization processing on the deep feature information; the feedforward neural network is used to perform nonlinear activation on the data output by the post-processing model; the posture estimation network is used to identify key points based on the data output by the feedforward neural network and output skeleton key points.

[0028] In one embodiment, the arm key points include wrist nodes and elbow nodes; each human arm key point is matched with a tracking frame corresponding to each hand to obtain a hand recognition result for each human, including:

[0029] Calculate the distance between the center point of the tracking frame corresponding to each hand and the wrist node of each human body to obtain the distance matrix;

[0030] Search the minimum value from the distance matrix to determine the matching wrist node and hand tracking box;

[0031] For each human body, the arm posture line is determined according to the wrist node and elbow node of the current human body; it is determined whether the arm posture line passes through the tracking frame that matches the wrist node of the current human body; if so, the left hand recognition result or the right hand recognition result of the current human body is determined according to the tracking frame that matches the wrist node of the current human body.

[0032] In a second aspect, the present application also provides a video-based two-hand recognition device, comprising:

[0033] The target detection module is used to extract several frame image data from the current video data, detect the human body and hands in the frame image data using the target detection algorithm, and output the human body detection frame and the hand detection frame;

[0034] The target tracking module is used to track the human body and hand in the current video data using the target tracking algorithm, and output the human body tracking frame and the hand tracking frame;

[0035] A fusion module is used to fuse the human body detection frame and the human body tracking frame to determine a human body tracking queue; the human body tracking queue includes multiple tracking frames corresponding to human bodies; a hand detection frame and a hand tracking frame are fused to determine a hand tracking queue; the hand tracking queue includes multiple tracking frames corresponding to hands;

[0036] The key point detection module is used to traverse the human body tracking queue, identify the key points of the traversed tracking frame, and determine the skeleton key points of each human body; the skeleton key points include the arm key points;

[0037] The two-hand recognition module is used to match the key points of each human arm with the tracking frame corresponding to each hand to obtain the two-hand recognition results of each human body.

[0038] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in the first aspect above are implemented.

[0039] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in the first aspect above are implemented.

[0040] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect above.

[0041] The above-mentioned video-based two-hand recognition method, device, computer equipment, computer-readable storage medium and computer program product extract several frame image data from the current video data, use the target detection algorithm to detect the human body and hands in the frame image data, and output the human body detection frame and the hand detection frame; use the target tracking algorithm to track the human body and hands in the current video data, and output the human body tracking frame and the hand tracking frame; fuse the human body detection frame and the human body tracking frame to determine the human body tracking queue; the human body tracking queue includes tracking frames corresponding to multiple human bodies; fuse the hand detection frame and the hand tracking frame to determine the hand tracking queue; the hand tracking queue includes tracking frames corresponding to multiple hands; traverse the human body tracking queue, identify key points of the traversed tracking frames, and determine the skeletal key points of each human body; the skeletal key points include arm key points; match the arm key points of each human body with the tracking frames corresponding to each hand to obtain the two-hand recognition results of each human body. Through the above method, human body detection and tracking are used to identify human body key points, and human hands are identified based on arm key points. Since the human body frame has a larger proportion in the image than the hand frame, the left and right information it represents has higher accuracy, and can distinguish left and right hands when the hands are blurred, improving recognition accuracy and solving the problem of insufficient left and right hand recognition accuracy in scenes such as hand angle side or fist posture. Combining human body recognition and hand recognition, both hands can be attributed to specific individuals, supporting both hands recognition in multi-person scenes, and improving recognition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments of the present application or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 FIG. 1 is an application environment diagram of a video-based two-hand recognition method in one embodiment;

[0044] Figure 2 is a flow chart of a method for two-hand recognition based on video in one embodiment;

[0045] Figure 3 is a schematic diagram of a processing flow in an embodiment;

[0046] Figure 4 is a schematic diagram of a processing result in an embodiment;

[0047] Figure 5 Schematic diagram of the network architecture of a human body detection model and a hand detection model in one embodiment;

[0048] Figure 6 A schematic diagram of a network architecture of a bounding box regression network model in one embodiment;

[0049] Figure 7 A schematic diagram of a network architecture of a posture estimation model in one embodiment;

[0050] Figure 8 A schematic diagram for comparing processing results in one embodiment;

[0051] Fig. 9 is a schematic diagram of arm key points and hand frames in one embodiment;

[0052] Fig.10 A schematic diagram of the geometric relationship between the key points of the arm and the hand frame in one embodiment;

[0053] Fig.11 is a structural block diagram of a video-based two-hand recognition device in one embodiment;

[0054] Fig.12 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] The video-based two-hand recognition method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 transmits the collected video data to the server 104, and the server 104 processes the video data according to the video-based two-hand recognition method provided in this application. It is understandable that the server 104 can perform special effects rendering, etc. according to the processing results, and transmit the rendered data to the terminal 102 or other devices connected to the terminal 102. The data storage system stores the intermediate data generated during the processing of the server 104, and also stores algorithm-related data. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head mounted device may be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0057] In an optional implementation, the terminal 102 collects video data, processes the video data according to the video-based two-hand recognition method provided in the present application, and performs gesture remote control based on the processing result.

[0058] In an exemplary embodiment, Figure 2 As shown in FIG. 1 , a video-based two-hand recognition method is provided, and the method is applied to Figure 1 The server 104 in FIG. 1 is used as an example to illustrate, including:

[0059] Step 202, extracting several frame image data from the current video data, using the target detection algorithm to detect the human body and hands in the frame image data, and outputting the human body detection frame and the hand detection frame.

[0060] Among them, for the current video data, frame image data is collected once every N frames, and N is a parameter set in advance according to actual needs. For the collected frame image data, the target detection algorithm is used to detect the human body and hand, and the human body detection frame and hand detection frame corresponding to each frame image data are output. Optionally, the target detection algorithm includes a human body detection algorithm and a hand detection algorithm, and the human body detection algorithm is used to detect the human body in each frame image data, and the human body detection frame is output, and the hand detection algorithm is used to detect the hand in each frame image data, and the hand detection frame is output.

[0061] Step 204: Track the human body and hand in the current video data using a target tracking algorithm, and output a human body tracking frame and a hand tracking frame.

[0062] Step 206: fuse the human body detection frame and the human body tracking frame to determine a human body tracking queue; the human body tracking queue includes tracking frames corresponding to multiple human bodies.

[0063] Step 208: The hand detection frame and the hand tracking frame are merged to determine a hand tracking queue; the hand tracking queue includes tracking frames corresponding to multiple hands.

[0064] Among them, in order to reduce the time consumption of target detection and improve the accuracy of human body detection and two-hand recognition, a target tracking algorithm is used to track the human body and hands in the current video data. Optionally, the target tracking algorithm includes a human body tracking algorithm and a hand tracking algorithm. The human body tracking algorithm is used to track the human body in the current video data, and the hand tracking algorithm is used to track the hands in the current video data. By fusing the human body detection frame and the human body tracking frame, the detection algorithm can be used to supplement the tracking algorithm to avoid missing the human body. Similarly, by fusing the hand detection frame and the hand tracking frame, the detection algorithm can be used to supplement the tracking algorithm to avoid missing the hand.

[0065] In an optional implementation, for the human body or hand, the intersection over union (IOU) of the detection box and the tracking box is calculated. If the IOU is greater than a preset threshold, the detection box and the tracking box are considered to be the same target, and the corresponding element in the tracking queue is updated with the information of the detection box; if the IOU is not greater than the preset threshold, it is considered to be a new target, and the new target information is added to the tracking queue.

[0066] Step 210, traverse the human body tracking queue, perform key point recognition on the traversed tracking frame, and determine the skeleton key points of each human body; the skeleton key points include arm key points.

[0067] Among them, the key point detection algorithm is used to detect 17 skeletal key points of the human body in the image corresponding to each tracking frame, and three key points of each arm are retained (a total of six key points), namely the arm key points, including: wrist node, elbow node and shoulder node.

[0068] Step 212, matching the key points of each human arm with the tracking frame corresponding to each hand, to obtain the recognition results of both hands of each human body.

[0069] Among them, the left hand frame and right hand frame of each human body are determined by matching the geometric relationship between the arm key points and the tracking frame of the hand. Based on this, all the hand frames of all the human bodies in the video are classified into left and right hands, and the individual information of the left and right hands is clarified.

[0070] In an optional implementation, refer to Figure 3 The processing flow includes: for the current video data, performing body tracking, body detection, hand detection and hand tracking respectively; fusing the results of body tracking and body detection; fusing the results of hand detection and hand tracking; identifying key points based on the body frame; using the geometric relationship between the arm key points and the hand tracking frame, identifying the left and right hands of each body, and outputting the left and right hand frames.

[0071] In an optional implementation, refer to Figure 4 , Figure 4 A processing result is shown, which performs recognition on the current video data. The output results include a human body frame, a hand frame, and 17 human body key points. It is suitable for multi-person scenes and can identify the detection results of each human body and the recognition results of the left and right hands of the human body.

[0072] In the above-mentioned video-based two-hand recognition method, several frame image data in the current video data are extracted, and the human body and hands in the frame image data are detected using the target detection algorithm, and the human body detection frame and the hand detection frame are output; the human body and hands in the current video data are tracked using the target tracking algorithm, and the human body tracking frame and the hand tracking frame are output; the human body detection frame and the human body tracking frame are fused to determine the human body tracking queue; the human body tracking queue includes tracking frames corresponding to multiple human bodies; the hand detection frame and the hand tracking frame are fused to determine the hand tracking queue; the hand tracking queue includes tracking frames corresponding to multiple hands; the human body tracking queue is traversed, and the key points of the traversed tracking frames are identified to determine the skeletal key points of each human body; the skeletal key points include arm key points; the arm key points of each human body are matched with the tracking frames corresponding to each hand to obtain the two-hand recognition results of each human body. Through the above method, human body detection and tracking are used to identify human body key points, and human hands are identified based on arm key points. Since the human body frame has a larger proportion in the image than the hand frame, the left and right information it represents has higher accuracy, and can distinguish left and right hands when the hands are blurred, improving recognition accuracy and solving the problem of insufficient left and right hand recognition accuracy in scenes such as hand angle side or fist posture. Combining human body recognition and hand recognition, both hands can be attributed to specific individuals, supporting both hands recognition in multi-person scenes, and improving recognition efficiency.

[0073] In an exemplary embodiment, a target detection algorithm is used to detect a human body and a hand in frame image data, and a human body detection frame and a hand detection frame are output, including: inputting the frame image data into a human body detection model and a hand detection model respectively; performing human body detection using the human body detection model and outputting a human body detection frame; performing hand detection using the hand detection model and outputting a hand detection frame; the human body detection model and the hand detection model use the same network architecture, including: multiple convolutional layers, reshaping layers, fusion layers and output layers; wherein the reshaping layer is used to reshape the detection frame head and the category head output by the convolutional layer; the fusion layer is used to fuse the results output by the reshaping layer to generate detection frame fusion data and category fusion data; the output layer is used to output the detection frame result according to the detection frame fusion data, and output the classification result according to the category fusion data.

[0074] Among them, Figure 5As shown in the figure, the network architecture used by the target detection algorithm includes: multiple convolutional layers, reshaping layers, fusion layers, and output layers; the detection frame header and category header are the information obtained by the convolutional layer extracting features from the frame image data and predicting the detection frame and category. The reshaping layer includes multiple detection frame reshaping and multiple category reshaping, which reshape the detection frame header and category header output by the convolutional layer respectively. The fusion layer (concat) includes detection frame fusion and category fusion, which fuses the results output by the reshaping layer to generate detection frame fusion data and category fusion data respectively. The output layer outputs the detection frame result and the classification result.

[0075] In an optional implementation, the detection box fusion is connected to the detection box loss (Box loss), and the category fusion is connected to the category loss (Class loss), which can achieve the purpose of parallel computing in reasoning.

[0076] It is understandable that this embodiment is based on the improvement of SSD (Single Shot MultiBox Detector) for the following purposes: first, to optimize the network structure and speed up the forward reasoning speed of the model; second, to pay more attention to the information of the target area during training, a weak semantic segmentation auxiliary structure is added to the network structure; third, to balance positive and negative samples, the category loss function is changed from the original class loss to focal loss. Figure 5 The detection heads in the original network structure include boxhead (detection box head) and class head (class head) as well as the following network. Each detection head is followed by a loss output. The structure adopted in this embodiment is to perform a reshape operation and a concat operation on each detection head, and finally connect the loss output, thereby achieving the purpose of parallel computing in reasoning.

[0077] In an exemplary embodiment, the target tracking algorithm includes a bounding box regression network model trained using human body training data and a bounding box regression network model trained using hand training data.

[0078] The target tracking algorithm of this embodiment adopts a deep learning coordinate point regression method, namely a bounding box regression network model, and its network architecture is as follows: Figure 6 As shown in Figure 1, it includes: backbone network (backbone), fully connected layer (connect) and output layer. The size of the input image image is , b is batchsize (batch size), h is image height, w is image width, 3 is RGB three channels; the final output size is , 5 is the relative offset and confidence of the upper left and lower right points of the target tracking box (bbox) ((xmin, ymin), (xmax, ymax), confidence)). Optionally, use human training data to train Figure 6 When training the bounding box regression network model shown in , the mean square error loss function is used to measure the difference between the predicted value and the true value, and the parameters of the bounding box regression network model are adjusted until the number of iterations reaches the set number or the loss function value is less than the set value, and the trained bounding box regression network model is obtained for human tracking. Similarly, the hand training data is used to train Figure 6 When the bounding box regression network model shown is trained, the mean square error loss function is used to measure the difference between the predicted value and the true value, and the parameters of the bounding box regression network model are adjusted until the number of iterations reaches the set number or the loss function value is less than the set value, and a trained bounding box regression network model is obtained for hand tracking.

[0079] In an exemplary embodiment, step 206 includes: calculating the intersection-and-union ratio between the human body detection frame and the human body tracking frame; if the intersection-and-union ratio is greater than a preset threshold, updating the element corresponding to the human body tracking frame in the human body tracking queue according to the information of the human body detection frame; if the intersection-and-union ratio is less than or equal to the preset threshold, adding a new element to the human body tracking queue according to the information of the human body detection frame.

[0080] Among them, the preset threshold is a parameter set in advance according to the actual situation, and this embodiment does not limit the specific value. Calculate the intersection over union (IOU) between the human detection frame and the human tracking frame. If the IOU is greater than the preset threshold, it indicates that the human detection frame and the human tracking frame are the same target, and the information of the human detection frame is used to update the element corresponding to the human tracking frame in the human tracking queue. If the IOU is not greater than the preset threshold, it indicates that the human detection frame detects a new target, and the information of the new target is added to the human tracking queue.

[0081] In an optional implementation, step 208 includes: calculating the intersection-and-union ratio between the hand detection frame and the hand tracking frame; if the intersection-and-union ratio is greater than a preset threshold, updating the element corresponding to the hand tracking frame in the hand tracking queue according to the information of the hand detection frame; if the intersection-and-union ratio is less than or equal to the preset threshold, adding a new element to the hand tracking queue according to the information of the hand detection frame.

[0082] In an exemplary embodiment, key point recognition is performed on the traversed tracking frame to determine the skeletal key points of each human body, including: using a posture estimation model to recognize the key points of the traversed tracking frame to determine the skeletal key points of each human body; the posture estimation model includes a feature extraction module, an encoding module, a post-processing module, a feedforward neural network and a posture estimation network; wherein the feature extraction module is used to extract human key point information from the image captured by the traversed tracking frame; the encoding module uses a multi-head attention mechanism to extract deep feature information from the human key point information; the post-processing module is used to perform dimensionality reduction, residual connection processing and layer regularization processing on the deep feature information; the feedforward neural network is used to perform nonlinear activation on the data output by the post-processing model; the posture estimation network is used to recognize key points based on the data output by the feedforward neural network and output skeletal key points.

[0083] In this embodiment, the skeleton key points are 17 human key points. The posture estimation network can be a commonly used network for human posture estimation, such as DeepPose, PoseNet, etc. Figure 7 ,This embodiment adopts the Simcc (also called Simdr) algorithm, and adds a transformer layer on the basis of its ,network architecture, which can effectively capture the relationship between visual entities ,elements, and learn the statistical constraint relationship between key points from a ,large amount of data, thereby improving the accuracy of human key point ,detection.

[0084] It is understandable that the traditional Heatmap-base method generates Gaussian heatmaps as labels through 2D Gaussian distribution, supervises model output, and optimizes through L2 loss. However, the size of the heatmap obtained in this way is often smaller than the original size of the image. Finally, the coordinates obtained by argmax (optimization function) are enlarged back to the original image, which will introduce inevitable quantization errors.

[0085] The Simdr algorithm uses two independent one-dimensional vectors to represent the x and y coordinates of key points. A scaling factor is used to constrain the length of the one-dimensional vector to the size of the image, thus avoiding the upsampling operation in the heatmap method and saving network computation without reducing the algorithm performance indicators. However, even so, Simdr does not solve the problem of inaccurate key point positioning due to occlusion of some key points. For example, when an individual wears thicker clothes, the positioning of the elbow key point is prone to drift. Figure 7 , add a transformer layer before the Simdr head structure. The specific steps are as follows:

[0086] 1) The human detector detects the human body frame body_box, cuts the image in the frame to get the input image (image_frame), first passes through a prefix feature extraction module (neural network), infers 17 human key point information, and the dimension information is [17, h, w]. h and w are flattened to get a two-dimensional tensor of [17, d] (where ), as the word embedding of the transformer layer.

[0087] 2) The transformer layer includes an encoder module, a dimension conversion module, a residual connection, and layer regularization. The encoder module uses a multi-head attention mechanism (for example, a 4-head attention mechanism) to obtain more potential subspace information. After the multi-head attention mechanism, the original d-dimensional features become 4d features, and then through the dimension conversion module, they are transformed into d-dimensional features, and residual connections and layer regularization (layer norm) are performed. Finally, they are nonlinearly activated through a feedforward neural network. At this time, the output dimension is consistent with the input, which is [17, d]. This output is used as the input of the Simdr head.

[0088] 3) After receiving the input, Simdr head projects it through the mlp layer (linear projection layer LinearProjection) of the x-axis and y-axis respectively to obtain a one-dimensional vector x_d (dimension is ) and y_d (dimension is ), decode x_d and y_d through the Decoding layer, and calculate the final output (dimension is [17, 2]) through the output layer.

[0089] For example, refer to Figure 8 , Figure 8 The figure is a schematic diagram for comparing the processing results in an embodiment; in an occluded scene, the original posture estimation network has poor reasoning ability for the invisible part, while the network using the transformer structure has more accurate inference ability for the invisible part, which can improve the accuracy of key point detection. In the specific implementation, 2,000 multi-person images were selected from COCO, MPII and Human3.6M (all public data sets of 3D human posture) as tests, of which 800 were images of scenes with partial human body occlusion; the test results show that in the occluded scene, the network with the transformer structure has a mAP (mean Average Precision, the average accuracy of all categories) higher than the original network by about 3.2%.

[0090] In an exemplary embodiment, the arm key points include a wrist node and an elbow node; step 212 includes: respectively calculating the distance between the center point of the tracking frame corresponding to each hand and the wrist node of each human body to obtain a distance matrix; searching for the minimum value from the distance matrix to determine the matching wrist node and hand tracking frame; for each human body, determining the arm posture line according to the wrist node and elbow node of the current human body; judging whether the arm posture line passes through the tracking frame matching the wrist node of the current human body; if so, determining the left hand recognition result or the right hand recognition result of the current human body according to the tracking frame matching the wrist node of the current human body.

[0091] After determining the key points of the arm, the known key point numbers can be used to determine whether the wrist node and elbow node correspond to the left or right hand. By simply matching the hand tracking frame with the key points of the left and right arms of each person, the left and right hand frames of each person can be determined. Fig. 9 The key points of the left arm include the shoulder node L1, the elbow node L2 and the wrist node L3, and the key points of the right arm include the shoulder node R1, the elbow node R2 and the wrist node R3. The tracking box of the hand includes BBox1 and BBox2, and their center points are C1 and C2 respectively.

[0092] It can be understood that the steps of matching the arm key points with the tracking frame of the hand (hereinafter referred to as the hand frame) include:

[0093] 1) Calculate the distance between all hand BBox center points (C1 / C2) and wrist nodes (L3 / R3) to get the distance matrix . Where m corresponds to the wrist node, and n corresponds to the center point of the hand BBox.

[0094] 2) Search for the minimum value and record the corresponding row and column number (p1, c1) as the best match between the wrist node and the hand tracking frame. Remove the elements with the same row and column number to get a new distance matrix. . Perform the same operation as before on the new matrix to find a series of matching points: (p2,c2),…,(pi,ci). Find the match between the wrist node and the hand box through the distance relationship as a candidate match.

[0095] 3) When more than two human hands are not fully detected, the above operation will most likely lead to mismatching, and additional judgment strategies are required. In normal activities, the forearm and hand are almost in the same straight line, so the forearm and hand frame have a fixed geometric relationship: Fig.10 As shown, the straight line L0 formed by the elbow node to the wrist node must pass through the tracking frame of the hand.

[0096] Optionally, in order to simplify calculation, the process of determining whether the straight line L0 formed by the elbow node and the wrist point passes through the hand frame can be simplified as follows: calculating whether the straight line L0 intersects with the four line segments of the hand frame.

[0097] Optionally, the positional relationship between the hand frame and the normal line of the straight line L0 can be used to determine whether the straight line L0 passes through the hand frame. Specifically, it is detected whether at least one of the four boundary points of the hand frame is on the same side of the normal line L1 as the wrist node. The mathematical expression is:

[0098] ;

[0099] Among them, a1, b1, c1 are the parameters of the normal L1, (pt1_x, pt1_y) is the wrist node, and (xi, yi) is the boundary point of the hand box.

[0100] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0101] Based on the same inventive concept, the embodiment of the present application also provides a video-based two-hand recognition device for implementing the video-based two-hand recognition method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations of one or more video-based two-hand recognition device embodiments provided below can refer to the limitations of the video-based two-hand recognition method above, and will not be repeated here.

[0102] In an exemplary embodiment, Fig.11 As shown, a video-based two-hand recognition device is provided, comprising:

[0103] The target detection module 1102 is used to extract several frame image data from the current video data, detect the human body and hands in the frame image data using the target detection algorithm, and output the human body detection frame and the hand detection frame.

[0104] The target tracking module 1104 is used to track the human body and hand in the current video data using a target tracking algorithm, and output a human body tracking frame and a hand tracking frame.

[0105] The fusion module 1106 is used to fuse the human body detection frame and the human body tracking frame to determine the human body tracking queue; the human body tracking queue includes multiple tracking frames corresponding to the human bodies; the hand detection frame and the hand tracking frame are fused to determine the hand tracking queue; the hand tracking queue includes multiple tracking frames corresponding to the hands.

[0106] The key point detection module 1108 is used to traverse the human body tracking queue, identify the key points of the traversed tracking frame, and determine the skeletal key points of each human body; the skeletal key points include the arm key points.

[0107] The two-hand recognition module 1110 is used to match the key points of each human arm with the tracking frame corresponding to each hand to obtain the two-hand recognition result of each human body.

[0108] In the above-mentioned video-based two-hand recognition device, human body detection and tracking are used to identify human body key points, and human hands are identified based on arm key points. Since the human body frame has a larger proportion in the image than the hand frame, the left and right information it represents has higher accuracy, and can distinguish between left and right hands when the hands are blurred, improving recognition accuracy and solving the problem of insufficient left and right hand recognition accuracy in scenes such as hand angle side or fist posture. Combining human body recognition and hand recognition, both hands can be attributed to specific individuals, supporting two-hand recognition in multi-person scenes, and improving recognition efficiency.

[0109] In an exemplary embodiment, the target detection module 1102 is also used to input the frame image data into the human body detection model and the hand detection model respectively; use the human body detection model to perform human body detection and output the human body detection frame; use the hand detection model to perform hand detection and output the hand detection frame; the human body detection model and the hand detection model use the same network architecture, including: multiple convolutional layers, reshaping layers, fusion layers and output layers; wherein the reshaping layer is used to reshape the detection frame head and category head output by the convolutional layer; the fusion layer is used to fuse the results output by the reshaping layer to generate detection frame fusion data and category fusion data; the output layer is used to output the detection frame result according to the detection frame fusion data, and output the classification result according to the category fusion data.

[0110] In an exemplary embodiment, the target tracking algorithm includes a bounding box regression network model trained using human body training data and a bounding box regression network model trained using hand training data.

[0111] In an exemplary embodiment, the fusion module 1106 is also used to calculate the intersection-and-union ratio between the human body detection frame and the human body tracking frame; if the intersection-and-union ratio is greater than a preset threshold, the element corresponding to the human body tracking frame in the human body tracking queue is updated according to the information of the human body detection frame; if the intersection-and-union ratio is less than or equal to the preset threshold, a new element is added to the human body tracking queue according to the information of the human body detection frame.

[0112] In an exemplary embodiment, the key point detection module 1108 is also used to use the posture estimation model to identify the key points of the traversed tracking frame and determine the skeletal key points of each human body; the posture estimation model includes a feature extraction module, an encoding module, a post-processing module, a feedforward neural network and a posture estimation network; wherein the feature extraction module is used to extract human key point information from the image captured by the traversed tracking frame; the encoding module adopts a multi-head attention mechanism to extract deep feature information from the human key point information; the post-processing module is used to perform dimensionality reduction, residual connection processing and layer regularization processing on the deep feature information; the feedforward neural network is used to perform nonlinear activation on the data output by the post-processing model; the posture estimation network is used to identify the key points according to the data output by the feedforward neural network and output the skeletal key points.

[0113] In an exemplary embodiment, the arm key points include a wrist node and an elbow node; the two-hand recognition module 1110 is also used to respectively calculate the distance between the center point of the tracking frame corresponding to each hand and the wrist node of each human body to obtain a distance matrix; search the minimum value from the distance matrix to determine the matching wrist node and hand tracking frame; for each human body, determine the arm posture line according to the wrist node and elbow node of the current human body; determine whether the arm posture line passes through the tracking frame that matches the wrist node of the current human body; if so, determine the left hand recognition result or the right hand recognition result of the current human body according to the tracking frame that matches the wrist node of the current human body.

[0114] Each module in the above video-based two-hand recognition device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0115] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.12As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video-based two-hand recognition method is implemented.

[0116] Those skilled in the art will understand that Fig.12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0117] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: extracting several frame image data in the current video data, detecting the human body and hands in the frame image data using a target detection algorithm, and outputting a human body detection frame and a hand detection frame; tracking the human body and hands in the current video data using a target tracking algorithm, and outputting a human body tracking frame and a hand tracking frame; fusing the human body detection frame and the human body tracking frame to determine a human body tracking queue; the human body tracking queue includes tracking frames corresponding to multiple human bodies; fusing the hand detection frame and the hand tracking frame to determine a hand tracking queue; the hand tracking queue includes tracking frames corresponding to multiple hands; traversing the human body tracking queue, identifying key points of the traversed tracking frames, and determining the skeletal key points of each human body; the skeletal key points include arm key points; matching the arm key points of each human body with the tracking frames corresponding to each hand to obtain the recognition results of both hands of each human body.

[0118] In one embodiment, when the processor executes the computer program, the following steps are also implemented: inputting the frame image data into the human body detection model and the hand detection model respectively; performing human body detection using the human body detection model and outputting a human body detection frame; performing hand detection using the hand detection model and outputting a hand detection frame; the human body detection model and the hand detection model use the same network architecture, including: multiple convolutional layers, reshaping layers, fusion layers and output layers; wherein the reshaping layer is used to reshape the detection frame head and category head output by the convolutional layer; the fusion layer is used to fuse the results output by the reshaping layer to generate detection frame fusion data and category fusion data; the output layer is used to output the detection frame result according to the detection frame fusion data, and output the classification result according to the category fusion data.

[0119] In one embodiment, when the processor executes the computer program, the following steps are also implemented: calculating the intersection-and-union ratio between the human body detection frame and the human body tracking frame; if the intersection-and-union ratio is greater than a preset threshold, updating the element corresponding to the human body tracking frame in the human body tracking queue according to the information of the human body detection frame; if the intersection-and-union ratio is less than or equal to the preset threshold, adding a new element to the human body tracking queue according to the information of the human body detection frame.

[0120] In one embodiment, when the processor executes the computer program, the following steps are also implemented: using the posture estimation model to identify key points of the traversed tracking frame to determine the skeletal key points of each human body; the posture estimation model includes a feature extraction module, an encoding module, a post-processing module, a feedforward neural network and a posture estimation network; wherein the feature extraction module is used to extract human key point information from the image captured by the traversed tracking frame; the encoding module uses a multi-head attention mechanism to extract deep feature information from the human key point information; the post-processing module is used to perform dimensionality reduction, residual connection processing and layer regularization processing on the deep feature information; the feedforward neural network is used to perform nonlinear activation on the data output by the post-processing model; the posture estimation network is used to identify key points based on the data output by the feedforward neural network and output skeletal key points.

[0121] In one embodiment, the processor also implements the following steps when executing the computer program: respectively calculating the distance between the center point of the tracking frame corresponding to each hand and the wrist node of each human body to obtain a distance matrix; searching for the minimum value from the distance matrix to determine the matching wrist node and tracking frame of the hand; for each human body, determining the posture line of the arm according to the wrist node and elbow node of the current human body; judging whether the posture line of the arm passes through the tracking frame matching the wrist node of the current human body; if so, determining the left hand recognition result or the right hand recognition result of the current human body according to the tracking frame matching the wrist node of the current human body.

[0122] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: extracting a number of frame image data in current video data, detecting a human body and a hand in the frame image data using a target detection algorithm, and outputting a human body detection frame and a hand detection frame; tracking a human body and a hand in the current video data using a target tracking algorithm, and outputting a human body tracking frame and a hand tracking frame; fusing the human body detection frame and the human body tracking frame to determine a human body tracking queue; the human body tracking queue includes tracking frames corresponding to multiple human bodies; fusing the hand detection frame and the hand tracking frame to determine a hand tracking queue; the hand tracking queue includes tracking frames corresponding to multiple hands; traversing the human body tracking queue, identifying key points of the traversed tracking frames, and determining the skeletal key points of each human body; the skeletal key points include arm key points; matching the arm key points of each human body with the tracking frames corresponding to each hand to obtain the recognition results of both hands of each human body.

[0123] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: inputting the frame image data into the human body detection model and the hand detection model respectively; performing human body detection using the human body detection model and outputting a human body detection frame; performing hand detection using the hand detection model and outputting a hand detection frame; the human body detection model and the hand detection model use the same network architecture, including: multiple convolutional layers, reshaping layers, fusion layers and output layers; wherein the reshaping layer is used to reshape the detection frame head and category head output by the convolutional layer; the fusion layer is used to fuse the results output by the reshaping layer to generate detection frame fusion data and category fusion data; the output layer is used to output the detection frame result according to the detection frame fusion data, and output the classification result according to the category fusion data.

[0124] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: calculating the intersection-and-union ratio between the human body detection frame and the human body tracking frame; if the intersection-and-union ratio is greater than a preset threshold, updating the element corresponding to the human body tracking frame in the human body tracking queue according to the information of the human body detection frame; if the intersection-and-union ratio is less than or equal to the preset threshold, adding a new element to the human body tracking queue according to the information of the human body detection frame.

[0125] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: using the posture estimation model to identify key points of the traversed tracking frame to determine the skeletal key points of each human body; the posture estimation model includes a feature extraction module, an encoding module, a post-processing module, a feedforward neural network and a posture estimation network; wherein the feature extraction module is used to extract human key point information from the image captured by the traversed tracking frame; the encoding module uses a multi-head attention mechanism to extract deep feature information from the human key point information; the post-processing module is used to perform dimensionality reduction, residual connection processing and layer regularization processing on the deep feature information; the feedforward neural network is used to perform nonlinear activation on the data output by the post-processing model; the posture estimation network is used to identify key points based on the data output by the feedforward neural network and output skeletal key points.

[0126] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: respectively calculating the distance between the center point of the tracking frame corresponding to each hand and the wrist node of each human body to obtain a distance matrix; searching for the minimum value from the distance matrix to determine the matching wrist node and tracking frame of the hand; for each human body, determining the posture line of the arm according to the wrist node and elbow node of the current human body; judging whether the posture line of the arm passes through the tracking frame matching the wrist node of the current human body; if so, determining the left hand recognition result or the right hand recognition result of the current human body according to the tracking frame matching the wrist node of the current human body.

[0127] In one embodiment, a computer program product is provided, including a computer program, which implements the following steps when executed by a processor: extracting several frame image data in current video data, detecting human bodies and hands in the frame image data using a target detection algorithm, and outputting a human body detection frame and a hand detection frame; tracking human bodies and hands in the current video data using a target tracking algorithm, and outputting a human body tracking frame and a hand tracking frame; fusing the human body detection frame and the human body tracking frame to determine a human body tracking queue; the human body tracking queue includes tracking frames corresponding to multiple human bodies; fusing the hand detection frame and the hand tracking frame to determine a hand tracking queue; the hand tracking queue includes tracking frames corresponding to multiple hands; traversing the human body tracking queue, identifying key points of the traversed tracking frames, and determining the skeletal key points of each human body; the skeletal key points include arm key points; matching the arm key points of each human body with the tracking frames corresponding to each hand to obtain the recognition results of both hands of each human body.

[0128] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: inputting the frame image data into the human body detection model and the hand detection model respectively; performing human body detection using the human body detection model and outputting a human body detection frame; performing hand detection using the hand detection model and outputting a hand detection frame; the human body detection model and the hand detection model use the same network architecture, including: multiple convolutional layers, reshaping layers, fusion layers and output layers; wherein the reshaping layer is used to reshape the detection frame head and category head output by the convolutional layer; the fusion layer is used to fuse the results output by the reshaping layer to generate detection frame fusion data and category fusion data; the output layer is used to output the detection frame result according to the detection frame fusion data, and output the classification result according to the category fusion data.

[0129] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: calculating the intersection-and-union ratio between the human body detection frame and the human body tracking frame; if the intersection-and-union ratio is greater than a preset threshold, updating the element corresponding to the human body tracking frame in the human body tracking queue according to the information of the human body detection frame; if the intersection-and-union ratio is less than or equal to the preset threshold, adding a new element to the human body tracking queue according to the information of the human body detection frame.

[0130] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: using the posture estimation model to identify key points of the traversed tracking frame to determine the skeletal key points of each human body; the posture estimation model includes a feature extraction module, an encoding module, a post-processing module, a feedforward neural network and a posture estimation network; wherein the feature extraction module is used to extract human key point information from the image captured by the traversed tracking frame; the encoding module uses a multi-head attention mechanism to extract deep feature information from the human key point information; the post-processing module is used to perform dimensionality reduction, residual connection processing and layer regularization processing on the deep feature information; the feedforward neural network is used to perform nonlinear activation on the data output by the post-processing model; the posture estimation network is used to identify key points based on the data output by the feedforward neural network and output skeletal key points.

[0131] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: respectively calculating the distance between the center point of the tracking frame corresponding to each hand and the wrist node of each human body to obtain a distance matrix; searching for the minimum value from the distance matrix to determine the matching wrist node and tracking frame of the hand; for each human body, determining the posture line of the arm according to the wrist node and elbow node of the current human body; judging whether the posture line of the arm passes through the tracking frame matching the wrist node of the current human body; if so, determining the left hand recognition result or the right hand recognition result of the current human body according to the tracking frame matching the wrist node of the current human body.

[0132] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0133] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0134] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0135] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A video-based two-hand recognition method, characterized in that: The method comprises: Extracting a plurality of frame image data from the current video data, detecting a human body and a hand in the frame image data using a target detection algorithm, and outputting a human body detection frame and a hand detection frame; Tracking a human body and a hand in the current video data using a target tracking algorithm, and outputting a human body tracking frame and a hand tracking frame; The human body detection frame and the human body tracking frame are merged to determine a human body tracking queue; the human body tracking queue includes tracking frames corresponding to multiple human bodies; The hand detection frame and the hand tracking frame are merged to determine a hand tracking queue; the hand tracking queue includes tracking frames corresponding to multiple hands; Traversing the human body tracking queue, performing key point recognition on the traversed tracking frame, and determining the skeleton key points of each human body; the skeleton key points include arm key points; The key points of each human arm are matched with the tracking frame corresponding to each hand to obtain the recognition results of both hands of each human.

2. The method according to claim 1, characterized in that The method of detecting a human body and a hand in the frame image data by using a target detection algorithm and outputting a human body detection frame and a hand detection frame includes: Inputting the frame image data into a human body detection model and a hand detection model respectively; Performing human body detection using the human body detection model and outputting a human body detection frame; The hand detection model is used to perform hand detection and output a hand detection frame; the human body detection model and the hand detection model use the same network architecture, including: multiple convolutional layers, reshaping layers, fusion layers and output layers; Among them, the reshaping layer is used to reshape the detection box head and category head output by the convolution layer; the fusion layer is used to fuse the results output by the reshaping layer to generate detection box fusion data and category fusion data; the output layer is used to output the detection box result according to the detection box fusion data, and output the classification result according to the category fusion data.

3. The method according to claim 1, characterized in that The target tracking algorithm includes a bounding box regression network model trained using human body training data and a bounding box regression network model trained using hand training data.

4. The method according to claim 1, characterized in that: The fusing the human body detection frame and the human body tracking frame to determine a human body tracking queue includes: Calculating an intersection-over-union ratio between the human body detection frame and the human body tracking frame; If the intersection-over-union ratio is greater than a preset threshold, updating the element corresponding to the human body tracking frame in the human body tracking queue according to the information of the human body detection frame; If the intersection-over-union ratio is less than or equal to the preset threshold, a new element is added to the human body tracking queue according to the information of the human body detection frame.

5. The method according to claim 1, characterized in that The key point recognition is performed on the traversed tracking frame to determine the key points of each human skeleton, including: Using a posture estimation model to identify key points of the traversed tracking frame to determine the skeletal key points of each human body; the posture estimation model includes a feature extraction module, an encoding module, a post-processing module, a feedforward neural network and a posture estimation network; Among them, the feature extraction module is used to extract human key point information from the image captured by the traversed tracking frame; the encoding module adopts a multi-head attention mechanism to extract deep feature information from the human key point information; the post-processing module is used to perform dimensionality reduction, residual connection processing and layer regularization processing on the deep feature information; the feedforward neural network is used to perform nonlinear activation on the data output by the post-processing model; the posture estimation network is used to perform key point recognition based on the data output by the feedforward neural network and output skeleton key points.

6. The method according to any one of claims 1 to 5, characterized in that The arm key points include wrist nodes and elbow nodes; The arm key points of each human body are matched with the tracking frame corresponding to each hand to obtain the recognition results of both hands of each human body, including: Calculate the distance between the center point of the tracking frame corresponding to each hand and the wrist node of each human body to obtain the distance matrix; Searching for a minimum value from the distance matrix to determine a matching wrist node and a tracking frame of the hand; For each human body, the posture line of the arm is determined according to the wrist node and elbow node of the current human body; it is determined whether the posture line of the arm passes through the tracking frame that matches the wrist node of the current human body; if so, the left hand recognition result or the right hand recognition result of the current human body is determined according to the tracking frame that matches the wrist node of the current human body.

7. A video-based two-hand recognition device, characterized in that: The device comprises: The target detection module is used to extract a plurality of frame image data from the current video data, detect the human body and hand in the frame image data using the target detection algorithm, and output the human body detection frame and the hand detection frame; A target tracking module, used for tracking the human body and hand in the current video data by using a target tracking algorithm, and outputting a human body tracking frame and a hand tracking frame; A fusion module, configured to fuse the human body detection frame and the human body tracking frame to determine a human body tracking queue; the human body tracking queue includes a plurality of tracking frames corresponding to human bodies; and to fuse the hand detection frame and the hand tracking frame to determine a hand tracking queue; the hand tracking queue includes a plurality of tracking frames corresponding to hands. A key point detection module is used to traverse the human body tracking queue, identify key points of the traversed tracking frame, and determine the skeleton key points of each human body; the skeleton key points include arm key points; The two-hand recognition module is used to match the key points of each human arm with the tracking frame corresponding to each hand to obtain the two-hand recognition results of each human body.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images