Gesture recognition and trajectory tracking method and device based on deep learning model

Abstract

The invention discloses a gesture recognition and trajectory tracking method and device based on a deep learning model, and the method comprises the steps: 1, collecting a video image, and carrying out the preprocessing of the image; step 2, inputting the preprocessed image into a neural network, performing gesture detection on the image by the neural network, and outputting a candidate bounding box set containing a gesture position; 3, confidence screening and deduplication detection are carried out on the candidate bounding box set, an effective gesture bounding box is obtained, and the effective gesture bounding box comprises a plurality of gesture categories; step 4, carrying out association matching on the recognized gesture category and the character, and distributing a matched gesture for each detected character; and step 5, calculating a movement vector of the gesture to determine a movement direction of the gesture, and calculating a total movement distance and an instantaneous movement distance at the same time to realize tracking of a gesture space movement track. According to the method, gestures are recognized through the deep learning model, and the spatial movement track of the gestures is accurately tracked.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent control, in particular to a gesture recognition and trajectory tracking method and device based on a deep learning model. BACKGROUND

[0002] With the rapid development of smart home and human-computer interaction technology, users have higher convenience and flexibility requirements for the control method of sound equipment. The traditional sound control method mainly relies on physical keys, remote controllers or mobile phone applications, which have obvious limitations: physical keys require close contact with the device, remote controllers are easy to lose and operation depends on vision, and mobile phone applications need to unlock the device and open specific software, with a cumbersome operation process.

[0003] To solve the above problems, a sound control scheme based on gesture recognition appears in the prior art, which attempts to achieve control through non-contact gestures. However, in the prior art, for example, in the patent with publication number CN222563920U, there are the following deficiencies: high hardware cost, dependence on special equipment, resulting in significant increase in equipment cost and difficulty in popularization; complex gestures, high user learning cost, design of multiple gesture categories to achieve different functions, resulting in the need for users to remember a large number of gesture rules, high learning cost and easy confusion; 3. Lack of precise control mechanism for spatial movement, gesture control schemes mostly recognize static gestures or lack quantitative tracking of spatial movement of dynamic gestures, resulting in low control accuracy. SUMMARY

[0004] The purpose of the present application is to overcome the defects in the prior art of sound control scheme based on gesture recognition, such as high hardware cost, dependence on special equipment, complex gestures, high user learning cost and lack of precise tracking mechanism for spatial movement trajectory, and to provide a gesture recognition and trajectory tracking method and device based on a deep learning model, which accurately tracks the spatial movement trajectory of gestures through a deep learning model.

[0005] The purpose of the present application is achieved by the following technical solutions: The gesture recognition and trajectory tracking method based on a deep learning model comprises the following steps: Step 1, collect video images and pre-process the images; Step 2, input the pre-processed images into a neural network, the neural network detects gestures for the images and outputs a candidate bounding box set containing gesture positions; Step 3, perform confidence screening and duplicate detection on the candidate bounding box set to obtain effective gesture bounding boxes, the effective gesture bounding boxes including several gesture categories; Step 4, the identified gesture category is matched with the person, and a matched gesture is assigned to each detected person; Step 5, by establishing a position history mechanism within a sliding time window, a moving vector of the gesture is calculated to determine the moving direction of the gesture, and a total moving distance and an instantaneous moving distance are calculated to realize tracking of the spatial moving track of the gesture.

[0006] As preferred, the step 1 is specifically: Sub-step 1.1, the camera collects a YUYV format video stream of a fixed resolution to obtain an original image sequence; Sub-step 1.2, the YUYV format image is converted into a BGR format image; Sub-step 1.3, the BRG format image is subjected to bilinear interpolation scaling to adjust the BRG format image to an image size suitable for input into the neural network; Sub-step 1.4, the scaled BRG image is subjected to normalization processing to normalize the pixel value from the integer range of 0-255 to the floating-point number range of 0-1.

[0007] As preferred, in the step 2, the neural network is a neural network based on the YOLOv5 architecture, and the neural network model comprises a backbone feature extraction network, a feature pyramid fusion network and a multi-scale detection head, and the step 2 is specifically: Sub-step 2.1, the backbone feature extraction network performs feature extraction, and the feature mapping calculation of the lth layer is: Fl=σ(BN(Convl(Fl-1))) Wherein, σ is an activation function, BN is batch normalization, and Conv is a convolution operation; Sub-step 2.2, the feature pyramid fusion network performs multi-scale feature fusion, and a bottom-up and top-down double fusion strategy is adopted: Pi=Concat(Upsample(Pi+1),Fi) Ni=Concat(Downsample(Ni-1),Pi) Wherein, Concat represents a feature map splicing operation, Pi represents a fusion feature map obtained by splicing the bottom-up feature map Fi through top-down upsampling, Ni represents a further fusion feature map obtained by splicing the previous layer Ni-1 through bottom-up downsampling on the basis of Pi, and i∈{3,4,5} represents different scale levels; Sub-step 2.3, the detection head outputs prediction results at several different scales.

[0008] As preferred, in the step 3, the confidence screening of the candidate bounding box set is specifically: confidence×max(pclass)≥θ_conf wherein, confidence represents the confidence of the candidate bounding box, max(pclass) represents the predicted probability distribution of the gesture class, and θ_conf is a basic confidence threshold value; The de-duplication detection is specifically: IoU(boxi,boxj) = Area(boxi∩boxj) / Area(boxi∪boxj) wherein, boxi,boxj represents two compared candidate bounding boxes, Area(boxi∩boxj) represents the intersection area of the two candidate bounding boxes, Area(boxi∪boxj) represents the area of the union part of the two bounding boxes, and IoU represents the intersection-over-union ratio.

[0009] Preferably, the step 4 is specifically: Sub-step 4.1, calculating the spatial distance between the gesture bounding box and the person bounding box; Sub-step 4.2, establishing a distance constraint matching condition; Sub-step 4.3, assigning the detected task with the gesture class that is closest in distance and meets the matching condition.

[0010] Preferably, the step 5 is specifically: Sub-step 5.1, establishing a position history record mechanism within a sliding time window to maintain a sequence of gesture center positions for consecutive N frames; Sub-step 5.2, calculating the moving vector within the gesture class value time window by using the first and last position difference method; Sub-step 5.3, determining four basic moving directions based on the principal component of the moving vector; Sub-step 5.4, calculating the total moving distance and the instantaneous moving distance.

[0011] The gesture recognition and trajectory tracking device based on a deep learning model is applicable to a gesture recognition and trajectory tracking method based on a deep learning model, and comprises: A video acquisition and preprocessing module is configured to acquire video images and pre-process the images; A gesture detection module is configured to input the pre-processed images into a neural network, perform gesture detection on the images by using the neural network, and output a candidate bounding box set containing gesture positions; A gesture screening and classification module is configured to perform confidence screening and de-duplication detection on the candidate bounding box set to obtain effective gesture bounding boxes, wherein the effective gesture bounding boxes comprise several gesture classes; A gesture and person association matching module is configured to associate and match the recognized gesture classes with persons, and assign a matched gesture to each detected person; The gesture trajectory tracking module is used for tracking the spatial movement trajectory of the gesture by establishing a position history mechanism within a sliding time window, calculating a movement vector of the gesture to determine a movement direction of the gesture, and simultaneously calculating a total movement distance and an instantaneous movement distance.

[0012] Preferably, the video acquisition and preprocessing module comprises: The YUYV is a video stream acquisition unit, which is used for acquiring a fixed resolution YUYV format video stream through a camera to obtain an original image sequence. The format conversion unit is used for converting the YUYV format image into a BGR format image. The size adjustment unit is used for performing bilinear interpolation scaling on the BGR format image to adjust the BGR format image to an image size suitable for input into the neural network. The normalization processing unit is used for performing normalization processing on the scaled BGR image.

[0013] Preferably, the gesture trajectory tracking module comprises: The position history recording unit is used for establishing a position history mechanism within a sliding time window to maintain a gesture center position sequence of consecutive N frames. The movement vector calculation unit is used for calculating a movement vector of the gesture within the time window by using a first and last position difference method. The movement direction determination unit is used for determining four basic movement directions based on a principal component of the movement vector. The movement distance calculation unit is used for calculating a total movement distance and an instantaneous movement distance.

[0014] A storage medium, wherein the storage medium stores computer executable instructions, and when the computer executable instructions are loaded and executed by a processor, the steps of the gesture recognition and trajectory tracking method based on a deep learning model are implemented.

[0015] The present application has the following advantages: through the complete process of video acquisition and preprocessing, neural network detection, screening and deduplication, person association matching and sliding time window trajectory tracking, the present application realizes accurate tracking of gesture recognition and spatial movement trajectory based on a deep learning model, and provides a reliable technical foundation for gesture control function. The use of the neural network with YOLOv5 architecture and mechanisms such as confidence screening and non-maximum suppression deduplication improves the accuracy and effectiveness of gesture detection and reduces the interference of redundant detection boxes. Through gesture and person association matching, accurate gesture distribution in a multi-person simultaneous control scene can be realized, different person gestures are avoided from being confused, and the feasibility of multi-user interaction is enhanced. The quantitative calculation of the moving direction, total moving distance and instantaneous moving distance in gesture trajectory tracking provides a quantitative basis for the accurate triggering of control functions such as volume adjustment and song switching, and improves the precision of control and user experience. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a flowchart of the present application. DETAILED DESCRIPTION

[0017] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example implementations to those skilled in the art.

[0018] Moreover, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the

[0019] The block diagrams shown in the accompanying drawings are merely functional entities, and do not necessarily have to correspond to physically independent entities. That is, these functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0020] The flowcharts shown in the accompanying drawings are merely exemplary illustrations, and do not necessarily include all contents and operations / steps, nor do they have to be executed in the order described. For example, some operations / steps can be further broken down, and some operations / steps can be combined or partially combined, so the actual execution order can be changed according to actual conditions.

[0021] Embodiments: The gesture recognition and trajectory tracking method based on a deep learning model, as shown in Figure 1 includes the following steps: Step 1: Collect video images and pre-process the images; Step 2: Input the pre-processed images into a neural network, and the neural network detects gestures in the images and outputs a candidate bounding box set containing gesture positions; Step 3: Confidence screening and deduplication detection are performed on the candidate bounding box set to obtain valid gesture bounding boxes, which include several gesture categories; In step 4, the identified gesture categories are associated and matched with the characters, and a matching gesture is assigned to each detected character. In step 5, a position history recording mechanism is established within a sliding time window to calculate the movement vector of the gesture to determine the movement direction of the gesture, and the total movement distance and instantaneous movement distance are calculated to track the spatial movement trajectory of the gesture.

[0022] The step 1 is specifically as follows: Sub-step 1.1: the camera captures a YUYV format video stream with a fixed resolution to obtain an original image sequence; Sub-step 1.2: Convert the YUYV format image to BGR format image. The conversion formula is: Ibgr(u,v)=ColorSpaceYUYV→BGR(Iyuyv(u,v) The ColorSpace transformation uses the YUV to RGB conversion matrix of the ITU-R BT.601 standard. Ibgr(u,v) is the pixel value at the coordinate (u,v) of the output BGR format image, Iyuyv(u,v) is the pixel value at the coordinate (u,v) of the input YUYV format image, ColorSpaceYUYV→BGR represents the color space conversion function, and (u,v) is the pixel coordinate position in the image. In substep 1.3, bilinear interpolation is performed on the BRG format image to adjust the BRG format image to an image size of 640×640 suitable for input to the neural network: Iresized(u,v)=BilinearResize(Ibgr,640,640) Among them, Iresized represents the converted BGR format image, BilinearResize represents the scaling operation, (u,v)∈[0,639]×[0,639]; In substep 1.4, normalize the scaled BRG image, converting pixel values ​​from integers (0–255) to floating-point values ​​(0–1), which is the standard input format for deep learning models. Normalized data has a smaller numerical range, which helps the gradient descent algorithm converge faster.

[0023] Inorm(u,v,c)=Iresized(u,v,c) / 255.0 Among them, Inorm(u,v,c) represents the pixel value of the normalized image at coordinate (u,v) and color channel c, and c∈{R,G,B} represents the color channel.

[0024] In step 2, the neural network is a neural network based on the YOLOv5 architecture, and the neural network model includes a backbone feature extraction network, a feature pyramid fusion network, and a multi-scale detection head. Step 2 is specifically: Sub-step 2.1, the backbone feature extraction network performs feature extraction, and the feature mapping calculation of the l-th layer is: Fl=σ(BN(Convl(Fl-1))) Where σ is the activation function, BN is the batch normalization, and Conv is the convolution operation. Sub-step 2.2, the feature pyramid fusion network performs multi-scale feature fusion, using a bottom-up and top-down double fusion strategy: Pi=Concat(Upsample(Pi+1),Fi) Ni=Concat(Downsample(Ni-1),Pi) Where Concat represents the feature map concatenation operation, Pi represents the fusion feature map obtained by concatenating the top-down upsampling and the bottom-up feature map Fi, Ni represents the further fusion feature map obtained by concatenating the bottom-up downsampling and the previous layer Ni-1 based on Pi, and i∈{3,4,5} represents different scale levels. Sub-step 2.3, the detection head outputs prediction results at three scales, and each grid cell (i,j) predicts K candidate bounding boxes: bboxk=[xcenter,ycenter,width,height,confidence,pclass] Where the calculation formula of the bounding box parameters is the center coordinates: xcenter=(i+σ(tx)) / Gx, ycenter=(j+σ(ty)) / Gy; size prediction: width=anchor_w×e^tw, height=anchor_h×e^th; confidence: confidence=σ(tconf), pclass=softmax(tclass); class probability: pclass=softmax(tclass), σ(x)=1 / (1+e^(-x)) is the sigmoid activation function, (i,j) is the grid cell coordinate, (Gx,Gy) is the grid size, (tx,ty,tw,th) is the original prediction value output by the network, (anchor_w,anchor_h) is the preset anchor box size, tconf is the confidence original output, and tclass is the class prediction original output.

[0025] In step 3, the confidence screening of the candidate bounding box set is specifically: confidence x max(pclass) > 0_conf wherein confidence represents the confidence of the candidate bounding box, max(pclass) represents the predicted probability distribution of the gesture class, and 0_conf is a basic confidence threshold; The de-duplication detection is specifically: IoU(boxi, boxj) = Area(boxi∩boxj) / Area(boxi∪boxj) wherein boxi, boxj represent two compared candidate bounding boxes, Area(boxi∩boxj) represents the area of intersection of the two candidate bounding boxes, Area(boxi∪boxj) represents the area of the union part of the two bounding boxes, and IoU represents the intersection-over-union ratio.

[0026] When IoU > 0_iou = 0.45, the detection box with lower confidence is deleted.

[0027] The retained detection boxes are classified into two gesture classes: C = {OK(0), STOP(1)}.

[0028] The step 4 is specifically: Sub-step 4.1, the spatial distance between the gesture bounding box and the person bounding box is calculated, and the Euclidean distance is used for measurement: wherein center_p and center_g are the coordinates of the geometric center points of the person box and the gesture box, respectively; Sub-step 4.2, a matching condition of distance constraint is established, d ≤ Dmax, wherein the threshold value Dmax = 200 pixels; Sub-step 4.3, for each detected task, a gesture class that is closest in distance and satisfies the matching condition is assigned: matched_gesture = argmin(d(person_box, gesture_box)) wherein gesture ∈ G, and G is the set of all gestures detected in the current frame.

[0029] The step 5 is specifically: Sub-step 5.1, a position history recording mechanism in a sliding time window is established, which is used to maintain a sequence of gesture center positions for consecutive N frames: Position_History = {pos_t, pos_t-1, …, pos_t-N+1} wherein N = 10 is the window size of the history frame number, and pos_t is the gesture center position of each frame. Sub-step 5.2, the gesture class value time window is calculated by using the first and last position difference method to calculate the movement vector movement = center_newest - center_oldest movement = (dx, dy) = (x_new - x_old, y_new - y_old) Wherein, newest and oldest represent the latest and earliest positions in the time window, respectively; Sub-step 5.3, based on the principal component of the movement vector, four basic movement directions are determined: According to the principal component analysis of the movement vector (dx, dy), four direction determination rules are established: When |dx|>|dy| and dx>0, the LEFT direction is determined; When |dx|>|dy| and dx<0, the RIGHT direction is determined; When |dx|≤|dy| and dy<0, the UP direction is determined; When |dx|≤|dy| and dy>0, the DOWN direction is determined; Other cases are determined as NONE direction.

[0030] Sub-step 5.4, the total movement distance and the instantaneous movement distance are calculated, and the total movement distance calculation formula is: The instantaneous movement distance calculation formula is:

[0031] In this embodiment, the YOLOv5 model uses a loss function optimized for two categories, and the total loss function is: L_total = λ_coord × L_coord + λ_conf × L_conf + λ_class × L_class_binary Wherein, L_total represents the total loss function of the neural network, λ_coord represents the weight coefficient of the bounding box coordinate loss, L_coord represents the bounding box coordinate loss, λ_conf represents the weight coefficient of the confidence loss, L_conf represents the confidence loss, λ_class represents the weight coefficient of the class loss, and the binary classification loss function is simplified as: Wherein, S 2 represents all grid units, I_i^obj represents an indicator function, p_i(c) represents the probability that the model predicts that the gesture in the i-th grid unit belongs to class c, represents the true label of the gesture in the i-th grid unit belonging to class c.

[0032] In this embodiment, the spatial movement tracking algorithm contains an automatic correction mechanism for camera mirror effect: The mirror transformation matrix is: M_mirror = [[-1,0,width],[0,1,0],[0,0,1]]; Corrected direction: direction_corrected = Transform(direction_raw, M_mirror) Among them, direction_corrected is the gesture movement direction after mirror transformation correction, Transform represents the mirror transformation function, direction_raw represents the original direction of gesture movement, and M_mirror represents the mirror transformation matrix.

[0033] A gesture recognition and trajectory tracking device based on a deep learning model, applicable to a gesture recognition and trajectory tracking method based on a deep learning model, including: Video acquisition and preprocessing module, used to acquire video images and preprocess the images; The gesture detection module is used to input the preprocessed image into the neural network, perform gesture detection on the image through the neural network, and output a set of candidate bounding boxes containing the gesture location; A gesture screening and classification module is used to perform confidence screening and deduplication detection on a set of candidate bounding boxes to obtain valid gesture bounding boxes, which include several gesture categories; A gesture and person association and matching module is used to associate and match the recognized gesture categories with people and assign a matching gesture to each detected person; The gesture trajectory tracking module is used to calculate the movement vector of the gesture to determine the movement direction of the gesture by establishing a position history recording mechanism within a sliding time window, and at the same time calculate the total movement distance and instantaneous movement distance to track the spatial movement trajectory of the gesture.

[0034] The video acquisition and preprocessing module includes: YUYV is a video stream acquisition unit that is used to acquire a fixed-resolution YUYV format video stream through a camera to obtain an original image sequence; A format conversion unit, used for converting a YUYV format image into a BGR format image; A size adjustment unit is used to perform bilinear interpolation scaling on the BGR format image so as to adjust the BGR format image to an image size suitable for input to the neural network; The normalization processing unit is used to perform normalization processing on the scaled BGR image.

[0035] The gesture trajectory tracking module includes: a position history unit configured to establish a position history mechanism within a sliding time window to maintain a sequence of gesture center positions of N consecutive frames; a movement vector calculation unit configured to calculate a movement vector of the gesture within the time window using a first and last position difference method; a movement direction determination unit configured to determine four basic movement directions based on a principal component of the movement vector; a movement distance calculation unit configured to calculate a total movement distance and an instantaneous movement distance.

[0036] Specifically, taking a gesture-controlled Bluetooth sound system as an example, one specific embodiment of matching a plurality of gestures to a character is as follows: the system simultaneously detects two users (character A and character B) and their respective gestures (character A makes an OK gesture and character B makes a STOP gesture).

[0037] Calculate the Euclidean distance between the gesture box and the character box: wherein the distance d1 between the gesture box of character A and the character A box is 80 pixels, and the distance d2 between the gesture box of character A and the character B box is 300 pixels, and since d1≤200 pixels (Dmax=200), the OK gesture is matched to character A; similarly, the STOP gesture is matched to character B.

[0038] Establish independent state trackers for the two people to record parameters such as current_gesture and frame_count, and the frame_count of character A is incremented with each consecutive OK gesture, and the frame_count of character B is incremented with each consecutive STOP gesture, while simultaneously tracking the gesture spatial trajectory of character A and character B, and a 10-frame sliding time window (N=10) is established for the OK gesture of character A, and a circular buffer is used to store the sequence of center positions Position_History={pos_t,pos_t-1,...,pos_t-9} of each frame.

[0039] Calculate the movement vector: take the first frame (pos_t-9=(x_old,y_old)) and the last frame (pos_t=(x_new,y_new)) within the window to obtain dx=x_new-x_old and dy=y_new-y_old.

[0040] Determine the direction: since |dx|=50 pixels and |dy|=20 pixels (|dx|>|dy|), and dx>0, the preliminary determination is the LEFT direction; after correction by the mirror transformation matrix M_mirror=[[-1,0,640],[0,1,0],[0,0,1]], the direction_corrected is the RIGHT direction.

[0041] Calculate the distance: the total movement distance distance_total=√(502 +20 2 )≈53.85 pixels; instantaneous moving distance Where distance_frame = 8 pixels (since > 5 pixels minimum threshold, included in calculation).

[0042] Person A's OK gesture keeps moving right, system starts accumulating horizontal distance: current direction is RIGHT, consistent with accumulated direction, so distance_accumulated += |dx_frame| (dx_frame is horizontal component per frame).

[0043] Time window constraint: the interval from t_direction_start to t_current is 1500ms ≤ 2000ms (T_window = 2000ms), window is valid (window_valid = true).

[0044] When distance_accumulated = 65 pixels ≥ 60 pixels (D_threshold = 60), and window_valid = true, trigger the song change instruction NEXT_TRACK.

[0045] Through the above description of the embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present application can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to make a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) execute the method according to the embodiments of the present application.

[0046] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the embodiments disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such

[0047] It should be understood that the present application is not limited to the precise construction that has been described above and illustrated in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the present application. The scope of the present application is limited only by the appended claims.

Claims

1. A gesture recognition and trajectory tracking method based on a deep learning model, characterized by: The following steps are involved: Step 1: Capture video images and preprocess the images; Step 2: Input the preprocessed image into the neural network, which performs gesture detection on the image and outputs a set of candidate bounding boxes containing the gesture location. Step 3: Confidence screening and deduplication detection are performed on the candidate bounding box set to obtain valid gesture bounding boxes, which include several gesture categories; Step 4: Associating and matching the identified gesture categories with the characters, assigning a matching gesture to each detected character; Step 5: By establishing a position history recording mechanism within the sliding time window, the movement vector of the gesture is calculated to determine the movement direction of the gesture, and the total movement distance and instantaneous movement distance are calculated at the same time to track the spatial movement trajectory of the gesture.

2. The gesture recognition and trajectory tracking method based on a deep learning model according to claim 1 is characterized in that: The step 1 is specifically as follows: Sub-step 1.1: the camera captures a fixed-resolution YUYV format video stream to obtain an original image sequence; Sub-step 1.2: Convert the YUYV format image to BGR format image; Sub-step 1.3, performing bilinear interpolation scaling on the BRG format image to adjust the BRG format image to an image size suitable for input to the neural network; Sub-step 1.4, normalize the scaled BRG image, normalizing the pixel values ​​from the integer range of 0-255 to the floating point range of 0-1.

3. The gesture recognition and trajectory tracking method based on a deep learning model according to claim 1, characterized in that: In step 2, the neural network is a neural network based on the YOLOv5 architecture. The neural network model includes a backbone feature extraction network, a feature pyramid fusion network, and a multi-scale detection head. Step 2 is specifically as follows: Sub-step 2.1, the backbone feature extraction network performs feature extraction, and the feature map of the lth layer is calculated as: Fl = σ(BN(Convl(Fl-1))) Among them, σ is the activation function, BN is batch normalization, and Conv is the convolution operation; In sub-step 2.2, the feature pyramid fusion network performs multi-scale feature fusion, adopting a dual fusion strategy of bottom-up and top-down: Pi=Concat(Upsample(Pi+1),Fi) Ni=Concat(Downsample(Ni-1),Pi) Among them, Concat represents the feature map concatenation operation, Pi represents the fused feature map obtained by concatenating the top-down upsampling with the bottom-up feature map Fi, Ni represents the further fused feature map obtained by concatenating the bottom-up downsampling with the previous layer Ni-1 on the basis of Pi, and i∈{3,4,5} represents different scale levels; In sub-step 2.3, the detection head outputs prediction results at several different scales.

4. The gesture recognition and trajectory tracking method based on a deep learning model according to claim 1, characterized in that: In step 3, the confidence screening of the candidate bounding box set is specifically as follows: confidence×max(pclass)≥θ_conf Among them, confidence represents the confidence of the candidate bounding box, max(pclass) represents the predicted probability distribution of the gesture category, and θ_conf is the basic confidence threshold; Deduplication detection is specifically as follows: IoU(boxi,boxj)=Area(boxi∩boxj) / Area(boxi∪boxj) Among them, boxi, boxj represent two candidate bounding boxes with comparison, Area(boxi∩boxj) represents the area of ​​the intersection of the two candidate bounding boxes, Area(boxi∪boxj) represents the area of ​​the combined part of the two bounding boxes, and IoU represents the intersection-over-union ratio.

5. The gesture recognition and trajectory tracking method based on a deep learning model according to claim 1, characterized in that: The step 4 is specifically as follows: Sub-step 4.1, calculating the spatial distance between the gesture bounding box and the person bounding box; Sub-step 4.2, establishing distance-constrained matching conditions; In sub-step 4.3, each detected task is assigned the gesture category that is closest and satisfies the matching conditions.

6. The gesture recognition and trajectory tracking method based on a deep learning model according to claim 1, characterized in that: The step 5 is specifically as follows: Sub-step 5.1, establish a position history recording mechanism within the sliding time window to maintain a sequence of gesture center positions for N consecutive frames; Sub-step 5.2, using the first and last position difference method to calculate the movement vector within the gesture category value time window; Sub-step 5.3, determining four basic movement directions based on the principal components of the movement vector; Sub-step 5.4, calculate the total moving distance and instantaneous moving distance.

7. A gesture recognition and trajectory tracking device based on a deep learning model, applicable to the gesture recognition and trajectory tracking method based on a deep learning model as claimed in any one of claims 1 to 6, characterized in that: include: Video acquisition and preprocessing module, used to acquire video images and preprocess the images; The gesture detection module is used to input the preprocessed image into the neural network, perform gesture detection on the image through the neural network, and output a set of candidate bounding boxes containing the gesture location; A gesture screening and classification module is used to perform confidence screening and deduplication detection on a set of candidate bounding boxes to obtain valid gesture bounding boxes, which include several gesture categories; A gesture and person association and matching module is used to associate and match the recognized gesture categories with people and assign a matching gesture to each detected person; The gesture trajectory tracking module is used to calculate the movement vector of the gesture to determine the movement direction of the gesture by establishing a position history recording mechanism within a sliding time window, and at the same time calculate the total movement distance and instantaneous movement distance to track the spatial movement trajectory of the gesture.

8. The gesture recognition and trajectory tracking device based on a deep learning model according to claim 7, characterized in that: The video acquisition and preprocessing module includes: The YUYV video stream acquisition unit is used to acquire a YUYV format video stream with a fixed resolution through a camera to obtain an original image sequence; A format conversion unit, used for converting a YUYV format image into a BGR format image; A size adjustment unit is used to perform bilinear interpolation scaling on the BGR format image so as to adjust the BGR format image to an image size suitable for input to the neural network; The normalization processing unit is used to perform normalization processing on the scaled BGR image.

9. The gesture recognition and trajectory tracking device based on a deep learning model according to claim 7, characterized in that: The gesture trajectory tracking module includes: The position history recording unit is used to establish a position history recording mechanism within a sliding time window to maintain a sequence of gesture center positions for N consecutive frames; A motion vector calculation unit, configured to calculate the motion vector of the gesture within the time window using a first-end position difference method; a moving direction determination unit, configured to determine four basic moving directions based on the principal components of the moving vector; The moving distance calculation unit is used to calculate the total moving distance and the instantaneous moving distance.

10. A storage medium, characterized in that: The storage medium stores computer-executable instructions, which, when loaded and executed by the processor, implement the steps of the gesture recognition and trajectory tracking method based on a deep learning model as described in any one of claims 1 to 6.

Images