A hand gesture-based identity recognition method, system, terminal and storage medium

Abstract

The application discloses a hand gesture-based identity recognition method and system, a terminal and a storage medium. The method comprises the following steps: acquiring a target video, pre-processing video frames to obtain a plurality of pre-processed video frames, and acquiring contour maps and skeleton maps according to a contour extraction network and a skeleton extraction model; encoding each contour map to obtain a plurality of contour features, and encoding each skeleton map to obtain a plurality of skeleton features; using global maximum pooling and global average pooling to obtain spatial feature embedding and time feature embedding; inputting the spatial feature embedding into a multi-modal spatial fusion module to obtain spatial feature representation, and inputting the time feature embedding into a multi-modal time fusion module to obtain time feature representation; obtaining a classification result according to the spatial feature representation and the time feature representation, and performing identity recognition according to the classification result. The application can obtain accurate hand gesture classification that can be used for identity recognition, and then perform identity recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a method, system, terminal and storage medium for identity recognition based on hand gestures. Background Art

[0002] Currently, identity recognition methods include those based on knowledge, human biometrics, and objects. Each of these methods has its own advantages and disadvantages and is suitable for different application scenarios. Biometric recognition, among others, confirms identity by detecting and analyzing unique physiological or behavioral characteristics, offering high levels of uniqueness and security. Common biometric recognition methods include fingerprint recognition, facial recognition, iris recognition, voiceprint recognition, and gait recognition. Gesture recognition is commonly used in applications such as human-computer interaction and sign language expression.

[0003] However, in the current identification methods based on biometrics, since biometrics are easy to be forged, the corresponding identification methods are prone to fraud and cannot guarantee the safety of users.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0005] The main purpose of the present invention is to provide an identity recognition method, system, terminal and computer-readable storage medium based on hand gestures, aiming to solve the problem in the existing biometric-based recognition methods that biometric features are easily forged, resulting in the corresponding recognition methods being easily fraudulent and unable to guarantee user safety.

[0006] To achieve the above object, the present invention provides an identity recognition method based on hand gestures, the identity recognition method based on hand gestures comprising the following steps:

[0007] Acquire a target video, and preprocess each video frame of the target video to obtain a plurality of preprocessed video frames, and obtain a contour map and a skeleton map of each preprocessed video frame according to a contour extraction network and a skeleton extraction model;

[0008] Encode each contour image according to the contour encoding module to obtain multiple contour features, and encode each skeleton image according to the skeleton encoding module to obtain multiple skeleton features;

[0009] Using global maximum pooling and global average pooling on all the contour features and all the skeleton features to obtain spatial feature embedding and temporal feature embedding, inputting the spatial feature embedding into a multimodal spatial fusion module to obtain spatial feature representation, and inputting the temporal feature embedding into a multimodal temporal fusion module to obtain temporal feature representation;

[0010] A classification result is obtained according to the spatial feature representation and the temporal feature representation, and identity recognition is performed according to the classification result.

[0011] Optionally, the acquiring of a target video, preprocessing each video frame of the target video to obtain a plurality of preprocessed video frames, and acquiring a contour map and a skeleton map of each preprocessed video frame according to a contour extraction network and a skeleton extraction model specifically includes:

[0012] Obtain a target video, obtain corresponding video frames according to the target video, crop each of the video frames, and adjust the resolution to obtain multiple pre-processed video frames;

[0013] Inputting all the pre-processed video frames into a contour extraction network to extract features and perform classification to obtain a contour map of each video frame;

[0014] All the pre-processed video frames are input into a skeleton extraction model to extract features, and a skeleton graph of each video frame is obtained.

[0015] Optionally, the skeleton extraction model includes a pyramid feature extractor, a feature enhancement module, an anchor interaction module and an anchor weight offset estimation module;

[0016] The step of inputting all the pre-processed video frames into a skeleton extraction model to extract features and obtain a skeleton graph of each video frame specifically includes:

[0017] Inputting all the pre-processed video frames into the pyramid feature extractor in sequence to extract corresponding pyramid features;

[0018] Inputting the pyramid features into a feature enhancement module, wherein the feature enhancement module enhances the pyramid features through deformable multi-head attention to obtain enhanced features, wherein the feature enhancement module includes 6 encoder layers, and each encoder layer includes one deformable multi-head attention;

[0019] Inputting the enhanced features into the anchor interaction module, the anchor interaction module processes the enhanced features through 6 decoder layers to obtain a decoded output;

[0020] The decoded output is input to the anchor weight offset estimation module, which estimates the 3D offset and weight of the hand joint corresponding to each anchor point in the decoded output separately, and performs weighted fusion to obtain a skeleton graph corresponding to each preprocessed video frame.

[0021] Optionally, the contour encoding module includes a channel attention module, a local time fusion module, and a global to local motion perception module;

[0022] Each contour image is encoded according to the contour encoding module to obtain multiple contour features, including:

[0023] Inputting the contour map into the channel attention for compression to obtain a 4-dimensional tensor;

[0024] Input the 4-dimensional tensor into the local time fusion module to perform convolution operation to obtain convolution features;

[0025] The convolutional features are input into the global-to-local motion perception module to obtain corresponding contour features.

[0026] Optionally, encoding each skeleton image according to the skeleton encoding module to obtain multiple skeleton features specifically includes:

[0027] Inputting the skeleton image into a skeleton encoding module to obtain a plurality of skeleton features;

[0028] The skeleton coding module is composed of multiple Bottlenecks, each Bottleneck includes a spatial block, a temporal block and a residual link.

[0029] Optionally, the spatial feature embedding includes a first spatial feature and a second spatial feature;

[0030] The spatial feature embedding is input into the multimodal spatial fusion module to obtain the spatial feature representation, which specifically includes:

[0031] Inputting the first spatial feature into the first cross attention of the multimodal spatial fusion module, inputting the second spatial feature into the second cross attention of the multimodal spatial fusion module, and connecting the corresponding outputs to obtain a multimodal spatial fusion connection;

[0032] The multimodal spatial fusion connection is processed and fused through the Norm and FNN layers to obtain the spatial feature representation.

[0033] Optionally, the time feature embedding includes a first time feature and a second time feature;

[0034] The embedding of the time feature into a multimodal time fusion module to obtain a time feature representation specifically includes:

[0035] Inputting the first time feature into a first cross attention in the multimodal time fusion module, and inputting the second time feature into a second cross attention in the multimodal time fusion module;

[0036] The outputs of the first cross attention and the second cross attention are processed through the Norm and FNN layers respectively, and connected to obtain the temporal feature representation.

[0037] In addition, to achieve the above-mentioned purpose, the present invention further provides an identity recognition system based on hand gestures, wherein the identity recognition system based on hand gestures comprises:

[0038] An extraction module is used to obtain a target video, preprocess each video frame of the target video to obtain a plurality of preprocessed video frames, and obtain a contour map and a skeleton map of each preprocessed video frame according to a contour extraction network and a skeleton extraction model;

[0039] an encoding module, configured to encode each contour image according to the contour encoding module to obtain a plurality of contour features, and to encode each skeleton image according to the skeleton encoding module to obtain a plurality of skeleton features;

[0040] A spatiotemporal fusion module, configured to perform global maximum pooling and global average pooling on all the contour features and all the skeleton features to obtain spatial feature embedding and temporal feature embedding, input the spatial feature embedding into a multimodal spatial fusion module to obtain spatial feature representation, and input the temporal feature embedding into a multimodal temporal fusion module to obtain temporal feature representation;

[0041] A result output module is used to obtain a classification result based on the spatial feature representation and the temporal feature representation, and perform identity recognition based on the classification result.

[0042] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a hand gesture-based identity recognition program stored on the memory and runnable on the processor, and when the hand gesture-based identity recognition program is executed by the processor, the steps of the hand gesture-based identity recognition method as described above are implemented.

[0043] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a hand gesture-based identity recognition program, and when the hand gesture-based identity recognition program is executed by the processor, the steps of the hand gesture-based identity recognition method as described above are implemented.

[0044] In the present invention, a target video is obtained, and each video frame of the target video is preprocessed to obtain multiple preprocessed video frames, and a contour map and a skeleton map of each preprocessed video frame are obtained according to a contour extraction network and a skeleton extraction model; each contour map is encoded according to a contour encoding module to obtain multiple contour features, and each skeleton map is encoded according to a skeleton encoding module to obtain multiple skeleton features; global maximum pooling and global average pooling are used for all the contour features and all the skeleton features to obtain spatial feature embedding and temporal feature embedding, the spatial feature embedding is input into a multimodal spatial fusion module to obtain spatial feature representation, and the temporal feature embedding is input into a multimodal temporal fusion module to obtain temporal feature representation; a classification result is obtained according to the spatial feature representation and the temporal feature representation, and identity recognition is performed according to the classification result. The present invention adopts two independent coding modules, including the contour coding module (CCM) and the skeleton coding module (SCM), to capture the unique discriminant information of each modality, and proposes a global-local motion-aware module (GLMAM), which not only focuses on the global information of the hand, but also pays attention to the connection between local regions. It also proposes a multimodal spatial fusion module (MSFM) and a multimodal time fusion module (MTFM), so that accurate hand posture classification that can be used for identity recognition can be obtained, and then identity recognition can be performed. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of a preferred embodiment of the identity recognition method based on hand gestures of the present invention;

[0046] Figure 2 Schematic diagram of the structure of the contour extraction network in the hand gesture-based identity recognition method of the present invention;

[0047] Figure 3 Schematic diagram of the structure of the skeleton extraction model in the hand gesture-based identity recognition method of the present invention;

[0048] Figure 4 1 is a schematic structural diagram of a pyramid feature extractor of a skeleton extraction model in a method for identifying an individual by hand gesture according to the present invention;

[0049] Figure 5 Schematic diagram of the structure of each encoder in the feature enhancement module of the skeleton extraction model in the hand gesture-based identity recognition method of the present invention;

[0050] Figure 6 Schematic diagram of the structure of each decoder in the anchor interaction module of the skeleton extraction model in the hand gesture-based identity recognition method of the present invention;

[0051] Figure 7 Schematic diagram of the structure of the contour encoding module of the skeleton extraction model in the hand gesture-based identity recognition method of the present invention;

[0052] Figure 8 2 is a schematic diagram of the structure of the channel attention of the contour encoding module of the skeleton extraction model in the hand gesture-based identity recognition method of the present invention;

[0053] Figure 9 It is a schematic structural diagram of the global to local motion perception module of the contour encoding module of the skeleton extraction model in the identity recognition method based on hand gestures of the present invention;

[0054] Figure 10 The template function is applied to S in the contour encoding module of the skeleton extraction model in the identity recognition method based on hand gesture of the present invention. p Schematic diagram of each moment;

[0055] Figure 11 The contour encoding module of the skeleton extraction model in the hand gesture-based identity recognition method of the present invention obtains S m Schematic diagram of;

[0056] Figure 12 Schematic diagram of the structure of the skeleton encoding module in the hand gesture-based identity recognition method of the present invention;

[0057] Figure 13 Schematic diagram of the structure of the multimodal spatial fusion module in the hand gesture-based identity recognition method of the present invention;

[0058] Figure 14 Schematic diagram of the structure of the multimodal time fusion module in the hand gesture-based identity recognition method of the present invention;

[0059] Figure 15 This is a schematic diagram of an implementation flow of an embodiment of the hand gesture-based identity recognition method of the present invention;

[0060] Figure 16 is a structural diagram of a preferred embodiment of the hand gesture-based identity recognition system of the present invention;

[0061] Figure 17 FIG. 4 is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0063] Currently, identity recognition methods include those based on knowledge, human biometrics, and objects. Each of these methods has its own advantages and disadvantages and is suitable for different application scenarios. Biometric recognition confirms identity by detecting and analyzing unique physiological or behavioral characteristics of the human body, offering high levels of uniqueness and security. Common biometric recognition methods include fingerprint recognition, facial recognition, iris recognition, voiceprint recognition, and gait recognition. Gesture recognition is commonly used in applications such as human-computer interaction and sign language. However, current biometric-based recognition methods are susceptible to fraud due to the ease with which biometrics can be forged, making them vulnerable to fraud and unable to guarantee user security.

[0064] In response to one or more of the above problems, the present invention obtains a target video and preprocesses each video frame of the target video to obtain multiple preprocessed video frames, and obtains a contour map and a skeleton map of each preprocessed video frame according to a contour extraction network and a skeleton extraction model; encodes each contour map according to a contour encoding module to obtain multiple contour features, and encodes each skeleton map according to a skeleton encoding module to obtain multiple skeleton features; uses global maximum pooling and global average pooling for all the contour features and all the skeleton features to obtain spatial feature embedding and temporal feature embedding, inputs the spatial feature embedding into a multimodal spatial fusion module to obtain spatial feature representation, and inputs the temporal feature embedding into a multimodal temporal fusion module to obtain temporal feature representation; obtains a classification result according to the spatial feature representation and the temporal feature representation, and performs identity recognition based on the classification result.

[0065] The preferred embodiment of the present invention is a method for identifying an individual based on hand gestures, as shown in FIG. Figure 1 As shown, the hand gesture-based identity recognition method includes the following steps:

[0066] Step S10: Obtain a target video, and preprocess each video frame of the target video to obtain multiple preprocessed video frames, and obtain a contour map and a skeleton map of each preprocessed video frame according to a contour extraction network and a skeleton extraction model.

[0067] Specifically, in the present invention, for a target video, all video frames are correspondingly extracted after acquisition and preprocessed, and then a contour map and a skeleton map corresponding to each frame of the video are extracted.

[0068] The step of obtaining a target video, preprocessing each video frame of the target video to obtain a plurality of preprocessed video frames, and obtaining a contour map and a skeleton map of each preprocessed video frame according to a contour extraction network and a skeleton extraction model specifically includes:

[0069] Obtain a target video, obtain corresponding video frames according to the target video, crop each of the video frames, and adjust the resolution to obtain multiple pre-processed video frames;

[0070] Inputting all the pre-processed video frames into a contour extraction network to extract features and perform classification to obtain a contour map of each video frame;

[0071] All the pre-processed video frames are input into a skeleton extraction model to extract features, and a skeleton graph of each video frame is obtained.

[0072] Specifically, in the present invention, a target video is acquired and each frame of the target video is preprocessed, that is, the image is cropped to focus on the hand area. This can be accomplished using a detection-based region of interest method. The cropped image is then resized to a 256×256 resolution. After obtaining the corresponding preprocessed video frames, they are input into the contour extraction network and the skeleton extraction model, respectively, to obtain the corresponding contour map and skeleton map.

[0073] Among them, the contour map is extracted through the FCN network, that is, the contour extraction network. The FCN network architecture is as follows Figure 2 As shown, c represents the channel dimension, (h, w) represents the height and width of the input video frame, a×a represents the size of the convolution kernel, sc represents the step size c, and num_cls represents the number of categories for the final classification; FCN uses VGG16 as the backbone, changes the last fully connected layer to a convolutional layer, and performs upsampling through transposed convolution to restore the feature map to the input image size and output the heat map of the category. For the present invention, FCN only needs to judge two categories: hand and background, thereby generating a hand contour map. In binary classification training, the loss is calculated for each pixel, and the average is calculated after aggregation as the final loss of the batch. The definition of the loss function is shown in formula (1):

[0074] L i =-y i log(p i )-(1-y i )log(1-p i ); (1)

[0075] Among them, L i is the loss of the i-th pixel, y i is the true label of the i-th pixel (0 or 1), p iis the probability that the model predicts that the i-th pixel is a positive class. The loss of each pixel is shown in formula (2):

[0076] L total =∑ i L i ; (2)

[0077] The final loss function is shown in formula (3):

[0078]

[0079] Furthermore, the skeleton extraction model includes a pyramid feature extractor, a feature enhancement module, an anchor interaction module and an anchor weight offset estimation module;

[0080] The step of inputting all the pre-processed video frames into a skeleton extraction model to extract features and obtain a skeleton graph of each video frame specifically includes:

[0081] Inputting all the pre-processed video frames into the pyramid feature extractor in sequence to extract corresponding pyramid features;

[0082] Inputting the pyramid features into a feature enhancement module, wherein the feature enhancement module enhances the pyramid features through deformable multi-head attention to obtain enhanced features, wherein the feature enhancement module includes 6 encoder layers, and each encoder layer includes one deformable multi-head attention;

[0083] Inputting the enhanced features into the anchor interaction module, the anchor interaction module processes the enhanced features through 6 decoder layers to obtain a decoded output;

[0084] The decoded output is input to the anchor weight offset estimation module, which estimates the 3D offset and weight of the hand joint corresponding to each anchor point in the decoded output separately, and performs weighted fusion to obtain a skeleton graph corresponding to each preprocessed video frame.

[0085] Specifically, in the present invention, when obtaining the skeleton graph, the A2J-Transformer model is used to extract the hand skeleton graph of the video frame. The core idea of ​​the A2J-Transformer model, that is, the skeleton extraction model, is to use a set of anchor points as local estimators, and each anchor point predicts its position offset to all hand joints. These offsets are then used to estimate the final 3D position of the joints. The framework diagram of A2J-Transformer is shown in the figure below. Figure 3 As shown in Figure 2, n represents the number of nodes in the hand joint. The A2J-Transformer model is divided into four parts: pyramid feature extractor, feature enhancement module, anchor interaction module, and anchor weight offset estimation module.

[0086] Among them, pyramid feature extractors such as Figure 4 As shown in Figure 2, the pyramid feature extractor uses multi-scale features to capture global information from the input image while preserving details. The pyramid feature extractor uses ResNet-50 as the backbone network to extract pyramid features from the input video frames. Specifically, the pyramid features are obtained by extracting feature maps from blocks 2-4 and the last layer after block 4 in ResNet-50.

[0087] Multi-scale features are useful for capturing global clues and recovering local details. A feature enhancement module is added to the A2J-Transformer model to enhance multi-scale features by the self-attention module. It consists of six encoder layers. The first encoder layer of this module is as follows Figure 5 As shown, the input features of the remaining encoder layers are the output of the previous layer. pyr +p,k=ref(x pyr ), v = x pyr , p is the position encoding, ref(·) represents the input feature x pyr , which is used to generate a set of keys for attention calculation.

[0088] In the anchor interaction module, the A2J-Transformer first densely distributes uniformly distributed 3D anchors on the original image. The planar coordinates of the 3D anchors are evenly distributed on the plane of the input image. The planar stride of the anchors is set to 16. For an input image resolution of 256×256, a total of 16×16=256 anchors fill the image plane. Each anchor then takes multiple depth values ​​centered at the depth of the root joint, representing different depth positions. Specifically, the center depth is set to 0, and two depth values ​​are set forward and backward within ±100 mm. Therefore, each planar anchor corresponds to three depth values. The model then directly estimates the hand joints using these 3D anchors. Using local anchors for hand pose estimation has two advantages. First, dense anchors effectively capture local image details, making them suitable for processing detailed information of hands with strong interactions. Second, the cross-attention module establishes interactions between local anchors, capturing global cues and helping to address occlusion issues. Based on the above two points, A2J-Transformer designs an anchor interaction module consisting of 6 decoder layers to link various anchors so that each anchor can obtain global information. Figure 6 As shown, for the first decoder layer, the decoder embedding will be replaced by the encoder output. Where k = q = d + p d ,v=d,p d =MLP(PE(a q )), a qrepresents the coordinates of the qth anchor point, PE(·) represents the position encoding of the sinusoidal embedding generated from the floating-point number, and d represents the encoding of the initially set anchor point.

[0089] After each anchor point passes through the anchor interaction module in the A2J-Transformer module, local details can be restored and global information can be perceived. Subsequently, the anchor offset weight estimation model estimates the 3D offset and weight of each anchor point relative to the hand joint separately. Each anchor point acts as a local estimator, and finally the results of all anchor points are fused by weighted summation to obtain the final position of the hand joint. Joint estimation loss and anchor surrounding loss are used to train A2J-Transformer. The expression of joint estimation loss is shown in formula (4):

[0090]

[0091] The parameter α defaults to 0.5 to balance the loss between the in-plane and depth offset estimation tasks. and are the estimated plane coordinates and depth coordinates of joint j, and are the real plane coordinates and depth coordinates of joint j, L τ2 (·) is the smoothL1-like loss function.

[0092] The expression of Anchor surrounding loss is shown in formula (5):

[0093]

[0094] Where A is the anchor point set, a∈A, W' i / j (a) represents the normalized weight of anchor point a to i / j, C i (a) and C d (a) represents the in-plane and depth coordinates of the anchor point a. The hand joint skeleton graph K can be obtained through A2J-Transformer.

[0095] Step S20: Encode each contour image according to the contour encoding module to obtain a plurality of contour features, and encode each skeleton image according to the skeleton encoding module to obtain a plurality of skeleton features.

[0096] The overall network for hand gesture-based identity recognition in this paper is based on the Transformer model. This model completely abandons the sequential processing structure of RNNs and adopts a parallel encoder-decoder design. It also uses a self-attention mechanism to flexibly capture global information in the input sequence. It uses CCM and SCM to encode skeleton sequences and contour sequences, respectively, targeting information from different modalities.

[0097] Furthermore, the contour encoding module includes a channel attention module, a local time fusion module, and a global to local motion perception module;

[0098] Each contour image is encoded according to the contour encoding module to obtain multiple contour features, including:

[0099] Inputting the contour map into the channel attention for compression to obtain a 4-dimensional tensor;

[0100] Input the 4-dimensional tensor into the local time fusion module to perform convolution operation to obtain convolution features;

[0101] The convolutional features are input into the global-to-local motion perception module to obtain corresponding contour features.

[0102] Build the encoder CCM to encode the contour map. The innovation of this module lies in the design of a global-local motion perception module. The specific CCM module is as follows Figure 7 As shown in Figure 2, CCM consists of three parts: channel attention (CA), local temporal fusion module (LTFM), and global-local motion-aware module (GLMAM).

[0103] The channel attention mechanism, namely channel attention, compresses the feature map in the spatial dimension to obtain a 4-dimensional tensor with dimensions of c×t×h×w. Figure 8 As shown in Figure 2, when compressing in the spatial dimension, both average pooling and maximum pooling are considered. Average pooling and maximum pooling are used to aggregate the spatial information of the feature map and compress its spatial dimension to generate a channel attention map. Average pooling provides feedback to all pixels in the feature map, while maximum pooling only provides gradient feedback to the pixel with the largest response. The channel attention mechanism can be expressed as formula (6):

[0104] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))); (6)

[0105] In LTFM, this layer can integrate the temporal information of the local sequence and maintain more spatial information. is the input of local temporal aggregation, where C1 is the number of channels, T1 is the length of the hand action sequence, and (H1, W1) is the image size of each frame. The process can be expressed as shown in formula (7):

[0106]

[0107] in, represents a 3D convolution operation with kernel size a and time step b. is the output of the LTFM operation.

[0108] Gesture recognition needs to focus on the global and local connections of the hand. Therefore, the present invention proposes a GLMAM to extract features, which can utilize global and local information. Figure 9 As shown in Figure 2, GLMAM is implemented by a conv3d layer and a local action builder, which contains global and local feature extractors. The global feature extractor captures the overall information of the hand, while the local feature extractor mines the detailed features, where T represents the length of the sequence.

[0109] The local micro-motion features of the hand in a short time interval are the most discriminative features of the hand posture, and the micro-motion state at any moment is determined by itself and its adjacent frames. Therefore, in GLMAM, the present invention designs a local motion builder (LMB) to map the frame-level local information feature vector to the micro-motion feature vector. Referring to the method of using average frames in GEI in gait recognition, the present invention uses two statistical functions as instantiations of the builder function (BuilderFunc). Assuming that the feature map obtained by conv3d output is First, reshape it into Where S = H3 × W3, then at each moment of T3 there is a two-dimensional tensor S p , the dimensions are channel and space, respectively, such as Figure 10 As shown, the template function is applied to S p every moment.

[0110] like Figure 11 As shown, first perform 2D average pooling and 2D maximum pooling with a kernel size of 2r+1, where r represents the r frames before and after the current frame to be fused. In this way, the motion feature vector S m The sequence can be expressed as shown in formula (8):

[0111] Sm =Avgpool2d(S p )+Maxpool2d(S p ); (8)

[0112] Finally, we get the feature F of the contour map L s ∈R C×T×S .

[0113] Furthermore, the skeleton encoding module encodes each skeleton image to obtain multiple skeleton features, specifically including:

[0114] Inputting the skeleton image into a skeleton encoding module to obtain a plurality of skeleton features;

[0115] The skeleton coding module is composed of multiple Bottlenecks, each Bottleneck includes a spatial block, a temporal block and a residual link.

[0116] Build the encoder SCM to encode the skeleton graph. The overall framework of the SCM module is as follows: Figure 12 As shown in Figure 1, the module consists of a ResGCN block, using a residual connection in a graph convolution and an optional bottleneck. The output channel of the last block is set to 128, the same as the output of CCM.

[0117] The parameters of the skeleton encoding module are [B1, N2, N3], which correspond to the modules and the number of modules in different model parts. Specifically, B1 represents a ResGCN module with a basic block, and N2 / N3 represents two / three ResGCN modules with a bottleneck block. Each module in the network consists of a spatial block, a temporal block, and a residual link. The temporal block slides along the time dimension through one-dimensional convolution to extract the dynamic change pattern and contextual dependency of the time series, capturing the motion trajectory characteristics of the hand joints at different time steps. The temporal convolution operation can be expressed as shown in formula (9):

[0118] f t =ReLU(Batch Norm(Conv1D(X))); (9)

[0119] The size of the convolution kernel L is a hyperparameter that defines the range of the time window (e.g. 3, 5, 9, etc.), such as the operation of the LMB mentioned above. The spatial block uses graph convolution operations to extract features from spatial relationships, using the adjacency matrix A and the feature Figure X , perform graph convolution to update the features of each node. The formula of spatial graph convolution is shown in formula (10):

[0120]

[0121] in, is the normalized adjacency matrix, and D is the degree matrix. Meanwhile, in the present invention, three types of residual links are described in ResGCN, namely, block residual, module residual, and dense residual.

[0122] Furthermore, the hand skeleton graph is represented as G = (V, E), where V = {v1, ..., v n} is a set of N joint nodes, E is the adjacency matrix A∈R N×N The set of edges of the captured skeleton. The hand pose graph is a feature set of graph sequence x = {x t，n ∈R C |t, n∈Z,1≤t≤T,1≤n≤N} is represented as the feature tensor X∈R C×T×N , where x t，n =X t，n , is the node v n The C-dimensional feature vector (such as 2D coordinates and confidence) at time t. The input skeleton graph can be described by A to describe the structure and X to describe the features. t ∈R N×C Represents a posture at time t, where N is the number of joints, and the first layer of learnable matrix in the entire network is recorded as Among them C l Indicates the number of input channels of this layer, C l+1 Indicates the number of output channels of this layer. On the skeleton input, the layer-by-layer update rule of graph convolution can be applied to the features at time t, and the update formula is as follows:

[0123]

[0124] It is a skeleton graph with self-looping, the purpose is to preserve the identity features, yes The diagonal matrix of , б(·) is the activation function, The normalization operation can be intuitively interpreted as an approximate spatial mean feature aggregation performed through messages passed by direct neighbors. The skeleton graph is passed through ResGCN to obtain the feature F k ∈R C×T×N .

[0125] Step S30: Use global maximum pooling and global average pooling on all the contour features and all the skeleton features to obtain spatial feature embedding and temporal feature embedding, input the spatial feature embedding into a multimodal spatial fusion module to obtain spatial feature representation, and input the temporal feature embedding into a multimodal temporal fusion module to obtain temporal feature representation.

[0126] Specifically, the contour feature F is obtained s and skeleton features F k Afterwards, the global maximum pooling is used on the time axis to obtain spatial feature embedding, that is, and The maximum pooling on the time axis can be expressed as formula (12):

[0127]

[0128] i represents different features, when F i is the contour feature F s When the first spatial feature is obtained When F i is the skeleton feature F k When the second spatial feature

[0129] Global average pooling is used on the spatial axis to obtain temporal feature embedding, i.e. the first spatial feature and the second spatial features Using global average pooling on the spatial axis can be expressed as formula (13):

[0130]

[0131] Get spatial feature embedding and and temporal feature embedding and Finally, MSFM and MTFM are used to fuse the spatiotemporal features of different modalities respectively, and a common attention mechanism is used in MSFM and MTFM to perform fine-grained feature extraction.

[0132] Further, the spatial feature embedding includes a first spatial feature and a second spatial feature;

[0133] The spatial feature embedding is input into the multimodal spatial fusion module to obtain the spatial feature representation, which specifically includes:

[0134] Inputting the first spatial feature into the first cross attention of the multimodal spatial fusion module, inputting the second spatial feature into the second cross attention of the multimodal spatial fusion module, and connecting the corresponding outputs to obtain a multimodal spatial fusion connection;

[0135] The multimodal spatial fusion connection is processed and fused through the Norm and FNN layers to obtain the spatial feature representation.

[0136] Spatial co-attention aggregation uses a co-attention fusion module to establish various spatial relationships between contour parts and skeleton joints to exploit the complementary advantages of the two data modalities. Figure 13 As shown, the spatial features are embedded into the input cross attention structure for complementary information fusion, and finally the feedforward network (FFN) layer generates the spatial feature representation Y s ∈R C×S Specifically, embedding spatial features into and Input the cross attention separately, in the left and right different cross attention and Denote Q, K and V respectively, MSFM fuses each contour part and each skeleton node in a fine-grained manner and obtains the spatial feature representation Y s .

[0137] The time feature embedding includes a first time feature and a second time feature;

[0138] The embedding of the time feature into a multimodal time fusion module to obtain a time feature representation specifically includes:

[0139] Inputting the first time feature into a first cross attention in the multimodal time fusion module, and inputting the second time feature into a second cross attention in the multimodal time fusion module;

[0140] The outputs of the first cross attention and the second cross attention are processed through the Norm and FNN layers respectively, and connected to obtain the temporal feature representation.

[0141] Specifically, the present invention uses MTFM to aggregate the temporal features of the two modalities and uses period position embedding to model and align the temporal information of the two modal sequences. Similar to MSFM, the temporal features are embedded into and They are input into the cross attention respectively. The difference is that the temporal features of the two modalities are very different. Therefore, the MTFM module is Figure 14 As shown in Figure 2, two independent FFN layers are used to map the unique temporal features of the two modalities. MTFM models the temporal relationship and fuses the long-term feature information of each modality for temporal feature representation Y t .

[0142] Step S40: Obtain a classification result based on the spatial feature representation and the temporal feature representation, and perform identity recognition based on the classification result.

[0143] Specifically, Y s and Yt The concat splicing is performed as the final representation Y of the hand posture sequence. The present invention uses multiple separate fully connected layers to further aggregate the information from the Y channel to obtain Y out For the final classification. The feature map can be defined as shown in formula (14):

[0144] Y out =F sfc (Y)∈R C×(T+S) ; (14)

[0145] After obtaining the classification results, identity recognition is performed accordingly.

[0146] Furthermore, in the present invention, for the overall network structure, a combined loss function is used during training, wherein the combined loss function includes triple loss and cross entropy loss. Triple loss can improve the inter-class distance and reduce the intra-class distance, which helps the cross entropy loss to identify human IDs. During the training phase, Y out Input into the combined loss function and calculate the loss independently. Combined loss function L combined It can be defined as shown in formula (15):

[0147] L combined =L tri +L csl ; (15)

[0148] Among them, L tri and L csl They represent triplet loss and cross entropy loss respectively. L tri It can be defined as shown in formula (16):

[0149] L tri =[D(F(i),F(k))-D(F(i),F(j))+m] + ; (16)

[0150] where i and j are samples from the same class A, and k represents samples from class B. F(·) represents the feature extraction and mapping operations of the proposed method. D(d1,d2) is the Euclidean distance between d1 and d2. m is the margin of the triplet loss. Operation [·] + Equal to max(γ,0).

[0151] And L csl It can be defined as shown in formula (17):

[0152]

[0153] where y ij represents the true prediction of the jth class of the i-th sample, p ijDenotes the prediction of the jth class for the i-th sample, and N is the number of samples. Since the present invention adopts the Batch All strategy, P identities are randomly selected, and for each identity, K samples are randomly selected to form a batch, and each batch contains P×K samples.

[0154] As for training data, in one embodiment of the present invention, a USB high-definition camera (default resolution 1280x720) was used to collect five groups of video sequences of 124 left and right hands, each containing 11 gesture sequences from different angles (0°-180°, with a sampling interval of 18°). Therefore, 12×10 (groups)×11 (viewing angles) = 13,640 hand gesture sequences were collected. Each hand gesture sequence was divided into a training set and a test set. During the training phase, three training settings were configured according to different training scales: small-scale training (ST), medium-scale training (MT), and large-scale training (LT). For these three settings, 24, 62, and 74 hand gestures were selected as training sets, and the remaining 100, 62, and 50 hand gestures were tested, respectively. All hand gesture data in the training sets was used to train the model during the training phase. First, the FCN and A2J-Transformer models were trained to extract the contour and skeleton images of the video frames. Before training, the video frames were cropped to the same size and input into the two models for training. After training FCN and A2J-Transformer, the contour map and skeleton map of the video frame are obtained respectively; the entire network is trained through the trained FCN and A2J-Transformer.

[0155] Furthermore, if Figure 15 As shown, the input video enters the model to obtain the RGB image of each frame, which is then input into FCN and A2J-Transformer to obtain the contour map and skeleton map respectively, and then input into the model proposed by this invention to obtain the spatial feature embedding respectively. and and temporal feature embedding and Then it is input into MSFM and MTFM to obtain the representation Y of the hand posture sequence, and after passing through multiple separate fully connected layers, Y out Perform the final classification and then obtain the identification result.

[0156] The present invention obtains a target video, and preprocesses each video frame of the target video to obtain multiple preprocessed video frames, and obtains a contour map and a skeleton map of each preprocessed video frame according to a contour extraction network and a skeleton extraction model; encodes each contour map according to a contour encoding module to obtain multiple contour features, and encodes each skeleton map according to a skeleton encoding module to obtain multiple skeleton features; uses global maximum pooling and global average pooling for all the contour features and all the skeleton features to obtain spatial feature embedding and temporal feature embedding, inputs the spatial feature embedding into a multimodal spatial fusion module to obtain spatial feature representation, and inputs the temporal feature embedding into a multimodal temporal fusion module to obtain temporal feature representation; obtains a classification result according to the spatial feature representation and the temporal feature representation, and performs identity recognition according to the classification result. The present invention adopts two independent encoding modules including a contour encoding module and a skeleton encoding module to capture the unique discriminant information of each modality, and proposes a global-to-local motion perception module, which not only focuses on the global information of the hand, but also pays attention to the connection between local areas. It also proposes a multimodal spatial fusion module and a multimodal temporal fusion module, so that accurate hand posture classification that can be used for identity recognition can be obtained, and then identity recognition can be performed.

[0157] Furthermore, if Figure 16 As shown, based on the above-mentioned hand gesture-based identity recognition method, the present invention also provides a hand gesture-based identity recognition system, wherein the hand gesture-based identity recognition system includes:

[0158] An extraction module 161 is used to obtain a target video, preprocess each video frame of the target video to obtain multiple preprocessed video frames, and obtain a contour map and a skeleton map of each preprocessed video frame based on a contour extraction network and a skeleton extraction model;

[0159] an encoding module 162 for encoding each contour image according to the contour encoding module to obtain a plurality of contour features, and encoding each skeleton image according to the skeleton encoding module to obtain a plurality of skeleton features;

[0160] a spatiotemporal fusion module 163 for applying global maximum pooling and global average pooling to all the contour features and all the skeleton features to obtain spatial feature embedding and temporal feature embedding, inputting the spatial feature embedding into a multimodal spatial fusion module to obtain spatial feature representation, and inputting the temporal feature embedding into a multimodal temporal fusion module to obtain temporal feature representation;

[0161] The result output module 164 is configured to obtain a classification result based on the spatial feature representation and the temporal feature representation, and perform identity recognition based on the classification result.

[0162] Furthermore, if Figure 17 As shown, based on the above-mentioned hand gesture-based identity recognition method and system, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 17 Only some of the components of the terminal are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0163] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a hand gesture-based identity recognition program 40 is stored on the memory 20, and the hand gesture-based identity recognition program 40 can be executed by the processor 10, thereby realizing the hand gesture-based identity recognition method of the present invention.

[0164] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program codes or process data stored in the memory 20, such as executing the hand gesture-based identity recognition method.

[0165] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0166] In one embodiment, when the processor 10 executes the hand gesture-based identity recognition program 40 in the memory 20 , the steps of the above hand gesture-based identity recognition method are implemented.

[0167] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a hand gesture-based identity recognition program, and when the hand gesture-based identity recognition program is executed by a processor, the steps of the hand gesture-based identity recognition method described above are implemented.

[0168] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal comprising the element.

[0169] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When the program is executed, it can include the processes in the above-described method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0170] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for identifying an individual based on hand gestures, characterized in that: The hand gesture-based identity recognition method comprises: Acquire a target video, and preprocess each video frame of the target video to obtain a plurality of preprocessed video frames, and obtain a contour map and a skeleton map of each preprocessed video frame according to a contour extraction network and a skeleton extraction model; Encode each contour image according to the contour encoding module to obtain multiple contour features, and encode each skeleton image according to the skeleton encoding module to obtain multiple skeleton features; Using global maximum pooling and global average pooling on all the contour features and all the skeleton features to obtain spatial feature embedding and temporal feature embedding, inputting the spatial feature embedding into a multimodal spatial fusion module to obtain spatial feature representation, and inputting the temporal feature embedding into a multimodal temporal fusion module to obtain temporal feature representation; The global maximum pooling and global average pooling are used for all the contour features and all the skeleton features to obtain spatial feature embedding and temporal feature embedding, specifically including: Applying global maximum pooling to the contour features and the skeleton features on the time axis to obtain spatial feature embedding; Applying global average pooling to the contour features and the skeleton features on the spatial axis respectively to obtain temporal feature embedding; The spatial feature embedding includes a first spatial feature and a second spatial feature; The embedding of the spatial features into a multimodal spatial fusion module to obtain a spatial feature representation specifically includes: Inputting the first spatial feature into the first cross attention of the multimodal spatial fusion module, inputting the second spatial feature into the second cross attention of the multimodal spatial fusion module, and connecting the corresponding outputs to obtain a multimodal spatial fusion connection; Processing and fusing the multimodal spatial fusion connection through the Norm and FNN layers to obtain the spatial feature representation; The time feature embedding includes a first time feature and a second time feature; The embedding of the time feature into a multimodal time fusion module to obtain a time feature representation specifically includes: Inputting the first time feature into a first cross attention in the multimodal time fusion module, and inputting the second time feature into a second cross attention in the multimodal time fusion module; The outputs of the first cross attention and the second cross attention are processed by the Norm and FNN layers respectively, and connected to obtain the temporal feature representation; A classification result is obtained according to the spatial feature representation and the temporal feature representation, and identity recognition is performed according to the classification result.

2. The identity recognition method based on hand gesture according to claim 1, characterized in that: The step of obtaining a target video, preprocessing each video frame of the target video to obtain a plurality of preprocessed video frames, and obtaining a contour map and a skeleton map of each preprocessed video frame according to a contour extraction network and a skeleton extraction model specifically includes: Obtain a target video, obtain corresponding video frames according to the target video, crop each of the video frames, and adjust the resolution to obtain multiple pre-processed video frames; Inputting all the pre-processed video frames into a contour extraction network to extract features and perform classification to obtain a contour map of each video frame; All the pre-processed video frames are input into a skeleton extraction model to extract features, and a skeleton graph of each video frame is obtained.

3. The identity recognition method based on hand gesture according to claim 2, characterized in that: The skeleton extraction model includes a pyramid feature extractor, a feature enhancement module, an anchor interaction module and an anchor weight offset estimation module; The step of inputting all the pre-processed video frames into a skeleton extraction model to extract features and obtain a skeleton graph of each video frame specifically includes: Inputting all the pre-processed video frames into the pyramid feature extractor in sequence to extract corresponding pyramid features; Inputting the pyramid features into a feature enhancement module, wherein the feature enhancement module enhances the pyramid features through deformable multi-head attention to obtain enhanced features, wherein the feature enhancement module includes 6 encoder layers, and each encoder layer includes one deformable multi-head attention; Inputting the enhanced features into the anchor interaction module, the anchor interaction module processes the enhanced features through 6 decoder layers to obtain decoded output; The decoded output is input to the anchor weight offset estimation module, which estimates the 3D offset and weight of the hand joint corresponding to each anchor point in the decoded output separately, and performs weighted fusion to obtain a skeleton graph corresponding to each preprocessed video frame.

4. The method for identifying an individual based on hand gestures according to claim 1, wherein: The contour encoding module includes a channel attention module, a local time fusion module and a global to local motion perception module; Each contour image is encoded according to the contour encoding module to obtain multiple contour features, including: Inputting the contour map into the channel attention for compression to obtain a 4-dimensional tensor; Input the 4-dimensional tensor into the local time fusion module to perform convolution operation to obtain convolution features; The convolutional features are input into the global-to-local motion perception module to obtain corresponding contour features.

5. The method for identifying an individual based on hand gestures according to claim 3, wherein: The skeleton encoding module encodes each skeleton image to obtain multiple skeleton features, specifically including: Inputting the skeleton image into a skeleton encoding module to obtain a plurality of skeleton features; The skeleton coding module is composed of multiple Bottlenecks, each Bottleneck includes a spatial block, a temporal block and a residual link.

6. A hand gesture-based identity recognition system, wherein the hand gesture-based identity recognition system is applied to the hand gesture-based identity recognition method according to any one of claims 1 to 5, characterized in that: The hand gesture-based identity recognition system includes: An extraction module is used to obtain a target video, preprocess each video frame of the target video to obtain a plurality of preprocessed video frames, and obtain a contour map and a skeleton map of each preprocessed video frame according to a contour extraction network and a skeleton extraction model; an encoding module, configured to encode each contour image according to the contour encoding module to obtain a plurality of contour features, and to encode each skeleton image according to the skeleton encoding module to obtain a plurality of skeleton features; A spatiotemporal fusion module, configured to perform global maximum pooling and global average pooling on all the contour features and all the skeleton features to obtain spatial feature embedding and temporal feature embedding, input the spatial feature embedding into a multimodal spatial fusion module to obtain spatial feature representation, and input the temporal feature embedding into a multimodal temporal fusion module to obtain temporal feature representation; A result output module is used to obtain a classification result based on the spatial feature representation and the temporal feature representation, and perform identity recognition based on the classification result.

7. A terminal, characterized in that: The terminal includes: a memory, a processor, and a hand gesture-based identity recognition program stored in the memory and runnable on the processor. When the hand gesture-based identity recognition program is executed by the processor, the steps of the hand gesture-based identity recognition method as described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a hand gesture-based identity recognition program, which, when executed by a processor, implements the steps of the hand gesture-based identity recognition method according to any one of claims 1 to 5.

Images