Point cloud gesture recognition method based on extended Point LSTM model

Abstract

The invention relates to the technical field of computer vision, in particular to a point cloud gesture recognition method based on an extended Point LSTM (Long Short Term Memory) model, which takes the Point LSTM model as a reference, combines state information of past adjacent points with current features, updates a current state through a weight sharing LSTM layer, introduces an xlstm model and strengthens information propagation. Meanwhile, a stacked network is designed based on two training methods, and a hierarchical feature fusion method is designed for the characteristics of the network and is used for learning different network depth information in a point cloud sequence recognition network. Experimental verification is carried out through several widely used data sets, information transmission between frames of the point cloud sequence can be more effectively carried out so as to improve the information acquisition amount of the time dimension of the point cloud sequence, meanwhile, the understanding ability of the model to the point cloud sequence is improved, and the recognition ability of actions such as gestures is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a point cloud gesture recognition method based on an extended PointLSTM model. Background Art

[0002] Action recognition is an important research direction in computer vision. It aims to analyze, identify, and predict actions by analyzing video sequences or sequences in various representations, such as RGB images, infrared images, skeleton points, depth maps, and point clouds. It has been widely used in various application areas, such as video understanding and human-computer interaction. As a foundational work for these tasks, human action recognition and prediction hold considerable research potential. Many applications, such as autonomous driving and robotic manipulation, require systems to understand their 3D environment. With the rapid development of high-precision sensors such as LiDAR and Kinect, point clouds have become the primary data format for representing the 3D world. Among various 3D geometric representations, point clouds are the closest to the raw sensory data from LiDAR or RGB-D cameras. Compared to RGB data, point clouds more accurately describe the underlying geometric structure and distance information of object surfaces, providing complementary clues for gesture recognition.

[0003] How to learn the rich spatial information in point clouds remains a major challenge. In the past few years, static point cloud data analysis methods have become increasingly mature, such as point cloud object classification in static environments, environment reconstruction, point cloud registration, etc. Instead of representing point clouds as voxels or multi-view formats, Qi et al. proposed the PointNet architecture to extract structural information directly from the original point cloud, which is a pioneering work. PointNet++ extends PointNet by applying hierarchical grouping and sampling operations to capture local structural information. Some recent works have modified the grouping operation to extract motion and structural features from spatiotemporal neighbors. However, these methods only focus on short-term modeling and are not enough to capture long-term relationships.

[0004] The success of recurrent neural networks (RNNs) and long short-term memory (LSTMs) in sequence models has important implications for point cloud video processing. Point cloud data is disordered; the points in the data have no specific order, and each point exists independently. This characteristic requires algorithms that process point cloud data to be able to adapt to disordered data. Unlike skeletal points, which maintain the stability of the relative position and motion relationships between body parts, it is difficult to find corresponding points between adjacent frames in a point cloud sequence. More mature point cloud registration methods are more targeted at static environmental point clouds. Aligning point clouds across multiple frames is highly challenging, as points between different frames lack strong correlation and are expensive to align. The aligned data lacks temporal continuity, so features are still acquired from neighboring points in the past.

[0005] The existing PointLSTM method offers a new approach to point cloud action recognition. It can leverage long-term spatiotemporal relationships in irregular sequence data while preserving the spatial structure of the irregular sequence recognition problem, effectively capturing human motion information. However, because it uses the original LSTM as the update principle, it cannot achieve good recognition results. Summary of the Invention

[0006] The purpose of this invention is to provide a point cloud gesture recognition method based on an extended PointLSTM model, improve the PointLSTM model and design a stacked network to enhance feature extraction capabilities and perform efficient action recognition.

[0007] To achieve the above objectives, the present invention provides a point cloud gesture recognition method based on an extended PointLSTM model, comprising the following steps:

[0008] Step 1: Preprocess the original point cloud data, specify the number of input time frames and the number of points per frame to obtain the input point cloud data;

[0009] Step 2: Input the point cloud data into PointNet++ and PointLSTM in turn, and extract features from the neighboring points of the previous frame and the next frame respectively;

[0010] Step 3: The features extracted by PointNet++ and PointLSTM are fused through HFFN to obtain fused features;

[0011] Step 4: Feed the fused features and the features extracted by the backbone network into the classifier to obtain the action classification.

[0012] Optionally, the PointLSTM model in step 2 converts each point Assumed to have independent hidden states and cell status Update the state of the current point by searching for neighboring points in the past frame, that is, each point has an independent state, and the state is updated based on the current input and state of the neighborhood in the previous frame;

[0013] Among them, point is a T-frame point cloud sequence (P (1) ,P (2) ,...,P (T) ) Each frame contains n t Point p.

[0014] Optionally, when PointLSTM changes its update rule in step 2, the exponential gate, normalization gate, and stabilization gate of the sLSTM model are introduced, and combined with the mLSTM model, the original LSTM gating technology is broadcast to the new architecture, using the additional state m t Stable door.

[0015] Optionally, the feature fusion process through HFFN in step 3 is to interleave the updated PointLSTM model and the PointNet model repeatedly, and the two layers form a group, which is stacked three times. After that, a PointNet++ layer is connected to aggregate the features of the previous, next and current frames. Only the first layer of PointNet++ aggregates the neighboring point information within the current frame, and the others are used to obtain adjacent frames and the adjacent points of the current frame.

[0016] The present invention provides a point cloud gesture recognition method based on an extended PointLSTM model. Using the PointLSTM model as a benchmark, the method combines the state information of past adjacent points with current features, updates the current state through a weight-sharing LSTM layer, and introduces the xLSTM model to enhance the information propagation method. A stacked network is designed based on two training methods, and a hierarchical feature fusion method is designed based on the characteristics of the network to learn different network depth information in the point cloud sequence recognition network. Experimental verification using several widely used datasets demonstrates that the present invention can more effectively transfer information between frames of a point cloud sequence, thereby increasing the amount of information acquired in the time dimension of the point cloud sequence. This also improves the model's ability to understand the point cloud sequence and enhances the ability to recognize actions such as gestures. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is a schematic diagram of the principle of a point cloud gesture recognition method based on an extended PointLSTM model of the present invention.

[0019] Figure 2 This is a schematic diagram of the PointLSTM update rule flow of a point cloud gesture recognition method based on an extended PointLSTM model of the present invention.

[0020] Figure 3This is a schematic diagram of the hierarchical feature fusion network module structure in a point cloud gesture recognition method based on an extended PointLSTM model of the present invention.

[0021] Figure 4 Schematic diagram of the confusion matrix of the NvGesture dataset according to a specific embodiment of the present invention.

[0022] Figure 5 FIG. 4 is a schematic diagram of a confusion matrix of the SHREC'17 dataset according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0024] The present invention provides a point cloud gesture recognition method based on an extended PointLSTM model, comprising the following steps:

[0025] Step 1: Preprocess the original point cloud data, specify the number of input time frames and the number of points per frame to obtain the input point cloud data;

[0026] Step 2: Input the point cloud data into PointNet++ and PointLSTM processing in sequence, and extract features from the neighboring points of the previous frame and the next frame respectively;

[0027] Step 3: The features extracted by PointNet++ and PointLSTM are fused through HFFN to obtain fused features;

[0028] Step 4: Feed the fused features and the features extracted by the backbone network into the classifier to obtain the action classification.

[0029] For the specific implementation process, please refer to Figure 1 The acquired point cloud data is processed by PointNet and PointLSTM in turn. The output of each pair is input into HFFN to obtain different hierarchical features. This is repeated three times. The output of each time and the output of the last layer of maximum pooling are input into the classification head together to obtain the action classification result.

[0030] HFFN Figure 3 The designed module shown is the abbreviation of Hierarchical Feature Fusion Network, which is a hierarchical feature fusion module used to obtain and fuse features at different levels.

[0031] The following is a further explanation of the relevant terms:

[0032] 1. PointLSTM

[0033] A complete point cloud sequence has T frames, which can be expressed as (P (1) ,P (2) ,...,P (T) ), each frame contains n t A point p can be expressed as Each point It consists of two parts: coordinate vector and eigenvector m-dimf i (t) Due to occlusion and other reasons, it is difficult for each point in the point cloud sequence to find the corresponding point in other frames like other data types, so the information is obtained from the neighboring points in the adjacent frames. Indicates a point In P (t +Δt) The adjacent points in .

[0034] PointLSTM uses LSTM as the basic update rule, namely:

[0035] i (t) =σ(U (i) y (t) +W (i) h (t-1) +b (i) ),

[0036] f (t) =σ(U (f) y (t) +W (f) h (t-1) +b (f) ),

[0037] o (t) =σ(U (o) y (t) +W (o) h (t-1) +b (o) ),

[0038]

[0039] h (t) =o (t) ⊙c (t) ,

[0040] Can be abbreviated as:

[0041]

[0042] PointLSTM transforms each point Assumed to have independent hidden states and cell status Update the state of the current point by searching for neighboring points in the past frame. Figure 2 As shown, the point status information of the previous frame can be transferred to the next frame. in It expresses the update mechanism as:

[0043]

[0044] Among them, [.;.] represents the connection operation, It's right Virtual hidden and cell states, the final state is updated by:

[0045]

[0046] in Corresponding to point To update the hidden units and unit states, g is a symmetric function implemented as a maximum pooling layer.

[0047] 2. xLSTM

[0048] PointLSTM uses the original LSTM as the basic update rule, which cannot meet the higher information acquisition requirements. It is necessary to introduce more efficient update rules. The following are two new update rules provided by xLSTM:

[0049] sLSTM

[0050] To enable LSTM to modify its memory decisions, exponential gates as well as normalization and stabilization gates are introduced. In particular, the input gate and forget gate can have exponential activation functions. For normalization, a normalizer state is introduced that adds the product of the input gate multiplied by all future forget gates. The forward pass of the sLSTM is:

[0051] C t =f t C t-1 +i t z t

[0052] n t =f t n t-1 +i t

[0053]

[0054] mLSTM

[0055] In order to improve the storage capacity of LSTM, the storage unit of LSTM is increased from scalar C∈R to matrix C∈R d×d At time t, a pair of stored vectors is described using transformer terms: key k t ∈R d , value v t ∈R d , at the next time t+τ, by the query vector q t+τ Retrieve value v t . For this matrix memory, the normalizer state is a weighted sum of key vectors, where each key vector is weighted by the input gate and all future forget gates. Similarly, the normalizer state keeps 4 records of the gate strengths. The dot product between the query and the normalizer state is close to zero, and the absolute value of this dot product is used and floored by a threshold (usually 1.0). The mLSTM forward pass is:

[0056] C t =f t C t-1 +i t v t k t

[0057] n t =f t n t-1 +i t k t

[0058]

[0059] q t =W q x t +b q

[0060]

[0061] v t =W v x t +b v

[0062]

[0063] The original LSTM gating technique (i.e. input and / or hidden dependency gates plus the bias term) is broadcasted to the new architecture. Exponential activation functions can lead to large values ​​that can cause overflow. Therefore, an additional state mt is used to stabilize the gate:

[0064] m t =max(log(f t )+m t-1 ,log(it ))

[0065]

[0066] f t ′=exp(log(f t )+m t-1 -m t )

[0067] In the forward pass, use f t ′ replace f t , and use i t ′ replace i t Neither the output of the entire network nor the derivative of the loss with respect to the parameters is changed.

[0068] In this paper, both update rules are used, and experiments are conducted based on the NvGesture dataset to study the effectiveness of the two update rules. The results are shown in Table 4, namely, mLSTM and sLSTM in Table 4. √ is used in the table to represent which update rule is used.

[0069] 3. Hierarchical feature fusion

[0070] The present invention designs a hierarchical feature fusion module, such as Figure 3 As shown, and as Figure 1 A new network is designed as shown.

[0071] In the deep network, the point cloud data before passing the specified layer is recorded as D past , the point cloud data after the specified layer calculation is recorded as D after The point cloud data before and after processing are subjected to maximum pooling and average pooling according to the point number dimension, and the obtained and Afterwards and and and The splicing operations are performed along the feature dimensions as follows:

[0072]

[0073] d1′, d3′=conv(d1), conv(d3)

[0074] d2′=conv(Rule(conv(d2)))

[0075] Among them, d1′ and d3′ are reduced in dimension by d1 and d2 respectively through 1×1 shared weight feature channel convolution, and d2′ is reduced in dimension by d2 through 1×1 feature channel convolution, and then an additional Rule and convolution operation is performed to obtain the subsequent output.

[0076] W=sigmoid(conv(d1′+d2′+d3′))

[0077] In the calculation of hierarchical feature weights, the transformed d1′, d2′, and d3′ are superimposed, and then the feature weights are calculated using the sigmoid function after channel convolution conversion.

[0078]

[0079] The calculated weights W and 1-W are respectively compared with and Multiply and add the original and Get the final output, splice it to the final classification head, and calculate the final classification result.

[0080] During the network construction process, the updated PointLSTM and PointNet are used as the main parts of the network, and the two networks are placed alternately, such as Figure 3 As shown in the module designed in [1], two layers are stacked three times to form a group, followed by a PointNet++ layer that aggregates features from the previous, next, and current frames. Only the first PointNet++ layer aggregates information about neighboring points within the current frame; the remaining layers are used to obtain information about adjacent frames and the current frame's immediate neighbors.

[0081] Finally, the main network output part and the hierarchical feature fusion part are spliced ​​together to perform the classification task.

[0082] In the early stage of network design, the feature dimensions of each stage are designed to be consistent so that features at different levels can be more conveniently integrated in subsequent operations.

[0083] For further information, see Figure 4 and Figure 5 This paper demonstrates the effectiveness of the xLSTM used in the update network by performing recognition on two challenging dynamic gesture datasets, NVGesture and SHREC'17. It also experimentally explores the effectiveness of the proposed xLSTM module and the constructed hierarchical feature fusion update network. Finally, an ablation experiment is performed to demonstrate the effectiveness of the proposed method.

[0084] 1) Dataset information

[0085] NVGesture. The NVIDIA Dynamic Gesture Dataset is a challenging dataset for vehicle human-machine interfaces. This dataset provides multiple modalities, including RGB, depth, and IR images. A total of 1,532 videos across 25 categories were divided into 1,050 training videos and 482 test videos by the subjects.

[0086] SHREC'17. The SHREC'17Track dataset is a public dynamic gesture dataset for ShREC'17Track. Gestures in SHREC'17 are defined by the gesture action or the shape of the hand, corresponding to coarse gestures and fine gestures. The dataset contains 2800 videos of 14 gesture classes, each gesture is performed in two ways: using one finger or the whole hand. It is divided into 1960 training sequences (70%) and 840 test sequences (30%). The dataset also provides the coordinates of 22 hand joints in 3D world space, which are widely used in skeleton-based gesture recognition.

[0087] 2) Verification experiment

[0088] Recurrent Neural Networks. This example first tests the effectiveness of the newly proposed xLSTM update rule. PointLSTM uses the original LSTM update rule. Since the introduction of LSTM, many researchers have conducted experiments in this area and proposed many effective long-short-term memory networks, which also provides a lot of experimental space for subsequent researchers.

[0089] This example uses xLSTM, the latest update strategy proposed by the creators of LSTM. This update strategy provides an additional numerical stabilization strategy, and special experiments have been conducted on this strategy. The subsequent method of using xLSTM as the update rule for long-short-term memory networks is called extended PointLSTM.

[0090] Table 1 Performance comparison after updating PointLSTM rules on NvGesture dataset

[0091]

[0092] In this phase of the experiment, we used PointLSTM-middle as the baseline. We then replaced its update rule with sLSTM and conducted experiments on its stabilization layer. As shown in Table 1, the proposed sLSTM surpassed the performance of the original LSTM, and its numerical stability also demonstrated its effectiveness. Subsequent ablation experiments demonstrated that sLSTM outperformed mLSTM, so detailed experimental data for mLSTM is presented here.

[0093] Table 2 Comparison of action recognition performance of different methods on the NvGesture dataset

[0094]

[0095] Table 3 Comparison of gesture recognition performance of different methods on the SHREC'17 dataset

[0096]

[0097] Comparison with other methods. This example compares the proposed method with several state-of-the-art methods on two datasets. The results are shown in Tables 2 and 3. Table 2 shows that the proposed method achieves a respectable performance of 89.4% on NVGesture, only 0.2% behind the state-of-the-art method, M2PSC, and achieves the highest accuracy in supervised learning.

[0098] The device used in this embodiment is an NVIDIA A5000 24G graphics card.

[0099] 3) Ablation experiment

[0100] This example also conducted an ablation study of different pretext tasks on the NVGesture dataset. Table 4 lists the results for different pretext tasks. The effectiveness of different pretext tasks varies. The hierarchical feature fusion module and the xLSTM update strategy outperform other tasks and can help point clouds better understand point cloud sequences. To better guide model learning, this example conducted three pretext tasks, including stacked networks, hierarchical feature fusion, and the xLSTM update strategy. The xLSTM includes two update strategies, each of which was tested to explore their impact and differences.

[0101] Table 4. Ablation study of different pretext tasks

[0102]

[0103]

[0104] The original PointLSTM network is more suitable for single placement in a network, and exhibits poor performance when placed in multiple locations. However, the updated PointLSTM is better adapted to large-scale networks. The proposed stacked network and hierarchical feature fusion are more suitable for working with networks that carry more information. Compared to mLSTM and LSTM, sLSTM has greater information carrying capacity and can more accurately transmit inter-frame information.

[0105] In summary, the update method of the present invention can more effectively transfer information between frames in a point cloud sequence, thereby increasing the amount of information acquired in the temporal dimension of the point cloud sequence. The hierarchical fusion component can effectively fuse information from different layers and obtain training information at all depths of the network. Experimental validation using several widely used datasets also demonstrates that the method of the present invention can significantly improve the model's understanding of point cloud sequences.

[0106] The above disclosure is merely one or more preferred embodiments of the present invention, and certainly cannot be used to limit the scope of the present invention. A person skilled in the art can understand that all or part of the processes of the above embodiments and equivalent changes made in accordance with the claims of the present invention still fall within the scope of the invention.

Claims

1. A point cloud gesture recognition method based on an extended PointLSTM model, characterized in that: The following steps are involved: Step 1: Preprocess the original point cloud data, specify the number of input time frames and the number of points per frame to obtain the input point cloud data; Step 2: Input the point cloud data into PointNet++ and PointLSTM processing in sequence, and extract features from the neighboring points of the previous frame and the next frame respectively; Step 3: The features extracted by PointNet++ and PointLSTM are fused through HFFN to obtain fused features; Step 4: Feed the fused features and the features extracted by the backbone network into the classifier to obtain the action classification.

2. The point cloud gesture recognition method based on the extended PointLSTM model according to claim 1, characterized in that: The PointLSTM in step 2 transforms each point Assumed to have independent hidden states and cell status Update the state of the current point by searching for neighboring points in the past frame, that is, each point has an independent state, and the state is updated based on the current input and state of the neighborhood in the previous frame; Among them, point is a T-frame point cloud sequence (P (1) ,P (2) ,...,P (T) ) Each frame contains n t Point p.

3. The point cloud gesture recognition method based on the extended PointLSTM model according to claim 2, characterized in that: When PointLSTM changes the update rule in step 2, the exponential gate, normalization gate and stabilization gate of the sLSTM model are introduced, and combined with the mLSTM model, the original LSTM gating technology is broadcast to the new architecture, using the additional state m t Stable door.

4. The point cloud gesture recognition method based on the extended PointLSTM model according to claim 3, characterized in that: In step 3, the feature fusion process using HFFN is to alternately place the updated PointLSTM model and the PointNet model. The two layers form a group and are stacked three times. This is followed by a PointNet++ layer that aggregates the features of the previous, next, and current frames. Only the first layer of PointNet++ aggregates the neighboring point information within the current frame, while the others are used to obtain adjacent frames and the adjacent points of the current frame.

Images