Point Cloud Sequence Generation Method Based on Transformer and Self-Attention

Abstract

The present invention discloses a method for generating point cloud sequences based on Transformer and self-attention, comprising: 1) Selecting a point cloud sequence, a point cloud shape label representing this sequence, and an action label from a dataset and inputting them into PointEncode to obtain a point cloud sequence in the feature space; 2) Using TransformerEncoder to obtain the hidden vectors of the point cloud sequence; 3) Inputting the hidden vectors into TransformerDecoder to obtain the reconstructed point cloud sequence in the feature space, and then using PointDecoder to restore it to a point cloud sequence for training; 4) Sampling hidden vectors from a standard Gaussian distribution and successively using the trained TransformerDecoder and PointDecoder to generate a point cloud sequence from the hidden vectors. The present invention can be trained with limited optical motion capture data, generate more optically reasonable and realistic motion capture data, and achieve better results than previous methods.

Description

Technical Field

[0001] The present invention relates to the technical field of sequential data generation, and in particular to a method for generating point cloud sequences based on Transformer and self-attention. Background Art

[0002] Optical motion capture is a technology widely used in the game and film industries. It uses multiple infrared sensors with different perspectives and calibrated to track the reflective or luminous points arranged on the surface of a person to obtain point cloud sequence data representing the person's movement, and finally resolves to obtain the person's skeleton. Since optical motion capture devices are expensive and the process of motion capture requires a large amount of labor and time, human motion generation has been widely studied in recent years, generating human motions that meet certain conditions through certain conditional constraints such as motion labels, text, or music. These methods all use the human skeleton as the representation of the pose, and use a series of skeletons to represent a person's motion. Therefore, these methods cannot directly process the three-dimensional point cloud sequence on the surface of a person obtained through optical motion capture. The point cloud or mesh model itself is the representation of the pose, and the skeleton can be regarded as an agent for the surface mesh or point cloud of the model.

[0003] Obtaining point cloud sequence data requires a large amount of manpower, material resources, and time. The current methods for generating point clouds are all used for the generation task of a single point cloud. The generation network of point clouds can be roughly divided into methods based on GANs and probability-based methods. These methods are trained on static single-frame point cloud datasets to generate single-frame point clouds different from those in the dataset. They all perform generation on single-frame point clouds and there is no method for generating on point cloud sequences. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and propose a method for generating point cloud sequences based on Transformer and self-attention, which can be trained with limited optical motion capture data to generate more optically reasonable and realistic motion capture data. The point cloud sequence can be the point cloud sequence on the surface of an object or the point cloud sequence of skeleton joints. Compared with the previous human motion generation network based on the skeleton with actions as conditions, the method in this paper has achieved the best results.

[0005] To achieve the above purpose, the technical solution provided by the present invention is: A method for generating point cloud sequences based on Transformer and self-attention, including the following steps:

[0006] 1) Select a point cloud sequence, the point cloud shape label and action label representing this sequence from the dataset and input them into the point cloud encoding network PointEncoder to obtain the point cloud sequence in the feature space;

[0007] 2) Use the sequence encoding network TransformerEncoder to encode the point cloud sequence in the feature space obtained in step 1) to obtain a hidden vector as the implicit representation of the entire point cloud sequence, and use the KL divergence constraint to constrain the distribution of the hidden vector in the standard Gaussian distribution;

[0008] 3) Input the hidden vector obtained in step 2) into the sequence decoding network TransformerDecoder to obtain the reconstructed point cloud sequence in the feature space, and then use the point cloud decoding network PointDecoder to restore the reconstructed point cloud sequence in the feature space to the reconstructed point cloud sequence. Construct a loss function with the reconstructed point cloud sequence as the predicted value and the true value of the dataset for training;

[0009] 4) Sample a random vector from the standard Gaussian distribution as the hidden vector, and successively use the trained sequence decoding network TransformerDecoder and point cloud decoding network PointDecoder in step 3) to restore the sampled hidden vector to the generated point cloud sequence.

[0010] Furthermore, in step 1), the point cloud sequence refers to a series of continuous point clouds, and the entire sequence represents an action of an object within a specific time length, expressed as:

[0011]

[0012] where A refers to the point cloud sequence, p t is the t-th point cloud in the point cloud sequence A, and a sequence has a total of T point clouds. The point cloud is a set of three-dimensional points, expressed as:

[0013]

[0014] where x i is the i-th three-dimensional point in the point cloud p t , and the point cloud p t has a total of n three-dimensional points;

[0015] The shape label represents the shape of the point cloud sequence, denoted by the scalar s; the action label represents the action performed by the point cloud, denoted by the scalar m;

[0016] Input each frame of the point cloud p i of the point cloud sequence A into the point cloud encoding network PointEncoder. The point cloud encoding network PointEncoder consists of 3 multi-layer perceptron layers with shared weights and 1 max pooling layer. The point cloud encoding network PointEncoder outputs a point cloud feature vector z i , and after encoding each frame of the point cloud of the point cloud sequence A using the point cloud encoding network PointEncoder, a set formula is as follows:

[0017]

[0018] wherein, nz×n represents the dimension of z i is nz, and there are n z's in the set Z i , z n represents the nth point cloud feature vector, represents the set of real numbers;

[0019] The action label m and the point cloud shape label s are respectively converted into parameters that the network can learn and nz indicates that the dimension of these parameters is nz. Add their respective components, and the formula is as follows:

[0020]

[0021]

[0022] wherein, and represent intermediate parameters that can be learned in the network, encoding the shape and action information of the point cloud, and represent intermediate parameters that can be learned in the network, encoding the action information of the point cloud, and represent intermediate parameters that can be learned in the network, encoding the shape information of the point cloud; Concatenate with the set Z to obtain the set Z′ represents the point cloud sequence in the feature space, nz×(n + 2) indicates that the dimension of each element of the set Z′ is nz, and there are n + 2 elements in the set Z′.

[0023] Furthermore, in step 2), the sequence encoding network TransformerEncoder is the encoder part of the Transformer network. After using the sequence encoding network TransformerEncoder to encode the point cloud sequence Z′ in the feature space, the set Z″ represents the set of latent variables encoding the entire sequence relationship, μ″ represents the network latent variable obtained by the sequence encoding network TransformerEncoder encoding , and ∑” represents the network latent variable obtained by the sequence encoding network TransformerEncoder encoding , z nThe n-th vector in the set Z″ excluding μ× and ∑″, where nz×(n + 2) indicates that each element of the set Z″ has a dimension of nz and there are n + 2 elements in the set Z″; select the first two vectors μ″ and ∑″ from the set Z″, and after reparameterizing μ″ and ∑″, we get Z A represents the low-dimensional representation of the point cloud sequence A in the latent space, and nz represents the dimension of Z A is nz; use the KL divergence to constrain the distribution of Z A so that the distribution of Z A tends to satisfy the standard Gaussian distribution as much as possible.

[0024] Furthermore, in step 3), the sequence decoding network TransformerDecoder is the decoder part of the Transformer network, which converts the action label m and the point cloud shape label s into parameters that the network can learn and represent intermediate parameters that can be learned in the network and encode the action information of the point cloud represent intermediate parameters that can be learned in the network and encode the shape information of the point cloud. nz indicates that the dimension of these parameters is nz. Add their respective components, and the formula is as follows

[0025]

[0026] In the formula represents intermediate parameters that can be learned in the network and encode the shape and action information of the point cloud. Add to the latent variable Z of the point cloud sequence A so that Z A is biased into the representations of the action label m and the point cloud shape label s. Finally, input the biased Z A and the positional encoding into the TransformerDecoder to obtain the implicit representation of the point cloud sequence is a set, and nz×n indicates that each element of the set has a dimension of nz, and there are n elements in the set ; represents the n-th element in;

[0027] The point cloud decoding network PointDecoder is used to decode the implicit representation of the point cloud into the point cloud PointDecoder uses a structure similar to the self-attention network Input a 4-layer fully connected layer, and then input the obtained variables into four different fully connected layers respectively to obtain four different vectors Among them, Q i ', K i ', V i ', O i ' all represent intermediate parameters that can be learned in the network, and nd k represents the vector parameter Q i ', K i ', V i ', O i '. After expanding and arranging the vector parameters Q i ', K i ', V i ', O i ' into matrices, matrix variables are obtained Among them, Q i , K i , V i , O i are all intermediate parameters that can be learned in the network and exist in the form of matrices. n is the number of rows of the matrix, and d k represents the number of columns of the matrix. Input O i into a multi-layer perceptron with shared weights to reduce its channel dimension to 3, and then perform Attention operations on Q i , K i , V i to obtain matrix variables. The Attention operation formula is as follows:

[0028]

[0029] In the formula, softmax is a normalization function, is the obtained matrix variable, n is the number of rows of the matrix, and d k represents the number of columns of the matrix. Input Attention(Q i , K i , V i ) into a multi-layer perceptron with shared weights to reduce its d k dimension to 3; input O i into a multi-layer perceptron with shared weights to reduce its d k dimension to 3. Add these two n×3 matrices to obtain the reconstructed point cloud Each frame of point cloud is stitched together to form a reconstructed point cloud sequence Use the latent vector z A , A, as the construction of the loss function to train the network. The total loss function is as follows:

[0030]

[0031]

[0032]

[0033] In the formula, represents the total loss function, and the total loss function includes the reconstruction loss of the point cloud sequence and the KL divergence loss function L KL , the reconstruction loss of the point cloud sequence is the L2 loss of each point cloud in the point cloud sequence ; L KL is the KL divergence loss function of z A and , aiming to make the distributions of z A and as identical as possible. is an nz-dimensional vector subject to the standard Gaussian distribution, N(0, I) represents the standard Gaussian distribution, and λ KL is the weight parameter of the KL divergence loss function.

[0034] Furthermore, in step 4), sample an nz-dimensional random vector from the standard Gaussian distribution as the latent vector z A , and input the latent vector z A , the action label m, and the point cloud shape label s into the trained sequence decoding network TransformerDecoder and the point cloud decoding network PointDecoder to obtain the point cloud sequence

[0035] where the generated point cloud shape label s represents the action label m.

[0036] 1. The present invention proposes a network architecture (including the sequence encoding network TransformerEncoder, the cloud encoding network PointEncoder, the sequence decoding network TransformerDecoder, and the point cloud decoding network PointDecoder) for generating the entire point cloud sequence using the point cloud sequence as training data. By inputting the predefined action label m and the point cloud shape label s, a point cloud action sequence with the semantics of the point cloud shape label s and the action label m is generated, and the local details of the point cloud are well maintained during the movement process.

[0037] 2. The present invention directly processes the point cloud sequence, which can be the point cloud sequence of the object surface or the point cloud sequence of the skeleton joints. Compared with the previous human action generation networks conditional on actions, the method in this paper has achieved the best results.

[0038] 3. The point cloud sequence generation method based on Transformer and self-attention can restore the point cloud from a latent variable. Compared with the previous point cloud generation methods using the point cloud distance as the loss function, the method in this paper achieves better results. Description of the Drawings

[0039] Figure 1 It is a schematic flow diagram of the method of the present invention.

[0040] Figure 2 It is a schematic diagram of an example of the point cloud sequence generated by the present invention. Detailed Embodiments

[0041] The present invention will be further described in detail below in conjunction with the embodiments and the drawings, but the embodiments of the present invention are not limited thereto.

[0042] As Figure 1 and Figure 2 shown, this embodiment provides a point cloud sequence generation method based on Transformer and self-attention, which includes the following steps:

[0043] 1) Use farthest point sampling from the mesh model sequence in the dataset HumanAct12 to obtain a point cloud sequence of 1024 surface points as the training data. The point cloud sequence refers to a series of continuous point clouds, and the entire sequence represents an action of an object within a specific time length, expressed as:

[0044]

[0045] In the formula, A refers to the point cloud sequence, p t is the t-th point cloud in the point cloud sequence A, and there are T point clouds in a sequence. In the embodiment, T = 60. The point cloud is a set of three-dimensional points, expressed as:

[0046]

[0047] In the formula, x i is the i-th three-dimensional point in the point cloud p t , and there are n three-dimensional points in the point cloud p t . In the embodiment, n = 1024.

[0048] The shape label represents the shape of the point cloud sequence, represented by the scalar s. In the embodiment, s = 1 represents "a person of medium build"; the action label represents the action performed by the point cloud, represented by the scalar m. In the embodiment, m = 0 represents "drinking water";

[0049] For each frame of the point cloud p of the point cloud sequence A iIt is input into the point cloud encoding network PointEncoder, which consists of 3 multi-layer perceptron layers with shared weights and 1 max pooling layer. The point cloud encoding network PointEncoder outputs a point cloud feature vector z with nz dimensions. i , in the embodiment, nz = 256. After encoding each frame of the point cloud sequence A using the point cloud encoding network PointEncoder, a set is formed as follows:

[0050]

[0051] In the formula, nz × n represents z i The dimension of is nz, and there are n z's in the set Z. i , z n represents the nth point cloud feature vector. represents the set of real numbers;

[0052] The action label m and the point cloud shape label s are respectively converted into parameters that the network can learn and nz indicates that the dimensions of these parameters are all nz. Add their respective components, and the formula is as follows:

[0053]

[0054]

[0055] In the formula, and represent the learnable intermediate parameters in the network, encoding the shape and action information of the point cloud. and represent the learnable intermediate parameters in the network, encoding the action information of the point cloud. and represent the learnable intermediate parameters in the network, encoding the shape information of the point cloud. Concatenate and the set Z to obtain the set Z′ represents the point cloud sequence in the feature space, and nz × (n + 2) indicates that each element of the set Z′ has a dimension of nz, and there are n + 2 elements in the set Z′.

[0056] 2) The sequence encoding network TransformerEncoder is the encoder part of the Transformer network. After encoding the point cloud sequence Z′ in the feature space using the sequence encoding network TransformerEncoder, the set $Z''$ represents the set of latent variables encoding the entire sequence relationship, and $\mu''$ represents the network latent variable obtained after the sequence encoding network Transformer Encoder encodes After encoding, $\sum''$ represents the network latent variable obtained after the sequence encoding network Transformer Encoder encodes After encoding, $z n ''$ represents the $n$-th vector in the set $Z''$ excluding $\mu''$ and $\sum''$. $n_z\times(n + 2)$ indicates that each element of the set $Z''$ has a dimension of $n_z$, and there are $n + 2$ elements in the set $Z''$; Select the first two vectors $\mu''$ and $\sum''$ from the set $Z''$, and after reparameterizing $\mu''$ and $\sum''$, we get $Z A which represents the low-dimensional representation of the point cloud sequence $A$ in the latent space. $n_z$ represents that the dimension of $Z A is $n_z$; Use the KL divergence to constrain the distribution of $Z A so that the distribution of $Z A tends to satisfy the standard Gaussian distribution as much as possible.

[0057] 3) The sequence decoding network Transformer Decoder is the decoder part of the Transformer network, which converts the action label $m$ and the point cloud shape label $s$ into parameters that the network can learn and represents the learnable intermediate parameter in the network, encoding the action information of the point cloud, represents the learnable intermediate parameter in the network, encoding the shape information of the point cloud. $n_z$ indicates that the dimension of these parameters is $n_z$. Add their respective components, and the formula is as follows:

[0058]

[0059] In the formula, represents the learnable intermediate parameter in the network, encoding the shape and action information of the point cloud. Add to the latent variable $Z$ of the point cloud sequence A so that $Z A is biased towards the representations of the action label $m$ and the point cloud shape label $s$. Finally, input the biased $Z A and the positional encoding into the Transformer Decoder to obtain the implicit representation of the point cloud sequence is a set. $n_z\times n$ indicates that each element of the set has a dimension of $n_z$, and there are $n$ elements in the set ; represents the $n$-th element in;

[0060] The point cloud decoding network PointDecoder is used to decode the implicit representation of the point cloud into a point cloud PointDecoder uses a structure similar to the self-attention network. Input four fully connected layers, and then input the obtained variables into four different fully connected layers to obtain four different vectors Among them, Q i ', K i ', V i ', O i ' all represent learnable intermediate parameters in the network. nd k represents the vector parameter Q i ', K i ', V i ', O i 's dimension. After expanding and arranging the vector parameters Q i ', K i ', V i ', O i ' into matrices, matrix variables are obtained Among them, Q i , K i , V i , O i are all learnable intermediate parameters in the network and exist in the form of matrices. n is the number of rows of the matrix, and d k represents the number of columns of the matrix. Input O i into a multi-layer perceptron with shared weights to reduce its channel dimension to 3. Then perform the Attention operation on Q i , K i , V i to obtain a matrix variable. The Attention operation formula is as follows:

[0061]

[0062] In the formula, softmax is a normalization function. is the obtained matrix variable. n is the number of rows of the matrix, and d k represents the number of columns of the matrix. Input Attention(Q i , K i , V i ) into a multi-layer perceptron with shared weights to reduce its d k dimension to 3; input O i into a multi-layer perceptron with shared weights to reduce its d k dimension to 3. Add these two n×3 matrices to obtain the reconstructed point cloud Each frame of point cloud is stitched and combined into a reconstructed point cloud sequence Use the latent vector z A 、A、 as a construction loss function to train the network. The total loss function is as follows:

[0063]

[0064]

[0065]

[0066] In the formula, represents the total loss function. The total loss function includes the reconstruction loss of the cloud sequence and the KL divergence loss function L KL . The reconstruction loss of the point cloud sequence is the L2 loss of each point cloud in the point cloud sequence ; L KL is the KL divergence loss function of z A and . The purpose is to make the distribution of z A as similar as possible to the distribution of . is an nz-dimensional vector sampled from the standard Gaussian distribution. N(0, I) represents the standard Gaussian distribution, and λ KL is the weight parameter of the KL divergence loss function. In the embodiment, λ KL = 0.01. In the embodiment, Transformer uses the module of Pytorch, with 6 layers of Transformer layers, the multi-head attention set to 4 heads, the dropout set to 0.1, the dimension of the intermediate feedforward network set to 1024 dimensions, and the batch size set to 10.

[0067] 4) Sample an nz-dimensional random vector from the standard Gaussian distribution as the latent vector z A . Input the latent vector z A , the action label m, and the point cloud shape label s into the trained sequence decoding network TransformerDecoder and the point cloud decoding network PointDecoder to obtain the point cloud sequence

[0068] where the generated point cloud shape label s represents the action label mThe above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and are all included in the protection scope of the present invention.

Claims

1. A method for generating a point cloud sequence based on Transformer and self-attention, characterized in that, Including the following steps: 1) Select a point cloud sequence, the point cloud shape label and the action label representing this sequence from the dataset and input them into the point cloud encoding network PointEncoder to obtain the point cloud sequence in the feature space; The point cloud sequence refers to a series of continuous point clouds. The entire sequence represents an action of an object within a specific time length, expressed as: where A is the point cloud sequence, and p t is the t-th point cloud in the point cloud sequence A. There are a total of T point clouds in a sequence. A point cloud is a set of three-dimensional points and is represented as: where x i is the i-th three-dimensional point in the point cloud p t , and there are n three-dimensional points in the point cloud p t ; The shape label represents the shape of the point cloud sequence, denoted by the scalar s; The action label represents the action performed by the point cloud, denoted by the scalar m; For each frame of point cloud p in the point cloud sequence A i input it into the point cloud encoding network PointEncoder, which consists of 3 multi-layer perceptron layers with shared weights and 1 max pooling layer. The point cloud encoding network PointEncoder outputs a point cloud feature vector z with nz dimensions i , after encoding each frame of the point cloud sequence A using the point cloud encoding network PointEncoder, a set formula is formed as follows: where, \(n_z\times n\) represents that the dimension of \(z\) i is \(n_z\), and there are \(n\) \(z\)'s in the set \(Z\) i , and \(z\) n represents the \(n\)th point cloud feature vector, represents the set of real numbers; Convert the action label m and the point cloud shape label s into parameters that can be learned by the network respectively and nz indicates that the dimensions of these parameters are all nz. Add their respective components, and the formula is as follows: wherein, and represent intermediate parameters that can be learned in the network, encoding the shape and motion information of the point cloud, and represent intermediate parameters that can be learned in the network, encoding the motion information of the point cloud, and represent intermediate parameters that can be learned in the network, encoding the shape information of the point cloud; concatenate and the set Z to obtain the set Z′ represents a sequence of point clouds in the feature space, nz×(n + 2) indicates that each element of the set Z′ has a dimension of nz, and there are n + 2 elements in the set Z′; 2) Use the sequence encoding network TransformerEncoder to encode the point cloud sequence in the feature space obtained in step 1) to obtain a hidden vector as the implicit representation of the entire point cloud sequence, and use the KL divergence constraint to constrain the distribution of the hidden vector in the standard Gaussian distribution; 3) Input the hidden vector obtained in step 2) into the sequence decoding network TransformerDecoder to obtain the reconstructed point cloud sequence in the feature space, and then use the point cloud decoding network PointDecoder to restore the reconstructed point cloud sequence in the feature space to the reconstructed point cloud sequence. Construct a loss function with the reconstructed point cloud sequence as the predicted value and the true value of the dataset for training; 4) Sample a random vector from the standard Gaussian distribution as the hidden vector, and successively use the trained sequence decoding network TransformerDecoder and point cloud decoding network PointDecoder in step 3) to restore the sampled hidden vector to the generated point cloud sequence.

2. The method for generating a point cloud sequence based on Transformer and self-attention according to claim 1, characterized in that, In step 2), the sequence encoding network TransformerEncoder is the encoder part of the Transformer network. After encoding the point cloud sequence Z′ in the feature space using the sequence encoding network TransformerEncoder, a set is obtained Z″ represents the set of latent variables encoding the entire sequence relationship, and μ″ represents the network latent variable obtained by encoding using the sequence encoding network TransformerEncoder. ∑” represents the network latent variable obtained by encoding using the sequence encoding network TransformerEncoder. z n ” represents the nth vector in the set Z″ excluding μ″ and Σ”. nz×(n + 2) indicates that each element of the set Z″ has a dimension of nz, and there are n + 2 elements in the set Z″. Select the first two vectors μ″ and ∑” from the set Z″, and after reparameterizing μ″ and ∑”, we get Z A which represents the low-dimensional representation of the point cloud sequence A in the latent space. nz represents the dimension of Z A being nz. Use the KL divergence to constrain the distribution of Z A so that the distribution of Z A satisfies the standard Gaussian distribution.

3. The method for generating a point cloud sequence based on Transformer and self-attention according to claim 2, wherein In step 3), the sequence decoding network Transformer Decoder is the decoder part of the Transformer network, which converts the action label m and the point cloud shape label s into parameters that the network can learn respectively. and denotes the intermediate parameters that can be learned in the network and encodes the action information of the point cloud. denotes the intermediate parameters that can be learned in the network and encodes the shape information of the point cloud. nz represents that the dimension of these parameters is nz. Add their respective components, and the formula is as follows: In the formula, represents the intermediate parameters that can be learned in the network, encoding the shape and action information of the point cloud, and is added to the latent variable Z of the point cloud sequence, A biasing Z A into the representations of the action label m and the point cloud shape label s. Finally, the biased Z A and the positional encoding are input into the Transformer Decoder to obtain the implicit representation of the point cloud sequence is a set, and nz×n represents that each element of the set has a dimension of nz, and there are n elements in the set represents the nth element in The Point Cloud Decoder network is used to decode the implicit representation of the point cloud into a point cloud PointDecoder uses the structure of the self-attention network. Input 4 fully connected layers, and then input the obtained variables into four different fully connected layers respectively to obtain four different vectors Among them, Q i ′, K i ′, V i ′, O i ′ all represent intermediate parameters that can be learned in the network. nd k represents the vector parameters Q i ′, K i ′, V i ′, O i ′. After expanding and arranging the vector parameters Q i ′, K i ′, V i ′, O i ' into matrices, matrix variables are obtained Among them, Q i 、K i 、V i 、O i are all intermediate parameters that can be learned in the network and exist in the form of matrices. n is the number of rows of the matrix, and d k represents the number of columns of the matrix. Input O i into a multi-layer perceptron with shared weights to reduce its channel dimension to 3, and then perform Attention operations on Q i 、K i 、V i to obtain matrix variables. The Attention operation formula is as follows: In the formula, softmax is the normalization function, is the obtained matrix variable, n is the number of rows of the matrix, and d k represents the number of columns of the matrix. The Attention(Q i , K i , V i ) is input into the multi-layer perceptron with shared weights to reduce the dimension of d k to 3 dimensions; O i is input into the multi-layer perceptron with shared weights to reduce the dimension of d k to 3 dimensions. These two n×3 matrices are added together to obtain the reconstructed point cloud Each frame of point cloud is stitched and combined into a reconstructed point cloud sequence The latent vector z A , A, are used to construct the loss function to train the network. The total loss function is as follows: In the formula, represents the total loss function, and the total loss function includes the reconstruction loss of the point cloud sequence and the KL divergence loss function L KL . The reconstruction loss of the point cloud sequence is the L2 loss of each point cloud p in the point cloud sequence A i , . L KL is the KL divergence loss function of z A and . The purpose is to make the distribution of z A the same as the distribution of . is an nz-dimensional vector subject to the standard Gaussian distribution, N(0, I) represents the standard Gaussian distribution, and λ KL is the weight parameter of the KL divergence loss function.

4. The method for generating a point cloud sequence based on Transformer and self-attention according to claim 3, wherein In step 4), sample an nz-dimensional random vector from the standard Gaussian distribution as the latent vector z A , and input the latent vector z A , the action label m, and the point cloud shape label s into the trained sequence decoding network TransformerDecoder and the point cloud decoding network PointDecoder to obtain the point cloud sequence of the generated point cloud shape label s representing the action label m

Images