Model training method, skeleton action recognition method, device and storage medium

Abstract

The application provides a model training method based on skeleton action recognition, a skeleton action recognition method, a skeleton action recognition device and a computer readable storage medium. The model training method comprises: using a global information modeling module of a skeleton action recognition model to perform convolution operation on first local features, second local features and third local features in other feature dimensions respectively to obtain first global features corresponding to the first local features, second global features corresponding to the second local features and third global features corresponding to the third local features; fusing the first global features, the second global features and the third global features to obtain global features, and fusing the local features and the global features to obtain fused features. In the above manner, the skeleton action recognition device can comprehensively and effectively mine the global information of the skeleton data space-time domain through the novel multi-view global information modeling module, and improve the training effect and recognition effect of the skeleton action recognition model.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a model training method, a skeleton action recognition method, a skeleton action recognition device, and a computer-readable storage medium based on skeleton action recognition. Background Technology

[0002] Graph convolutions, due to their excellent topological representation capabilities on non-Euclidean structured data, have demonstrated outstanding performance in skeleton-based action recognition tasks. However, the operations of graph convolutions are limited to local neighborhoods, which restricts their ability to capture global information. This lack of global information makes graph convolution-based skeleton action recognition methods prone to confusing locally similar actions. To overcome this drawback of graph convolutions, researchers have proposed many solutions.

[0003] To capture longer-range temporal dependencies, Long Short-Term Memory (LSTM) networks have been introduced into graph convolutional networks. To simultaneously capture global information in the spatiotemporal domain of the skeleton, self-attention mechanisms have been incorporated into graph convolutions; for example, the attention matrix computed in the spatiotemporal dimension by non-local operations is used as the adjacency matrix of the graph convolution. Transformer-based methods have been introduced into the skeleton action recognition field, using multi-head self-attention to capture global information in the spatiotemporal domain of the skeleton. However, due to its step-by-step processing, LSTMs do not acquire sufficient global information. While transformer-based methods can capture global information well, they suffer from computational complexity quadratically proportional to the number of tokens, making them impractical for devices with limited computing resources. Summary of the Invention

[0004] This application provides a model training method, a skeleton action recognition method, a skeleton action recognition device, and a computer-readable storage medium based on skeleton action recognition.

[0005] This application provides a model training method based on skeleton action recognition, the model training method comprising:

[0006] Obtain the skeleton video to be trained;

[0007] The local information modeling module of the skeleton action recognition model is used to extract local features of the skeleton video to be trained;

[0008] Obtain the first local feature, the second local feature, and the third local feature corresponding to the three feature dimensions of the local feature;

[0009] The global information modeling module of the skeleton action recognition model is used to perform convolution operations of other feature dimensions on the first local feature, the second local feature and the third local feature respectively to obtain the first global feature corresponding to the first local feature, the second global feature corresponding to the second local feature and the third global feature corresponding to the third local feature;

[0010] The first global feature, the second global feature, and the third global feature are fused to obtain a global feature, and the local feature and the global feature are fused to obtain a fused feature.

[0011] The fused features are input into the classifier of the skeleton action recognition model to obtain the predicted category of the skeleton video to be trained;

[0012] The skeleton action recognition model is trained based on the predicted category and label category of the skeleton video to be trained.

[0013] Among them, the convolution operations of the other feature dimensions are position-sensitive convolution operations;

[0014] The position-sensitive convolution operation is as follows: determine the self-feature dimension of the second local feature, as well as other feature dimensions; obtain the position encoding of the skeleton joints on the self-feature dimension and the convolution operation dimension; use the position encoding on the self-feature dimension and the convolution operation dimension and the concatenated feature of the second local feature to perform convolution processing to obtain the second global feature corresponding to the second local feature.

[0015] The step of obtaining the position encoding of the skeleton joints in the dimension of their own features and the dimension of convolution operation includes:

[0016] Obtain the position index of the convolution operation dimension and the length of its own feature dimension;

[0017] Obtain the position index of the skeleton joint in its own feature dimension;

[0018] The positional encoding of the skeleton joints is obtained according to the position index of the convolution operation dimension, the length of the self-feature dimension, and the position index of the self-feature dimension.

[0019] The step of performing convolution processing using the positional encoding on the self-feature dimension and the convolution operation dimension, along with the concatenated features of the second local features, to obtain the second global feature corresponding to the second local feature includes:

[0020] Based on the feature dimension of the second local feature, the positional encoding on the feature dimension of the feature itself and the convolution operation dimension is copied to obtain a copied positional encoding with the same feature dimension length as the feature dimension length of the second local feature.

[0021] The second local feature is added to the copy position encoding to obtain the position-sensitive second local feature. The position-sensitive second local feature is then concatenated with itself according to the direction of the convolution operation dimension to obtain the concatenated local feature.

[0022] A one-dimensional convolution kernel with an orientation along the convolutional dimension and a size equal to the length of the convolutional dimension on the second local feature is used to perform convolution processing on the spliced ​​local features to obtain the second global feature corresponding to the second local feature.

[0023] The other feature dimensions include the first feature dimension and the second feature dimension of the second local feature in addition to its own feature dimension;

[0024] The step of performing convolution processing using the positional encoding on the self-feature dimension and the convolution operation dimension, along with the concatenated features of the second local features, to obtain the second global feature corresponding to the second local feature includes:

[0025] The second local sub-feature is obtained by performing convolution processing using the positional encoding of the first feature dimension and its own feature dimension, as well as the concatenated features of the second local features.

[0026] The second local output feature is obtained by performing convolution processing using the positional encoding of the second feature dimension and its own feature dimension, as well as the concatenated features of the second local sub-features.

[0027] The second local feature and the second local output feature are fused to obtain the second global feature corresponding to the second local feature.

[0028] The step of fusing the local features and the global features to obtain the fused features includes:

[0029] Using the fusion modeling module of the skeleton action recognition model, the first local features and global features of the local features are obtained;

[0030] Obtain the first fused feature output from the initial fusion of the local features and the global features;

[0031] Obtain the second fused feature output by fusing the first local feature with the first fused feature, and obtain the first global feature of the second fused feature;

[0032] Obtain the third fusion feature output by fusing the global feature and the first fusion feature, and obtain the secondary local features of the third fusion feature;

[0033] The first global feature and the second local feature are fused together to obtain the fused feature.

[0034] The step of fusing the initial global features and the secondary local features to obtain the fused features includes:

[0035] The concatenated features of the first global feature and the second local feature are subjected to a 1×1 convolution to obtain the fused feature.

[0036] The step of fusing the local features and the global features to obtain the fused features includes:

[0037] The local features and the global features are fused to obtain the fused features;

[0038] The fusion features are sequentially input into the local information modeling module, the fusion modeling module, and the local information modeling module to obtain high-level fusion features;

[0039] The step of inputting the fused features into the classifier of the skeleton action recognition model to obtain the predicted category of the skeleton video to be trained includes:

[0040] The local features, the fused features, and the high-level fused features are fused together, and the feature fusion result is input into the classifier of the skeleton action recognition model to obtain the predicted category of the skeleton video to be trained.

[0041] The step of sequentially inputting the fused features into the local information modeling module, the fusion modeling module, and the local information modeling module to obtain high-level fused features includes:

[0042] The fused features are input into the local information modeling module, so that the fused features are downsampled by the local information modeling module to obtain the first fused features;

[0043] The initial fusion features are sequentially input into the fusion modeling module and the local information modeling module to obtain the high-level fusion features.

[0044] The local information modeling module of the skeleton action recognition model extracts local features from the skeleton video to be trained, including:

[0045] The initial features of the skeleton video to be trained are extracted using the local information modeling module of the skeleton action recognition model.

[0046] Perform depth-level graph convolution on the skeleton video to be trained to obtain a skeleton feature map;

[0047] Perform a one-dimensional convolution operation along the time dimension on the skeleton feature map to obtain local features in the spatiotemporal domain;

[0048] The initial features and the spatiotemporal local features are added together to obtain the spatiotemporal fused local features;

[0049] The spatiotemporal fusion local features are input into the feedforward neural network to obtain feedforward local features;

[0050] The local features are obtained by adding the spatiotemporal domain fusion local features and the feedforward local features.

[0051] The step of obtaining the skeleton video to be trained includes:

[0052] Obtain the skeleton video;

[0053] The skeleton video is equally divided into several video segments along the time axis;

[0054] The same number of skeleton video frames are extracted from each video segment to form the skeleton video to be trained.

[0055] This application also provides a skeleton motion recognition method, the skeleton motion recognition method comprising:

[0056] Obtain the skeleton video;

[0057] The skeleton video is input into a pre-trained skeleton action recognition model, wherein the skeleton action recognition model is trained by the model training method described above.

[0058] Obtain the skeleton action category output by the skeleton action recognition model.

[0059] This application also provides a skeleton motion recognition device, which includes a processor and a memory. The memory stores program data, and the processor executes the program data to implement the model training method and / or skeleton motion recognition method as described above.

[0060] This application also provides a computer-readable storage medium for storing program data, which, when executed by a processor, is used to implement the above-described model training method and / or skeleton motion recognition method.

[0061] The beneficial effects of this application are as follows: The skeleton action recognition device acquires a skeleton video to be trained; using the local information modeling module of the skeleton action recognition model, it extracts local features of the skeleton video to be trained; it acquires a first local feature, a second local feature, and a third local feature corresponding to the three feature dimensions of the local features; using the global information modeling module of the skeleton action recognition model, it performs convolution operations on the first local feature, the second local feature, and the third local feature respectively to obtain a first global feature corresponding to the first local feature, a second global feature corresponding to the second local feature, and a third global feature corresponding to the third local feature; it fuses the first global feature, the second global feature, and the third global feature to obtain a global feature, and fuses the local features and the global feature to obtain a fused feature; it inputs the fused feature into the classifier of the skeleton action recognition model to obtain the predicted category of the skeleton video to be trained; and it trains the skeleton action recognition model based on the predicted category and label category of the skeleton video to be trained. Through the above method, the skeleton action recognition device, through a novel multi-view global information modeling module, can comprehensively, effectively, and efficiently mine the global information in the spatiotemporal domain of skeleton data, improving the training and recognition effects of the skeleton action recognition model. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0063] Figure 1 This is a flowchart illustrating an embodiment of the model training method based on skeleton action recognition provided in this application;

[0064] Figure 2 This is a schematic diagram of the overall process of the model training method based on skeleton action recognition provided in this application;

[0065] Figure 3 This is a schematic diagram of the local information modeling module provided in this application;

[0066] Figure 4 This is a schematic diagram of the feedforward neural network provided in this application;

[0067] Figure 5 This is a schematic diagram of the global information modeling module provided in this application;

[0068] Figure 6 This application provides ParC V A schematic diagram of the operation process;

[0069] Figure 7 yes Figure 1 The diagram shows the specific process flow of step S15 in the model training method based on skeleton action recognition.

[0070] Figure 8 This is a schematic diagram of the local and global information fusion module provided in this application;

[0071] Figure 9 This is a flowchart illustrating another embodiment of the model training method based on skeleton action recognition provided in this application;

[0072] Figure 10 This is a schematic diagram of the skeleton joint division provided in this application;

[0073] Figure 11 This is a flowchart illustrating an embodiment of the skeleton motion recognition method provided in this application;

[0074] Figure 12 This is a schematic diagram of the structure of an embodiment of the skeleton motion recognition device provided in this application;

[0075] Figure 13 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0076] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0077] To address the problems of existing technologies, this application transfers the ParC (position-sensitive convolution) operation from the image domain to the skeleton data domain. Based on the unique characteristics of skeleton data, it proposes a global information modeling module suitable for skeleton data. Compared with previous methods, this module can capture the global information of skeleton data in the spatiotemporal domain more effectively and efficiently. Furthermore, it explores more effective ways to fuse local salient features and global semantic features, and constructs a computationally friendly and high-performance skeleton action recognition method.

[0078] Please refer to the details. Figure 1 and Figure 2 , Figure 1 This is a flowchart illustrating an embodiment of the model training method based on skeleton action recognition provided in this application. Figure 2 This is a schematic diagram of the overall process of the model training method based on skeleton action recognition provided in this application.

[0079] The model training method of this application is applied to a skeleton motion recognition device. This skeleton motion recognition device can be a server or a system consisting of a server and a terminal device working together. Accordingly, the various parts of the skeleton motion recognition device, such as units, subunits, modules, and submodules, can all be located in the server, or they can be located separately in the server and the terminal device.

[0080] Furthermore, the aforementioned server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules, such as software or software modules used to provide distributed servers, or as a single software program or software module; no specific limitation is made here. In some possible implementations, the sensitive information processing method of this application embodiment can be implemented by a processor calling computer-readable instructions stored in memory.

[0081] Specifically, such as Figure 1 As shown, the model training method based on skeleton action recognition in this application embodiment specifically includes the following steps:

[0082] Step S11: Obtain the skeleton video to be trained.

[0083] In this embodiment, the skeleton motion recognition device acquires skeleton video, including: acquiring skeleton data using a depth camera or advanced pose estimation algorithm; and annotating the skeleton data. Assuming the skeleton video input is... N is the batch size during training, C represents the number of coordinate feature channels, T represents the length of the skeleton video, and V represents the number of skeleton joints.

[0084] To reduce computation, the skeleton video is equally divided into S video segments along the timeline. Then, a frame is randomly selected from each video segment to form a new skeleton video as input, where S is much smaller than T. If the new skeleton video is less than S frames, the process repeats until S frames are reached. This process is represented as follows:

[0085] X′ in =Preprocess(X in (1)

[0086] Step S12: Use the local information modeling module of the skeleton action recognition model to extract local features of the skeleton video to be trained.

[0087] In this application embodiment, the Local Modelling Block (LMB) is first introduced, namely... Figure 2The local modeling is shown below. The skeleton action recognition device uses the local information modeling module of the skeleton action recognition model to extract the initial features of the skeleton video to be trained; performs a depth-level graph convolution operation on the skeleton video to be trained to obtain a skeleton feature map; performs a one-dimensional convolution operation along the time dimension on the skeleton feature map to obtain spatiotemporal domain local features; adds the initial features and the spatiotemporal domain local features to obtain spatiotemporal domain fused local features; inputs the spatiotemporal domain fused local features into a feedforward neural network to obtain feedforward local features; and adds the spatiotemporal domain fused local features and the feedforward local features to obtain local features.

[0088] It should be noted that, except Figure 2 Apart from the first local information modeling module, the remaining local information modeling modules use the output features of the preceding modules as the input features of the local information modeling modules. The rest of the process is the same and will not be described again here.

[0089] Please refer to the details. Figure 3 , Figure 3 This is a schematic diagram of the local information modeling module provided in this application. For example... Figure 3 As shown, in order to reduce parameters, the design of the local information modeling module in this application borrows from the form of depthwise separable convolution, and its specific expression is as follows:

[0090]

[0091]

[0092] Where X′ represents the input to the LMB, i.e., the skeleton video to be trained or the initial features, Y′ o This is the output of LMB, i.e., local features. Norm is a commonly used batch normalization operation. GCN DW This is a depth-level graph convolution operation. This application uses depth-level graph convolution operations to extract the positional features, i.e., the skeleton feature map, of each video frame. TCN DW This application uses a one-dimensional convolution operation at the depth level along the time axis to extract the temporal features of the skeleton video to be trained, i.e., local features in the spatial and temporal domain. This indicates an element-wise addition operation.

[0093] FFN stands for Feed-forward Neural Network. In this application, the feed-forward local features of spatiotemporal fusion local features are extracted by the feed-forward neural network. Then, the local information modeling module adds the feed-forward local features and the spatiotemporal fusion local features element by element to obtain the local features.

[0094] It can be seen that (2) corresponds to the local information modeling of the skeleton data in the spatiotemporal domain, and (3) corresponds to the modeling between channels.

[0095] The structure of a feedforward neural network is as follows: Figure 4 As shown, Figure 4 This is a schematic diagram of the feedforward neural network provided in this application. The process of the feedforward neural network can be represented as follows:

[0096] Y F =1×1Conv(Dropout(Relu(1×1Conv(X F (4)

[0097] Among them, X F It is the input of FFN, that is, spatiotemporal fusion of local features, Y F This represents the output of FFN, i.e., feedforward local features. 1×1Conv is a convolution with a 1×1 kernel. ReLU is a commonly used activation function. Dropout is a common operation to prevent overfitting.

[0098] Step S13: Obtain the first local feature, the second local feature, and the third local feature corresponding to the three feature dimensions of the local feature.

[0099] Step S14: Using the global information modeling module of the skeleton action recognition model, perform convolution operations on the first local feature, the second local feature, and the third local feature respectively to obtain the first global feature corresponding to the first local feature, the second global feature corresponding to the second local feature, and the third global feature corresponding to the third local feature.

[0100] In this embodiment of the application, taking the position-sensitive convolution operation of the second local feature as an example, the specific process of the position-sensitive convolution operation is as follows: determine the self-feature dimension C of the second local feature, as well as other feature dimensions, namely the feature dimension V of the first local feature and the feature dimension T of the third local feature; obtain the position encoding of the skeleton joints in the self-feature dimension and the convolution operation dimension, and perform convolution processing using the position encoding in the self-feature dimension and the convolution operation dimension and the concatenated feature of the second local feature to obtain the second global feature corresponding to the second local feature.

[0101] The calculation process for the position encoding of the skeleton joints in their own feature dimension and convolutional action dimension is as follows: obtain the position index of the convolutional action dimension and the length of the own feature dimension; obtain the position index of the skeleton joint in its own feature dimension; obtain the position encoding of the skeleton joint in its own feature dimension and convolutional action dimension according to the position index of the convolutional action dimension, the length of the own feature dimension and the position index in its own feature dimension.

[0102] Specifically, steps 13 to 14 above can be implemented by the Global Modelling Block (GMB) provided in this application, such as... Figure 5 As shown, Figure 5 This is a schematic diagram of the global information modeling module provided in this application. For example... Figure 5 As shown, the specific expression form of the global information modeling module is as follows:

[0103] X V X C X T =Permute(X) (5)

[0104]

[0105]

[0106]

[0107]

[0108]

[0109]

[0110]

[0111] Where X is the input of GMB, i.e., local features. In equation (5), the Permute operation represents the exchange of feature dimensions of X, such as... Figure 5 As shown, the local feature X has a feature dimension of C, and input data X is generated with V, C, and T as their respective feature dimensions. V (First local feature), X C (Second local feature), X T (Third local feature). The Permute operation in equations (9), (10), and (11) means restoring the feature dimensions of the first, second, and third local features to the same feature dimension order as the input local features, i.e., using C as its own feature dimension. Norm here also refers to the batch normalization operation.

[0112] ParC represents position-sensitive circular convolution. In this embodiment, the specific process of ParC operating on dimension V is shown when the feature dimension is C, denoted as ParC. V Please refer to the details. Figure 6 , Figure 6 This application provides ParC V A schematic diagram of the operation process.

[0113] like Figure 6 As shown, PE V This represents the positional encoding of skeleton joints in the convolutional dimension V and their own feature dimension C, which is generated using commonly used sine and cosine functions, as shown in the following formula:

[0114] PE V (p, 2i) = sin(p / 1000) 2i / d (13)

[0115] PE V (p,2i+1)=Cos(p / 1000 2i / d (14)

[0116] Where p represents the index on the convolutional dimension V, d is the length of the feature dimension C, and i is the result of taking the index on feature dimension C modulo 2. 2i corresponds to an even index on feature dimension C, and 2i+1 corresponds to an odd index on feature dimension C. For example, if the feature dimension C can be represented as (x, y, z), then the length of the feature dimension C is 3. When the index of the feature dimension C is 0, it corresponds to the x-axis, and i = 0; when the index is 1, it corresponds to the y-axis, and i = 0; when the index is 2, it corresponds to the z-axis, and i = 1.

[0117] Furthermore, ParC V Location encoding PE based on feature dimension T V Perform dimensional copying to obtain the feature dimension length and the second local feature X. C Copy position encoding PE with the same feature dimension length V ′;The second local feature X C With the copy location encoding PE V The addition yields the position-sensitive second local feature X. PV The location-sensitive second local feature X PV The convolutional features are concatenated with the convolutional features along the direction of the convolutional dimension V to obtain concatenated local features. Then, a pre-defined convolutional kernel is used to perform convolution processing on these concatenated local features to obtain the second local feature X. C The corresponding second global feature Y V .

[0118] Specifically, the above process is manifested as follows:

[0119] PE V =Expand(PE) V (15)

[0120]

[0121] Y V =WV *Stack V (X) PV (17)

[0122] Among them, X C ParC V The input is "Expand". "Expand" indicates that the PE file is copied. V Dimensions and X C Align. X C and PE V′ X is obtained by adding elements one by one. PV X PV It can be seen that it includes position-sensitive positional encoding features. Stack V This means that two identical copies of X will be used. PV (i.e. X) PV The concatenation operation (with itself) is performed along dimension V. To prevent duplication in subsequent convolution operations, the feature corresponding to the last index of dimension V after concatenation is removed. Therefore, the length of the feature dimension V of the concatenated features is (2V-1). * represents the convolution operation, W V These are the parameters of the corresponding convolution kernel. This convolution operation can be viewed as a depth-level convolution with a kernel size of 1×V. V It is ParC V The output of .

[0123] It should be noted that the specific processes of ParC when the convolutional operation dimensions are C and T are respectively denoted as ParC. C and ParC T Their specific implementations refer to ParC. V And so on.

[0124] Please continue reading. Figure 5 Taking the first local feature as an example, the global information modeling module will use the first local feature X... V After batch normalization, the data is then sequentially input into ParC. C and ParC T The result is compared with the first local feature X. V The first local feature X′ is obtained by adding the elements together. V ; Continue to apply the first local feature X′ V Through batch normalization and feedforward neural network (FFN) processing, feedforward local features are obtained. These feedforward local features are then combined with the first local feature X′. V Add the elements together, and then swap the feature dimensions of the sum to obtain the first global feature X″ whose feature dimensions are restored to the dimensional order of the local feature X. V .

[0125] It should be noted that the FFNs in equations (9), (10), and (11) are independent of each other. Specifically, the FFN in equation (9) is used for modeling between feature dimensions V. The FFN in equation (10) is used for modeling between feature dimensions C. The FFN in equation (11) is used for modeling between feature dimensions T.

[0126] Furthermore, the global information modeling module fuses the first, second, and third global features to obtain the global features. Here, the fusion weights α, β, and γ represent three learnable parameters of the network, used to weigh the importance of these three different perspective branches. g This is the final output of GMB.

[0127] Step S15: Fuse the first global feature, the second global feature, and the third global feature to obtain the global feature, and fuse the local features and the global features to obtain the fused feature.

[0128] In this embodiment of the application, the skeleton motion recognition device can fuse the local features and global features obtained in the above steps by adding elements together to obtain fused features, or it can fuse the local features and global features according to a conventional feature fusion algorithm to obtain fused features.

[0129] In one specific implementation, the skeleton motion recognition device may also employ a local and global information fusion module (Fusing Block, abbreviated as FB), i.e. Figure 2 The fusion module shown enables the fusion of local and global features.

[0130] Please refer to the details. Figure 7 and Figure 8 , Figure 7 yes Figure 1 The diagram shows the detailed process of step S15 in the model training method based on skeleton action recognition. Figure 8 This is a schematic diagram of the local and global information fusion module provided in this application.

[0131] Specifically, such as Figure 7 As shown, the model training method based on skeleton action recognition in this application embodiment specifically includes the following steps:

[0132] Step S151: Use the fusion modeling module of the skeleton action recognition model to obtain the first local features of the local features and the global features.

[0133] Step S152: Obtain the first fused feature output from the initial fusion of local and global features.

[0134] Step S153: Obtain the second fused feature output by fusing the first local feature with the first fused feature, and obtain the first global feature of the second fused feature.

[0135] Step S154: Obtain the third fusion feature output by fusing the global feature and the first fusion feature, and obtain the secondary local features of the third fusion feature.

[0136] Step S155: Fuse the first global features and the second local features to obtain the fused features.

[0137] In this embodiment of the application, the specific processes of steps S151 to S155 are illustrated by the following formulas:

[0138] X′ l =LMB(X′) f (18)

[0139] X′ g =GMB(X′) f (19)

[0140] X fusion =1×1conv(Concat(X′) l , X′ g (20)

[0141]

[0142]

[0143] Y fusion =1×1conv(Concat(X″) l ,X″ g )) (twenty three)

[0144] Where, X′ f It is the input to FB, i.e., local features; X′ l The output corresponding to the first LMB, i.e., the initial local feature; X′ g This corresponds to the output of the first GMB, i.e., the global features. `Concat` indicates a concatenation operation along the feature dimension C. X fusion It is the fusion feature output from the first fusion, i.e., the first fusion feature. X″ represents the second fused feature output by fusing the first local feature with the first fused feature. g This represents the first global feature of the second fusion feature. X″ represents the third fusion feature output by fusing the global feature and the first fusion feature. l Y represents a secondary local feature representing the third fusion feature. fusionThis represents the fused output of the initial global features and the secondary local features, and is also the final fused feature output by FB. It can be seen that FB implements both parallel and serial fusion methods.

[0145] Therefore, Figure 2 The entire feature extraction stage in the process can be represented as follows:

[0146] X s 1 = LMB(LMB(Preprocess(X) in ))) (twenty four)

[0147] X s2 =FB(X) s1 (25)

[0148] X s3 =LMB(FB(LMB(X) s2 (26)

[0149] Step S16: Input the fused features into the classifier of the skeleton action recognition model to obtain the predicted category of the skeleton video to be trained.

[0150] In the embodiments of this application, such as Figure 2 As shown, the skeleton motion recognition device inputs the fused features into the classifier, and the classifier outputs the predicted category of the skeleton video to be trained. At this time, the fused features can be the fused features calculated in step 15, i.e. Figure 2 The first fusion module outputs the fusion feature shown.

[0151] In another specific implementation, the fusion feature can also be a fusion feature obtained by fusing local features or fusion features at different scales. Please refer to further details. Figure 9 The model training process is shown.

[0152] Step S17: Train the skeleton action recognition model based on the predicted category and label category of the skeleton video to be trained.

[0153] In this embodiment, Softmax can be used as the classifier, and Y is the class probability output by the final classification network. Classification uses the commonly used cross-entropy loss function for supervised training, as shown in the following equation:

[0154]

[0155] Where N represents the batch size, Y i y represents the final output of the classification network. i The label corresponding to the input sample.

[0156] In this embodiment, the skeleton action recognition device acquires a skeleton video to be trained; extracts local features of the skeleton video using the local information modeling module of the skeleton action recognition model; acquires a first local feature, a second local feature, and a third local feature corresponding to the local features in three feature dimensions; performs convolution operations on the first local feature, the second local feature, and the third local feature using the global information modeling module of the skeleton action recognition model, respectively, to obtain a first global feature corresponding to the first local feature, a second global feature corresponding to the second local feature, and a third global feature corresponding to the third local feature; fuses the first global feature, the second global feature, and the third global feature to obtain a global feature; fuses the local features and the global feature to obtain a fused feature; inputs the fused feature into the classifier of the skeleton action recognition model to obtain the predicted category of the skeleton video to be trained; and trains the skeleton action recognition model based on the predicted category and label category of the skeleton video to be trained. Through the above method, the skeleton action recognition device, with its novel multi-view global information modeling module, can comprehensively, effectively, and efficiently mine the global information in the spatiotemporal domain of skeleton data, improving the training and recognition effects of the skeleton action recognition model.

[0157] Please continue reading. Figure 9 , Figure 9 This is a flowchart illustrating another embodiment of the model training method based on skeleton action recognition provided in this application.

[0158] Specifically, such as Figure 9 As shown, the model training method based on skeleton action recognition in this application embodiment specifically includes the following steps:

[0159] Step S21: Obtain the skeleton video to be trained.

[0160] Step S22: Use the local information modeling module of the skeleton action recognition model to extract local features from the skeleton video to be trained.

[0161] Step S23: Obtain the first local feature, the second local feature, and the third local feature corresponding to the three feature dimensions of the local feature.

[0162] Step S24: Using the global information modeling module of the skeleton action recognition model, perform position-sensitive convolution operations of other feature dimensions on the first local feature, the second local feature, and the third local feature respectively to obtain the first global feature corresponding to the first local feature, the second global feature corresponding to the second local feature, and the third global feature corresponding to the third local feature.

[0163] Step S25: Fuse the first global feature, the second global feature, and the third global feature to obtain the global feature.

[0164] Step S26: Fuse local features and global features to obtain fused features.

[0165] In this embodiment of the application, steps S21 to S26 are... Figure 1 Steps S11 to S15 are basically the same, and will not be repeated here.

[0166] Step S27: Input the fused features into the local information modeling module, the fusion modeling module, and the local information modeling module in sequence to obtain high-level fused features.

[0167] In the embodiments of this application, such as Figure 2 As shown, the skeleton motion recognition device uses features at three different scales for fusion classification, which can be specifically represented as follows:

[0168] X o1 =FC(GAP(X) s1 (28)

[0169] X o2 =FC(GAP(X) s2 (29)

[0170] X o3 =FC(GAP(X) s3 )) (30)

[0171]

[0172] Here, a, b, and c represent three learnable parameters of the network, used to adjust the weights of features at three scales. FC stands for fully connected operation. It should be noted that the FC operations in equations (28), (29), and (30) are independent of each other. GAP stands for Global Average Pooling.

[0173] Additionally, it should be noted that, such as Figure 2 As shown, the LMB between the two FBs requires downsampling of the skeleton data. During this process, the residual connections in the LMB are removed. Specifically, the time domain uses the TCN in the LMB. DW For 2x downsampling, GCN will be used in the spatial domain. DW The skeletal data is divided into 5 body parts, as shown in the diagram below. Figure 10 As shown, the skeleton data is divided into five parts: torso, left arm, right arm, left leg, and right leg. This is done so that the features output by the two FB (Future Frame) systems can form hierarchical features. Furthermore, GMB (Gross Memory Management) only uses positional encoding information before downsampling. After downsampling, GMB does not use positional encoding information.

[0174] Step S28: Perform feature fusion on local features, fused features, and high-level fused features, and input the feature fusion result into the classifier of the skeleton action recognition model to obtain the predicted category of the skeleton video to be trained.

[0175] In the embodiments of this application, Figure 2 The local features in branch a can be regarded as the low-level features of the skeleton video to be trained. Figure 2 The fused features in branch b can be regarded as mid-level features of the skeleton video to be trained. Figure 2 The high-level fused features in the c branch can be regarded as high-level features of the skeleton video to be trained. The skeleton action recognition device achieves optimal classification results by fusing and classifying features of three different scales: low-level features, mid-level features, and high-level features.

[0176] Step S29: Train the skeleton action recognition model based on the predicted category and label category of the skeleton video to be trained.

[0177] In this embodiment, ParC from the image domain is introduced into the skeleton data domain. Based on the unique characteristics of skeleton data, a novel multi-view global information modeling module is proposed, which can comprehensively, effectively and efficiently mine the global information in the spatiotemporal domain of skeleton data. A novel fusion module is proposed, which can fully fuse local salient features and global semantic features in both parallel and serial manner. A multi-scale fusion classification scheme is used, which can improve the accuracy of skeleton action classification by utilizing features at different levels.

[0178] Please continue reading. Figure 11 , Figure 11 This is a flowchart illustrating an embodiment of the skeleton motion recognition method provided in this application.

[0179] Specifically, such as Figure 11 As shown, the skeleton motion recognition method of this application embodiment specifically includes the following steps:

[0180] Step S31: Obtain the skeleton video.

[0181] Step S32: Input the skeleton video into the pre-trained skeleton action recognition model.

[0182] Among them, the skeleton action recognition model can be Figure 1 or Figure 9 The skeleton action recognition model trained by the model training method shown is not described in detail here.

[0183] Step S33: Obtain the skeleton action category output by the skeleton action recognition model.

[0184] The skeleton action recognition method constructed in this application effectively integrates local and global information of skeleton data in the spatiotemporal domain in a computationally friendly manner, achieving excellent performance and facilitating practical deployment and use.

[0185] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0186] To implement the model training method and / or skeleton action recognition method based on skeleton action recognition described in the above embodiments, this application also proposes a skeleton action recognition device, which can be found in the details below. Figure 12 , Figure 12 This is a schematic diagram of an embodiment of the skeleton motion recognition device provided in this application.

[0187] The skeleton motion recognition device 400 of this application embodiment includes a memory 41 and a processor 42, wherein the memory 41 and the processor 42 are coupled together.

[0188] The memory 41 is used to store program data, and the processor 42 is used to execute the program data to implement the skeleton motion recognition method described in the above embodiments.

[0189] In this embodiment, processor 42 can also be referred to as a CPU (Central Processing Unit). Processor 42 may be an integrated circuit chip with signal processing capabilities. Processor 42 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, or processor 42 can be any conventional processor.

[0190] To implement the skeleton action recognition method of the above embodiments, this application also provides a computer-readable storage medium, such as... Figure 13 As shown, the computer-readable storage medium 500 is used to store program data 51, which, when executed by the processor, is used to implement the skeleton motion recognition method as described in the above embodiments.

[0191] This application also provides a computer program product, wherein the computer program product includes a computer program operable to cause a computer to perform the sensitive information processing method as described in the embodiments of this application. The computer program product can be a software installation package.

[0192] The sensitive information processing method described in the above embodiments of this application, when implemented as a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0193] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A model training method based on skeleton action recognition, characterized in that, The model training method includes: Obtain the skeleton video to be trained; The local information modeling module of the skeleton action recognition model is used to extract local features of the skeleton video to be trained; Obtain the first local feature, the second local feature, and the third local feature corresponding to the three feature dimensions of the local feature; The global information modeling module of the skeleton action recognition model is used to perform convolution operations of other feature dimensions on the first local feature, the second local feature and the third local feature respectively to obtain the first global feature corresponding to the first local feature, the second global feature corresponding to the second local feature and the third global feature corresponding to the third local feature; The first global feature, the second global feature, and the third global feature are fused to obtain a global feature, and the local feature and the global feature are fused to obtain a fused feature. The fused features are input into the classifier of the skeleton action recognition model to obtain the predicted category of the skeleton video to be trained; The skeleton action recognition model is trained based on the predicted category and label category of the skeleton video to be trained. The convolution operations for the other feature dimensions are position-sensitive convolution operations; The position-sensitive convolution operation is as follows: determine the self-feature dimension of the second local feature and other feature dimensions; obtain the position encoding of the skeleton joints on the self-feature dimension and the convolution operation dimension; use the position encoding on the self-feature dimension and the convolution operation dimension and the concatenated feature of the second local feature to perform convolution processing to obtain the second global feature corresponding to the second local feature. The step of performing convolution processing using the positional encoding on the self-feature dimension and the convolution operation dimension, along with the concatenated features of the second local features, to obtain the second global feature corresponding to the second local feature includes: Based on the feature dimension of the second local feature, the positional encoding on the feature dimension of the feature itself and the convolution operation dimension is copied to obtain a copied positional encoding with the same feature dimension length as the feature dimension length of the second local feature. The second local feature is added to the copy position encoding to obtain the position-sensitive second local feature. The position-sensitive second local feature is then concatenated with itself according to the direction of the convolution operation dimension to obtain the concatenated local feature. A one-dimensional convolution kernel with an orientation along the convolutional dimension and a size equal to the length of the convolutional dimension on the second local feature is used to perform convolution processing on the spliced ​​local features to obtain the second global feature corresponding to the second local feature.

2. The model training method according to claim 1, characterized in that, The acquisition of the position encoding of the skeleton joints in its own feature dimension and the convolution operation dimension includes: Obtain the position index of the convolution operation dimension and the length of its own feature dimension; Obtain the position index of the skeleton joint in its own feature dimension; The positional encoding of the skeleton joints is obtained according to the position index of the convolution operation dimension, the length of the self-feature dimension, and the position index of the self-feature dimension.

3. The model training method according to claim 1, characterized in that, The other feature dimensions include the first feature dimension and the second feature dimension of the second local feature in addition to its own feature dimension; The step of performing convolution processing using the positional encoding on the self-feature dimension and the convolution operation dimension, along with the concatenated features of the second local features, to obtain the second global feature corresponding to the second local feature includes: The second local sub-feature is obtained by performing convolution processing using the positional encoding of the first feature dimension and its own feature dimension, as well as the concatenated features of the second local features. The second local output feature is obtained by performing convolution processing using the positional encoding of the second feature dimension and its own feature dimension, as well as the concatenated features of the second local sub-features. The second local feature and the second local output feature are fused to obtain the second global feature corresponding to the second local feature.

4. The model training method according to claim 1, characterized in that, The process of fusing the local features and the global features to obtain the fused features includes: Using the fusion modeling module of the skeleton action recognition model, the first local features and global features of the local features are obtained; Obtain the first fused feature output from the initial fusion of the local features and the global features; Obtain the second fused feature output by fusing the first local feature with the first fused feature, and obtain the first global feature of the second fused feature; Obtain the third fusion feature output by fusing the global feature and the first fusion feature, and obtain the secondary local features of the third fusion feature; The first global feature and the second local feature are fused together to obtain the fused feature.

5. The model training method according to claim 4, characterized in that, The step of fusing the initial global features and the secondary local features to obtain the fused features includes: The concatenated features of the first global feature and the second local feature are subjected to a 1×1 convolution to obtain the fused feature.

6. The model training method according to claim 1, characterized in that, The process of fusing the local features and the global features to obtain the fused features includes: The local features and the global features are fused to obtain the fused features; The fusion features are sequentially input into the local information modeling module, the fusion modeling module, and the local information modeling module to obtain high-level fusion features; The step of inputting the fused features into the classifier of the skeleton action recognition model to obtain the predicted category of the skeleton video to be trained includes: The local features, the fused features, and the high-level fused features are fused together, and the feature fusion result is input into the classifier of the skeleton action recognition model to obtain the predicted category of the skeleton video to be trained.

7. The model training method according to claim 6, characterized in that, The step of sequentially inputting the fused features into the local information modeling module, the fusion modeling module, and the local information modeling module to obtain high-level fused features includes: The fused features are input into the local information modeling module, so that the fused features are downsampled by the local information modeling module to obtain the first fused features; The initial fusion features are sequentially input into the fusion modeling module and the local information modeling module to obtain the high-level fusion features.

8. The model training method according to claim 1, characterized in that, The local information modeling module of the skeleton action recognition model extracts local features from the skeleton video to be trained, including: The initial features of the skeleton video to be trained are extracted using the local information modeling module of the skeleton action recognition model. Perform depth-level graph convolution on the skeleton video to be trained to obtain a skeleton feature map; Perform a one-dimensional convolution operation along the time dimension on the skeleton feature map to obtain local features in the spatiotemporal domain; The initial features and the spatiotemporal local features are added together to obtain the spatiotemporal fused local features; The spatiotemporal fusion local features are input into the feedforward neural network to obtain feedforward local features; The local features are obtained by adding the spatiotemporal domain fusion local features and the feedforward local features.

9. The model training method according to claim 8, characterized in that, The process of obtaining the skeleton video to be trained includes: Obtain the skeleton video; The skeleton video is equally divided into several video segments along the time axis; The same number of skeleton video frames are extracted from each video segment to form the skeleton video to be trained.

10. A skeleton motion recognition method, characterized in that, The skeleton motion recognition method includes: Obtain the skeleton video; The skeleton video is input into a pre-trained skeleton action recognition model, wherein the skeleton action recognition model is trained by the model training method according to any one of claims 1 to 9; Obtain the skeleton action category output by the skeleton action recognition model.

11. A skeleton motion recognition device, characterized in that, The skeleton motion recognition device includes a processor and a memory, wherein the memory stores program data, and the processor executes the program data to implement the model training method as described in any one of claims 1 to 9, and / or the skeleton motion recognition method as described in claim 10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program data, which, when executed by a processor, is used to implement the model training method according to any one of claims 1 to 9, and / or the skeleton motion recognition method according to claim 10.

Images