Cross-view action recognition method based on skeleton self-similarity

A technology of self-similarity and action recognition, applied in biometric feature recognition, neural learning methods, character and pattern recognition, etc., can solve problems affecting action recognition and achieve good results and high structural stability

Pending Publication Date: 2020-08-21
ZHEJIANG UNIV OF TECH
View PDF1 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The invention solves the problem that the change of human body appearance under the change of line of sight affects action recognition, proposes a cross-view action recognit...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Cross-view action recognition method based on skeleton self-similarity
  • Cross-view action recognition method based on skeleton self-similarity
  • Cross-view action recognition method based on skeleton self-similarity

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0057] This embodiment proposes a cross-view action recognition method based on skeleton self-similarity, including the following steps:

[0058] S1, acquire skeleton sequences at 3 scales in a fine-to-coarse manner;

[0059] S2, representing the skeleton self-similarity as a self-similar image SSI;

[0060] S3, constructing the space-time convolution module SCM;

[0061] S4, constructing the sequence encoding module SEM;

[0062] S5, a backbone network based on SCM and SEM, constructs a multi-stream neural network MSNN through multi-stream fusion.

[0063] The present invention considers that the use of skeleton information of different scales can effectively improve the recognition performance and deal with the occlusion situation well. Therefore, human skeletons at three scales are first acquired in a fine-to-coarse manner. On a finer scale, the corresponding human skeleton contains more joints; the proposal of the multi-scale SSI scheme makes the present invention more...

Embodiment 2

[0097] This embodiment proposes a cross-view action recognition method based on skeleton self-similarity. Compared with step S5 in Embodiment 1, the early fusion is replaced by late fusion. Refer to Figure 6 , step S5 specifically includes the following steps:

[0098] Using SCM and SEM as the backbone of building a multi-stream neural network MSNN, such as figure 1 shown. Among them, SCM is used to extract the spatio-temporal features of SSI images; the spatio-temporal features are further input into SEM to model the temporal dependencies between action sequences. Finally, action representations are learned from SSI images of different scales by fusing the three backbone networks to form a multi-stream neural network. Based on this backbone structure, the later fusion is to directly form an action combination representation U by concatenating the SEM output features of multiple streams;

Embodiment 3

[0100] For Examples 1 and 2, a batch normalization BN layer is added after U to eliminate the covariate shift in U; ​​finally, the normalized U is input to the SoftMax classifier, and according to the given U, The predicted probability of belonging to the i-th category is,

[0101]

[0102] in, Indicates the probability that U is the i-th category, where w s,i (w s,i ) represents the weight matrix W in the SoftMax layer s The i-th row (j-th column) of , C represents the total number of classes;

[0103] Express the final optimization objective function of MSNN as a cross-entropy loss function L with L2 norm regularization:

[0104]

[0105] where y=(y 1 ,y 2 ,...,y C ) is the ground truth label, Indicates the predicted probability that the sequence U belongs to the i-th action category; W indicates the global matrix of network weights, here merged into one matrix; L2 regularization is applied to W to reduce network overfitting. scalar lambda 1 The role of is ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a cross-view action recognition method based on skeleton self-similarity. The cross-view action recognition method comprises the following steps: S1, obtaining skeleton sequences of three scales in a fine-to-rough mode; S2, expressing the skeleton self-similarity as a self-similarity image SSI; S3, constructing a space-time convolution module SCM; S4, constructing a sequenceencoding module SEM; and S5, based on the backbone network of the SCM and the SEM, constructing a multi-stream neural network MSNN through multi-stream fusion. According to the method, the recognition performance can be effectively improved and the shielding situation can be well processed by considering the use of skeleton information of different scales. Therefore, human skeletons of three scales are obtained in a fine-to-rough mode. In a finer scale, the corresponding human skeleton comprises more joints; the multi-scale SSI scheme is provided so that the method has better robustness, andthe shielding problem caused by a side view angle is effectively prevented.

Description

technical field [0001] The invention relates to the technical field of action recognition in vision, in particular to a cross-view action recognition method based on skeleton self-similarity. Background technique [0002] Human action recognition has been an active research area due to its wide range of applications, such as video understanding, human-computer interaction, and assistive robotics. Under a fixed general viewpoint, most existing methods in this field can achieve action recognition well. However, their performance is limited or even unpredictable when the camera viewing angle changes. Action recognition in such so-called "cross-view" scenarios is a very challenging task, because the visual appearance of the same action captured by different sensors from various viewpoints may be completely different. Therefore, it is a major challenge in this field to design a cross-view action recognition method that can still perform well in the scene of view change. [000...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06K9/00G06K9/62G06N3/04G06N3/08
CPCG06N3/08G06V40/20G06V40/10G06N3/047G06N3/045G06F18/2415G06F18/241
Inventor 邵展鹏刘鹏胡超群周小龙
Owner ZHEJIANG UNIV OF TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap