Skeleton behavior recognition method and device based on improved CTR-GCN model and medium

Abstract

The invention relates to a skeleton behavior recognition method and device based on an improved CTR-GCN model, and a medium. The method comprises the following steps: constructing an SML-CTR-GCN model based on the CTR-GCN model, introducing a motion feature enhancement module, and calculating spatial change information of joint points between adjacent frames to obtain motion features of the joint points; a SimAM mechanism is added, a three-dimensional attention weight is generated for a feature map by using an energy function, and the key feature extraction capability of the model for large-amplitude behaviors is enhanced; and in combination with an LSTM network, a gating mechanism is adopted to effectively capture a long-time dependency relationship of a skeleton sequence, so that the recognition performance of the model on complex behaviors is improved. And the personnel behavior is identified by using the trained SML-CTR-GCN model. Compared with the prior art, the model provided by the invention has good recognition performance on complex behaviors, higher recognition accuracy and better stability and anti-interference capability.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a skeleton behavior recognition method, device and medium based on an improved CTR-GCN model. Background Art

[0002] In recent years, human behavior recognition technology has been widely used in many fields such as security monitoring, human-computer interaction and intelligent medical care. A large number of scholars have used RGB video data to analyze and identify human behavior, but the skeleton behavior recognition method based on RGB data requires high computing resources and will affect the recognition accuracy under conditions such as lighting changes and background interference. Therefore, many scholars have turned their research direction to skeleton data. Since skeleton data has low complexity and is not easily affected by factors such as environmental changes, skeleton behavior recognition methods based on human skeleton data have gradually become dominant.

[0003] With the continuous progress of deep learning, skeleton behavior recognition methods based on deep learning are gradually replacing methods based on manually designed features. These methods mainly include three categories: Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) and Graph Convolutional Network (GCN). The input of the CNN network usually converts the skeleton data into a 2D pseudo image, and then uses the convolution kernel to extract the features of the pseudo image for classification, but this method will inevitably lose time information and cannot fully mine feature information. RNN is good at modeling time series data. This type of method often overemphasizes the temporal characteristics of the behavior and is not ideal for extracting spatial information.

[0004] Graph convolutional neural networks are increasingly being used to model dynamic skeleton sequences. Representative algorithms include ST-GCN, 2s-AGCN, and CTR-GCN. Although these algorithms have more obvious advantages in modeling dynamic skeleton sequences, they still have some shortcomings. First, the existing GCN model has relatively weak processing of the time dimension and lacks the ability to capture long-term dependencies. Second, the motion features of skeleton sequences are insufficiently extracted and are easily affected by noise, which makes it impossible to effectively distinguish and identify complex and diverse behavior categories of people. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art such as insufficient motion feature extraction and susceptibility to noise interference, and to provide a skeleton behavior recognition method based on an improved CTR-GCN model.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] As a first aspect of the present invention, a skeleton behavior recognition method based on an improved CTR-GCN model is provided, the steps comprising:

[0008] Obtain behavioral data and process it to obtain skeleton sequence data of multiple modes;

[0009] The obtained skeleton sequence data of multiple modalities are input into the trained SML-CTR-GCN model to identify the human behavior;

[0010] The SML-CTR-GCN model is based on the CTR-GCN model, introduces a motion feature enhancement module, calculates the spatial change information of joints between adjacent frames to obtain the motion features of joints; adds the SimAM mechanism, uses the energy function to generate three-dimensional attention weights for the feature map; combines with the LSTM network, and uses the gating mechanism to capture the long-term dependency of the skeleton sequence.

[0011] As a preferred technical solution, the overall structure of the SML-CTR-GCN model includes:

[0012] The spatial module includes three parallel CTG-GC modules and one motion feature enhancement ME module. The spatial features of the joints are extracted through the CTG-GC module, and the motion feature enhancement ME module is used to capture the motion change information of the joints between adjacent frames. Subsequently, the key frames and key nodes are further focused through the SimAM mechanism.

[0013] The temporal module introduces LSTM after the spatial module to strengthen temporal context modeling and capture the long-term dependencies of skeleton sequences;

[0014] The features are processed through global average pooling and fully connected layers to output behavior categories.

[0015] As a preferred technical solution, the motion feature enhancement module is specifically as follows:

[0016] The motion characteristics of the joint points are obtained by calculating the spatial change information of the joint points between adjacent frames;

[0017] Connect the motion features of joints in the time dimension;

[0018] The motion features are average pooled and max pooled in the time dimension respectively, and the two features are fused through two-dimensional convolution;

[0019] Average pooling and maximum pooling are performed in the channel dimension, and then two-dimensional convolution is applied to reduce the number of channels from 2 to 1. The motion information weight is obtained by the activation function for the obtained feature map;

[0020] The motion information weights are combined with the original features through element-by-element multiplication, and a residual structure is added to keep the original spatial features from being lost, and finally the output is obtained.

[0021] As a preferred technical solution, the SimAM mechanism generates three-dimensional attention weights for feature maps through a minimum energy function, focusing on more discriminative key nodes and key frames. The minimum energy function is expressed as:

[0022]

[0023]

[0024]

[0025] Where E represents the minimum energy of each neuron, t represents the target neuron for calculating the minimum energy, and represents the mean and variance of all neurons on a single channel, λ represents the regularization coefficient, M=H×W represents the number of neurons on each channel, H and W represent the length and width of the feature map respectively; the lower the energy of a target neuron t, the higher its importance.

[0026] As a preferred technical solution, the LSTM network adopts a gating mechanism to effectively capture the long-term dependency of the skeleton sequence; the LSTM network consists of an input gate, a forget gate, an output gate and a memory unit, which are defined as follows:

[0027] f t =σ(W f [h t-1 ,x t ]+b f )

[0028] i t =σ(W i ×[h t-1 ,x t ]+b i )

[0029]

[0030]

[0031] o t =σ(W o [h t-1 ,x t ]+b o )

[0032] h t =o t ×tanh(Ct )

[0033] Among them, h t-1 Represents the output at the previous moment, x t represents the input at the current moment, f t is the generated weight vector between 0 and 1, σ represents the activation function sigmoid, W f and b f Respectively represent the weight matrix and bias term of the forget gate, W i 、b i Represent the weight and bias of the input gate respectively, W c 、b c Represent the weight and bias of the unit state respectively, represents a new cell state, h t The final output result is W o 、b o are the weight and bias of the output gate respectively.

[0034] As a preferred technical solution, the skeleton dataset is processed into data of four modalities: joints, bones, joint motions and bone motions as input to the SML-CTR-GCN model.

[0035] As a preferred technical solution, the SML-CTR-GCN model is trained using a multi-stream fusion framework: independent training is performed using data from four modalities: joints, bones, joint motion, and bone motion, to obtain output scores for the four branches, which are then fused by weighted summation, and the behavior prediction category is determined based on the fusion score.

[0036] As a preferred technical solution, the recognition accuracy is used as an evaluation indicator when the SML-CTR-GCN model is tested.

[0037] As a second aspect of the present invention, a skeleton behavior recognition device based on an improved CTR-GCN model is provided, comprising a memory, a processor, and a program stored in the memory, wherein when the processor executes the program, the skeleton behavior recognition method based on the improved CTR-GCN model as described above is implemented.

[0038] As a third aspect of the present invention, a storage medium is provided, on which a program is stored, and when the program is executed, the skeleton behavior recognition method based on the improved CTR-GCN model as described above is implemented.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1) The present invention constructs the SML-CTR-GCN model based on the CTR-GCN model, introduces a motion feature enhancement module, calculates the spatial change information of joints between adjacent frames to obtain the motion features of joints, and improves the model's ability to distinguish similar behaviors; adds the SimAM mechanism, uses the energy function to generate three-dimensional attention weights for the feature map, enhances the model's ability to extract key features of large-scale behaviors, and reduces the interference of noise information; combines the LSTM network, and adopts a gating mechanism to effectively capture the long-term dependency of skeleton sequences, thereby improving the model's recognition performance for complex behaviors.

[0041] 2) The present invention uses a human skeleton sequence as input, and uses data of four modalities, namely, joints, bones, joint movements and bone movements, for independent training. The behavior prediction category is judged based on the fusion score to ensure that the input data is lightweight and anti-interference, which can well realize personnel behavior recognition, has a high accuracy rate, and has the advantages of good stability and strong anti-interference ability, and can be applied to personnel recognition systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of the skeleton behavior recognition method based on the improved CTR-GCN model of the present invention;

[0043] Figure 2 A network structure diagram of a motion feature enhancement module in one embodiment of the present invention;

[0044] Figure 3 is a structural diagram of the SimAM mechanism in one embodiment of the present invention;

[0045] Figure 4 LSTM structure diagram in one embodiment of the present invention;

[0046] Figure 5 The overall structure diagram of the SML-CTR-GCN model in one embodiment of the present invention;

[0047] Figure 6 This is a diagram showing the skeleton behavior recognition effect in one embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0049] Example 1

[0050] This embodiment provides a skeleton behavior recognition algorithm based on an improved CTR-GCN model, and constructs an SML-CTR-GCN model based on the CTR-GCN model, which has a high accuracy rate for skeleton behavior recognition of people. Specifically, Figure 1 As shown, the following steps are included:

[0051] Step S100: construct an SML-CTR-GCN model based on the CTR-GCN model, wherein the SML-CTR-GCN model introduces a motion feature enhancement module to calculate the spatial change information of joints between adjacent frames to obtain the motion features of joints; adds a SimAM mechanism to generate three-dimensional attention weights for feature maps using an energy function; and combines with an LSTM network to use a gating mechanism to effectively capture the long-term dependencies of skeleton sequences.

[0052] Specifically, Figure 2 As shown in the figure, the motion feature enhancement module obtains the motion features of the joints by calculating the spatial change information of the joints between adjacent frames, thereby improving the model's ability to distinguish similar behaviors. The extracted motion features can be expressed as:

[0053] X motion (t) = X r (t+1)-X r (t),1≤t≤T-1

[0054] in, represents the node motion characteristics, X r (t+1) represents the node feature of the next time frame t+1, X r (t) represents the node features of the current time frame t.

[0055] Then the obtained motion features are connected in the time dimension:

[0056] X motion =[X motion (1),...,X motion (2),0]

[0057] in, is the node motion feature of T time frames.

[0058] Then the motion features are averaged and max-pooled in the time dimension, and the two features are fused through two-dimensional convolution:

[0059] X1=conv 1×1 (concat(MaxPool(X motion ),AvgPool(X motion )))

[0060] in, Represents the obtained fusion features.

[0061] Then, average pooling and maximum pooling are performed in the channel dimension, and then two-dimensional convolution is applied to reduce the number of channels from 2 to 1, obtaining a feature map X2 with a dimension of N×1×1×V, and extracting more comprehensive motion information, as shown in the following formula:

[0062] X2=conv 1×1 (concat(MaxPool(X1),AvgPool(X1)))

[0063] Among them, concat represents the concatenation operation.

[0064] Then, a motion information weight is obtained through the sigmoid activation function. Finally, the weight is combined with the original feature through element-by-element multiplication to fuse the motion feature into the original spatial feature. At the same time, a residual structure is added to keep the original spatial feature from being lost, and the output is finally obtained:

[0065] X out =X+X·σ(X2)

[0066] Among them, X represents the original spatial feature, X out represents the final output feature, and σ represents the sigmoid activation function.

[0067] like Figure 3 As shown in the figure, the SimAM mechanism generates three-dimensional attention weights for feature maps through the minimum energy function, which helps the model better understand the relationship between features and pay attention to more discriminative key nodes and key frames. The minimum energy function can be expressed as:

[0068]

[0069]

[0070]

[0071] Where E represents the minimum energy of each neuron, t represents the target neuron for calculating the minimum energy, and represents the mean and variance of all neurons on a single channel, λ represents the regularization coefficient, M = H × W represents the number of neurons on each channel, and H and W represent the length and width of the feature map, respectively. The lower the energy of a target neuron t, the more obvious its difference from neighboring neurons, and the higher its importance.

[0072] like Figure 4As shown in the figure, the LSTM network uses a gating mechanism to effectively capture the long-term dependencies of the skeleton sequence and improve the model's recognition performance for complex behaviors. The LSTM network consists of an input gate, a forget gate, an output gate, and a memory unit, which are defined as follows:

[0073] f t =σ(W f [h t-1 ,x t ]+b f )

[0074] i t =σ(W i ×[h t-1 ,x t ]+b i )

[0075]

[0076]

[0077] o t =σ(W o [h t-1 ,x t ]+b o )

[0078] h t =o t ×tanh(C t )

[0079] Among them, h t-1 Represents the output at the previous moment, x t represents the input at the current moment, f t is the generated weight vector between 0 and 1, σ represents the activation function sigmoid, W f and b f Respectively represent the weight matrix and bias term of the forget gate. i 、b i Represent the weight and bias of the input gate respectively, W c 、b c Represent the weight and bias of the unit state respectively, represents a new cell state, h t The final output result is W o 、b o are the weight and bias of the output gate respectively.

[0080] Step S200: preprocessing the public behavior datasets NTU RGB+D and NW-UCLA to obtain skeleton sequence data of four modes as input of the model;

[0081] Specifically, the skeleton dataset is processed into four modal data of joints, bones, joint motions and bone motions as the input of the model.

[0082] Step S300: Using a multi-stream fusion network framework, the processed four modal data are respectively input into the SML-CTR-GCN model for training and testing;

[0083] Specifically, the SML-CTR-GCN model is trained using a multi-stream fusion framework, and the recognition accuracy is used as the evaluation indicator during testing. The data of four modalities, joints, bones, joint movement, and bone movement, are used for independent training to obtain the output scores of the four branches, which are then fused by weighted summation, and the behavior prediction category is determined based on the fusion score.

[0084] Step S400: using the trained SML-CTR-GCN model to identify human behavior.

[0085] Specifically, Figure 5 The overall structure schematic diagram of the SML-CTR-GCN model of the present invention is shown. The overall structure of the SML-CTR-GCN model is based on the skeleton design of the CTR-GCN model, and the feature extraction and modeling capabilities are enhanced through a series of improved modules. The basic part of the CTR-GCN model includes a graph convolutional network and a temporal convolutional network for extracting skeleton spatial features. On this basis, the present application introduces a motion feature enhancement ME module, a SimAM mechanism and an LSTM module to improve the model performance. Among them, the spatial module consists of 3 parallel CTG-GC modules and 1 ME module, and the spatial features of the joint points are extracted by the CTG-GC module, and the motion change information of the joint points between adjacent frames is captured by the ME module. Subsequently, the SimAM mechanism further focuses on key frames and key nodes to enhance the discrimination ability. These modules are embedded in the L1-L10 basic modules of the backbone network, wherein the number of channels of each module is 64, 64, 64, 64, 128, 128, 128, 256, 256, 256, and features are extracted step by step from low to high levels. In terms of temporal modeling, the multi-scale temporal convolution module combines the dynamic feature capture capabilities of different time ranges, and introduces LSTM after the L10 module to strengthen temporal context modeling. Finally, the features are processed through global average pooling and a fully connected layer to output the behavior category. This model combines the ME module, SimAM mechanism, and LSTM network to build a behavior recognition model, which improves the accuracy of personnel behavior recognition.

[0086] The training of the behavior recognition model in this embodiment includes the following steps:

[0087] (1) Environment setting. The environment setting is Windows 10, 64-bit operating system, GPU model is NVIDIA Tesla V100, and the experimental running environment is python 3.8 and pytorch 1.9 learning framework.

[0088] (2) Dataset processing: The public skeleton datasets NTU RGB+D and NW-UCLA datasets are processed into four modalities of data: joints, bones, joint motion, and bone motion as input to the model.

[0089] (3) Model training. Model training was implemented using pytorch. For the NTU RGB+D and NW-UCLA datasets, the batch size was set to 64 and 16, the number of training rounds was set to 65, and a warm-up strategy was used in the first 5 epochs to ensure more stable training. The stochastic gradient descent algorithm SGD with a weight decay of 0.0004 was used for optimization. The initial learning rate was set to 0.1, and it was decayed by a decay factor of 0.1 after 35 and 55 training rounds.

[0090] Use the trained skeleton behavior recognition model to perform behavior recognition, and the output human behavior recognition results are as follows: Figure 6 shown.

[0091] Example 2

[0092] This embodiment provides a skeleton behavior recognition algorithm device based on an improved CTR-GCN model, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described in Embodiment 1 is implemented.

[0093] In one embodiment, the apparatus comprises:

[0094] Model building module: used to build the SML-CTR-GCN model. The SML-CTR-GCN model introduces a motion feature enhancement module to calculate the spatial change information of joints between adjacent frames to obtain the motion features of joints; adds the SimAM mechanism to generate three-dimensional attention weights for feature maps using energy functions; combines with the LSTM network and uses a gating mechanism to effectively capture the long-term dependencies of skeleton sequences;

[0095] Dataset processing module: pre-process the public behavior datasets NTU RGB+D and NW-UCLA to obtain skeleton sequence data of four modes as the input of the model;

[0096] Model training and testing module: Using a multi-stream fusion network framework, the processed four modal data are input into the SML-CTR-GCN model for training and testing;

[0097] Behavior recognition module: Use the trained SML-CTR-GCN model to identify human behavior.

[0098] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0099] Example 3

[0100] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0101] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A skeleton behavior recognition method based on an improved CTR-GCN model, characterized in that: The method comprises the following steps: acquiring behavior data and processing to obtain skeleton sequence data of multiple modes; inputting the obtained skeleton sequence data of multiple modes into the trained SML-CTR-GCN model to recognize the behavior of personnel; The SML-CTR-GCN model is based on the CTR-GCN model, introduces a motion feature enhancement module, calculates the spatial change information of joints between adjacent frames to obtain the motion features of joints; adds the SimAM mechanism, uses the energy function to generate three-dimensional attention weights for the feature map; combines with the LSTM network, and uses the gating mechanism to capture the long-term dependency of the skeleton sequence.

2. According to claim 1, a skeleton behavior recognition method based on an improved CTR-GCN model is characterized in that: The overall structure of the SML-CTR-GCN model includes: The spatial module includes three parallel CTG-GC modules and one motion feature enhancement ME module. The spatial features of the joints are extracted through the CTG-GC module, and the motion feature enhancement ME module is used to capture the motion change information of the joints between adjacent frames. Subsequently, the key frames and key nodes are further focused through the SimAM mechanism. The temporal module introduces LSTM after the spatial module to strengthen temporal context modeling and capture the long-term dependencies of skeleton sequences; The features are processed through global average pooling and a fully connected layer to output the behavior category.

3. According to claim 1, a skeleton behavior recognition method based on an improved CTR-GCN model is characterized in that: The motion feature enhancement module is specifically as follows: The motion characteristics of the joint points are obtained by calculating the spatial change information of the joint points between adjacent frames; Connect the motion features of joints in the time dimension; The motion features are average pooled and max pooled in the time dimension respectively, and the two features are fused through two-dimensional convolution; Average pooling and maximum pooling are performed in the channel dimension, and then two-dimensional convolution is applied to reduce the number of channels from 2 to 1. The motion information weight is obtained by the activation function for the obtained feature map; The motion information weights are combined with the original features through element-by-element multiplication, and a residual structure is added to keep the original spatial features from being lost, and finally the output is obtained.

4. According to claim 1, a skeleton behavior recognition method based on an improved CTR-GCN model is characterized in that: The SimAM mechanism generates three-dimensional attention weights for feature maps through the minimum energy function, focusing on more discriminative key nodes and key frames. The minimum energy function is expressed as: Where E represents the minimum energy of each neuron, t represents the target neuron for calculating the minimum energy, and represents the mean and variance of all neurons on a single channel, λ represents the regularization coefficient, M=H×W represents the number of neurons on each channel, H and W represent the length and width of the feature map respectively; the lower the energy of a target neuron t, the higher its importance.

5. According to claim 1, a skeleton behavior recognition method based on an improved CTR-GCN model is characterized in that: The LSTM network adopts a gating mechanism to effectively capture the long-term dependency of the skeleton sequence; the LSTM network consists of an input gate, a forget gate, an output gate and a memory unit, which are defined as follows: f t =σ(W f [h t-1 ,x t ]+b f ) i t =σ(W i ×[h t-1 ,x t ]+b i ) the t =σ(W o [h t-1 ,x t ]+b o ) h t =o t ×tanh(C t ) Among them, h t-1 Represents the output at the previous moment, x t represents the input at the current moment, f t is the generated weight vector between 0 and 1, σ represents the activation function sigmoid, W f and b f Respectively represent the weight matrix and bias term of the forget gate, W i 、b i Represent the weight and bias of the input gate respectively, W c 、b c denote the weight and bias of the unit state respectively, represents a new cell state, h t The final output result is W o 、b o are the weight and bias of the output gate respectively.

6. The skeleton behavior recognition method based on the improved CTR-GCN model according to claim 1 is characterized in that: The skeleton dataset is processed into four modal data of joints, bones, joint motions and bone motions as input to the SML-CTR-GCN model.

7. The skeleton behavior recognition method based on the improved CTR-GCN model according to claim 6 is characterized in that: The SML-CTR-GCN model is trained using a multi-stream fusion framework: data from four modalities, namely joints, bones, joint motion and bone motion, are used for independent training to obtain output scores of four branches, which are then fused by weighted summation to determine the behavior prediction category based on the fusion score.

8. The skeleton behavior recognition method based on the improved CTR-GCN model according to claim 7 is characterized in that: The SML-CTR-GCN model uses recognition accuracy as an evaluation indicator when testing.

9. A skeleton behavior recognition device based on an improved CTR-GCN model, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the skeleton behavior recognition method based on the improved CTR-GCN model as described in any one of claims 1-8 is implemented.

10. A storage medium having a program stored thereon, characterized in that: When the program is executed, the skeleton behavior recognition method based on the improved CTR-GCN model as described in any one of claims 1 to 8 is implemented.

Images