An infrared human behavior recognition method based on ViT and ST-GCN

Abstract

The present invention provides an infrared human behavior recognition method based on ViT and ST-GCN, which includes acquiring infrared video data; constructing a Lite-ViTAPose posture estimation module, using the Lite-ViTAPose posture estimation module to extract the human posture of each frame in the infrared video, and processing and converting the extracted human posture to form a skeleton sequence as a network input; wherein the Lite-ViTAPose posture estimation module includes a MobileViT module based on multi-scale feature fusion and a multi-person posture estimator DAs-MPPE based on a residual dual-head attention mechanism; constructing a 2s-ViTGCN behavior estimation module, based on the input skeleton sequence, using 2s-ViTGCN to extract the spatiotemporal features of the skeleton sequence, thereby identifying the movement of the human body; wherein the 2s-ViTGCN behavior estimation module includes a dual-stream neural network, a ViTGCN block and a ViTGCN layer. The present invention combines the Lite-ViTAPose module and the 2s-ViTGCN behavior estimation module for infrared human behavior recognition, effectively overcoming the drawbacks existing in the existing infrared human behavior recognition technology.

Description

Technical Field

[0001] The present invention relates to the technical field of infrared image recognition, and in particular to an infrared human behavior recognition method based on ViT and ST-GCN. Background Art

[0002] With the development of science and technology and the improvement of people's living standards, people's demands for material and spiritual aspects are also increasing. Life safety and convenience have become issues that people pay more and more attention to. At the same time, with the acceleration of the aging process, falls of the elderly have become a danger that cannot be ignored. It is imperative to seek an efficient and accurate detection method to prevent and promptly deal with minor injuries, serious injuries, and even life-threatening situations. In this context, infrared human behavior recognition technology came into being. This technology uses infrared videos collected by sensors to analyze and identify human postures and behaviors, and can be widely used in intelligent monitoring, intelligent security, medical care and other fields.

[0003] Behavior recognition based on visible light has been extensively studied, but there are still deficiencies in actual video surveillance systems. First, visible light video can only work properly during the day and in good weather, and loses its function at night and in bad weather, thus failing to meet the requirements of 24-hour uninterrupted monitoring. Second, visible light video cannot guarantee the privacy of the person being photographed, and there is a certain risk of privacy leakage. In comparison, infrared video can work not only at night, but also in bad weather such as fog, dust, rain and snow, which is more conducive to uninterrupted monitoring day and night. In addition, infrared video only captures the outline information of the person being photographed, which is more conducive to protecting the privacy of the person being photographed. However, due to the manufacturing cost of infrared equipment, early behavior recognition technology was only used in the military field. With the continuous deepening of infrared technology research, it has gradually been applied to the civilian field, but they should mostly belong to video-based behavior recognition methods.

[0004] Compared with the video-based behavior recognition method, the skeleton-based behavior recognition method has low computational consumption and good robustness in the face of complex backgrounds, multi-scale changes and perspective changes. More and more researchers have begun to apply this method to the infrared field. For example, in the field of security protection, through the skeleton-based infrared human behavior recognition technology, the daily movement trajectories of the elderly and disabled can be analyzed 24 hours a day and night, and charts can be drawn, thereby providing more help and attention to the health status of special groups. In addition, in the field of public security monitoring, the skeleton-based infrared human behavior recognition technology is also widely used. For example, by monitoring crowded areas such as shopping malls, schools or airports, abnormal or suspected criminal behaviors can be judged at night or in bad weather, thereby greatly improving social security. At the same time, the skeleton-based infrared human behavior recognition technology can also be used for personal identity authentication. For example, by monitoring human walking posture and body shape, personal identity information can be verified without causing the risk of privacy leakage. Therefore, the skeleton-based infrared human behavior recognition has important research value. However, it also faces many challenges in practical applications. These challenges mainly appear in the acquisition of skeleton data and the extraction of skeleton features. For example, due to the poor quality of infrared images, the accuracy of posture recognition based on infrared images is often low and the scale of the human body changes; in addition, there is also the problem of information being blocked.

[0005] Infrared human behavior recognition is a behavior discrimination technology that uses infrared human images or videos as research objects and extracts infrared human behavior features. Because it is not affected by natural factors such as light, it has gradually become the focus of human behavior recognition technology. At present, infrared human behavior recognition technology basically draws on methods based on visible light, and there are relatively few methods based on infrared. Therefore, infrared-based human behavior methods can also be classified into video-based methods and bone-based methods according to different input sequences. These methods combine the advantages of infrared imaging technology and make up for the shortcomings of visible light methods to a certain extent. For example, in the monitoring system for elderly fall detection, the behavior of the elderly can be monitored 24 hours a day and night, and the relevant medical staff can be quickly informed of the elderly's situation; in the monitoring system of intelligent security, criminal incidents often occur at night or in bad weather. Infrared human behavior recognition technology can monitor the occurrence of criminal incidents in the above environment and alarm in time. Summary of the invention

[0006] The purpose of the present invention is to provide an infrared human behavior recognition method based on ViT and ST-GCN, which combines the Lite-ViTAPose module and the 2s-ViTGCN behavior estimation module to perform infrared human behavior recognition, effectively overcoming the drawbacks of the existing infrared human behavior recognition technology.

[0007] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0008] An infrared human behavior recognition method based on ViT and ST-GCN, the method comprising the following steps:

[0009] Acquire infrared video data;

[0010] Construct a Lite-ViTAPose posture estimation module, use the Lite-ViTAPose posture estimation module to extract the human posture of each frame in the infrared video, process and transform the extracted human posture into a skeleton sequence as the network input; wherein the Lite-ViTAPose posture estimation module includes a MobileViT module based on multi-scale feature fusion and a multi-person posture estimator DAs-MPPE based on the residual dual-head attention mechanism;

[0011] A 2s-ViTGCN behavior estimation module is constructed. Based on the input skeleton sequence, the spatiotemporal features of the skeleton sequence are extracted using 2s-ViTGCN to identify human body movements. The 2s-ViTGCN behavior estimation module includes a two-stream neural network, a ViTGCN block, and a ViTGCN layer.

[0012] According to the infrared human behavior recognition method based on ViT and ST-GCN provided by the present invention, when extracting human posture, a high-resolution and semantically rich feature F∈R is extracted through the MobileViT module. C×W×H , as the input of the multi-person pose estimator DAs-MPPE;

[0013] DAs-MPPE generates L to encode the association between limbs, and then generates S to predict the position of joints; where L = (L1; L2; ...; L c ) represents the PAF of all limbs, which is used to describe the position and direction information of the limbs. c ∈R W×H×2 represents the PAF of limb c, c∈{1...C} represents the index of the limb; S=(S1; S2; ...; S j ) represents the PCM of all joint points, which aims to locate the position information of the joint points. j ∈R W×H represents the PCM of joint point j, j∈{1...J} represents the index of the joint point;

[0014] Combining PAF and PCM, the joint points and limb positions of all people in the infrared video are output through the bipartite graph maximum weight matching algorithm.

[0015] According to an infrared human behavior recognition method based on ViT and ST-GCN provided by the present invention, the MobileViT module includes:

[0016] MobileViT unit, the MobileViT unit includes a MobileViT block and MobileNetV2, the MobileViT block uses fewer parameters to model local and global information, and includes a local information encoding module, a global information encoding module and a feature fusion module, the local information encoding module is used to extract local feature information, the global information encoding module is used to extract global feature information, and the feature fusion module is used to fuse local and global feature information; the MobileNetV2 block is used to downsample the feature map, and adopts a downsampling strategy of first increasing and then decreasing the number of channels.

[0017] The FPN unit includes a bottom-up forward feature extraction process, a top-down upsampling process and a lateral connection fusion process. The bottom-up forward feature extraction process generates a series of feature maps of different scales by using the MobileViT backbone network; the lateral connection fusion process is used to unify the number of channels of deep features and shallow features, and the top-down upsampling process is used to fuse shallow features and upsampled deep features.

[0018] According to an infrared human behavior recognition method based on ViT and ST-GCN provided by the present invention, the multi-person posture estimator DAs-MPPE includes:

[0019] An improved single-branch OpenPose module, the single-branch OpenPose module includes a backbone network, a PAF module and a PCM module, the backbone network is the first 10 layers of the VGG19 convolutional neural network; the PAF module consists of 5 convolution blocks and 2 1×1 convolutions, wherein each convolution block has 3 3×3 convolutions, and the output of each convolution is directly transmitted to the next layer and directly transmitted to the end of the convolution block at the same time; the PCM module has the same network structure as the PAF module, and the number of output channels of the PAF module is twice that of the PCM module;

[0020] The convolution block of OpenPose includes the OpenPose convolution block module and the DAsNet module. The DAsNet module includes the position attention mechanism module, the channel attention module and the residual structure.

[0021] SMU activation function,Replace the original ReLU activation function in DAs-MPPE with the SMU activation function.

[0022] According to an infrared human behavior recognition method based on ViT and ST-GCN provided by the present invention, the dual-stream neural network includes two branches, the first branch is used to extract the spatiotemporal feature information of joint points, and the second branch is used to extract the spatiotemporal feature information of bones. The network structure of each branch is the same, including a BN layer, five adaptive spatiotemporal graph convolution blocks, five ViTGCN blocks and a global average pooling layer; wherein the BN layer is used to normalize the input; the global average pooling layer is used to pool the feature maps of different samples to the same size; J-ViTGCN represents the network structure of the joint point stream, and B-ViTGCN represents the network structure of the bone stream.

[0023] According to an infrared human behavior recognition method based on ViT and ST-GCN provided by the present invention, the joint points close to the center of gravity of the human body are defined as source joint points, and the joint points far from the center of gravity of the human body are defined as target joint points. The joint point information is represented by I n ∈(X,Y,S n ), where X, Y represent the absolute coordinates of the joint point in the image, S n Represents the confidence score of the joint point; the skeleton information is used Indicates that Represents the vector from the source joint point to the target joint point. S e =(S n1 +S n2 ) / 2;

[0024] Among them, the human neck is taken as the central joint point, so that the input dimensions of the joint flow and the bone flow are the same.

[0025] According to an infrared human behavior recognition method based on ViT and ST-GCN provided by the present invention, the ViTGCN block includes a graph convolution module, a temporal convolution module and a residual module, the graph convolution module includes a ViTGCN layer, a BN layer and a ReLU activation function, which is used to extract the features of the skeleton sequence in the spatial dimension; the temporal convolution module includes a convolution layer, a BN layer and a ReLU activation function, which is used to extract the features of the skeleton sequence in the temporal dimension; the residual module fuses the input and output features through a convolution to improve the performance of the network.

[0026] According to an infrared human behavior recognition method based on ViT and ST-GCN provided by the present invention, the ViTGCN layer divides the adjacency matrix of the skeleton graph into adjacency matrix A in the graph convolution of ST-GCN j ∈R N×N , weighted adjacency matrix B j ∈R N ×N and the self-attention adjacency matrix Cj ∈R N×N , where A j and B j The implementation method is consistent with ST-GCN. j The implementation method uses the MobiletViT-v3 block, and the formula is as follows:

[0027] f out =∑ j W j f in (A j +B j +C j )

[0028] in, represents the output feature map, C out Represents the output dimension of the joint point, T represents the number of bone sequences, and N represents the number of joint points; represents the input feature map, C in represents the output dimension of the joint point; j∈{0, 1, 2} refers to the space partitioning strategy of ST-GCN, dividing the skeleton graph into three subsets.

[0029] According to an infrared human behavior recognition method based on ViT and ST-GCN provided by the present invention, MobileViT-v3 blocks are used to implement C j , first replace the 3×3 convolution layer of the MobileViT-v1 block with a 1×1 convolution layer, then fuse the features of the local representation block and the global representation block instead of fusing the input and the global representation block; finally, in the local representation block, replace the regular 3×3 convolution with a depthwise 3×3 convolution.

[0030] According to an infrared human behavior recognition method based on ViT and ST-GCN provided by the present invention, the network structure of the MobileViT-v3 block output is improved to generate an adjacency matrix C j :

[0031] For a given input feature X∈R C×T×N , first get the features through the MobileViT-v3 block And reshape the result into N×C e T, where C e represents the number of channels output by the MobileViT-v3 block; then, we get the feature and reshape the result into C e T×N; then, perform matrix multiplication on the reshaped feature M and the reshaped feature O to obtain S∈R N×N ; Finally, take the matrix S as input and generate the adjacency matrix C through the Softmax functionj .

[0032] It can be seen that, compared with the prior art, the present invention provides an end-to-end infrared human behavior recognition network (ViTGCN-IRNet) based on ViT and ST-GCN, which aims to use infrared video as input to identify the human body's actions in the video. First, input a segment of infrared video, and then use Lite-ViTAPose to extract the human body posture of each frame in the video. Then, these human body postures are further processed and converted to form a skeleton sequence as the input of the next network. Finally, based on this skeleton sequence, 2s-ViTGCN is used to identify the human body's actions.

[0033] In the field of security protection, the infrared human behavior recognition technology based on skeletons can analyze the daily movement trajectories of the elderly and disabled people 24 hours a day and night, and draw charts, so as to provide more help and attention to the health status of special groups. In addition, in the field of public security monitoring, the infrared human behavior recognition technology based on skeletons is also widely used. For example, by monitoring crowded areas such as shopping malls, schools or airports, abnormal or suspected criminal behavior can be judged at night or in bad weather, thereby greatly improving social security. At the same time, the infrared human behavior recognition technology based on skeletons can also be used for personal identity authentication. For example, by monitoring human walking posture and body shape, personal identity information can be verified without causing the risk of privacy leakage. Therefore, the infrared human behavior recognition based on skeletons has important value.

[0034] The present invention also provides an electronic device, comprising:

[0035] a memory storing computer executable instructions;

[0036] a processor configured to execute the computer executable instructions,

[0037] Among them, when the computer executable instructions are executed by the processor, the steps of any one of the above-mentioned infrared human behavior recognition methods based on ViT and ST-GCN are implemented.

[0038] The present invention also provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is used to implement the steps of any one of the above-mentioned infrared human behavior recognition methods based on ViT and ST-GCN.

[0039] It can be seen that the present invention also provides an electronic device and storage medium for an infrared human behavior recognition method based on ViT and ST-GCN, which includes: one or more memories and one or more processors. The memory is used to store program codes and intermediate data generated during program operation, storage of model output results, and storage of models and model parameters; the processor is used for processor resources occupied by code operation and multiple processor resources occupied when training models.

[0040] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flow chart of an embodiment of an infrared human behavior recognition method based on ViT and ST-GCN of the present invention.

[0042] Figure 2 It is a schematic diagram of a Lite-ViTAPose posture estimation module and a 2s-ViTGCN behavior estimation module in an embodiment of an infrared human behavior recognition method based on ViT and ST-GCN of the present invention.

[0043] Figure 3 It is a network structure principle diagram of an infrared human posture estimation module based on lightweight ViT and attention mechanism in an embodiment of an infrared human behavior recognition method based on ViT and ST-GCN of the present invention.

[0044] Figure 4 It is a network structure principle diagram of the MSF-MobileViT module in an embodiment of an infrared human behavior recognition method based on ViT and ST-GCN of the present invention.

[0045] Figure 5 It is a network structure principle diagram of the MobieViT block in an embodiment of an infrared human behavior recognition method based on ViT and ST-GCN of the present invention.

[0046] Figure 6 It is a network structure principle diagram of the MobieNetV2 block in an embodiment of an infrared human behavior recognition method based on ViT and ST-GCN of the present invention.

[0047] Figure 7 It is a network structure principle diagram of a single-branch OpenPose in an embodiment of an infrared human behavior recognition method based on ViT and ST-GCN of the present invention.

[0048] Figure 8 It is a network structure principle diagram of a convolution block based on DAsNet in an embodiment of an infrared human behavior recognition method based on ViT and ST-GCN of the present invention.

[0049] Fig. 9 It is a network structure principle diagram of 2s-ViTGCN in an embodiment of an infrared human behavior recognition method based on ViT and ST-GCN of the present invention.

[0050] Fig.10 It is a network structure principle diagram of the ViTGCN block in an embodiment of an infrared human behavior recognition method based on ViT and ST-GCN of the present invention.

[0051] Fig.11 It is a network structure principle diagram of the ViTGCN layer in an embodiment of an infrared human behavior recognition method based on ViT and ST-GCN of the present invention.

[0052] Fig.12 It is a network structure principle diagram of the MobileViT-v3 block in an embodiment of an infrared human behavior recognition method based on ViT and ST-GCN of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0055] See also Figure 1 and Figure 2 The present invention provides an infrared human behavior recognition method based on ViT and ST-GCN, the method comprising the following steps:

[0056] Step S1, acquiring infrared video data;

[0057] Step S2, constructing a Lite-ViTAPose posture estimation module, using the Lite-ViTAPose posture estimation module to extract the human posture of each frame in the infrared video, and processing and converting the extracted human posture into a skeleton sequence as a network input; wherein the Lite-ViTAPose posture estimation module includes a MobileViT module based on multi-scale feature fusion and a multi-person posture estimator DAs-MPPE based on a residual dual-head attention mechanism;

[0058] Step S3, constructing a 2s-ViTGCN behavior estimation module, based on the input skeleton sequence, using 2s-ViTGCN to extract the spatiotemporal features of the skeleton sequence, so as to identify the human body's movements; wherein the 2s-ViTGCN behavior estimation module includes a two-stream neural network, a ViTGCN block and a ViTGCN layer.

[0059] like Figure 3 As shown, Figure 3 The specific structure of the Lite-ViTAPose pose estimation module is composed of MobileViT based on multi-scale feature fusion (MSF-MobileViT) and Multi-person Pose Estimator based on Residual Dual Attention (DAs-MPPE). When extracting human pose, a high-resolution and semantically rich feature F∈R is extracted through the MobileViT module. C×W×H , as the input of the multi-person pose estimator DAs-MPPE. Secondly, DAs-MPPE generates L to encode the association between limbs, and then generates S to predict the position of joints; where L = (L1; L2; ...; L c ) represents the PAF of all limbs, which is used to describe the position and direction information of the limbs. c ∈R W×H×2 represents the PAF of limb c, c∈{1...C} represents the index of the limb; S=(S1; S2; ...; S j ) represents the PCM of all joint points, which aims to locate the position information of the joint points. represents the PCM of joint point j, j∈{1...J} represents the index of the joint point;

[0060] Finally, PAF and PCM are combined to output the joint points and limb positions of all people in the infrared video through the bipartite graph maximum weight matching algorithm.

[0061] In order to solve the problems of low accuracy and large changes in posture scale caused by poor quality of infrared images in the recognition process, this embodiment designs a MobileViT backbone network based on multi-scale feature fusion, which consists of MobileViT and FPN. Its network structure is as follows: Figure 4 As shown in Figure 2, MobileViT aims to generate a series of multi-scale pyramids with global and local features, and feature pyramid FPN aims to aggregate these multi-scale pyramids.

[0062] CNN has the advantages of spatial inductive bias and low sensitivity to data enhancement, ViT has the advantages of input adaptive weighting and global processing, and MobileViT combines the advantages of CNN and ViT, which can encode local and global information of images with fewer parameters and is a lightweight network model.

[0063] In this embodiment, the MobileViT module includes a MobileViT unit and a FPN unit:

[0064] The MobileViT unit includes the MobileViT block and MobileNetV2. The MobileViT block uses fewer parameters to model local and global information. It includes a local information encoding module, a global information encoding module and a feature fusion module. The local information encoding module is used to extract local feature information, the global information encoding module is used to extract global feature information, and the feature fusion module is used to fuse local and global feature information. Its network structure is as follows: Figure 5 As shown in the figure, in order to improve the detection accuracy, the MobileNetV2 block is used to downsample the feature map, and the downsampling strategy of first increasing and then decreasing the number of channels is adopted. However, too few channels will lead to insufficient feature information. Therefore, MobileNetV2 makes a trade-off between the amount of computation and feature expression, while achieving high detection accuracy, it also maintains a relatively small amount of computation. Its network structure is shown in the figure. Figure 6 shown.

[0065] Specifically, for a given infrared image X∈R C×H×W , the MobileViT block first encodes local information through 3×3 standard convolution and projects the input channel C into the high-dimensional space d using 1×1 point-wise convolution to obtain X L ∈R d×H×W Secondly, in order to learn global features with spatial inductive bias, X is folded by L Transform to X U ∈R d×P×N , through Transformer encoding each patch feature X G ∈R d×P×N , through the fold operation, X GTransform to X F ∈R d×H×H , where P = wh, N represents the number of patches in the image, w and h represent the width and height of the patch, respectively. Then, the input channel d is projected into the low-dimensional space C through 1×1 point-by-point convolution, and the result is concatenated with the input X. Finally, the local and global features are fused through 3×3 standard convolution, as shown in formula (1):

[0066] X G (p) = Transformer(X U (p)), 1≤p≤P (1)

[0067] In neural networks, the network structure from shallow to deep can extract features at different levels. Shallow features can capture detailed information in the image, such as edges, pixels, and colors; while deep features have richer semantic information. FPN integrates deep and shallow features, which makes it have a good detection effect on multi-scale targets. Its network structure is as follows Figure 4 As shown. In this embodiment, the FPN unit includes a bottom-up forward feature extraction process, a top-down upsampling process, and a lateral connection fusion process. The bottom-up forward feature extraction process generates a series of feature maps of different scales by using the MobileViT backbone network; the lateral connection fusion process is used to unify the number of channels of deep features and shallow features, and the top-down upsampling process is used to fuse shallow features and upsampled deep features, thereby injecting high semantic information into the shallow feature map, thereby improving the semantic information of the shallow feature map. Figure 4 As shown in the figure, FPN first outputs the feature map of each MobileViT block, which are {C3, C4, C5}, respectively. Their resolutions are {1 / 8, 1 / 16, 1 / 32} of the input image resolution, and the downsampling multiples are {8, 16, 32} of the original image. Then, a 1×1 convolution is used to unify the number of channels of {C3, C4, C5} to 256, and {P3, P4, P5} is obtained. Finally, the upsampled deep feature map is added element by element to the shallow feature map, and the deep semantic information is injected into the shallow feature map through a 3×3 convolution to make up for the lack of semantic information of the shallow feature.

[0068] In order to solve the problem of human body occlusion during the recognition process, this embodiment designs a multi-person pose estimator based on the residual dual-head attention mechanism, which is improved on the basis of the single-branch OpenPose. Figure 3As shown in the figure, DAs-MPPE first abandons the backbone network of the single-branch OpenPose. Then, a residual dual-head attention mechanism network (DAsNet) is attached to the convolutional block of the single-branch OpenPose to make the model pay more attention to the occluded joints. Finally, the SMU activation function is used to improve the performance of the network.

[0069] Specifically, the multi-person pose estimator DAs-MPPE based on the residual dual-head attention mechanism includes an improved single-branch OpenPose module, an OpenPose convolutional block, and a SMU activation function.

[0070] The single-branch OpenPose first refines the PAF in multi-stage cascades, and then refines the PCM in multi-stage cascades, which greatly reduces the calculation. In addition, the single-branch OpenPose replaces the 7×7 convolution kernel with 3 consecutive 3×3 convolution kernels, and connects the 3 consecutive convolution kernels according to the structure of DenseNet, which not only reduces the amount of calculation while retaining the receptive field, but also can integrate high-level and low-level features. Its network structure is as follows Figure 7 shown.

[0071] Among them, the single-branch OpenPose module includes a backbone network, a PAF module, and a PCM module. The backbone network is the first 10 layers of the VGG19 convolutional neural network; the PAF module consists of 5 convolution blocks and 2 1×1 convolutions, where each convolution block has 3 3×3 convolutions, and the output of each convolution is directly transmitted to the next layer and directly transmitted to the end of the convolution block at the same time; the PCM module has the same network structure as the PAF module, and the number of output channels of the PAF module is twice that of the PCM module. Specifically, the output feature map of the backbone network is first used as the input of the first stage to obtain the PAF of the first stage. In each subsequent stage of the PAF, the prediction of the previous stage and the output feature map of the backbone network are spliced ​​together, and the results of the PAF are continuously refined, so as to more accurately predict the skeleton information. Then, the output feature map of the backbone network and the output feature map of the last stage of PAF are used as the input of the first stage PCM to obtain the first stage PCM. In each subsequent stage of PCM, the prediction of the previous stage, the output feature map of the backbone network and the feature map of the last stage of PAF are spliced ​​together to continuously refine the results of PCM, thereby more accurately predicting the joint point information. The above operations can be expressed as formulas (2)-(5):

[0072] L 1 =φ 1 (F) (2)

[0073]

[0074]

[0075]

[0076] Where: φ t and ρ t T represents the CNN of the t-th stage reasoning; P Indicates the total number of PAF stages; T C Indicates the total number of PCM stages; L t represents the PAF of the tth stage; S t Represents the PCM of the tth stage.

[0077] The convolutional block of OpenPose is as follows Figure 7 As shown in the figure, it has two problems: one is that it is difficult to learn the global contextual relationship, and the other is that it cannot use the relevant feature information to further refine the key point position. Therefore, the present invention adds DAsNet to the OpenPose convolutional block. By modeling the semantic dependency relationship in the channel dimension and the spatial dimension, it can not only better describe the spatial and channel-level relationship between pixels, but also help improve the focusing effect, so that the model has stronger perception and expression capabilities, thereby recovering the occluded body parts. Its network structure is as follows Figure 8 As shown in the figure. The convolution block includes the OpenPose convolution block module and the DAsNet module. The DAsNet module includes the Position Attention Model (PAM), the Channel Attention Model (CAM) and the residual structure. Among them, the PAM module can more extensively use context information to encode local features and improve the representation of local features; the CAM module can enable the network to adaptively focus on certain feature channels to improve the feature expression ability of specific semantics; the residual network can reduce information duplication while maintaining high-dimensional feature representation, alleviate the gradient diffusion problem, and increase the feature expression ability and stability of the model.

[0078] Specifically, the OpenPose convolutional block first obtains the input features A of PAM through two 3×3 convolutions. p ∈R C ×H×W and CAM input feature A c ∈R C×H×W , where C represents the number of channels, H represents the height of the image, and W represents the width of the image. p First, three 1×1 convolutions are used to obtain the features B∈R C×H×W , C∈R C×H×W and D∈R C×H×W Then the reshaped and transposed features B∈R N×C With the reshaped feature C∈RC×N Perform matrix multiplication to obtain the spatial attention matrix S∈R N×N , where N = H × W represents the number of pixels in the image. Then the reshaped features Perform matrix multiplication with S to obtain the attention feature O p ∈R N×N , and reshape the result into C×H×W. Finally, O p Multiply the scale parameter α and feature A p Perform element-by-element summation to obtain the final output E∈R C×H×W , as shown in formulas (6), (7) and (8). For CAM, firstly transform the reshaped A c ∈R C×N With input feature A c Perform matrix multiplication to obtain the channel attention matrix X∈R C×C Then X and the reshaped A c ∈R C×N Perform matrix multiplication to get the attention feature O c ∈R C×N , and reshape the result into C×H×W. Finally, O c Multiply by the scale parameter β and add to A c Perform element-by-element summation to obtain the final output E c ∈R C×H×W , as shown in equations (9), (10) and (11). Finally, the features of PAM, CAM and OpenPose convolutional blocks are fused through the residual network. The above operations can be expressed as equations (6)-(11):

[0079] S=θ(B) T θ(C) (6)

[0080] O p =θ(D)S (7)

[0081] E p =αθ(O p )+B p (8)

[0082] X=A c θ(A c ) (9)

[0083] O c =θ(A c )S (10)

[0084] E c =βθ(O c )+A c (11)

[0085] Among them: α and β are initialized to 0 and gradually learn to assign more weights; S represents the spatial attention matrix; X represents the channel attention matrix; O p and O c represents attention features; B, C and D represent A p Feature matrix after convolution; θ(.) represents the Reshape function.

[0086] In this embodiment, the SMU activation function is a smooth approximation function of ReLU, which enables the neural network to capture more information; it can effectively solve the gradient vanishing and gradient exploding problems; it can improve the training speed of the neural network, and at the same time improve the generalization performance of the neural network. Therefore, in this embodiment, the original ReLU activation function in DAs-MPPE is replaced by the SMU activation function, and the calculation formula of the SMU activation function is as shown in formulas (12)-(15):

[0087]

[0088]

[0089]

[0090]

[0091] In order to extract the spatiotemporal features of the skeleton sequence, this embodiment designs a spatiotemporal graph convolutional network based on a lightweight ViT, and its network structure is as follows: Fig. 9 It can not only learn the adjacency matrix of the skeleton graph adaptively in an end-to-end manner, but also capture the long-term and short-term dependencies of joint points in the spatiotemporal dimensions, thereby improving the flexibility and performance of the model.

[0092] In this embodiment, the two-stream neural network is a flexible, robust, high-precision and interpretable neural network model, which has broad application prospects in the field of video-based behavior recognition. It can obtain more comprehensive and accurate behavior features by simultaneously utilizing information from two modalities. Fig. 9 As shown in the figure, the two-stream neural network includes two branches. The first branch is used to extract the spatiotemporal feature information of joints, and the second branch is used to extract the spatiotemporal feature information of bones. The network structure of each branch is the same, including a BN layer, five adaptive spatiotemporal graph convolution blocks, five ViTGCN blocks and a global average pooling layer (Global Average Pooled, GAP); among them, the BN layer is used to normalize the input; the global average pooling layer is used to pool the feature maps of different samples to the same size; J-ViTGCN represents the network structure of the joint point stream, and B-ViTGCN represents the network structure of the bone stream.

[0093] In this embodiment, the joint points close to the center of gravity of the human body are defined as source joint points, and the joint points far from the center of gravity of the human body are defined as target joint points. n ∈(X,Y,S n ), where X, Y represent the absolute coordinates of the joint point in the image, S n Represents the confidence score of the joint point; the skeleton information is used Indicates that Represents the vector from the source joint point to the target joint point. S e =(S n1 +S n2 ) / 2. In this embodiment, the human neck is used as the central joint point, so that the input dimensions of the joint stream and the bone stream are the same, thereby simplifying the design of the network.

[0094] Specifically, the pose estimation result of Lite-ViTAPose is converted into I n and I e Secondly, the joint point flow is based on I n As input, J-ViTGCN is used to obtain the spatiotemporal feature information of the joint points; the skeleton flow is based on I e As input, B-ViTGCN is used to obtain the spatiotemporal feature information of the skeleton. Finally, the Softmax scores of the two streams are added to obtain the fused score and predict the action classification label.

[0095] In this embodiment, the network structure of the ViTGCN block is as follows: Fig.10 As shown in the figure, the ViTGCN block includes a graph convolution module, a temporal convolution module and a residual module. The graph convolution module includes a ViTGCN layer, a BN layer and a ReLU activation function, which is used to extract the features of the skeleton sequence in the spatial dimension; the temporal convolution module includes a convolution layer, a BN layer and a ReLU activation function, which is used to extract the features of the skeleton sequence in the temporal dimension; the residual module fuses the input and output features through a convolution to improve the performance of the network.

[0096] Specifically, given an input feature map Where T in represents the length of the input sequence, N represents the number of nodes, C in Represents the number of channels of the input node. First, the spatial feature map of the skeleton sequence is calculated by formula (16): Among them C out Represents the number of channels of the spatial feature map. Secondly, the obtained spatial feature map is input into the Dropout network to prevent overfitting. Then, the temporal feature map of the skeleton sequence is calculated by formula (18): Among them, T out Indicates the sequence length of the temporal feature map. Finally, the temporal feature map and the input feature map are fused through a residual connection to obtain the output feature map

[0097] F1=σ(BN(G * (X))) (16)

[0098] F2=Droutout(F1) (17)

[0099] F3=σ(BN(T * (F2))) (18)

[0100] Y=F3+Conv(X) (19)

[0101] Where: G * represents the ViTGCN layer; T * represents one-dimensional convolution; BN(·) represents batch normalization function; σ(·) represents ReLU activation function.

[0102] In order to adaptively learn different adjacency matrices in different GCN layers, the present invention designs a spatiotemporal graph convolutional network layer based on lightweight ViT, which is improved on the basis of ST-GCN. Its network structure is as follows: Fig.11 As shown, the ViTGCN layer divides the adjacency matrix of the skeleton graph into the adjacency matrix A in the graph convolution of ST-GCN. j ∈R N×N , weighted adjacency matrix B j ∈R N×N and the self-attention adjacency matrix C j ∈R N×N , where A j and B j The implementation method is consistent with ST-GCN. j The implementation method uses the MobiletViT-v3 block, as shown in formula (20):

[0103] f out =∑ j W j f in (A j +B j +C j ) (20)

[0104] in, represents the output feature map, C out Represents the output dimension of the joint point, T represents the number of bone sequences, and N represents the number of joint points; represents the input feature map, C inrepresents the output dimension of the joint point; j∈(0, 1, 2} refers to the space partitioning strategy of ST-GCN, dividing the skeleton graph into three subsets.

[0105] In this embodiment, B j It can not only represent the connection relationship of joints, but also the strength of the connection. In order to make the model adaptively learn the adjacency matrix of the skeleton graph and capture the long-term and short-term dependencies of joints in the spatiotemporal dimension, this embodiment uses the MobileViT-v3 block to implement C j , the network structure of MobileViT-v3 block is as follows Fig.12 As shown in Figure 2. The MobileViT-v3 block is an improvement on the MobileViT-v1 block. First, the 3×3 convolution layer of the MobileViT-v1 block is replaced with a 1×1 convolution layer. Then, the features of the local representation block and the global representation block are fused together instead of fusing the input and the global representation block together. Finally, in the local representation block, the regular 3×3 convolution is replaced with a depthwise 3×3 convolution. These changes result in the MobileViT-v3 block having fewer parameters and computational overhead than the MobileViT-v1 block.

[0106] The reasoning process of the MobileViT-v3 block is the same as that of the MobileViT-v1 block. The input is divided into several patches, and the height and width of each patch are represented by h and w respectively. In addition, the Transformer inside the MobileViT-v3 block can also enable the model to capture the long-term and short-term dependencies in the skeleton sequence, thereby further improving the performance of the model.

[0107] In this embodiment, the network structure output by the MobileViT-v3 block is also improved to generate the adjacency matrix C j :

[0108] like Fig.12 Specifically, for a given input feature X∈R C×T×N , first get the features through the MobileViT-v3 block And reshape the result into N×C e T, where C e represents the number of channels output by the MobileViT-v3 block; then, we get the feature and reshape the result into C e T×N; then, perform matrix multiplication on the reshaped feature M and the reshaped feature O to obtain S∈R N×N ; Finally, take the matrix S as input and generate the adjacency matrix C through the Softmax function j , expressed as formula (21)-(24):

[0109] M=MViT3(X) (21)

[0110] O=Conv(X) (22)

[0111] S=θ(M)θ(O) (23)

[0112] C j =Softmax(S) (24)

[0113] Where: θ(.) represents the Reshape function.

[0114] In summary, this embodiment provides an end-to-end infrared human behavior recognition network (ViTGCN-IRNet) based on ViT and ST-GCN, which aims to use infrared video as input to identify the human body's actions in the video. First, input a segment of infrared video, and then use Lite-ViTAPose to extract the human body posture of each frame in the video. Then, these human body postures are further processed and converted to form a skeleton sequence as the input of the next network. Finally, based on this skeleton sequence, 2s-ViTGCN is used to identify the human body's actions.

[0115] In the field of security protection, the infrared human behavior recognition technology based on skeletons can analyze the daily movement trajectories of the elderly and disabled people 24 hours a day and night, and draw charts, so as to provide more help and attention to the health status of special groups. In addition, in the field of public security monitoring, the infrared human behavior recognition technology based on skeletons is also widely used. For example, by monitoring crowded areas such as shopping malls, schools or airports, abnormal or suspected criminal behavior can be judged at night or in bad weather, thereby greatly improving social security. At the same time, the infrared human behavior recognition technology based on skeletons can also be used for personal identity authentication. For example, by monitoring human walking posture and body shape, personal identity information can be verified without causing the risk of privacy leakage. Therefore, the infrared human behavior recognition based on skeletons has important value.

[0116] In one embodiment, an electronic device is provided, which may be a server. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store data. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an infrared human behavior recognition method based on ViT and ST-GCN is implemented.

[0117] Those skilled in the art will understand that the electronic device structure shown in this embodiment is only a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than those shown in this embodiment, or combine certain components, or have a different arrangement of components.

[0118] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0119] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0120] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0121] It can be seen that this embodiment also provides an electronic device and a storage medium for an infrared human behavior recognition method based on ViT and ST-GCN, which includes: one or more memories and one or more processors. The memory is used to store program codes and intermediate data generated during program operation, storage of model output results, and storage of models and model parameters; the processor is used for processor resources occupied by code operation and multiple processor resources occupied when training models.

[0122] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] The above-mentioned embodiments are only preferred embodiments of the present invention and cannot be used to limit the scope of protection of the present invention. Any non-substantial changes and substitutions made by technicians in this field on the basis of the present invention shall fall within the scope of protection required by the present invention.

Claims

1. An infrared human behavior recognition method based on ViT and ST-GCN, characterized in that: The method comprises the following steps: Acquire infrared video data; Construct a Lite-ViTAPose posture estimation module, use the Lite-ViTAPose posture estimation module to extract the human posture of each frame in the infrared video, process and transform the extracted human posture into a skeleton sequence as the network input; wherein the Lite-ViTAPose posture estimation module includes a MobileViT module based on multi-scale feature fusion and a multi-person posture estimator DAs-MPPE based on the residual dual-head attention mechanism; Wherein, the MobileViT module includes: MobileViT unit, the MobileViT unit includes a MobileViT block and MobileNetV2, the MobileViT block uses fewer parameters to model local and global information, and includes a local information encoding module, a global information encoding module and a feature fusion module, the local information encoding module is used to extract local feature information, the global information encoding module is used to extract global feature information, and the feature fusion module is used to fuse local and global feature information; the MobileNetV2 block is used to downsample the feature map, and adopts a downsampling strategy of first increasing and then decreasing the number of channels; An FPN unit, wherein the FPN unit includes a bottom-up forward feature extraction process, a top-down upsampling process, and a lateral connection fusion process, wherein the bottom-up forward feature extraction process generates a series of feature maps of different scales by using a MobileViT backbone network; the lateral connection fusion process is used to unify the number of channels of deep features and shallow features, and the top-down upsampling process is used to fuse shallow features and upsampled deep features; Constructing a 2s-ViTGCN behavior estimation module, based on the input skeleton sequence, using 2s-ViTGCN to extract the spatiotemporal features of the skeleton sequence, so as to recognize the human body's actions; wherein the 2s-ViTGCN behavior estimation module includes a two-stream neural network, a ViTGCN block and a ViTGCN layer; The ViTGCN block includes a graph convolution module, a temporal convolution module and a residual module. The graph convolution module includes a ViTGCN layer, a BN layer and a ReLU activation function, which is used to extract the features of the bone sequence in the spatial dimension; the temporal convolution module includes a convolution layer, a BN layer and a ReLU activation function, which is used to extract the features of the bone sequence in the temporal dimension; the residual module fuses the input and output features through a convolution to improve the performance of the network; The ViTGCN layer divides the adjacency matrix of the skeleton graph into adjacency matrix A in the graph convolution of ST-GCN. j ∈R N×N , weighted adjacency matrix B j ∈R N×N and the self-attention adjacency matrix C j ∈R N×N .

2. The method according to claim 1, characterized in that: When extracting human posture, a high-resolution and semantically rich feature F∈R is extracted through the MobileViT module. C×W×H , as the input of the multi-person pose estimator DAs-MPPE; DAs-MPPE generates L to encode the association between limbs, and then generates S to predict the position of joints; where L = (L1; L2; ...; L c ) represents the PAF of all limbs, which is used to describe the position and direction information of the limbs. c ∈R W×H×2 represents the PAF of limb c, c∈{1...C} represents the index of the limb; S=(S1; S2; ...; S j ) represents the PCM of all joint points, which aims to locate the position information of the joint points. j ∈R W×H represents the PCM of joint point j, j∈{1...J} represents the index of the joint point; Wherein, the multi-person posture estimator DAs-MPPE comprises: An improved single-branch OpenPose module, the single-branch OpenPose module includes a backbone network, a PAF module and a PCM module, the backbone network is the first 10 layers of the VGG19 convolutional neural network; the PAF module consists of 5 convolution blocks and 2 1×1 convolutions, wherein each convolution block has 3 3×3 convolutions, and the output of each convolution is directly transmitted to the next layer and directly transmitted to the end of the convolution block at the same time; the PCM module has the same network structure as the PAF module, and the number of output channels of the PAF module is twice that of the PCM module; The convolution block of OpenPose includes the OpenPose convolution block module and the DAsNet module. The DAsNet module includes the position attention mechanism module, the channel attention module and the residual structure. SMU activation function, replace the original ReLU activation function in DAs-MPPE with the SMU activation function; Combining the PAF module with the PCM module, the joint points and limb positions of all people in the infrared video are output through the bipartite graph maximum weight matching algorithm.

3. The method according to claim 1, characterized in that: The dual-stream neural network includes two branches, the first branch is used to extract the spatiotemporal feature information of the joint points, and the second branch is used to extract the spatiotemporal feature information of the skeleton. The network structure of each branch is the same, including a BN layer, five adaptive spatiotemporal graph convolution blocks, five ViTGCN blocks and a global average pooling layer; wherein the BN layer is used to normalize the input; the global average pooling layer is used to pool the feature maps of different samples to the same size; J-ViTGCN represents the network structure of the joint point stream, and B-ViTGCN represents the network structure of the skeleton stream; Among them, the posture estimation result of Lite-ViTAPose is converted into I n and I e ; Joint point flow is I n As input, J-ViTGCN is used to obtain the spatiotemporal feature information of the joint points; the skeleton flow is based on I e As input, B-ViTGCN is used to obtain the spatiotemporal feature information of the skeleton; finally, the Softmax scores of the two streams are added to obtain the fused score and predict the action classification label.

4. The method according to claim 3, characterized in that: The joint points close to the center of gravity of the human body are defined as source joint points, and the joint points far from the center of gravity of the human body are defined as target joint points. The joint point information is represented by I n ∈(X,Y,S n ), where X, Y represent the absolute coordinates of the joint point in the image, S n Represents the confidence score of the joint point; the skeleton information is used Indicates that Represents the vector from the source joint point to the target joint point. Among them, the human neck is taken as the central joint point, so that the input dimensions of the joint flow and the bone flow are the same.

5. The method according to claim 1, characterized in that: A j and B j The implementation method is consistent with ST-GCN. j The implementation method uses the MobiletViT-v3 block, and the formula is as follows: f out =Σ j W j f in (A j +B j +C j ) in, represents the output feature map, C out Represents the output dimension of the joint point, T represents the number of bone sequences, and N represents the number of joint points; represents the input feature map, C in Represents the output dimension of the joint point; j∈{0,1,2} refers to the space partitioning strategy of ST-GCN, dividing the skeleton graph into three subsets.

6. The method according to claim 5, characterized in that: Use MobileViT-v3 blocks to implement C j , first replace the 3×3 convolution layer of the MobileViT-v1 block with a 1×1 convolution layer, then fuse the features of the local representation block and the global representation block instead of fusing the input and the global representation block; finally, in the local representation block, replace the regular 3×3 convolution with a depthwise 3×3 convolution.

7. The method according to claim 6, characterized in that: The network structure output by the MobileViT-v3 block is improved to generate the adjacency matrix C j : For a given input feature X∈R C×T×N , first get the features through the MobileViT-v3 block And reshape the result into N×C e T, where C e represents the number of channels output by the MobileViT-v3 block; then, we get the feature and reshape the result into C e T×N; then, perform matrix multiplication on the reshaped feature M and the reshaped feature O to obtain S∈R N×N ; Finally, take the matrix S as input and generate the adjacency matrix C through the Softmax function j .

Images