A deep learning-based image personnel action analysis method

Abstract

The application relates to the field of personnel action analysis, and specifically provides an image personnel action analysis method based on deep learning, which is realized by adopting a posture perception network, the posture perception network comprises a posture branch residual module, a posture coding module and a posture interaction module; the method comprises the following steps: acquiring a first-view image shot by a first camera, processing the first-view image, inputting the first-view image into the posture branch residual module, and obtaining a mixed posture feature; acquiring a second-view image shot by a second camera, inputting the second-view image into the posture coding module, and obtaining posture coding features output by the posture coding module; fusing the mixed posture feature and the posture coding features, inputting the mixed posture feature and the posture coding features into the posture interaction module for interaction, and obtaining posture interaction features; and inputting the posture interaction features into a human action prediction module, detecting human actions, and identifying abnormal behaviors of personnel. The application can improve the detection precision of personnel behavior abnormalities.

Description

Technical Field

[0001] The present invention relates to the field of personnel motion analysis, and in particular to an image personnel motion analysis method based on deep learning. Background Art

[0002] Abnormal human action recognition is a natural extension of the image action behavior field to the video field. Currently, the accuracy of deep learning algorithms in image action classification has exceeded the level of ordinary people. However, the progress of deep learning in the field of abnormal action recognition is not as significant as in the field of image action classification.

[0003] At present, the application of abnormal human behavior is very widespread. For example, by installing abnormal behavior recognition alarm cameras in the monitoring area, banks can effectively obtain image data for timely and automatic analysis. Once an abnormal situation is found, relevant personnel can be notified in a timely and rapid manner to avoid the occurrence of some extreme situations and ensure the property safety of banks and people.

[0004] Most of the current abnormal behavior detection systems use a single camera to complete anomaly detection. For example, in public places such as banks, campuses, and Internet cafes, the performance of such detection methods is affected by challenging situations such as occlusion and tilted perspectives, and the detection accuracy is not high. Summary of the Invention

[0005] The purpose of the present invention is to overcome the defects of the prior art and provide an image personnel motion analysis method based on deep learning to improve detection accuracy.

[0006] The present invention is achieved in that:

[0007] A method for analyzing human motion in images based on deep learning is implemented using a posture perception network, wherein the posture perception network includes a posture branch residual module, a posture encoding module, and a posture interaction module; the method includes:

[0008] Acquire a first-perspective image captured by a first camera, process the first-perspective image, input the processed first-perspective image into a posture branch residual module, and obtain a mixed posture feature output by the posture branch residual module;

[0009] Obtaining a second-viewpoint image captured by a second camera, inputting the second-viewpoint image into a posture encoding module, and obtaining posture encoding features output by the posture encoding module; wherein the first camera and the second camera are used to capture the person at different angles, and the second-viewpoint image has the same resolution as the first-viewpoint image;

[0010] fusing the mixed posture feature and the posture coding feature, and inputting the resultant feature into a posture interaction module for interaction, thereby obtaining a posture interaction feature output by the posture interaction module;

[0011] The gesture interaction features are input into a human motion prediction module to detect human motions and identify abnormal behaviors of people.

[0012] Optionally, the first perspective image is processed as follows:

[0013] The image is downscaled using two convolutional layers with 3×3 and 5×5 kernels, respectively. The stride of the 3×3 convolutional layer is 2 and the patch size is 1, while the stride of the 5×5 convolutional layer is 3 and the patch size is 2.

[0014] Optionally, the posture branch residual module includes two branches in parallel, and the output ends of the two branches are summed, wherein one branch includes a convolution layer with a convolution kernel of 3×3, a BN layer and a Relu activation function cascaded in sequence, and the other branch includes a convolution layer with a convolution kernel of 5×5, a BN layer and a Relu activation function cascaded in sequence.

[0015] Optionally, the posture encoding module includes a ResNet18 network, and the ResNet18 network performs feature extraction on the second-view image to obtain an overall image resolution feature of the second-view image as the posture encoding feature.

[0016] Optionally, the gesture interaction module includes a space-channel perception interaction submodule and a depth-separable multi-layer perception submodule;

[0017] The space-channel perception interaction unit submodule includes a channel attention unit and a space attention unit;

[0018] The depth-separable multi-layer perception submodule includes a LayerNorm layer, a depth-separable 3×3 convolutional layer and a GELU activation function that are cascaded in sequence.

[0019] Optionally, in the gesture interaction module, a 1×1 convolution operation is performed on the input X to obtain a query sequence Q, a keyword sequence K, and a value sequence V;

[0020] Performing a dot product calculation on the query sequence Q and the transposed keyword sequence K to obtain a spatial perception feature N;

[0021] Perform matrix multiplication of the value sequence V and the channel attention unit to obtain the result S′;

[0022] Perform a dimension conversion operation on S′ to obtain S, and calculate S with the spatial perception feature N to obtain the channel perception interaction feature M; where M = Softmax(N)×S;

[0023] Performing a dimension conversion operation on the channel perception interaction feature M to obtain M', and performing matrix multiplication of M' and the spatial attention unit to obtain the spatial perception interaction feature W;

[0024] The spatial perception interaction feature W and the channel perception interaction feature M are positionally added to obtain a spatial-channel perception interaction attention value;

[0025] The space-channel perception interaction attention value is input into the depth-separable multi-layer perception submodule, and the features of different channels and spaces of the space-channel perception interaction feature are processed to obtain the posture interaction feature.

[0026] Optionally, the mathematical expression of the channel attention unit is as follows:

[0027]

[0028]

[0029] y1=fc1(y s_averag )

[0030] y2=fc2(GELU(BN(y1)))

[0031] y c =soft(y2)

[0032]

[0033] Among them, fc3 represents depth-separable 3×3 convolution, average represents average pooling layer, fc1 represents 1×1 convolution layer, fc2 represents 1×1 convolution, BN represents BN layer, GELU represents activation function, and Soft represents sigmoid function. It is the dot multiplication sign.

[0034] Optionally, the mathematical expression of the spatial attention unit is as follows:

[0035]

[0036]

[0037]

[0038] Among them, fc4 represents a 1×1 convolution layer, fc5 represents a 1×1 convolution, BN represents a BN layer, GELU represents the activation function, s represents the sigmoid function, and · is the corresponding element multiplication sign.

[0039] Optionally, during the training process of the posture perception network, the human posture estimation module uses mean square error loss to calculate the predicted human posture skeleton features and the true value of the human posture skeleton to obtain Loss2;

[0040] The human motion prediction module uses cross entropy loss to calculate the detected human motion and the true value of the human motion to obtain Loss1;

[0041] Use the following formula to perform reverse gradient backpropagation to complete the training process

[0042] Loss=pLoss1+dLoss2

[0043] Where p and d represent coefficients.

[0044] The present invention has the following beneficial effects:

[0045] 1. The present invention adopts a dual-view method to solve the occlusion and angle problems existing in the current single-view detection method. Two cameras at different angles are used to detect the same user. The picture taken by one camera is used to fuse local and global posture information using a posture branch residual module. This module can effectively avoid the generation of ambiguous postures. The picture taken by the other camera is used to obtain posture coding features using a posture coding module. The accuracy of occluded joint points is effectively improved through the second-view image, and then both are provided to the spatial-channel perception interactive attention to improve the joint point recognition accuracy of the model.

[0046] 2. To enhance the attention unit (e.g. Figure 4 In order to enhance the modeling capability of the attention unit and the local channel and spatial dimensions, the present invention proposes spatial-channel-aware interactive attention to enhance the modeling capability of the attention unit and the local channel and spatial dimensions. The information in the depthwise separable convolution 3×3 convolution branch flows to the other branch through channel interaction, which enhances the modeling capability in the channel dimension. At the same time, the spatial attention unit enables spatial relationships to flow from one attention unit branch to another. Therefore, the two-way interaction proposed by the present invention provides complementary information to each other.

[0047] 3. Auxiliary loss supervision is used to predict human pose joints, assisting in behavior prediction and making abnormal behavior detection more accurate. By constraining the human pose loss function, the accuracy of human joints is effectively improved. The auxiliary loss function training strategy increases the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 A structural diagram of a posture perception network provided by an embodiment of the present invention;

[0050] Figure 2 A flowchart of a method for analyzing human motion in images based on deep learning provided by an embodiment of the present invention;

[0051] Figure 3 A structural diagram of a posture branch residual module provided in an embodiment of the present invention;

[0052] Figure 4 A schematic diagram of spatial-channel-aware interactive attention provided by an embodiment of the present invention;

[0053] Figure 5 Schematic diagram of the Dep-MLP submodule of the depth-separable multi-layer perception provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0055] The embodiment of the present invention provides a method for analyzing human motion in images based on deep learning. The method is implemented using a posture perception network. The block diagram of the posture perception network is as follows: Figure 1 As shown in the figure, the network mainly consists of three parts: posture branch residual module, posture encoding module and posture interaction module.

[0056] Please refer to Figure 2 The image personnel action analysis method based on deep learning includes the following steps:

[0057] S101, obtaining a first-view image captured by a first camera, processing the first-view image, inputting the processed first-view image into a posture branch residual module, and obtaining a mixed posture feature output by the posture branch residual module;

[0058] S102, obtaining a second-viewpoint image captured by a second camera, inputting the second-viewpoint image into a posture encoding module, and obtaining posture encoding features output by the posture encoding module; wherein the first camera and the second camera are used to capture the person at different angles, and the second-viewpoint image has the same resolution as the first-viewpoint image;

[0059] S103, fusing the mixed posture feature and the posture coding feature, and then inputting the mixed posture feature and the posture coding feature into a posture interaction module for interaction, thereby obtaining a posture interaction feature output by the posture interaction module;

[0060] S104: Input the gesture interaction features into a human motion prediction module to detect human motions and identify abnormal behaviors of people.

[0061] The present invention utilizes two cameras set at different angles indoors to perform dual-camera photography on people in the room, which can improve the accuracy of detecting abnormal behavior of people.

[0062] In the present invention, Figure 1 As shown, the first-perspective image is processed as follows: the image is reduced using two convolution layers with convolution kernels of 3×3 and 5×5, respectively, which are cascaded in sequence. The step size of the 3×3 convolution layer is 2 and the patch size is 1, and the step size of the 5×5 convolution layer is 3 and the patch size is 2. For example, the size of the first-perspective image and the second-perspective image is [H, W], and the size of the reduced image is [H / 8, W / 8]. The present invention processes the first-perspective image using convolution kernels of 3×3 and 5×5. After the image is processed through multiple convolutions, as much associated information of the image as possible can be obtained. Because the receptive field of the feature map becomes larger after multiple convolution operations, it is conducive to extracting edge information, providing a favorable basis for accurately locating joint points for human posture estimation.

[0063] like Figure 3As shown in the figure, the pose inception-residual module consists of two parallel branches, whose outputs are summed. One branch consists of a cascade of 3×3 convolutional layers, batch normalization (BN) layers, and Relu activation functions, while the other branch consists of a cascade of 5×5 convolutional layers, batch normalization (BN) layers, and Relu activation functions. The first-person view image, reduced to [H / 8, W / 8], is fed into the 3×3 and 5×5 convolutions, respectively. The purpose of the 3×3 convolution is to extract local spatial information, which facilitates accurate joint location. The purpose of the 5×5 convolution is to obtain global information, increase the receptive field, and obtain global contextual information, which is more conducive to joint classification and avoids ambiguous pose generation.

[0064] The posture encoding module is Figure 1 The CNN shown includes a ResNet18 network, which extracts features from the second-view image to obtain overall image resolution features for the second-view image, which serve as the pose encoding features. The feature map output by the ResNet18 network has a size of [C, H / 8, W / 8], where C is the number of channels, H / 8 is the height of the feature map, and W / 8 is the width of the feature map.

[0065] The gesture interaction module includes a spatial channel perception multi-head cross-attention submodule (S-CMHCA) and a depth separable multi-layer perception submodule (Dep-MLP), respectively. Figure 4 and Figure 5 The spatial-channel perception interaction unit submodule includes a channel attention unit and a spatial attention unit; the depthwise separable multi-layer perception submodule includes a sequentially cascaded LayerNorm layer, a depthwise separable 3×3 convolutional layer, and a GELU activation function.

[0066] In the gesture interaction module, assume that the input is X∈R C×H / 8×W / 8 , where C is the number of channels, H / 8 is the height of the feature map, and W / 8 is the width of the feature map.

[0067] Apply 1×1 convolution operation to the input X to obtain the query sequence Q, keyword sequence K and value sequence V; where Q∈R C ×H / 8×W / 8 , K∈R C×H / 8×W / 8 、V∈R C×H / 8×W / 8 ;

[0068] The dot product calculation is performed on the query sequence Q and the transposed keyword sequence K to obtain a feature map, which is the spatial perception feature N; N = (QK 转置 ), N∈R (H / 8×W / 8)×(H / 8×W / 8) ;

[0069] Due to the lack of local information in the transformer, this paper proposes a separable channel and spatial enhancement module: a channel attention unit and a spatial attention unit, which respectively enhance the channel information and spatial information of local information and provide local information for the transformer. The details are as follows:

[0070] The result S′ is obtained by matrix multiplication of the value sequence V and the channel attention unit; wherein the mathematical expression of the channel attention unit is as follows:

[0071]

[0072]

[0073] y1=fc1(y s_averag )

[0074] y2=fc2(GELU(BN(y1)))

[0075] y c =soft(y2)

[0076]

[0077] Among them, fc3 represents depth-wise separable 3×3 convolution, average represents average pooling layer, fc1 represents 1×1 convolution layer, the input dimension is C, the output dimension is C / r, where r is the scaling ratio, fc2 represents 1×1 convolution, the input dimension is C / r, the output dimension is C, where r is the scaling ratio, BN represents BN layer, GELU represents activation function, Soft represents sigmoid function, It is the dot multiplication sign.

[0078] Perform a dimension conversion operation on S′ to obtain S, and calculate S with the spatial perception feature N to obtain the channel perception interaction feature M; where M = Softmax(N)×S, Softmax is the activation function, M∈R C×(H / 8×W / 8) ;

[0079] The channel-aware interaction feature M is transformed into M', M'∈R C×H / 8×W / 8 , after matrix multiplication of M' and the spatial attention unit, the spatial perception interaction feature W is obtained; wherein the mathematical expression of the spatial attention unit is as follows:

[0080]

[0081]

[0082]

[0083] Among them, fc4 represents a 1×1 convolution layer with an input dimension of C and an output dimension of 1. fc5 represents a 1×1 convolution with an input dimension of 1 and an output dimension of C. BN represents a BN layer, GELU represents an activation function, s represents a sigmoid function, and · represents the corresponding element-wise multiplication sign.

[0084] The spatial perception interaction feature W and the channel perception interaction feature M are positionally added to obtain a spatial-channel perception interaction attention value;

[0085] The spatial-channel perception interaction attention value is input into the depth-separable multi-layer perception submodule, and the features of different channels and spaces of the spatial-channel perception interaction feature are processed to obtain the posture interaction feature. The mathematical expression of the depth-separable multi-layer perception submodule is as follows:

[0086] D = (LayerNorm(W))

[0087] y3=fc6(D)

[0088] y GELU =GELU(y3)

[0089] Here, LayerNorm represents the LayerNorm layer, fc6 represents the depthwise separable 3×3 convolution, and GELU represents the GELU function.

[0090] In the present invention, the number of posture interaction modules can be multiple, connected in series. The specific number can be set as needed. Using multiple posture interaction modules connected in series can gradually stabilize the network training process. During a single epoch of training, the posture interaction modules can be stably trained multiple times, resulting in more robust joint point training features. For dense key point prediction tasks such as the present invention, the design of a neural network is essential.

[0091] The Posture Interaction Module is connected to the Human Pose Estimation Module, which uses a 1×1 convolutional layer to estimate human skeletal features based on the posture interaction features. The output dimension is the number of joints. In the Coco dataset, there are 17 joints. In self-built datasets, the number of joints can be set as needed.

[0092] The human motion prediction module is a 1×1 convolution connected after the posture interaction module. The output dimension can be defined independently. Specifically, the task can be divided into several types of abnormal actions, and the output dimension is defined as a number. For example, if there are two abnormal actions, the output dimension is set to 2.

[0093] During the training process of the posture perception network, the human posture estimation module uses the mean square error loss MeanSquared Error Loss to calculate the estimated human posture skeleton features and the true value of the human posture skeleton to obtain Loss2;

[0094] The human motion prediction module uses cross entropy loss to calculate the detected human motion and the true value of the human motion to obtain Loss1;

[0095] Use the following formula to perform reverse gradient backpropagation to complete the training process

[0096] Loss=pLoss1+dLoss2

[0097] Wherein, p and d represent coefficients. In one embodiment, p and d are both 0.5.

[0098] This paper uses auxiliary loss supervision to predict key points and assist in behavior prediction, making the detection of abnormal human behavior more accurate. By constraining the loss function of human posture, the accuracy of human joint points is effectively improved. The training strategy of the auxiliary loss function increases the robustness of the model.

[0099] In this invention, the gesture interaction module uses the first-person perspective image to remove ambiguous gesture features, improving the ability to accurately recognize a person's gestures. By acquiring the entire scene in the second-person perspective image, it prevents recognition errors caused by excessive gesture correction, completing the training process. Using spatial-channel-aware interactive attention within the gesture interaction module, the second-person perspective image and the first-person perspective image interact, and the spatial-channel-aware interactive module improves the spatial perception of the second-person perspective image.

[0100] Experimental results: Tests were conducted on a self-built dataset and an official action test dataset. The camera captured an image size of 256×192. Using the solution of the present invention, the overall model running speed was 112fps (average within one minute after the test stabilized), and the accuracy of the self-built dataset reached 91.8%. Furthermore, to further demonstrate the effectiveness of the present method on official datasets, the human pose estimation module of the present invention was tested on the COCO dataset using images of 256×192 size and dual cameras inputting images from the same perspective. The average accuracy (AP) reached 74.8, which has already exceeded the results of "FastCNN-Based Single-Person 2D Human Pose Estimation for Autonomous Systems." The user can specify the recognized action as an abnormal action. If falling down is specified as an abnormal action (a person crawling on the floor of a bank can be considered abnormal, and this item can also be applied to some small public places, such as campus corridors, Internet cafe corridors, etc.), then it will trigger a reminder to the back-end personnel that someone has fainted in the bank (due to the large open area of ​​the bank, the application scheme of this application has high adaptability and high recognition accuracy. The self-built data set of this application is mainly based on bank data, supplemented by campus corridors and Internet cafe corridors, but similar situations in campus corridors and Internet cafe corridors can also be detected by this application scheme), realizing abnormal action alarm.

[0101] The above description of the disclosed embodiments will enable one of ordinary skill in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one of ordinary skill in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is to be construed in the widest possible manner consistent with the principles and novel features disclosed herein.

Claims

1. A method for analyzing human motion in images based on deep learning, characterized in that: The method is implemented using a posture perception network, which includes a posture branch residual module, a posture encoding module, and a posture interaction module. The method includes: Acquire a first-perspective image captured by a first camera, process the first-perspective image, input the processed first-perspective image into a posture branch residual module, and obtain a mixed posture feature output by the posture branch residual module; Obtaining a second-viewpoint image captured by a second camera, inputting the second-viewpoint image into a posture encoding module, and obtaining posture encoding features output by the posture encoding module; wherein the first camera and the second camera are used to capture the person at different angles, and the second-viewpoint image has the same resolution as the first-viewpoint image; fusing the mixed posture feature and the posture coding feature, and inputting the resultant feature into a posture interaction module for interaction, thereby obtaining a posture interaction feature output by the posture interaction module; Inputting the gesture interaction features into a human motion prediction module to detect human motion and identify abnormal behaviors of people; The posture branch residual module includes two branches in parallel, and the output ends of the two branches are summed, wherein one branch includes a convolution layer with a 3×3 convolution kernel, a BN layer, and a Relu activation function cascaded in sequence, and the other branch includes a convolution layer with a 5×5 convolution kernel, a BN layer, and a Relu activation function cascaded in sequence; The posture encoding module includes a ResNet18 network, and the ResNet18 network performs feature extraction on the second-view image to obtain an overall image resolution feature of the second-view image as the posture encoding feature; The gesture interaction module includes a space-channel perception interaction submodule and a depth-separable multi-layer perception submodule; The space-channel perception interaction unit submodule includes a channel attention unit and a space attention unit; The depth-separable multi-layer perception submodule includes sequentially cascaded LayerNorm layer, depthwise separable 3×3 convolutional layer and Activation function; In the gesture interaction module, the input X Using 1×1 convolution operation, we get the query sequence based on transformer , keyword sequence Sum value sequence V ; The query sequence and the transposed keyword sequence Perform dot product calculation to obtain the feature map, which is the spatial perception feature N ; The value sequence V The result is obtained by matrix multiplication with the channel attention unit ; Will Perform dimension conversion operation to obtain S ,Will S With the spatial perception characteristics N Perform calculations to obtain channel perception interaction features ; The channel-aware interaction feature Perform dimension conversion operation to obtain ,Will After matrix multiplication with the spatial attention unit, the spatial perception interaction feature is obtained W ; The spatial perception interaction feature W Interaction characteristics with the channel perception M Perform position addition to obtain the spatial-channel perception interaction attention value; The space-channel perception interaction attention value is input into the depth-separable multi-layer perception submodule, and the features of different channels and spaces of the space-channel perception interaction feature are processed to obtain the posture interaction feature.

2. The method according to claim 1, characterized in that The first perspective image is processed as follows: The image is downscaled using two convolutional layers with 3×3 and 5×5 kernels, respectively. The stride of the 3×3 convolutional layer is 2 and the patch size is 1, while the stride of the 5×5 convolutional layer is 3 and the patch size is 2.

3. The method according to claim 1, characterized in that The mathematical expression of the channel attention unit is as follows: ； ； ； ； ； ； in, represents depth-wise separable 3×3 convolution, average represents the average pooling layer, represents a 1×1 convolutional layer, represents 1×1 convolution, BN represents the BN layer, GELU Represents the activation function, Soft represents the sigmoid function, It is the dot multiplication sign.

4. The method according to claim 3, characterized in that The mathematical expression of the spatial attention unit is as follows: ； ； ； in, represents a 1×1 convolutional layer, represents 1×1 convolution, BN represents the BN layer, GELU represents the activation function, s represents the sigmoid function, is the element-wise multiplication sign.

5. The method according to any one of claims 1 to 4, characterized in that: During the training process of the posture perception network, the human posture estimation module uses the mean square error loss to calculate the predicted human posture skeleton features and the true value of the human posture skeleton to obtain ; The human motion prediction module uses cross entropy loss to calculate the detected human motion and the true value of the human motion, and obtains ; Use the following formula to perform reverse gradient backpropagation to complete the training process ； in, and Represents the coefficient.

Images