Action Detection Method Based on Residual Shrinkage Structure and Non-local Attention

Abstract

The present invention relates to the field of behavior detection technology, and in particular to a behavior detection method based on residual shrinkage structure and non-local attention (CN‑R‑DRSN). The residual shrinkage structure is integrated into the residual module of a 3D‑ResNet convolutional network, and a 3D‑DRSN feature extraction network is designed to adaptively eliminate redundant behavior feature information such as noise and background in a behavior video. Layer-by-layer spatial convolution is embedded in a temporal candidate subnet to retain richer hierarchical spatial information in the sampling of a temporal feature map and the segmentation of temporal boundary segments. In a behavior classification subnet, a Soft‑NMS suppression strategy is used to screen temporal candidate segments and a non-local attention mechanism is introduced to obtain contextual spatiotemporal motion information between high-quality temporal segments. The behavior detection method provided by the present invention can effectively improve the accuracy of behavior action boundary detection and behavior classification. Compared with R‑C3D, the mAP@0.5 value of the CN‑R‑DRSN method proposed by the present invention is improved to 32.7%.

Description

Technical Field

[0001] The present invention relates to the technical field of behavior detection, and in particular to a behavior detection method based on residual shrinkage structure and non-local attention. Background Art

[0002] With robots widely used in natural human-computer interaction scenarios such as welcoming guests and explaining, accompanying the elderly, etc., how to make robots correctly understand human behavior and perceive human intentions has become a hot topic in human-computer interaction research. As the most important means to improve the quality of human-computer interaction, behavior detection has received great attention and wide attention from researchers. The main task of behavior detection is to automatically locate the action time boundary (i.e., the start time of the behavior and the end time of the behavior) and perform action recognition in a long video that has not been edited. Therefore, compared with the behavior recognition method with known action boundaries, exploring behavior detection methods is more suitable for the application of robots in natural human-computer interaction and other fields. In natural scenes, robot recognition of human behavior is easily restricted by factors such as noise, lighting, and occlusion. On the one hand, the extracted information contains more redundant information such as noise and background, thereby affecting the accuracy of behavior area positioning and behavior classification accuracy; on the other hand, due to the continuity and timing of video actions, the contextual motion information of the behavior is easily ignored, which in turn affects the positioning accuracy of the candidate behavior segment. Therefore, designing a temporal behavior detection network that can suppress redundant feature information and integrate behavior contextual motion information to improve behavior detection accuracy is the research focus of temporal behavior detection. The current methods for solving such tasks are mainly divided into single-stage, two-stage and behavior detection methods based on weakly supervised learning. The single-stage method generates action boundary positioning information on one-dimensional temporal features, directly generates time boundaries and identifies behavior categories in the original video. For example, Lin T et al. used one-dimensional convolution to predict action boundary positioning information and behavior categories after extracting spatiotemporal behavior features of the video; Long F et al. used Gaussian kernels to predict behavior position segments and solved the variability of behavior duration; Piergiovanni AJ et al. constructed a convolution module based on temporal Gaussian mixture to replace the traditional convolution layer, and used soft attention mechanism to learn the parameters of the mixed Gaussian kernel; Yuan Z et al. classified single-frame images through a two-dimensional convolutional network and predicted the probability of each behavior in the time interval. The single-stage method is difficult to obtain motion feature information on the time sequence due to the single-frame calculation method, and its robustness is poor in the scene of natural human-computer interaction of robots. The two-stage method first extracts a large number of candidate time sequence segments that may contain behaviors from the video, and then locates and classifies the time boundaries of the behaviors. The representative one is the R-C3D network, which consists of three parts: feature extraction subnet, temporal candidate subnet and behavior classification subnet. The network structure is as follows: Figure 1 shown.

[0003] R-C3D uses the convolutional layer of the C3D network to extract the spatiotemporal features of the video, and then draws on the idea of ​​RPN to obtain a series of temporal candidate segments, and finally fine-tunes the temporal boundaries of the behavior and the classification of the behavior. This method is highly dependent on the quality of the generated behavioral temporal candidate segments. How to generate high-quality candidate segments is the main improvement idea. For example, Shou et al. use a sliding window to slide on the time domain of the video to generate time windows of different lengths, but the size of the sliding window is fixed, and the generated candidate segments are difficult to contain the entire behavior; Zhao divides the video into multiple video segments with equal spacing, and classifies and scores the video segments, and aggregates high-scoring and continuous video segments into temporal candidate segments; Lin T et al. proposed a boundary matching mechanism BMN, which effectively improves the quality of behavioral temporal candidate segments. In addition, some people have proposed to improve the overall network structure, such as adding a deconvolution module after the basic backbone network of the CDC network, making fine-grained predictions at the frame level, and refining the boundaries of the time series segments for precise positioning; Chao et al. proposed a strategy to align the receptive field with the time span through a multi-tower structure and dilated convolution, which solved the problem that the candidate segments generated on the behavior with large changes in duration are difficult to contain long-term behaviors due to the fixed size of the receptive field of the feature map. In recent years, some researchers have also proposed behavior detection methods based on weakly supervised learning, which mainly make segment-level predictions based on existing models, and then obtain the time boundary of the behavior in the prediction through certain rules. For example, Wang Limin et al. proposed to first calculate the action probability at the segment level, and then weightedly fuse the scores of all segments as the predicted value of the behavior; Nguyen P et al. proposed to calculate the action boundary by calculating T-CAM to obtain the time segment where the action occurs; Lin Tianwei's team proposed to generate complementary time regions based on two-stage online erasing, integrate multi-scale information in the form of a time pyramid, and finally obtain the prediction of the behavior segment. Although weakly supervised learning methods do not require the labeling of specific time boundaries, they also face problems with detection accuracy and algorithm performance.

[0004] In summary, the development of behavior detection methods based on deep residual shrinkage structure and non-local attention is still a key issue that needs to be urgently solved in the field of behavior detection technology. Summary of the invention

[0005] In order to solve the above problems, the present invention provides a behavior detection method based on deep residual shrinkage structure and non-local attention (CN-R-DRSN), which can effectively improve the accuracy of behavior action boundary detection and behavior classification. Compared with R-C3D, the mAP@0.5 value of the CN-R-DRSN method proposed in the present invention is improved to 32.7%.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] The present invention provides a behavior detection method based on a deep residual shrinkage structure and non-local attention, comprising the following steps:

[0008] (1) Feature extraction subnet based on residual shrinkage structure. The specific process is as follows:

[0009] (11) The input feature X is obtained by batch normalization, ReLU activation function and convolution twice as the input vector of the residual shrinkage module, and is simplified into a one-dimensional vector X after absolute value and global mean pooling operations. h,w,l,c :

[0010] x h,w,l,c =GAP(|ReLU(BN(Conv(X)))|), where ReLU() represents the ReLU activation function, BN() represents batch normalization, GAP() represents global mean pooling, and h, w, l, and c represent the length, width, frame length, and number of channels of the feature map, respectively;

[0011] (12) The simplified vector feature X h,w,l,c They are sent to the threshold shrinkage coefficient calculation branch and the channel mean feature calculation branch respectively. In the threshold shrinkage coefficient calculation branch, two fully connected layers are used to calculate X h,w,l,c The threshold shrinkage coefficient of each channel is:

[0012] α c =δ(FC(FC(x h,w,l,c ))), where FC() is the fully connected layer, δ() is the Sigmoid activation function, and α c ∈(0,1);

[0013] In the channel mean feature calculation branch, the average value of each channel feature is calculated separately:

[0014] β c =average(|x h,w,l,c |), where average() means averaging; after obtaining the threshold shrinkage coefficient and feature mean, the shrinkage threshold τ of each channel feature c =α c β c (c∈[1,3]); By calculating the shrinkage threshold for each channel, the shrinkage threshold vector τ=(τ1,τ2,τ3) of the input vector x can be obtained;

[0015] (13) The calculated shrinkage threshold vector τ is used to soft-threshold the input vector x:

[0016] x s=soft(x,τ)=sign(x)max{x|-τ,0}, where x s represents the feature obtained after soft thresholding. When |x|<τ, x s Set to zero, |x|>τ, x s Shrink towards zero; soft thresholding to obtain feature x s After that, make a residual connection with the input feature X: X′=X+x s ;

[0017] The n residual shrinkage modules are cascaded to construct a 3D-DRSN network. The spatiotemporal feature map after suppression processing by the 3D-DRSN network can be expressed as: E = DRSN n (X), where DRSN n () represents the feature extraction subnetwork composed of n residual shrinkage modules;

[0018] (2) Temporal candidate subnetwork based on layer-by-layer spatial convolution. The specific process is as follows:

[0019] (21) Three convolutional layers and a 1×2×2 maximum pooling layer are used to transform the spatiotemporal feature map obtained by the residual shrinkage network into a feature map E′ containing only temporal information:

[0020] E′=Maxpooling(Conv3(Conv2(Conv1(E)))), where conv1(·), conv2(·), and conv3(·) represent three convolutional layers respectively;

[0021] (22) Through layer-by-layer spatial convolution, we obtain To the length of the video timing feature graph E', then, Extract the temporal candidate segments from the video temporal feature graph E′ of length:

[0022] proposal=Conv 1×1×1 (E′), where Conv 1×1×1 () represents a 1×1×1 convolutional layer, and proposal is the extracted temporal candidate segment;

[0023] (3) Behavior classification subnetwork based on non-local attention mechanism. The specific process is as follows:

[0024] (31) Use Soft-NMS suppression strategy instead of non-maximum suppression:

[0025] Where N represents the number of temporal candidate segments extracted by the temporal candidate subnet, M represents the temporal bounding box with the current highest score, and b iis the i-th temporal bounding box to be processed, s i for b i The temporal bounding box confidence score of , γ is the bounding box suppression threshold, iou(M,b i ) represents the temporal bounding box M and b i Intersection ratio:

[0026]

[0027] , where M start 、M end denote the start time and end time of the highest scoring temporal bounding box M, respectively, and b istart 、b iend Respectively represent the time series bounding box b to be processed i The start and end time of L M , Represent the temporal bounding box M and the temporal bounding box b respectively i Length of time;

[0028] (32) Sort the adjusted timing candidate segments according to the confidence scores, and select K (K≤N) timing candidate segments with higher scores as high-quality timing candidate segments;

[0029] (33) After obtaining K high-quality time series candidate segments through the Soft-NMS strategy, 3DROI pooling is used to map the time series candidate segments of unequal lengths to the corresponding positions of the feature map E:

[0030] P k =ROIPooling(softnms(prposal,k),E), where P k ∈R 512×1×4×4 (k∈[1,K]) is the feature map of the selected temporal candidate segments;

[0031] (34) Obtaining high-quality temporal candidate segment feature maps After that, first Features at different positions are paired for calculation:

[0032] Where W θ , W φ is the weight matrix; P i k (k∈[1,K]) represents the feature map P k The i-th (i∈[1,16]) position feature; Represents the feature map P l The jth (j∈[1,16]) position feature; Represents the feature map P kThe i-th position feature and feature map P l At the same time, the feature map P k ∈ 512×1×4×4 The channel features (512 dimensions) of the 16 positions in (k∈[1,K]) are weighted mapped to form a position vector:

[0033] Where W g is the weight matrix;

[0034] (35) Through the feature map P k With the feature map P l Similarity as well as Calculate the fusion vector F of the i-th position of the k-th feature map i k :

[0035] Where M represents the feature map P k The total number of pixels;

[0036] (36) The feature map P k The fusion feature F at the i-th position i k And the original feature P i k Make a residual connection:

[0037] Where W z ∈R is the weight matrix. Cascade the residual connections of all positions of K features And input to the fully connected layer to obtain the time boundary position and behavior classification of all fragments:

[0038] In the formula, Respectively represent the starting and ending positions of the boundaries of the kth temporal candidate segment; represents the probability that the kth temporal candidate segment is predicted to be the i-th type of behavior; m represents the type of behavior category.

[0039] (4) CN-R-DRSN network optimization, the specific process is as follows:

[0040] A classification and regression joint optimization strategy is adopted, using the cross entropy loss function L cls (·) Predict whether the candidate segment contains a behavior or predict a specific behavior category, and use a smooth L1 loss function L reg (·) Optimize the relative displacement between the candidate bounding box and the ground-truth bounding box:

[0041] In the formula, represents the probability that the kth temporal candidate segment or behavior is predicted as the true label, represents the true label (GT) of the corresponding behavior, It is expressed as the relative offset of predicting the kth temporal candidate segment, represents the real-time bounding box corresponding to the coordinate transformation, which is calculated as follows:

[0042] In the formula, and Respectively represent the center position and length of the candidate temporal segment, and Respectively represent the center position and length of GT, and the objective function of the classification and regression joint optimization strategy is:

[0043] Where N cls 、N reg It represents the number of temporal candidate segments that need to be trained. In the behavior classification subnet, its number is equal to the batch size, and λ is the loss balance factor.

[0044] The present invention is further configured such that in step (11), the one-dimensional vector X h,w,l,c The calculation formula is:

[0045] Where ReLU(·) represents the activation function ReLU, BN(·) represents batch normalization, GAP(·) represents global average pooling, and h, w, l, c represent the length, width, frame length and number of channels of the feature map, respectively.

[0046] The present invention is further configured such that in step (12), the threshold contraction coefficient is:

[0047] α c =δ(FC(FC(X h,w,l,c ))), where FC(·) is the fully connected layer, δ(·) is the Sigmoid activation function, and α c ∈(0,1) represents the contraction coefficient of the cth channel.

[0048] The present invention is further configured such that in step (12), the formula for respectively calculating the average value of each channel feature is:

[0049] β c =average(|X h,w,l,c |), where average(·) means finding the average value.

[0050] The present invention is further configured such that in step (12), the formula for the shrinkage threshold of each channel feature is:

[0051] τ c =α c β c (c∈[1,3]), where τ c is the shrinkage threshold of the cth feature channel.

[0052] The present invention is further configured that in step (3), in order to capture the global dependency of the K feature maps, a non-local attention mechanism is introduced before classification and regression in the behavior classification subnetwork. The specific process is as follows:

[0053] Will The features at different positions are paired and calculated. At the same time, the feature map P k =R 512×1×4×4 The channel features of the 16 positions in (k∈[1,K]) are weighted mapped to form a position vector;

[0054] Through the feature map P k With the feature map P l Similarity as well as Calculate the fusion vector F of the i-th position of the k-th feature map i k ;

[0055] The feature map P k The fusion feature F at the i-th position i k And the original feature P i k Make a residual connection: Where W z ∈R is the weight matrix;

[0056] Cascade the residual connections of all positions of K features And input to the fully connected layer to obtain the time boundary position and behavior classification of all fragments: In the formula, Respectively represent the starting and ending positions of the boundaries of the kth temporal candidate segment, represents the probability that the kth temporal candidate segment is predicted to be the i-th type of behavior, and m represents the type of behavior category.

[0057] The present invention is further configured such that the pairing calculation formula is:

[0058] Where W θ , W φ is the weight matrix, P i k (k∈[1,K]) represents the feature map P k The i-th (i∈[1,16]) position feature; Represents the feature map P l The jth (j∈[1,16]) position feature, Represents the feature map P k The i-th position feature and feature map P l The similarity of the features at the j-th position.

[0059] The present invention is further configured such that the position vector is:

[0060] Where W g is the weight matrix.

[0061] The present invention is further configured as follows: the fusion vector is:

[0062] Where M represents the total number of pixels in the feature map.

[0063] Beneficial Effects

[0064] Compared with the known public technology, the technical solution provided by the present invention has the following beneficial effects:

[0065] (1) The present invention combines the residual shrinkage structure and non-local attention to design a CN-R-DRSN network structure to suppress the redundancy of behavioral features and integrate the spatiotemporal context of the behavior, thereby improving the accuracy of action boundary positioning and behavior classification.

[0066] (2) The present invention embeds a shrinkage structure and a soft thresholding operation in the residual module of the residual network to construct a 3D-DRSN feature extraction network, and improves the effectiveness of feature extraction by adaptively eliminating redundant information.

[0067] (3) The present invention extracts hierarchical spatial information of candidate time series segments through layer-by-layer spatial convolution, overcomes the problem of spatial feature information loss caused by a single convolution strategy, introduces a Soft-NMS strategy to measure the intersection-over-union ratio of the time series bounding box and the maximum bounding box, and adjusts the confidence of the time series candidate segments according to the bounding box suppression threshold to avoid missed detection or misreduction of candidate time series segments, and introduces a non-local attention mechanism to calculate the long-distance dependency of the features of the selected high-quality time series candidate segments. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is the R-C3D network structure diagram;

[0069] Figure 2 It is the network structure diagram of CN-R-DRSN;

[0070] Figure 3 Schematic diagram of the residual shrinkage module of 3D-DRSN;

[0071] Figure 4 This is a diagram of the temporal candidate subnetwork network structure based on layer-by-layer spatial convolution;

[0072] Figure 5 This is the network structure diagram of the behavior classification subnet;

[0073] Figure 6 The detection results of Cricket on R-C3D and CN-R-DRSN networks are shown in Figure 2.

[0074] Figure 7 The detection results of Billiards on R-C3D and CN-R-DRSN networks;

[0075] Figure 8 This is the detection result diagram of High Jump on R-C3D and CN-R-DRSN networks. DETAILED DESCRIPTION

[0076] Aiming at the problem that behavior detection needs to consider both the high redundancy of feature extraction and the inaccurate positioning of behavior boundaries, this paper proposes an improved behavior detection method CN-R-DRSN based on the R-C3D network structure, combined with the residual shrinkage structure and non-local attention. Figure 2 As shown in the figure, in order to suppress the redundant information such as background and noise of the behavior features in the video, the 3D-ResNet convolutional network is adopted and the residual shrinkage structure and soft thresholding operation are integrated to construct the feature extraction network of 3D-DRSN. In the temporal candidate subnet, considering that the one-time convolution strategy of the R-C3D network will cause the loss of spatial feature information, a layer-by-layer spatial convolution strategy is adopted to retain the hierarchical spatial information of the behavior. In the behavior classification subnet, in order to provide richer behavior feature information for subsequent boundary regression and classification, the non-maximum suppression (Soft-NMS) strategy is adopted to screen the temporal candidate segments and the non-local attention mechanism is introduced to calculate the global dependency between high-quality temporal segments.

[0077] The present invention is further described in detail below with reference to implementation cases.

[0078] Embodiment 1:

[0079] The present invention provides a behavior detection method based on residual shrinkage structure and non-local attention, comprising the following steps:

[0080] It should be noted that the task of the feature extraction subnet is to extract the spatiotemporal feature information of the video. Effective extraction of motion feature information is beneficial to the subsequent segmentation of candidate temporal segments and action classification. In R-C3D, the feature extraction subnet is constructed by the convolutional layer of the C3D network. Its feature extraction structure is shallow and the extracted features are not compact. At the same time, considering the natural scene of human-computer interaction, the extracted features are affected by noise, environment and other factors and may contain more redundant information. Therefore, based on the deep 3D residual convolutional neural network and drawing on the idea of ​​deep residual shrinkage network, the shrinkage structure and soft thresholding are embedded in the residual module to design the 3D-DRSN structure. Its residual shrinkage module is as follows: Figure 3 shown.

[0081] The redundant information contained in each sample is quite different, so the shrinkage thresholds are different. In order to automatically learn the shrinkage threshold of the sample, the present invention embeds a shrinkage structure in the residual module of the 3D-Resnet network, and the specific process is shown in step (1).

[0082] (1) Feature extraction subnet based on deep residual contraction structure. The specific process is as follows:

[0083] (11) The input feature X is obtained by batch normalization, ReLU activation function and convolution twice as the input vector of the residual shrinkage module, and is simplified into a one-dimensional vector X after absolute value and global mean pooling operations. h,w,l,c .

[0084] Furthermore, the one-dimensional vector X h,w,l,c The calculation formula is:

[0085] Where ReLU(·) represents the activation function ReLU, BN(·) represents batch normalization, GAP(·) represents global average pooling, and h, w, l, c represent the length, width, frame length and number of channels of the feature map, respectively.

[0086] (12) The simplified vector feature X h,w,l,c They are sent to the threshold shrinkage coefficient calculation branch and the channel mean feature calculation branch respectively. In the threshold shrinkage coefficient calculation branch, two fully connected layers are used to calculate X h,w,l,c The threshold shrinkage factor.

[0087] Furthermore, the threshold shrinkage factor is:

[0088] α c =δ(FC(FC(X h,w,l,c ))), where FC(·) is the fully connected layer, δ(·) is the Sigmoid activation function, and α c ∈(0,1) represents the contraction coefficient of the cth channel.

[0089] In the channel mean feature calculation branch, the average value of each channel feature is calculated respectively.

[0090] Furthermore, the formula for calculating the average value of each channel feature is:

[0091] β c =average(|X h,w,l,c |), where average(·) means finding the average value;

[0092] After obtaining the threshold shrinkage coefficient and feature mean, the shrinkage threshold of each channel feature is

[0093] Furthermore, the formula for the shrinkage threshold of each channel feature is:

[0094] τ c =α c β c (c∈[1,3]), where τ c is the shrinkage threshold of the cth feature channel;

[0095] By calculating the shrinkage threshold for each channel, the shrinkage threshold vector τ = (τ1, τ2, τ3) of the input vector x can be obtained;

[0096] (13) In order to suppress redundant information irrelevant to the behavioral target, the calculated shrinkage threshold vector is used to soft-threshold the input vector x:

[0097] x s =soft(x,τ)=sign(x)max{x|-τ,0}, where x s represents the feature obtained after soft thresholding. When |x|<τ, x s Set to zero, |x|>τ, x s Contraction towards zero;

[0098] Soft thresholding to obtain feature x s After that, make a residual connection with the input feature X:

[0099] X′=X+x s ,Through the designed residual shrinkage module, a set of thresholds can be automatically learned and each feature channel can be soft-thresholded, so that redundant information can be adaptively eliminated;

[0100] The n residual shrinkage modules are cascaded and a 3D-DRSN network is constructed, as Figure 2 As shown in the feature extraction subnetwork, the spatiotemporal feature map after suppression processing by the 3D-DRSN network can be expressed as:

[0101] E=DRSN n (X), where DRSN n Represents the feature extraction subnetwork composed of residual shrinkage modules.

[0102] It should be noted that the task of the temporal candidate subnet is to segment the temporal boundary segments containing behavioral actions from the spatiotemporal feature graph obtained by the residual shrinkage network. When extracting candidate temporal segments, considering the problem that the R-C3D network directly changes the length and width of the feature graph to 1 through a single convolution, which will cause a large amount of spatial feature information to be lost, the present invention adopts layer-by-layer spatial convolution to retain more spatial features of the behavior. The improved temporal candidate subnet structure is as follows: Figure 4 The specific process is shown in step (2).

[0103] (2) Temporal candidate subnetwork based on layer-by-layer spatial convolution. The specific process is as follows:

[0104] (21) Three convolutional layers and a 1×2×2 maximum pooling layer are used to transform the spatiotemporal feature map obtained by the residual shrinkage network into a feature map E′ containing only temporal information:

[0105] E′=Maxpooling(Conv3(Conv2(Conv1(E)))), where Conv1(·), conv2(·), and conv3(·) represent three convolutional layers. For better explanation, the sizes of the three convolutional layers are shown in Table 1.

[0106] Table 1: Convolution kernel sizes used for layer-by-layer spatial convolution

[0107] input size stride padding output Conv1 512 (3,3,3) (1,1,1) (1,1,1) 512 Conv2 512 (3,3,3) (1,2,2) (1,1,1) 512 Conv3 512 (3,3,3) (1,2,2) (1,1,1) 512

[0108] (22) Through layer-by-layer spatial convolution, we obtain To the length of the video timing feature map, then, in Extract the temporal candidate segments from the video temporal feature graph E′ of length:

[0109] proposal=Conv 1×1×1 (E′), where Conv 1×1×1 (*) indicates a 1×1×1 convolutional layer, and proposal is the extracted temporal candidate segment, which contains the boundary of the candidate segment and the confidence of the behavior. Layer-by-layer spatial convolution is used instead of single-layer convolution, so that the temporal candidate subnet can retain the hierarchical spatial information of the behavior in the feature map of temporal information and the segmentation of temporal boundary segments.

[0110] It should be noted that the task of the behavior classification subnet is to fine-tune the position and behavior classification of the video segments within each candidate boundary by combining the features extracted by the feature extraction subnet and the temporal candidate segments generated by the temporal candidate subnet. Since the temporal candidate subnet outputs a large number of candidate segments and there is a high degree of overlap between them, the R-C3D network uses a non-maximum suppression strategy to eliminate redundant duplicate segments, but it has two problems: (1) When the target boundary is close, the boundary with a low score is deleted due to the large overlapping area (2) Manually setting the threshold will cause missed detection or false reduction of the boundary. In order to further optimize the temporal boundary position and behavior classification tasks, Soft-NMS is used to reduce the confidence of the temporal bounding box to select the candidate segments, and the non-local attention mechanism is introduced into the behavior classification subnet to obtain the long-distance remote dependency between the selected candidate segments. The improved behavior classification subnet is shown in Figure 2. Figure 5 As shown, specifically as shown in step (3).

[0111] (3) Behavior classification subnetwork based on non-local attention mechanism. The specific process is as follows:

[0112] (31) To reduce the confidence of overlapping bounding boxes, the Soft-NMS strategy is used instead of non-maximum suppression:

[0113] Where N represents the number of temporal candidate segments extracted by the temporal candidate subnet, M represents the temporal bounding box with the current highest score, and b i is the i-th temporal bounding box to be processed, s i for b i The temporal bounding box confidence score of , γ is the bounding box suppression threshold, iou(M,b i ) represents the temporal bounding box M and b i Intersection ratio:

[0114]

[0115] , where M start 、M end denote the start time and end time of the highest scoring temporal bounding box M, respectively, and b istart 、b iend Respectively represent the time series bounding box b to be processed i The start and end time of L M , L bi Represent the temporal bounding box M and the temporal bounding box b respectively i Length of time;

[0116] (32) The adjusted temporal candidate segments are sorted according to the confidence scores, and K (K≤N) temporal candidate segments with higher scores are selected as high-quality temporal candidate segments. Compared with the non-maximum suppression strategy that directly removes the temporal candidate segments that are greater than the bounding box suppression threshold, the Soft-NMS strategy avoids missed detection and erroneous deletion of candidate temporal segments by adjusting the confidence scores.

[0117] (33) After obtaining K high-quality time series candidate segments through the Soft-NMS strategy, 3DROI pooling is used to map the time series candidate segments of unequal lengths to the corresponding positions of the feature map E:

[0118] P k =ROIPooling(softnms(prposal,k),E), where P k ∈R 512×1×4×4 (k∈[1,K]) is the feature map of the selected temporal candidate segments;

[0119] (34) Obtaining high-quality temporal candidate segment feature maps Finally, in order to capture the global dependencies of the K feature maps, a non-local attention mechanism is introduced before classification and regression in the behavior classification subnetwork. The specific process is as follows:

[0120] Will The features at different positions are paired and calculated. At the same time, the feature map P k =R 512×1×4×4 The channel features of the 16 positions in (k∈[1,K]) are weighted mapped to form a position vector;

[0121] Through the feature map P k With the feature map P l Similarity as well as Calculate the fusion vector F of the i-th position of the k-th feature map i k ;

[0122] The feature map P k The fusion feature F at the i-th position i k And the original feature P i k Make a residual connection: Where W z ∈R is the weight matrix;

[0123] Cascade the residual connections of all positions of K features And input to the fully connected layer to obtain the time boundary position and behavior classification of all fragments: In the formula, Respectively represent the starting and ending positions of the boundaries of the kth temporal candidate segment, represents the probability that the kth temporal candidate segment is predicted to be the i-th type of behavior, and m represents the type of behavior category.

[0124] The pairing calculation formula is:

[0125] Where W θ , W φ is the weight matrix, P i k (k∈[1,K]) represents the feature map P k The i-th (i∈[1,16]) position feature; Represents the feature map P l The jth (j∈[1,16]) position feature, Represents the feature map P k The i-th position feature and feature map P l The similarity of the features at the j-th position.

[0126] The position vector is:

[0127] Where W g is the weight matrix.

[0128] Fusion Vector Where M represents the total number of pixels in the feature map.

[0129] The Soft-NMS strategy is used to select the temporal candidate segments, and the non-local attention mechanism is introduced to enable the classification subnetwork to capture the global dependencies between the temporal candidate segments, thereby optimizing the boundary position and improving the classification accuracy.

[0130] It should be noted that, in order to further optimize the network training parameters, the present invention performs CN-R-DRSN network optimization, as shown in step (4).

[0131] (4) CN-R-DRSN network optimization, the specific process is as follows:

[0132] A classification and regression joint optimization strategy is adopted, using the cross entropy loss function L cls (·) Predict whether the candidate segment contains a behavior or predict a specific behavior category, and use a smooth L1 loss function L reg (·) Optimize the relative displacement between the candidate bounding box and the ground-truth bounding box:

[0133] In the formula, represents the probability that the kth temporal candidate segment or behavior is predicted as the true label, represents the true label (GT) of the corresponding behavior, It is expressed as the relative offset of predicting the kth temporal candidate segment, represents the real-time bounding box corresponding to the coordinate transformation, which is calculated as follows:

[0134] In the formula, and Respectively represent the center position and length of the candidate temporal segment, and Respectively represent the center position and length of GT, and the objective function of the classification and regression joint optimization strategy is:

[0135] Where N cls 、N reg It represents the number of temporal candidate segments that need to be trained. In the behavior classification subnet, its number is equal to the batch size, and λ is the loss balance factor.

[0136] Embodiment 2:

[0137] In order to train the parameters of the CN-R-DRSN model and verify the effectiveness of the model, this paper conducts experiments on the public THUMOS14 behavior dataset. The THUMOS14 dataset contains a large number of behavioral videos of human daily life and sports in real environments, which can be used for two types of tasks: action recognition and temporal action detection. Since this paper mainly solves the problem of behavior detection in natural scenes, unedited video data is used as the dataset for the temporal action detection task. The 3007 unedited behavior clips in the validation set are used as the training set, and the 3358 unedited behavior clips in the test set are used as the test set. On the Ubuntu18.04 operating system, the experiment uses a Tesla K40m graphics card and uses Pytorch to build and train the CN-R-DRSN network. For the improved method proposed in this paper, the binary cross entropy loss function and the smoothing loss function are used for the joint optimization strategy. The relevant parameters are shown in Table 2.

[0138] Table 2: Model parameter settings

[0139] Parameter name Reference value Optimizer Stochastic Gradient Descent (SGD) Maximum number of training rounds 8 Learning Rate <![CDATA[10 4 ]]> Video frame length 786 Bounding box suppression threshold γ 0.7 The number of high-quality time series candidate segments K 2000 Loss balance factor λ 1

[0140] In the THUMOS14 dataset, Table 3 summarizes the detection accuracy of the proposed method and other behavior detection methods between iou∈[0.1,0.7]. Table 3 shows that when iou≤0.3, the detection accuracy of CN-R-DRSN is lower than that of SSAD, R-C3D, SSN, DCC-R-C3D, but higher than that of S-CNN; and as iou increases, the detection accuracy of CN-R-DRSN is greatly improved and gradually exceeds that of other methods.

[0141] Table 3: mAP comparison of different methods

[0142]

[0143]

[0144] As can be seen from Table 3, when iou=0.5, the detection accuracy of CN-R-DRSN is 1.3-13.7 higher than other methods. To illustrate the classification effect of each behavior under iou=0.5, Table 4 statistics the AP and mAP values ​​of each method for 20 types of behaviors. Compared with CDC, R-C3D, and DCC-R-C3D methods, although CN-R-DRSN has lower detection accuracy in Frisbee Catch, Volleyball Spiking and other behaviors, it has improved the average accuracy of behavior detection (32.7%) as a whole, and has significantly improved the detection effects of Cliff Diving, Hammer Throw, High Jump, Pole Vault and other behaviors, reaching 58.5%, 59.6%, 38.1%, and 71.0% respectively.

[0145] Table 4 Comparison of AP (%) of each line of different methods

[0146]

[0147]

[0148] At the same time, in order to illustrate the division effect of the CN-R-DRSN timing boundary, this paper further compares it with the R-C3D network. Figure 6 The boundary demarcation effect of two types of actions, Cricket Bowling and Cricket Shot, is shown in the same time sequence. Figure 6 It can be seen that the temporal localization boundaries of Cricket Bowling and Cricket Shot overlap in the R-C3D network, but do not cross in the CN-R-DRSN network; Figure 7 The effect of Billiards action on boundary demarcation on R-C3D and CN-R-DRSN networks is shown. Figure 7 It shows that the R-C3D network includes non-motion information in the Billiards motion boundary division, while the CN-R-DRSN network can better distinguish non-motion information from motion information; Figure 8 The detection results of High Jump action on R-C3D and CN-R-DRSN networks are shown. Compared with the R-C3D network, the bounding box of the time sequence segment divided by the CN-R-DRSN network is closer to the real bounding box GT, and the action classification of each time sequence segment has a higher confidence.

[0149] In summary, in order to improve the accuracy of action boundary positioning and action classification, the present invention proposes a behavior detection method based on CN-R-DRSN network based on R-C3D network structure. In the feature extraction subnet, a 3D-DRSN feature extraction network is designed to adaptively eliminate redundant information such as noise and background in the behavior video; in the temporal candidate subnet, layer-by-layer spatial convolution is embedded to replace the convolution to retain richer hierarchical spatial information in the temporal feature map and temporal boundary segmentation; in the behavior classification subnet, the Soft-NMS strategy is adopted to avoid missed detection or misreduction of candidate temporal segments, and the non-local attention mechanism is introduced to calculate the global dependency between temporal segments. The experimental results show that the proposed method can improve the accuracy of action boundary positioning and behavior classification. Therefore, the proposed CN-R-DRSN behavior detection method can be applied to natural human-computer interaction scenarios such as welcoming explanations and elderly care, which is conducive to improving the accuracy of robot user behavior perception and intention understanding, thereby improving the quality of human-computer interaction.

[0150] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0151] References are as follows:

[0152] [1]Chen T, Lu S, Fan JS-CNN: Subcategory-aware convolutional networks for object detection[J]. IEEE transactions on pattern analysis and machine intelligence, 2017, 40(10): 2522-2528.

[0153] [2]Shou Z,Chan J,Zareian A,et al.Cdc:Convolutional-de-convolutionalnetworks for precise temporal action localization in untrimmed videos[C] / / Proceedings of the IEEE conference on computer vision and patternrecognition.2017:5734-5743.

[0154] [3]Lin T, Zhao X, Shou Z.Single shot temporal action detection[C] / / Proceedings of the 25th ACM international conference on Multimedia.2017:988-996.

[0155] [4]Xu H, Das A, Saenko KR-c3d:Region convolutional 3d network fortemporal activity detection[C] / / Proceedings of the IEEE internationalconference on computervision.2017:5783-5792.

[0156] [5]Qi T,Bayramli B,Ali U,et al.Spatial shortcut network for humanpose estimation[J].arXivpreprint arXiv:1904.03141,2019.

[0157] [6] Tian Xiang, Zhang Liang. Improved R-C3D temporal behavior detection network[J]. Signal Processing, 2021, 37(3): 447-455.

[0158] [7] Yuan Z, Stroud JC, Lu T, et al. Temporal action localization by structured maximal sums[C] / / Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition.2017:3684-3692.

[0159] [8] Zhao M, Zhong S, Fu X, et al. Deep residual shrinkage networks for fault diagnosis [J]. IEEE Transactions on Industrial Informatics, 2019, 16(7): 4681-4690.

[0160] [9] Gao Chenqiang, Chen Xu. A review of behavior detection methods based on deep learning[J]. Journal of Chongqing University of Posts and Telecommunications (Natural Science Edition), 2020, 32(6): 991-1002.

[0161]

[10] Li X.Human–robot interaction based on gesture and movementrecognition[J].Signal Processing:Image Communication,2020,81:115686.

Claims

1. A behavior detection method based on residual shrinkage structure and non-local attention, characterized in that: The following steps are involved: (1) Feature extraction subnet based on residual shrinkage structure. The specific process is as follows: (11) The input feature X is obtained by batch normalization, ReLU activation function and convolution twice as the input vector of the residual shrinkage module, and is simplified into a one-dimensional vector X after absolute value and global mean pooling operations. h,w,l,c :x h,w,l,c =GAP(|ReLU(BN(Conv(X)))|), where ReLU(·) represents the ReLU activation function, BN(·) represents batch normalization, GAP(·) represents global mean pooling, and h, w, l, c represent the length, width, frame length, and number of channels of the feature map, respectively; (12) The simplified vector feature X h,w,l,c They are sent to the threshold shrinkage coefficient calculation branch and the channel mean feature calculation branch respectively. In the threshold shrinkage coefficient calculation branch, two fully connected layers are used to calculate X h,w,l,c The threshold shrinkage coefficient of each channel is: α c =δ(FC(FC(x h,w,l,c ))), where FC(·) is the fully connected layer, δ(·) is the Sigmoid activation function, and α c ∈(0,1); In the channel mean feature calculation branch, the average value of each channel feature is calculated separately: β c =average(|x h,w,l,c |), where average(·) means averaging; after obtaining the threshold shrinkage coefficient and feature mean, the shrinkage threshold τ of each channel feature c =α c β c (c∈[1,3]); By calculating the shrinkage threshold for each channel, the shrinkage threshold vector τ=(τ1,τ2,τ3) of the input vector x can be obtained; (13) The calculated shrinkage threshold vector τ is used to soft-threshold the input vector x: x s =soft(x,τ)=sign(x)max{|x|-τ,0}, where x s represents the feature obtained after soft thresholding. When |x|<τ, x s Set to zero, |x|>τ, x s Shrink towards zero; soft thresholding to obtain feature x s After that, make a residual connection with the input feature X: X′=X+x s ; The n residual shrinkage modules are cascaded to construct a 3D-DRSN network. The spatiotemporal feature map after suppression processing by the 3D-DRSN network can be expressed as: E = DRSN n (X), where DRSN n (·) represents the feature extraction subnetwork composed of n residual shrinkage modules; (2) Temporal candidate subnetwork based on layer-by-layer spatial convolution. The specific process is as follows: (21) Three convolutional layers and a 1×2×2 maximum pooling layer are used to transform the spatiotemporal feature map obtained by the residual shrinkage network into a feature map E′ containing only temporal information: E′=Maxpooling(Conv3(Conv2(Conv1(E)))), where conv1(·), conv2(·), and conv3(·) represent three convolutional layers respectively; (22) Through layer-by-layer spatial convolution, we obtain To the length of the video timing feature graph E', then, Extract the temporal candidate segments from the video temporal feature graph E′ of length: proposal=Conv 1×1×1 (E′), where Conv 1×1×1 (·) represents a 1×1×1 convolutional layer, and proposal is the extracted temporal candidate segment; (3) Behavior classification subnetwork based on non-local attention mechanism. The specific process is as follows: (31) Use Soft-NMS suppression strategy instead of non-maximum suppression: Where N represents the number of temporal candidate segments extracted by the temporal candidate subnet, M represents the temporal bounding box with the current highest score, and b i is the i-th temporal bounding box to be processed, s i for b i The temporal bounding box confidence score of , γ is the bounding box suppression threshold, iou(M,b i ) represents the temporal bounding box M and b i Intersection ratio: ， Where M start 、M end denote the start time and end time of the highest scoring temporal bounding box M, respectively, and b istart , b iend Respectively represent the time series bounding box b to be processed i The start and end time of L M , Represent the temporal bounding box M and the temporal bounding box b respectively i Length of time; (32) Sort the adjusted timing candidate segments according to the confidence scores, and select K (K≤N) timing candidate segments with higher scores as high-quality timing candidate segments; (33) After obtaining K high-quality time series candidate segments through the Soft-NMS strategy, 3D ROI pooling is used to map the time series candidate segments of unequal lengths to the corresponding positions of the feature map E: P k =ROIPooling(softnms(prposal,k),E), where P k ∈R 512×1×4×4 (k∈[1,K]) is the feature map of the selected temporal candidate segments; (34) Obtaining high-quality temporal candidate segment feature maps After that, first Features at different positions are paired for calculation: Where W θ , W φ is the weight matrix; P i k (k∈[1,K]) represents the feature map P k The i-th (i∈[1,16]) position feature; Represents the feature map P l The jth (j∈[1,16]) position feature; Represents the feature map P k The i-th position feature and feature map P l The similarity of the feature at the jth position; at the same time, the feature map The channel features of the 16 positions in (k∈[1,K]) are weighted mapped to form a position vector: Where W g is the weight matrix; (35) Through the feature map P k With the feature map P l Similarity as well as Calculate the fusion vector F of the i-th position of the k-th feature map i k : Where M represents the feature map P k The total number of pixels; (36) The feature map P k The fusion feature F at the i-th position i k And the original feature P i k Make a residual connection: Where W z ∈R is the weight matrix; the residual connections of all positions of K features are cascaded And input to the fully connected layer to obtain the time boundary position and behavior classification of all fragments: In the formula, Respectively represent the starting and ending positions of the boundaries of the kth temporal candidate segment; represents the probability that the k-th temporal candidate segment is predicted to be the i-th type of behavior; m represents the type of behavior category; (4) CN-R-DRSN network optimization, the specific process is as follows: A classification and regression joint optimization strategy is adopted, using the cross entropy loss function L cls (·) Predict whether the candidate segment contains a behavior or predict a specific behavior category, and use a smooth L1 loss function L reg (·) Optimize the relative displacement between the candidate bounding box and the ground-truth bounding box: In the formula, represents the probability that the kth temporal candidate segment or behavior is predicted as the true label, represents the true label (GT) of the corresponding behavior, It is expressed as the relative offset of predicting the kth temporal candidate segment, represents the real-time bounding box corresponding to the coordinate transformation, which is calculated as follows: In the formula, and Respectively represent the center position and length of the candidate temporal segment, and Respectively represent the center position and length of GT, and the objective function of the classification and regression joint optimization strategy is: Where N cls 、N reg It represents the number of temporal candidate segments that need to be trained. In the behavior classification subnet, its number is equal to the batch size, and λ is the loss balance factor.

2. The behavior detection method based on residual shrinkage structure and non-local attention according to claim 1, characterized in that: In step (11), the one-dimensional vector X h,w,l,c The calculation formula is: Where ReLU(·) represents the activation function ReLU, BN(·) represents batch normalization, GAP(·) represents global average pooling, and h, w, l, c represent the length, width, frame length and number of channels of the feature map, respectively.

Images