Dual-path multi-label spatiotemporal action detection method based on foreground semantic enhancement

Abstract

The present invention discloses a dual-path multi-label spatiotemporal action detection method based on foreground semantic enhancement. The present invention first generates a video clip set and a background set by sampling a video, fuses background information through an action foreground semantic enhancement module, and obtains a video clip set with action foreground semantic enhancement; then uses a two-dimensional convolution and a three-dimensional convolution neural network to obtain a target bounding box, and extracts target-level spatiotemporal features and video spatiotemporal feature graphs; then, an interactive relationship encoder constructs a target interactive relationship to obtain target-level motion features that characterize the interactive relationship; finally, a dual-path action multi-label learning module is used to model the relationship between action tags and output action categories. The present invention not only reduces semantic deviation by enhancing action foreground semantics and target action category modeling, but also alleviates the problem of tag sparsity by characterizing the intrinsic association of different action tags, while minimizing the dual-path action classification loss to alleviate the problem of imbalance between positive and negative categories, thereby improving the accuracy of action detection.

Description

Technical Field

[0001] The invention belongs to the field of computer vision, in particular to the field of spatiotemporal action detection in video processing, and relates to a dual-path multi-label spatiotemporal action detection method based on foreground semantic enhancement. Background Art

[0002] With the rise of short video platforms, live streaming and self-media industries, a large number of videos are created and shared every day, which not only enriches people's entertainment life, but also provides valuable resources for scientific research and commercial applications. However, in the era of big data, how to effectively manage and utilize these massive videos has become an urgent problem to be solved; compared with other types of data such as text, images and audio, videos contain more complex visual information. Although humans can easily understand video content, it is extremely challenging for machines to achieve the same level of understanding. This is because videos not only contain static objects and scene information, but also dynamic time series information, such as character movements, object movement, etc.; spatiotemporal action detection technology is born to solve this problem. Its core task is to identify the time interval and corresponding category of the action in the video, and at the same time mark the spatial position of the target with the target bounding box within the spatial range. This has important application value in many fields. For example, in intelligent monitoring systems, real-time detection of dangerous actions (such as fighting, falling, etc.) is used to ensure public safety; in sports analysis, the details of athletes' movements are detected to assist coaches in tactical guidance; in smart home environments, support is provided for human-computer interaction by detecting human activity patterns.

[0003] At present, there are two problems in spatiotemporal action detection: the bias problem and the multi-label problem. The bias problem means that when training a spatiotemporal action detection model on a biased dataset, certain backgrounds and specific actions appear at the same time with a high frequency, causing the model to rely too much on these backgrounds for action detection during the learning process and fail to learn complex action foreground semantic features. For example, the model may detect that the target is swimming simply by relying on the video background being a swimming pool. The bias problem causes the model to perform well on independent and identically distributed data, but its generalization ability on non-independent and identically distributed data is reduced; the multi-label problem means that a sample may correspond to multiple action category labels. With the increase in action categories, the number of valid sample combinations in the multi-label dataset increases dramatically, which may cause the data distribution to become extremely sparse, that is, some label combinations rarely or never appear, which will affect the learning and generalization ability of the model. Existing spatiotemporal action detection methods use attribute supervision to solve the bias problem, using scene pseudo-labels and human masks to prevent the model from predicting scenes and identifying unattended actions, thereby alleviating the model's over-reliance on non-action features (such as background, clothing color, etc.); to address another problem, existing spatiotemporal action detection methods use the Sigmoid function as the activation function in the last layer of the classifier and use a weighted binary cross entropy loss function to achieve multi-label classification.

[0004] The shortcomings of the above-mentioned spatiotemporal action detection methods are mainly manifested in the following three aspects: 1) The method using scene pseudo-labels and human masks requires additional attribute labels. Additional attribute labels (such as object categories, action details, etc.) usually require experts to manually annotate, which is a time-consuming and expensive task; 2) The method relying on specific attribute labels may not adapt well to new scenes or unseen combinations, thus limiting the generalization ability of the model, because the model over-relies on specific labels (such as object categories, action details, etc.) during training, resulting in poor performance when facing new scenes or data without specific attribute labels, and cannot flexibly cope with diverse actual situations; 3) The method using Sigmoid activation function and weighted binary cross entropy loss function cannot fundamentally solve the problem of sample sparsity. As the number of action categories increases, the problem of sample sparsity still exists, making it difficult for the model to learn effective multi-label combinations. At the same time, the weighted binary cross entropy loss function will also bring about the problem of imbalance between positive and negative categories, resulting in a decrease in the overall performance of the model. Based on these considerations, it is urgent to design a spatiotemporal action detection method that can eliminate excessive reliance on non-action features and alleviate the problem of sample sparsity caused by multi-label action detection. Summary of the invention

[0005] The purpose of the present invention is to address the shortcomings of existing methods and provide a dual-path multi-label spatiotemporal action detection method based on foreground semantic enhancement, which can not only alleviate the bias problem, reduce the model's excessive dependence on the background, and capture the semantic features of the action foreground, but also alleviate the sparse sample problem caused by multi-label action detection and generate accurate action time intervals, action categories and target bounding boxes.

[0006] The present invention provides a dual-path multi-label spatiotemporal action detection method based on foreground semantic enhancement, comprising the following steps:

[0007] Step (1) constructing a video preprocessing module to preprocess the original video to obtain a background set and a video clip set;

[0008] Step (2) constructing an action foreground semantic enhancement module, the input of which is a background set and a video clip set, and the output of which is a video clip set with action foreground semantic enhancement;

[0009] Step (3) extracting features using two-dimensional convolutional and three-dimensional convolutional neural networks, with the input being a set of enhanced video clips and the output being a target bounding box, target-level motion features, and a video spatiotemporal feature map;

[0010] Step (4) constructing an interaction relationship encoder, the input of which is the target-level motion feature and the video spatiotemporal feature map, and the output of which is the target-level motion feature after the interaction relationship is constructed;

[0011] Step (5) constructing a dual-path action multi-label learning module, the input of which is the target-level motion feature embedded with the category label and constructed with the interaction relationship, and the output of which is the predicted action category; obtaining a spatiotemporal action detection model which includes a video preprocessing module, an action foreground semantic enhancement module, an interaction relationship encoder and a dual-path action multi-label learning module in sequence;

[0012] Step (6) training the spatiotemporal action detection model;

[0013] Calculate the target location and action classification losses, and then use the stochastic gradient descent algorithm to optimize the above spatiotemporal action detection model. Perform steps (1) to (5) on the new video sequence to obtain the spatial positions and action categories of all targets at different times until the model converges.

[0014] Step (7) obtains the video sequence to be detected and inputs it into the trained spatiotemporal action detection model to obtain the predicted target bounding box and action category probability, and then obtains the action start time, end time and action category.

[0015] Preferably, step (1) specifically includes the following steps:

[0016] (1-1) Construct a video preprocessing module, obtain the original video, input the video preprocessing module, and process it at a rate of The frame sampling rate, The value of is 30, the original video is sampled, and the number of frames obtained is A collection of video frame sequences ,in, Represents the first Frame video frame, represents the field of real numbers, and Respectively represent the height and width of the video frame, and 3 represents the RGB three channels;

[0017] (1-2) Using an object detector, the object detector Faster R-CNN model, to collect a set of video frame sequences Perform target detection and select a set of video frame sequences based on the detection results The frames that do not contain people are used as background frames, and the background frames are used to form a background set ,in, Represents the background collection The Background frames, Represents the background collection The number of background frames;

[0018] (1-3) From a video frame sequence collection The Frame starts, make sure Frame, last The last frame is not a key frame. Take a frame as the key frame for subsequent segmentation of video clips to obtain a key frame set ,in, Indicates the total duration of the original video in seconds. Indicates Keyframes of a video clip of 1 second, as a collection of keyframes Keyframes in As the center, from the video frame sequence collection The intercept length is Frame video clip , and finally get a collection of video clips ,in, Indicates seconds of video clip, from the key frame before Frame video frame, keyframe and keyframe after Frame video frame composition, Indicates half the length of the video clip.

[0019] Preferably, step (2) specifically includes the following steps:

[0020] Constructing an action foreground semantic enhancement module, which is used to fuse the original video clip with a static video clip composed of a randomly selected single background image, increase the diversity of action and background combinations, reduce the frequency of certain backgrounds and specific actions appearing at the same time, reduce the model's excessive dependence on the background, and enable the model to learn more complex action foreground semantic features, thereby alleviating the deviation problem; the action foreground semantic enhancement module is composed of a video fusion submodule and an action label smoothing module;

[0021] The video fusion submodule is used to perform the video fusion operation, including the following steps: Randomly select a frame as the background frame , copied along the time dimension to obtain a length of Frames of static background video , and with the video clip Linear addition is performed to achieve fusion and obtain the enhanced video clip ,in, represents the fusion ratio, Indicates a video clip The Frame video frame;

[0022] The motion tagging smoothing module is used to smooth the video clips. The action category mark of each target is smoothed to obtain the smoothed action category mark, which specifically includes the following steps:

[0023] Video Clip Middle The smoothed true action category label vector corresponding to the target is defined as , where the vector forward Dimensions express action categories, vector No. Dimensions Represents the background category, vector at last Dimensions Represents the mixed category consisting of the corresponding action category and the background category, vector The value range of each dimension is , the closer the dimension value is to 1, the greater the possibility of the existence of this category;

[0024] The video clip is calculated according to the following formula Middle The smoothed true action category label vector corresponding to the target ,in, is the uncertainty, , , and The categories corresponding to each dimension in Consistently, the categories corresponding to each dimension include: action category, background category and mixed category; represents the real action category label vector, forward The corresponding dimension in which an action exists takes the value of 1, and the values ​​of other dimensions are all 0. represents the background category label vector, No. The value of one dimension is 1, and the values ​​of other dimensions are 0. represents the action-centered fuzzy decision space label vector, at last The corresponding dimension of the mixed category with action in the dimension is 1, and the values ​​of other dimensions are 0. represents the fuzzy decision space label vector centered on the background, at last The value of one dimension is 1, and the values ​​of other dimensions are 0.

[0025] Preferably, step (3) specifically includes the following steps:

[0026] (3-1) Video clips Input into a 3D convolutional neural network to generate video clips The spatiotemporal feature map of ,in, , , are the height, width and number of channels of the spatiotemporal feature map, respectively. , , thus obtaining the spatiotemporal feature maps of all enhanced video clips;

[0027] (3-2) Convert the original video clip Keyframes Input into the target detection model based on 2D convolutional neural network, and Perform target detection on the people in the image and obtain a sequence set of target bounding boxes of the people. ,in, Represents a key frame The number of targets, Represents a key frame No. The object bounding box of the object, and Represents key frames No. The horizontal and vertical coordinates of the center point of the target bounding box of the target, and Represents key frames No. The width and height of the object bounding box;

[0028] (3-3) According to the target bounding box , by scaling the video clip The corresponding spatiotemporal feature map The object bounding box of the object ,in, , , , ;

[0029] (3-4) According to the corresponding target bounding box on the spatiotemporal feature map Perform a Region of Interest Align (RoIAlign) operation by aligning the target bounding box The corresponding feature map area is uniformly sampled using the bilinear interpolation algorithm points, among which Take the value as 7, and then sample The points are subjected to the global average pooling (GAP) operation to obtain the video clips. No. object-level motion features of objects ,in, , all object-level motion features belonging to the same video clip are stacked in the first dimension to obtain the video clip The target-level motion feature matrix .

[0030] Preferably, step (4) specifically includes the following steps:

[0031] (4-1) Construct an interactive relationship encoder, which consists of a feature concatenation module and It consists of multiple self-attention layers, and the input is a video clip. The spatiotemporal feature map of and the target-level motion feature matrix , the output is the target-level motion feature matrix after the interactive relationship is constructed ;

[0032] (4-2) Use the feature splicing module to splice the input spatiotemporal feature map and the target-level motion feature matrix :First, the spatiotemporal feature map Perform a Reshape operation to adjust the spatiotemporal feature map The shape of the video clip The serialized context spatiotemporal feature matrix ; Video clip The target-level motion feature matrix With the serialized context spatiotemporal feature matrix Stitching in the first dimension to get video clips The feature concatenation matrix ,in, It is a splicing operation;

[0033] (4-3) Construct a multi-head self-attention layer, each layer has self-attention head; The input of the layer is the feature concatenation matrix , the first Row eigenvector Input into three fully connected layers to calculate the query vector , key vector Sum value vector ,in, , , For the Layer The learnable matrix of self-attention heads, The number of channels representing query features and key features is taken , use the same method to obtain the first feature concatenation matrix Key vector corresponding to row features Sum value vector , the above are three feature vectors output by a self-attention head;

[0034] (4-4) Calculate the relationship weight between the target and the context according to the following formula: ,in, represents the exponential normalization (Softmax) function, represents the inner product, and Represents the vector index of two feature concatenation matrices, Layer The feature vector after the fusion interaction of the self-attention head output is ; The outputs of all self-attention heads in the layer are concatenated along the channel dimension and input into a fully connected layer to adjust the number of channels to obtain a new feature concatenation matrix , as the input of the next multi-head self-attention layer; the output of the last multi-head self-attention layer is , take the last The row feature vector is used as the target-level motion feature matrix after constructing the interaction relationship .

[0035] Preferably, step (5) specifically includes the following steps:

[0036] (5-1) constructing a dual-path action multi-label learning module to learn the relationship between category labels, wherein the dual-path action multi-label learning module is composed of a category correlation encoder and a dual-path action classification constraint submodule;

[0037] The category relevance encoder uses self-attention to learn the relationship between action category label embeddings, and the two-way action classification constraint submodule uses a two-way action classification loss including target level and category level to constrain the prediction results;

[0038] (5-2) The category relevance encoder is composed of three fully connected layers, and the input of the category relevance encoder is a randomly initialized category label embedding matrix , the output is the category label embedding matrix constructed with correlation , where the category label embedding matrix and Each row in represents the embedded features of an action category label. It is a learnable matrix that only needs to be randomly initialized once during the entire training process; first, the category label is embedded in the matrix No. Row eigenvector Input into three fully connected layers to calculate the query vector , key vector Sum value vector ,in, , , is the learnable matrix in the category relevance encoder, The number of channels representing query features and key features is taken ; Use the same method to obtain the category tag embedding No. Row eigenvector The corresponding key vector Sum value vector ;

[0039] (5-3) Calculate the relationship weight between category labels ,in, represents the exponential normalization (Softmax) function, represents the inner product, and represents the embedding vector index of two category labels, the output of the category relevance encoder The class label embedding feature vector is , concatenate all the category tag embedding feature vectors output by the category relevance encoder to obtain the category tag embedding matrix with constructed relevance ;

[0040] (5-4) The two-way action classification constraint submodule is composed of a fully connected layer and a two-way action classification loss, wherein the fully connected layer is used to adjust the number of feature channels, and the two-way action classification loss is used to constrain the prediction results;

[0041] First, embed the category labels with correlation into the matrix Input to a fully connected layer Adjust the number of channels in the matrix to obtain a category label embedding matrix with mixed categories and constructed correlations , the target-level motion feature matrix after constructing the interaction relationship obtained in step (4) Class label embedding matrix with mixed classes and constructed correlations The transposed matrix of is multiplied to obtain the logit matrix of the target action prediction of the fused category label embedding , each row in the matrix is , indicating the The action logit value of the target, forward Each dimension in the dimensions corresponds to an action category;

[0042] (5-5) Using dual-path action classification loss The prediction results are constrained from two paths. The first constraint is the target level loss constraint. The target-level loss constraint for each target is ,in, Indicates true mark The category corresponding to the nth dimension does not exist, and the categories corresponding to the nth dimension include: action category, background category and mixed category; Indicates true mark Middle The categories corresponding to the dimensions exist. The categories corresponding to the dimensions include: action category, background category and mixed category; express No. The logit value of an action, Represents the temperature coefficient, usually takes a value of 4, and the softplus function calculation formula is: , and The value range is ; The second constraint is the category-level loss constraint, The class-level loss constraint for each class is ,in, The value range is , and Respectively represent and The first target The logit value of an action, and Respectively represent and The first target The true value of an action;

[0043] (5-6) The logit matrix After Sigmoid activation function processing, the output prediction matrix , the prediction matrix The Behavior ,Pick forward The dimensions form the action category probability vector , where each dimension corresponds to the confidence of an action category. If the confidence is greater than 0.5, it means that the action exists. The action category probability vectors form an action category probability set , the action category probability set is used as the prediction result of the action category; and a spatiotemporal action detection model is obtained which includes a video preprocessing module, an action foreground semantic enhancement module, an interactive relationship encoder and a dual-path action multi-label learning module.

[0044] Preferably, step (6) specifically includes the following steps:

[0045] (6-1) Obtain a training video set, where each training video in the training video set includes: a complete video and a label, and the label includes: a set of real target bounding boxes , action start time, end time and real action category;

[0046] Inputting the training video into the spatiotemporal action detection model;

[0047] (6-2) Using the video preprocessing module, extract frames from the training video at a fixed frame rate and sample video clips to obtain the spatial positions of all targets at each moment represented by target bounding boxes and the corresponding action categories;

[0048] (6-3) Use the method described in step (3-2) to obtain a sequence set of human target bounding boxes , the sequence set of human target bounding boxes For each predicted box in , calculate its difference from the set of true target bounding boxes The intersection-over-union ratio between all ground-truth boxes ( Intersection over Union, IoU ) , get the IoU matrix , where the intersection-over-union ratio is the area of ​​the intersection of the predicted box and the true box divided by the area of ​​the union, Represents a key frame The true mark of The object bounding box of the object, and Represents a key frame The true mark of The horizontal and vertical coordinates of the center point of the target bounding box of the target, and Represents a key frame The true mark of The width and height of the object bounding box of each object;

[0049] (6-4) The matching box pairs are filtered out by using an IoU threshold of 0.5, and then each predicted box is assigned to a true target bounding box based on the highest IoU value. If no matching true target bounding box is found, it corresponds to the background. For a predicted box , and its corresponding true target bounding box is , the real action category is marked as ,in, and Represents a key frame No. The horizontal and vertical coordinates of the center point of the real target bounding box corresponding to the target bounding box of the target, and Represents a key frame No. The width and height of the ground-truth target bounding box corresponding to the target bounding box of each target;

[0050] (6-5) Using the method described in step (2), obtain The smoothed true mark corresponding to the target , using the method described in step (5-4) to obtain The action logit value of the target , the classification loss adopts the dual-path action classification loss , where the first constraint is the target-level loss constraint, The target-level loss constraint for each target is , Indicates true mark The category corresponding to the nth dimension does not exist, and the categories corresponding to the nth dimension include: action category, background category and mixed category; Indicates true mark Middle The categories corresponding to the dimensions exist. The categories corresponding to the dimensions include: action category, background category and mixed category; express No. The logit value of an action, Represents the temperature coefficient, usually takes a value of 4, and the softplus function calculation formula is: , and The value range is ; The second constraint is the category-level loss constraint, The class-level loss constraint for each class is ,in, The value range is , and Respectively represent and The first target The logit value of an action, and Respectively represent and The first target The true value of an action;

[0051] (6-6) Positioning loss using ,in, is the generalized intersection over union loss. express and The minimum bounding rectangle formed, Indicates the area enclosed by the box;

[0052] (6-7) Construct the total loss, the formula is as follows: ,in, and are the weight coefficients of classification loss and positioning loss respectively, and the above spatiotemporal action detection model is optimized by using stochastic gradient descent algorithm, and the model is iteratively trained until the model converges to obtain the optimized spatiotemporal action detection model;

[0053] Preferably, step (7) specifically includes the following steps:

[0054] (7-1) The spatiotemporal action detection model trained on the video input to be detected is used to obtain the predicted target bounding box and the class probability set , output Action categories corresponding to dimensions with probability greater than 0.5 ,in, represents the number of predicted targets, if Indicates a video clip The The target exists action, if , then the action does not exist;

[0055] (7-2) The target bounding box obtained according to (7-1) and action category , determine the start and end time of the action: from the video clip Start processing and add the target bounding box in this fragment As the initial tracking object, and record their action categories , for video clips , the current video clip middle Each object bounding box is associated with the next video segment middle The intersection and union ratio of each target bounding box is calculated and the Hungarian algorithm is used for matching. The input is and , the output is the optimal target matching set ,in, Indicates Seconds The object bounding box and Seconds A pair of object bounding boxes, Indicates Seconds video clip and The number of object bounding box pairs per second video clip;

[0056] The idea of ​​the Hungarian algorithm here is to construct a cost matrix (where the matrix elements represent the matching cost between the detection box in the current frame and the detection box in the next frame (such as 1-IoU)), and then find a minimum cost matching scheme so that each detection box matches only one target, thereby achieving the optimal association between targets;

[0057] Then, the optimal target matching set per second is According to the same target bounding box, the front and back connections are made to obtain a set of action pipelines ,in, Indicates the number of action pipelines, which are finally composed of action pipeline sets and action categories Determine the start and end times of all actions and the target bounding boxes at the corresponding times , start time The action pipeline is first classified as The moment of an action, the end time The last detected A moment of action, Indicates that from Seconds to Seconds The target bounding box corresponding to each action, For the The action category of the action, is the number of actions in the video sequence.

[0058] The present invention aims at the deviation and multi-label problems in the context of spatiotemporal action detection. The method has the following advantages: 1) The action foreground semantic enhancement module integrates the background, enhances the action foreground semantics, and models the target action category label, reducing the semantic deviation and improving the generalization performance of the model; 2) The dual-path action multi-label learning module depicts the intrinsic association of different action labels to alleviate the label sparsity problem, and the dual-path action classification loss (target-level constraint and category-level constraint) is used to constrain the prediction results, alleviating the problem of imbalance between positive and negative categories in multi-label classification. This provides an accurate spatiotemporal action detection method for practical application scenarios of intelligent monitoring, sports analysis, human-computer interaction, entertainment and games. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flow chart of the method of the present invention.

[0060] Figure 2 This is an example of the effect of the foreground semantic enhancement module.

[0061] Figure 3 This is an example of the effect of the dual-path action multi-label learning module.

[0062] Figure 4 This is the overall framework diagram of the dual-path multi-label spatiotemporal action detection method. DETAILED DESCRIPTION

[0063] The present invention provides a dual-path multi-label spatiotemporal action detection method based on foreground semantic enhancement, comprising the following steps:

[0064] Step (1) constructing a video preprocessing module to preprocess the original video to obtain a background set and a video clip set;

[0065] Step (2) constructing an action foreground semantic enhancement module, the input of which is a background set and a video clip set, and the output of which is a video clip set with action foreground semantic enhancement;

[0066] Step (3) extracting features using two-dimensional convolutional and three-dimensional convolutional neural networks, with the input being a set of enhanced video clips and the output being a target bounding box, target-level spatiotemporal features, and a video spatiotemporal feature map;

[0067] Step (4) constructing an interaction relationship encoder, the input of which is the target-level spatiotemporal features and the video spatiotemporal feature map, and the output of which is the target-level motion features after the interaction relationship is constructed;

[0068] Step (5) constructing a dual-path action multi-label learning module, the input of which is the target-level motion feature embedded with the category label and constructed with the interaction relationship, and the output of which is the predicted action category; obtaining a spatiotemporal action detection model which includes a video preprocessing module, an action foreground semantic enhancement module, an interaction relationship encoder and a dual-path action multi-label learning module in sequence;

[0069] Step (6) training the spatiotemporal action detection model;

[0070] Calculate the target location and action classification losses, and then use the stochastic gradient descent algorithm to optimize the spatiotemporal action detection model composed of the interactive relationship encoder and the dual-path action multi-label learning module. Perform steps (1) to (5) on the new video sequence to obtain the spatial position and action category of all targets at different times until the model converges.

[0071] Step (7) obtains the video sequence to be detected and inputs it into the trained spatiotemporal action detection model to obtain the predicted target bounding box and action category probability, and then obtains the action start time, end time and action category.

[0072] Further, step (1) is specifically:

[0073] (1-1) Construct a video preprocessing module, obtain the original video, input the video preprocessing module, and process it at a rate of The frame sampling rate, The value of is 30, the original video is sampled, and the number of frames obtained is A collection of video frame sequences ,in, Represents the first Frame video frame, represents the field of real numbers, and Respectively represent the height and width of the video frame, and 3 represents the RGB three channels;

[0074] (1-2) Using an object detector, the object detector Faster R-CNN model, to collect a set of video frame sequences Perform target detection and select a set of video frame sequences based on the detection results The frames that do not contain people are used as background frames, and the background frames are used to form a background set ,in, Represents the background collection The Background frames, Represents the background collection The number of background frames;

[0075] (1-3) From a video frame sequence collection The Frame starts, make sure Frame, last The last frame is not a key frame. Take a frame as the key frame for subsequent segmentation of video clips to obtain a key frame set ,in, Indicates the total duration of the original video (in seconds). Indicates Keyframes of a video clip of 1 second, as a collection of keyframes Keyframes in As the center, from the video frame sequence collection The intercept length is Frame video clip , and finally get a collection of video clips ,in, Indicates seconds of video clip, from the key frame before Frame video frame, keyframe and keyframe after Frame video frame composition, Indicates half the length of the video clip.

[0076] The video data set described in steps (1-1) to (1-3) is an AVA spatiotemporal action detection data set, and the data source comes from movie clips; the length of the divided video clips is The total number of action categories in the dataset is 80.

[0077] Further, step (2) is specifically:

[0078] Constructing an action foreground semantic enhancement module, which is used to fuse the original video clip with a static video clip composed of a randomly selected single background image, increase the diversity of action and background combinations, reduce the frequency of certain backgrounds and specific actions appearing at the same time, reduce the model's excessive dependence on the background, and enable the model to learn more complex action foreground semantic features, thereby alleviating the deviation problem; the action foreground semantic enhancement module is composed of a video fusion submodule and an action label smoothing module;

[0079] The video fusion submodule is used to perform the video fusion operation, including the following steps: Randomly select a frame as the background frame , copied along the time dimension to obtain a length of Frames of static background video , and with the video clip Linear addition is performed to achieve fusion and obtain the enhanced video clip ,in, represents the fusion ratio, Indicates a video clip The Frame video frame;

[0080] The motion tagging smoothing module is used to smooth the video clips. The action category mark of each target is smoothed to obtain the smoothed action category mark, which specifically includes the following steps:

[0081] Video Clip Middle The smoothed true action category label vector corresponding to the target is defined as , where the vector forward Dimensions express action categories, vector No. Dimensions Represents the background category, vector at last Dimensions Represents the mixed category consisting of the corresponding action category and the background category, vector The value range of each dimension is , the closer the dimension value is to 1, the greater the possibility of the existence of this category;

[0082] The video clip is calculated according to the following formula Middle The smoothed true action category label vector corresponding to the target ,in, is the uncertainty, , , and The categories corresponding to each dimension in Consistently, the categories corresponding to each dimension include: action category, background category and mixed category; represents the real action category label vector, forward The corresponding dimension in which an action exists takes the value of 1, and the values ​​of other dimensions are all 0. represents the background category label vector, No. The value of one dimension is 1, and the values ​​of other dimensions are 0. represents the action-centered fuzzy decision space label vector, at last The corresponding dimension of the mixed category with actions in the dimension is taken as 1, and the values ​​of other dimensions are all 0. represents the fuzzy decision space label vector centered on the background, at last The value of one dimension is 1, and the values ​​of other dimensions are 0.

[0083] The action foreground semantic enhancement module in step (2) merges the original video clip with a static video clip composed of a single randomly selected background frame, thereby increasing the diversity of action and background combinations, reducing the frequency of certain backgrounds and specific actions appearing at the same time, reducing the model's over-reliance on the background, and prompting the model to learn more complex action foreground semantic features, thereby alleviating the bias problem;

[0084] Furthermore, step (3) is specifically:

[0085] (3-1) Video clips Input into a 3D convolutional neural network to generate video clips The spatiotemporal feature map of ,in, , and are the height, width and number of channels of the spatiotemporal feature map, respectively. , , thus obtaining the spatiotemporal feature maps of all enhanced video clips;

[0086] The three-dimensional convolutional neural network used in the step (3-1) is the mainstream action recognition model SlowFast in the field. The three-dimensional convolutional neural network is used to obtain the enhanced spatiotemporal feature map of the video clip for subsequent action detection.

[0087] (3-2) Convert the original video clip Keyframes Input into the target detection model based on 2D convolutional neural network, and Perform target detection on the people in the image and obtain a sequence set of target bounding boxes of the people. ,in, Represents a key frame The number of targets, Represents a key frame No. The object bounding box of the object, and Represents key frames No. The horizontal and vertical coordinates of the center point of the target bounding box of the target, and Represents key frames No. The width and height of the object bounding box of each object;

[0088] The above step (3-2) selects the original video clip Keyframes in Performing target detection without selecting key frames in the enhanced video clip for target detection can ensure that target detection is not interfered by fused background frames; the two-dimensional convolutional neural network used is the mainstream target detection model Faster-RCNN in this field, and the target detector can detect all people in the video frame.

[0089] (3-3) According to the target bounding box , by scaling the video clip The corresponding spatiotemporal feature map The object bounding box of the object ,in, , , , ;

[0090] (3-4) According to the corresponding target bounding box on the spatiotemporal feature map Perform a Region of Interest Align (RoIAlign) operation by aligning the target bounding box The corresponding feature map area is uniformly sampled using the bilinear interpolation algorithm points, among which Take the value as 7, and then sample The points are subjected to the global average pooling (GAP) operation to obtain the video clips. No. object-level motion features of objects ,in, , all object-level motion features belonging to the same video clip are stacked in the first dimension to obtain the video clip The target-level motion feature matrix .

[0091] Steps (3-3) to (3-4) map the bounding box of the target in the original video clip to the spatiotemporal feature map of the video clip by scaling it proportionally to obtain the corresponding target bounding box; then, perform the RoIAlign operation on the feature map area corresponding to these bounding boxes, and use the bilinear interpolation algorithm to uniformly sample points, and then global average pooling (GAP) is applied to these sample points to extract a compact representation of each target - the target-level motion feature; finally, the target-level motion features of all targets in the same video clip are stacked along the first dimension to form the target-level motion feature matrix of the video clip; this process not only ensures the precise alignment of the features, but also enhances the robustness and representation ability of the features through global average pooling.

[0092] Going further, step (4) is specifically:

[0093] (4-1) Construct an interactive relationship encoder, which consists of a feature concatenation module and It consists of multiple self-attention layers, and the input is a video clip. The spatiotemporal feature map of and the target-level motion feature matrix , the output is the target-level motion feature matrix after the interactive relationship is constructed ;

[0094] (4-2) Use the feature splicing module to splice the input spatiotemporal feature map and the target-level motion feature matrix :First, the spatiotemporal feature map Perform a Reshape operation to adjust the spatiotemporal feature map The shape of the video clip The serialized context spatiotemporal feature matrix ; Video clip The target-level motion feature matrix With the serialized context spatiotemporal feature matrix Stitching in the first dimension to get video clips The feature concatenation matrix ,in, It is a splicing operation;

[0095] (4-3) Construct a multi-head self-attention layer. Each multi-head self-attention layer has self-attention head; The input of the layer is the feature concatenation matrix , the first Row eigenvector Input into three fully connected layers to calculate the query vector , key vector Sum value vector ,in, , , For the Layer The learnable matrix of self-attention heads, The number of channels representing query features and key features is taken , the same method is used to obtain the first feature concatenation matrix Key vector corresponding to row features Sum value vector , the above are three feature vectors output by a self-attention head;

[0096] (4-4) Calculate the relationship weight between the target and the context according to the following formula: ,in, represents the exponential normalization (Softmax) function, represents the inner product, and Represents the vector index of two feature concatenation matrices, Layer The feature vector after the fusion interaction of the self-attention head output is ; The outputs of all self-attention heads in the layer are concatenated along the channel dimension and input into a fully connected layer to adjust the number of channels to obtain a new feature concatenation matrix , as the input of the next multi-head self-attention layer; the output of the last multi-head self-attention layer is , take the last The row feature vector is used as the target-level motion feature matrix after constructing the interaction relationship .

[0097] The steps (4-1) to (4-4) described above are used to construct the interactive relationship between the target and the context. The interactive relationship encoder fuses the spatiotemporal features of the video clips with the target-level motion features through a feature splicing module and a multi-layer multi-head self-attention mechanism, thereby capturing the complex relationship between the target and the context. First, the spatiotemporal feature map is serialized and spliced ​​with the target-level motion feature matrix to form a feature splicing matrix. Then, the relationship weight between the target and the context is calculated through a multi-head self-attention layer, and these relationship information are fused to finally obtain an enhanced target-level motion feature matrix, thereby improving the understanding and representation capabilities of the interaction between targets.

[0098] Going further, step (5) is specifically:

[0099] (5-1) Construct a dual-path action multi-label learning module to learn the relationship between category labels. The module consists of a category relevance encoder and a dual-path action classification constraint submodule;

[0100] The category relevance encoder uses self-attention to learn the relationship between action category label embeddings, and the two-way action classification constraint submodule uses two-way action classification loss to constrain the prediction results from both the object level and the category level.

[0101] (5-2) The category relevance encoder is composed of three fully connected layers, and the input of the category relevance encoder is a randomly initialized category label embedding matrix , the output is the category label embedding matrix constructed with correlation , where the category label embedding matrix and Each row in represents the embedded features of an action category label. It is a learnable matrix that only needs to be randomly initialized once during the entire training process; first, the category label is embedded in the matrix No. Row eigenvector Input into three fully connected layers to calculate the query vector , key vector Sum value vector ,in, , , is the learnable matrix in the category relevance encoder, The number of channels representing query features and key features is taken ; Use the same method to obtain the category tag embedding No. Row eigenvector The corresponding key vector Sum value vector ;

[0102] (5-3) Calculate the relationship weight between category labels ,in, represents the exponential normalization (Softmax) function, represents the inner product, and represents the embedding vector index of two category labels, the output of the category relevance encoder The class label embedding feature vector is , concatenate all the category tag embedding feature vectors output by the category relevance encoder to obtain the category tag embedding matrix with constructed relevance ;

[0103] In the steps (5-2) to (5-3), the module generates query, key and value vectors through three fully connected layers, and uses the self-attention mechanism to calculate the relationship weights between category labels, thereby enhancing the correlation between categories; this process enables the model to better understand the intrinsic connections between different action categories and improve the accuracy of action classification.

[0104] (5-4) The two-way action classification constraint submodule is composed of a fully connected layer and a two-way action classification loss, wherein the fully connected layer is used to adjust the number of feature channels, and the two-way action classification loss is used to constrain the prediction results;

[0105] First, embed the category labels with correlation into the matrix Input to a fully connected layer Adjust the number of channels in the matrix to obtain a category label embedding matrix with mixed categories and constructed correlations , the target-level motion feature matrix after constructing the interaction relationship obtained in step (4) Class label embedding matrix with mixed classes and constructed correlations The transposed matrix of is multiplied to obtain the logit matrix of the target action prediction of the fused category label embedding , each row in the matrix is , indicating the The action logit value of the target, forward Each dimension in the dimensions corresponds to an action category;

[0106] (5-5) Using dual-path action classification loss The prediction results are constrained from two paths. The first constraint is the target level loss constraint. The target-level loss constraint for each target is ,in, Indicates true mark The category corresponding to the nth dimension does not exist, and the categories corresponding to the nth dimension include: action category, background category and mixed category; Indicates true mark Middle The categories corresponding to the dimensions exist. The categories corresponding to the dimensions include: action category, background category and mixed category; express No. The logit value of an action, Represents the temperature coefficient, usually takes a value of 4, and the softplus function calculation formula is: , and The value range is ; The second constraint is the category-level loss constraint, The class-level loss constraint for each class is ,in, The value range is , and Respectively represent and The first target The logit value of an action, and Respectively represent and The first target The true value of an action;

[0107] (5-6) The logit matrix After Sigmoid activation function processing, the output prediction matrix , the prediction matrix The Behavior ,Pick forward The dimensions form the action category probability vector , where each dimension corresponds to the confidence of an action category. If the confidence is greater than 0.5, it means that the action exists. The action category probability vectors form an action category probability set , the action category probability set is used as the prediction result of the action category; and a spatiotemporal action detection model is obtained which includes a video preprocessing module, an action foreground semantic enhancement module, an interactive relationship encoder and a dual-path action multi-label learning module.

[0108] In the steps (5-5) to (5-6), the dual-path action classification loss imposes constraints on the prediction results through two paths: the target level and the category level. The target level loss ensures that the prediction of each target is consistent with the true label, while the category level loss balances the prediction errors between different categories, thereby effectively alleviating the imbalance problem of positive and negative categories in multi-label action classification and improving the accuracy of action classification.

[0109] Furthermore, step (6) is specifically:

[0110] (6-1) Obtain a training video set, where each training video in the training video set includes: a complete video and a label, and the label includes: a set of real target bounding boxes , action start time, end time and real action category;

[0111] Inputting the training video into the spatiotemporal action detection model;

[0112] (6-2) Using the video preprocessing module, extract frames from the training video at a fixed frame rate and sample video clips to obtain the spatial positions of all targets at each moment represented by target bounding boxes and the corresponding action categories;

[0113] (6-3) Use the method described in step (3-2) to obtain a sequence set of human target bounding boxes , the sequence set of human target bounding boxes For each predicted box in , calculate its difference from the set of true target bounding boxes The Intersection over Union (IoU) between all real boxes is used to obtain the IoU matrix , where the intersection-over-union ratio is the area of ​​the intersection of the predicted box and the true box divided by the area of ​​the union, Represents a key frame The true mark of The object bounding box of the object, and Represents a key frame The true mark of The horizontal and vertical coordinates of the center point of the target bounding box of the target, and Represents a key frame The true mark of The width and height of the object bounding box of each object;

[0114] (6-4) The matching box pairs are filtered out by using an IoU threshold of 0.5, and then each predicted box is assigned to a true target bounding box based on the highest IoU value. If no matching true target bounding box is found, it corresponds to the background. For a predicted box , and its corresponding true target bounding box is , the real action category is marked as ,in, and Represents a key frame No. The horizontal and vertical coordinates of the center point of the real target bounding box corresponding to the target bounding box of the target, and Represents a key frame No. The width and height of the ground-truth target bounding box corresponding to the target bounding box of each target;

[0115] (6-5) Using the method described in step (2), obtain The smoothed true mark corresponding to the target , using the method described in step (5-4) to obtain The action logit value of the target , the classification loss adopts the dual-path action classification loss , where the first constraint is the target-level loss constraint, The target-level loss constraint for each target is , Indicates true mark The category corresponding to the nth dimension does not exist, and the categories corresponding to the nth dimension include: action category, background category and mixed category; Indicates true mark Middle The categories corresponding to the dimensions exist. The categories corresponding to the dimensions include: action category, background category and mixed category; express No. The logit value of an action, Represents the temperature coefficient, usually takes a value of 4, and the softplus function calculation formula is: , and The value range is ; The second constraint is the category-level loss constraint, The class-level loss constraint for each class is ,in, The value range is , and Respectively represent and The first target The logit value of an action, and Respectively represent and The first target The true value of an action;

[0116] (6-6) Positioning loss using ,in, is the generalized intersection over union loss. express and The minimum bounding rectangle formed, Indicates the area enclosed by the box;

[0117] (6-7) Construct the total loss, the formula is as follows: ,in, and are the weight coefficients of classification loss and positioning loss respectively, and the above spatiotemporal action detection model is optimized by using stochastic gradient descent algorithm, and the model is iteratively trained until the model converges to obtain the optimized spatiotemporal action detection model;

[0118] In the step (6), the performance evaluation index of the trained model is the frame-level mean average precision (f-mAP). mAP comprehensively considers the average precision (AP) of all categories at the frame level, and calculates the average AP of all categories to obtain the overall performance index.

[0119] Going further, step (7) is specifically:

[0120] (7-1) The spatiotemporal action detection model trained on the video input to be detected is used to obtain the predicted target bounding box and the class probability set , output Action categories corresponding to dimensions with probability greater than 0.5 ,in, represents the number of predicted targets, if Indicates a video clip The The target exists action, if , then the action does not exist;

[0121] (7-2) The target bounding box obtained according to (7-1) and action category , determine the start and end time of the action: from the video clip Start processing and add the target bounding box in this fragment As the initial tracking object, and record their action categories , for video clips , the current video clip middle Each object bounding box is associated with the next video segment middle The intersection and union ratio of each target bounding box is calculated and the Hungarian algorithm is used for matching. The input is and , the output is the optimal target matching set ,in, Indicates Seconds The object bounding box and Seconds A pair of object bounding boxes, Indicates Seconds video clip and The number of object bounding box pairs per second video clip;

[0122] The idea of ​​the Hungarian algorithm here is to construct a cost matrix (where the matrix elements represent the matching cost between the detection box in the current frame and the detection box in the next frame (such as 1-IoU)), and then find a minimum cost matching scheme so that each detection box matches only one target, thereby achieving the optimal association between targets;

[0123] Then, the optimal target matching set per second is According to the same target bounding box, the front and back connections are made to obtain a set of action pipelines ,in, Indicates the number of action pipelines, which are finally composed of action pipeline sets and action categories Determine the start and end times of all actions and the target bounding boxes at the corresponding times , start time The action pipeline is first classified as The moment of action, the end time The last detected A moment of action, Indicates that from Seconds to Seconds The target bounding box corresponding to each action, For the The action category of the action, is the number of actions in the video sequence.

[0124] The present invention proposes a dual-path multi-label spatiotemporal action detection method based on foreground semantic enhancement. The method has the following characteristics: 1) The action foreground semantic enhancement module is used to fuse the background to obtain a video clip with action foreground semantic enhancement. The action foreground semantics is enhanced and modeled with the target action category to reduce the semantic deviation and improve the generalization performance of the model; 2) The self-attention layer of the dual-path action multi-label learning module is used to characterize the intrinsic association of different action labels to alleviate the label sparsity problem. At the same time, in order to solve the positive and negative category imbalance problem caused by the binary cross entropy loss function, two constraints are added: target-level loss constraint and category-level loss constraint. The relative relationship between categories is processed by introducing the temperature parameter To adjust logits, thereby expanding the classification interval between positive and negative classes and improving the accuracy of action detection.

[0125] The present invention will be further described below in conjunction with the accompanying drawings. Figure 1 , a dual-path multi-label spatiotemporal action detection method based on foreground semantic enhancement, firstly, the given video is sampled through the video preprocessing module to obtain a set of video clips and a set of backgrounds; the background is fused through the action foreground semantic enhancement module to obtain a set of video clips with action foreground semantic enhancement; then the target bounding box is obtained by using two-dimensional convolution and three-dimensional convolution neural networks, and the target-level spatiotemporal features and video spatiotemporal feature maps are extracted; then, the interactive relationship encoder is used to construct the interactive relationship of the target to obtain the target-level motion features with the constructed interactive relationship; then, the relationship between the action tags is learned through the dual-path action multi-label learning module, and the predicted action category is output; finally, a new video is input and the previous steps are repeated until the model converges; the new video sequence is input into the trained model to obtain the predicted target bounding box and action category probability, and the action start time, end time and action category are obtained accordingly. This method uses the action foreground semantic enhancement module to fuse the background, enhance the action foreground semantics, model the target action category to reduce semantic deviation and improve the generalization performance of the model, and then uses the dual-path action multi-label learning module to characterize the internal association of different action tags to alleviate the problem of sparse tags.

[0126] like Figure 2As shown in the figure, red text and red bounding box represent real labels, blue text and blue bounding box represent predicted labels, and labels include action categories and target bounding boxes. The action of "swimming" in the training set will appear at the same time as the background of the water surface. During the training process, the model will over-rely on the background of the water surface to identify the action of "swimming", resulting in deviation problems. When tested on the test set, the model without adding the foreground semantic enhancement module predicted the action categories: rowing, sitting and swimming, and mistakenly judged that there was a "swimming" action in the current video clip; after adding the foreground semantic enhancement module, the model correctly detected the "rowing" and "sitting" actions of the target, and did not mistakenly judge that there was a "swimming" action in the current video clip; the foreground semantic enhancement module is used to alleviate the deviation problem and improve the generalization performance of the model;

[0127] like Figure 3 As shown in the figure, red text and red bounding box represent real labels, blue text and blue bounding box represent predicted labels, and labels include target bounding box and action category. The action of "answering the phone" is highly correlated with the action of "talking to people". When the dual-path action multi-label learning module is not added, the model predicts the action categories as answering the phone and sitting, and cannot detect the action of "talking to people". After adding the dual-path action multi-label learning module, the predicted action categories are: answering the phone, sitting and talking to people. The model learns the intrinsic association between different action labels, and correctly detects the existence of the target action of "talking to people" by detecting the action of "answering the phone". The relationship between action category labels is learned by the dual-path action multi-label learning module, and the dual-path action classification loss is used to constrain the prediction results from both the target level and the category level, alleviating the problem of imbalance between positive and negative categories in multi-label classification and improving the accuracy of action detection.

[0128] The contents described in this embodiment are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as being limited to the specific forms described in the embodiment. The protection scope of the present invention also extends to the equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. Dual-path multi-label spatiotemporal action detection method based on foreground semantic enhancement, It is characterized in that The following steps are involved: Step (1) constructing a video preprocessing module to preprocess the original video to obtain a background set and a video clip set; Step (2) constructing an action foreground semantic enhancement module, the input of which is a background set and a video clip set, and the output of which is a video clip set with action foreground semantic enhancement; Step (3) extracting features using two-dimensional convolutional and three-dimensional convolutional neural networks, with the input being a set of enhanced video clips and the output being a target bounding box, target-level motion features, and a video spatiotemporal feature map; Step (4) constructing an interactive relationship encoder, the input of which is the target-level motion feature and the video spatiotemporal feature map, and the output of which is the target-level motion feature after the interactive relationship is constructed; Step (5) construct a dual-path action multi-label learning module, the input is the category label embedding and the target-level motion features constructed with the interaction relationship, and the output is the predicted action category; A spatiotemporal action detection model is obtained which includes a video preprocessing module, an action foreground semantic enhancement module, an interaction relation encoder and a dual-path action multi-label learning module in sequence; Step (6) training the spatiotemporal action detection model using a stochastic gradient descent algorithm until the model converges; Step (7) obtaining a video sequence to be detected, inputting it into the trained spatiotemporal action detection model, obtaining a predicted target bounding box and action category probability, and then obtaining the action start time, end time and action category; The step (1) specifically comprises the following steps: (1-1) Construct a video preprocessing module, obtain the original video, input the video preprocessing module, and process it at N per second. f The sampling rate of the frame is used to sample the original video to obtain a video frame sequence set with a frame number T′. Among them, I t represents the tth video frame in a video frame sequence set, represents the real number domain, H and W represent the height and width of the video frame respectively, and 3 represents the RGB three channels; (1-2) Using the target detector, the video frame sequence is collected Perform target detection and select a set of video frame sequences based on the detection results The frames that do not contain people are used as background frames, and the background frames are used to form a background set Among them, u t 'Represents the background collection The t'th background frame in N u Represents the background collection The number of background frames; (1-3) From the video frame sequence set The Nth f Frame starts, make sure the Nth f Frame, last N f +1 frame and the last frame are not key frames. f Take a frame as the key frame for subsequent segmentation of video clips to obtain a key frame set Where T represents the total duration of the original video in seconds. Represents the key frame of the video clip at the sth second, as a key frame set Keyframes in As the center, from the video frame sequence collection Extract a video clip of length M' frames Finally, we get a collection of video clips in, Represents the s-second video clip, which consists of M video frames before the key frame, the key frame, and M-1 video frames after the key frame. Indicates half the length of the video clip; The step (2) specifically comprises the following steps: Constructing an action foreground semantic enhancement module, wherein the action foreground semantic enhancement module is composed of a video fusion submodule and an action labeling smoothing module; The video fusion submodule is used to perform the video fusion operation, including the following steps: Randomly select a frame as the background frame Copy along the time dimension to obtain a static background video with a length of M' frames And with video clips Linear addition is performed to achieve fusion and obtain the enhanced video clip Among them, α>0 indicates the fusion ratio, Indicates a video clip The t'th frame of the video frame; The motion tagging smoothing module is used to smooth the video clips. The action category mark of each target is smoothed to obtain the smoothed action category mark, which specifically includes the following steps: Video Clip The smoothed true action category label vector corresponding to the j-th target in is defined as Among them, the vector Top N C Dimensions N C action categories, vector Nth C +1 dimension Represents the background category, vector Last N C Dimensions Represents the mixed category consisting of the corresponding action category and the background category, vector The value range of each dimension is [0,1]; The video clip is calculated according to the following formula The smoothed true action category label vector corresponding to the jth target in Among them, u = min (α, 1-α) is the uncertainty, and The categories corresponding to each dimension in Consistently, the categories corresponding to each dimension include: action category, background category and mixed category; represents the real action category label vector, Top N C The corresponding dimension with action in the dimension is taken as 1, and the values ​​of other dimensions are all 0. s represents the background category label vector, b s Nth C +1 dimension takes value 1, and the other dimensions take value 0. represents the action-centered fuzzy decision space label vector, Last N C The corresponding dimension of the mixed category with action in the dimension is 1, and the values ​​of other dimensions are 0. represents the fuzzy decision space label vector centered on the background, Last N C +1 dimension takes value 1, and the other dimensions take value 0; The step (3) specifically comprises the following steps: (3-1) Video clips Input into a 3D convolutional neural network to generate video clips The spatiotemporal feature map of Among them, H', W', and C are the height, width, and number of channels of the spatiotemporal feature map, respectively. Thus, the spatiotemporal feature maps of all enhanced video clips are obtained; (3-2) The original video clip Keyframes Input into the target detection model based on 2D convolutional neural network, and Perform target detection on the people in the image and obtain a sequence set of target bounding boxes of the people. Among them, N s Represents a key frame The number of targets, Represents a key frame The target bounding box of the i-th target, and Represents key frames The horizontal and vertical coordinates of the center point of the target bounding box of the i-th target, and Represents key frames The width and height of the target bounding box of the i-th target; (3-3) According to the target bounding box By scaling the video clip The target bounding box of the i-th target corresponding to the spatiotemporal feature map in, (3-4) According to the corresponding target bounding box on the spatiotemporal feature map Perform region of interest alignment by aligning the bounding box of the target The bilinear interpolation algorithm is used to uniformly sample n'×n' points in the corresponding feature map area, where n' is 7, and then the global average pooling operation is performed on the sampled n'×n' points to obtain the video clip. The object-level motion features of the i-th object Where i = 1, 2, ..., N s , all object-level motion features belonging to the same video clip are stacked in the first dimension to obtain the video clip The target-level motion feature matrix The step (4) specifically comprises the following steps: (4-1) Construct an interactive relationship encoder, which consists of a feature concatenation module and N l It consists of multiple self-attention layers, and the input is a video clip. The spatiotemporal feature map F s and the target-level motion feature matrix H s , the output is the target-level motion feature matrix after the interactive relationship is constructed (4-2) Use the feature splicing module to splice the input spatiotemporal feature map F s and the target-level motion feature matrix H s :First, the spatiotemporal feature map F s Perform a Reshape operation to adjust the spatiotemporal feature map F s The shape of the video clip The serialized context spatiotemporal feature matrix Video Clip The target-level motion feature matrix H s and the serialized context spatiotemporal feature matrix F s 'Sticking in the first dimension to get the video clip The feature concatenation matrix Among them, [·,·] is the concatenation operation; (4-3) Construct a multi-head self-attention layer. Each multi-head self-attention layer has N h self-attention heads; the input of the lth layer is the feature concatenation matrix The mth feature concatenation matrix 1 Row eigenvector Input into three fully connected layers to calculate the query vector Key Vector Sum value vector in, is the learnable matrix of the hth self-attention head in the lth layer, d represents the number of channels of query features and key features, and d<C is taken. The mth feature concatenation matrix is ​​obtained in the same way. 2 Key vector corresponding to row features Sum value vector The above are three feature vectors output by a self-attention head; (4-4) Calculate the relationship weight between the target and the context according to the following formula Among them, softmax(·) represents the exponential normalization function, <·,·> represents the inner product, and m 1 and m 2 Represents the vector index of the two feature concatenation matrices. The feature vector after the fusion interaction relationship output by the h-th self-attention head of the l-th layer is The outputs of all self-attention heads in layer l are concatenated along the channel dimension and input into a fully connected layer to adjust the number of channels to obtain a new feature concatenation matrix As the input of the next multi-head self-attention layer; the output of the last multi-head self-attention layer is Take the last N s The row feature vector is used as the target-level motion feature matrix H' after constructing the interaction relationship s ; The step (5) specifically comprises the following steps: (5-1) constructing a dual-path action multi-label learning module to learn the relationship between category labels, wherein the dual-path action multi-label learning module is composed of a category correlation encoder and a dual-path action classification constraint submodule, wherein the category correlation encoder uses self-attention to learn the relationship between action category label embeddings, and the dual-path action classification constraint submodule uses a dual-path action classification loss to constrain the prediction results from two paths, namely, the target level and the category level; (5-2) The category relevance encoder is composed of three fully connected layers, and the input of the category relevance encoder is a randomly initialized category label embedding matrix The output is the category label embedding matrix constructed with correlation Among them, the category label embedding matrix L 0 and L 1 Each row in L represents the embedded features of an action category label. 0 It is a learnable matrix that only needs to be randomly initialized once during the entire training process; first, the category label is embedded in the matrix L 0 The mth 3 Row eigenvector Input into three fully connected layers to calculate the query vector Key Vector Sum value vector in, is the learnable matrix in the category relevance encoder, d represents the number of channels of query features and key features, and d < C is taken; the category label embedding L is obtained by the same method 0 The mth 4 Row eigenvector The corresponding key vector Sum value vector (5-3) Calculate the relationship weight between category labels Among them, softmax(·) represents the exponential normalization function, <·,·> represents the inner product, and m 3 and m 4 Represents the embedding vector index of two category labels, the mth output of the category relevance encoder 3 The class label embedding feature vector is Concatenate all the category mark embedding feature vectors output by the category relevance encoder to obtain the category mark embedding matrix L constructed with correlation 1 ; (5-4) The two-way action classification constraint submodule is composed of a fully connected layer and a two-way action classification loss, wherein the fully connected layer is used to adjust the number of feature channels, and the two-way action classification loss is used to constrain the prediction results; First, embed the category labels with built correlation into the matrix L 1 Input to a fully connected layer Adjust the number of channels in the matrix to obtain a category label embedding matrix with mixed categories and constructed correlations The target-level motion feature matrix H' obtained in step (4) after constructing the interactive relationship s Perform matrix multiplication with the transposed matrix of the category tag embedding matrix L with the constructed correlation and mixed categories to obtain the logit matrix of the target action prediction of the fused category tag embedding Each row in the matrix represents the action logit value of the i-th target, Top N C Each dimension in the dimensions corresponds to an action category; (5-5) Using dual-path action classification loss The prediction results are constrained from two paths. The first constraint is the target-level loss constraint. The target-level loss constraint of the i-th target is in, Indicates true mark The category corresponding to the nth dimension does not exist, and the categories corresponding to the nth dimension include: action category, background category and mixed category; Indicates true mark The category corresponding to the p-th dimension exists, and the category corresponding to the p-th dimension includes: action category, background category and mixed category; express The logit value of the nth action, τ represents the temperature coefficient, and the softplus function calculation formula is softplus[x]=1+e x , n and p are in the range of [1,N C ]; The second constraint is the category-level loss constraint. The category-level loss constraint for the c'th category is: Among them, the value range of c' is [1,N C ], and denote the logit values ​​of the c'th action for the ith and i'th targets, respectively, and denote the true value of the c'th action of the ith and i'th targets respectively; (5-6) The logit matrix M l After Sigmoid activation function processing, the output prediction matrix The prediction matrix M' l The i-th behavior in Pick Top N C The dimensions form the action category probability vector Each dimension corresponds to the confidence of an action category. If the confidence is greater than 0.5, it means that the action exists. s The action category probability vectors form an action category probability set The action category probability set is used as the prediction result of the action category; and a spatiotemporal action detection model is obtained, which sequentially includes a video preprocessing module, an action foreground semantic enhancement module, an interactive relationship encoder and a dual-path action multi-label learning module.

2. The dual-path multi-label spatiotemporal action detection method based on foreground semantic enhancement as claimed in claim 1, It is characterized in that Step (6) specifically includes the following steps: (6-1) Obtaining a training video set, each training video in the training video set includes: a complete video and a label, and the label includes: a set of real target bounding boxes Action start time, end time and real action category; Inputting the training video into the spatiotemporal action detection model; (6-2) using the video preprocessing module to extract frames from the training video at a fixed frame rate and sample video clips to obtain the spatial positions of all targets at each moment represented by target bounding boxes and the corresponding action categories; (6-3) Using the method described in step (3-2), obtain a sequence set of human target bounding boxes The sequence collection of the target bounding box of the person For each predicted box in , calculate its difference from the set of true target bounding boxes The intersection-over-union ratio between all real boxes is obtained by Among them, the intersection-over-union ratio is the area of ​​the intersection of the predicted box and the true box divided by the area of ​​the union. Represents a key frame The object bounding box of the jth object in the true label, and Represents a key frame The horizontal and vertical coordinates of the center point of the target bounding box of the jth target in the true label, and Represents a key frame The width and height of the object bounding box of the jth object in the true label; (6-4) The matching box pairs are filtered out by using an IoU threshold of 0.5, and then each predicted box is assigned to a true target bounding box based on the highest IoU value. If no matching true target bounding box is found, it corresponds to the background. For a predicted box The corresponding true target bounding box is The real action category is marked as in, and Represents a key frame The horizontal and vertical coordinates of the center point of the real target bounding box corresponding to the target bounding box of the i-th target, and Represents a key frame The width and height of the ground-truth target bounding box corresponding to the target bounding box of the i-th target; (6-5) Use the method described in step (2) to obtain the smoothed true mark corresponding to the i-th target Use the method described in step (5-4) to get the action logit value of the i-th target The classification loss adopts dual-path action classification loss Among them, the first constraint is the target-level loss constraint, and the target-level loss constraint of the i-th target is Indicates true mark The category corresponding to the nth dimension does not exist, and the categories corresponding to the nth dimension include: action category, background category and mixed category; Indicates true mark The category corresponding to the p-th dimension exists, and the category corresponding to the p-th dimension includes: action category, background category and mixed category; express The logit value of the nth action, τ represents the temperature coefficient, and the softplus function calculation formula is softplus[x]=1+e x , n and p are in the range of [1,N C ]; The second constraint is the category-level loss constraint. The category-level loss constraint for the c'th category is Among them, the value range of c' is [1,N C ], and denote the logit values ​​of the c'th action for the ith and i'th targets, respectively, and denote the true value of the c'th action of the ith and i'th targets respectively; (6-6) Positioning loss is adopted in, is the generalized intersection-combination loss, express and The minimum enclosing rectangular frame is formed, and |·| represents the area enclosed by the frame; (6-7) Construct the total loss, the formula is as follows: Among them, λ 1 and λ 2 are the weight coefficients of classification loss and positioning loss respectively. The stochastic gradient descent algorithm is used to optimize the above spatiotemporal action detection model. The model is iteratively trained until the model converges to obtain the optimized spatiotemporal action detection model.

3. The dual-path multi-label spatiotemporal action detection method based on foreground semantic enhancement as claimed in claim 2, It is characterized in that Step (7) specifically includes the following steps: (7-1) The spatiotemporal action detection model trained on the video input to be detected is used to obtain the predicted target bounding box and the class probability set Output Action categories corresponding to dimensions with probability greater than 0.5 Among them, N s represents the number of predicted targets, if Indicates a video clip The i-th goal in has the k-th action if Then the action does not exist; (7-2) The target bounding box obtained according to (7-1) and action category Determine the start and end time of an action: From video clips Start processing and add the target bounding box in this fragment As the initial tracking object, and record their action categories For video clips The current video clip middle Each object bounding box is associated with the next video segment middle The intersection and union ratio of each target bounding box is calculated and the Hungarian algorithm is used for matching. The input is and The output is the optimal target matching set in, N represents the pairing of the i'th target bounding box at the sth second and the i'th target bounding box at the s+1th second, s ” represents the number of object bounding box pairs between the s-th second video segment and the s+1-th second video segment; Then, the optimal target matching set per second is According to the same target bounding box, the front and back connections are made to obtain the action pipeline set Among them, N" represents the number of action pipelines, and finally the action pipeline set and action categories Determine the start and end times of all actions and the target bounding boxes at the corresponding times Start time The moment when the action pipeline is first classified as the "i" action, the end time is the time when the “i”th action was last detected, Indicates that from Seconds to The target bounding box corresponding to the i-th action in seconds, a i " is the action category of the i-th action, N a is the number of actions in the video sequence.

Images