Time sequence action detection method based on self-constraint multi-cognitive adapter
By introducing a self-constrained multi-cognitive adapter in timing action detection, a neural network model including feature extractor, self-constrained multi-cognitive adapter and classification head is built, which solves the problem of overhead in end-to-end training and improves detection accuracy.
Patent Information
- Application Number
- CN202411898065.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-23
- Publication Date
- 2025-04-15
AI Technical Summary
The existing timing action detection methods have huge memory and computational overhead in end-to-end training, and there are problems with tasks and solutions that are aligned in the application of adapter design in the field of computer vision.
A timing action detection method based on self-constrained multi-cognitive adapter is proposed. By constructing a neural network model including feature extractor, self-constrained multi-cognitive adapter and classification head, and freezing the parameters of the feature extractor during training to reduce overhead.
It effectively improves the accuracy of timing action detection, while reducing the overhead of end-to-end training. By extracting time multi-mode sensitive time information by self-constrained multi-cognitive adapter, the precise segmentation of the model in timing is realized.
Smart Images
Figure CN119832472A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a method for detecting a time sequence action, and in particular to a method for detecting a time sequence action based on a self-constrained multi-cognitive adapter. Background Art
[0002] Temporal action detection is an important task in the field of video understanding and has a wide range of applications in real life. The current mainstream method requires the use of a feature extractor to pre-extract video features, and on this basis, the designed classification head is used for temporal action detection. The overall technical route shows an obvious two-stage nature. Since the feature extractor is supervised and trained under the video understanding task, there is a significant task gap with the classification head, which is not conducive to the optimization of the final goal. Therefore, it becomes necessary to combine the feature extractor and the classification head for end-to-end training under a loss function. However, for the original end-to-end training, its memory and computational overhead are very huge. According to the analysis, most of the overhead is mainly concentrated in the feature extractor part, so using the adapter tuning method for tuning can well solve the memory and computational overhead problems mentioned above. The academic community has made great changes in the content design of the adapter tuning method. Directly migrating the adapter in the field of natural language processing to the field of computer vision may result in poor alignment between the task and the solution. The design of adapters in the field of temporal action detection has its own unique rules. Summary of the invention
[0003] In view of the limitations of existing methods in adapter design, the present invention proposes a temporal action detection method based on self-constrained multi-cognition, which effectively improves the accuracy of temporal action detection while reducing the cost of end-to-end training.
[0004] The technical solution adopted by the present invention is as follows: comprising the following steps:
[0005] 1. A temporal action detection method based on self-constrained multi-cognitive adapter
[0006] 1) Segment the acquired video into several image sequences of equal length and preprocess the images in each image sequence to obtain a training set;
[0007] 2) constructing a neural network model for sequential action detection, wherein the neural network model includes a feature extractor, a self-constrained multi-cognitive adapter, and a classification head connected in sequence; then training the neural network model using the training set to obtain a trained neural network model;
[0008] 3) Input the image sequence to be detected into the trained neural network model, and the model outputs the time-series action detection results.
[0009] The preprocessing includes image scaling, cropping, horizontal flipping, and adding random perturbations.
[0010] The self-constrained multi-cognitive adapter includes a downsampling projection layer, a cognitive convolution layer, a self-constrained projection layer, a channel mapping layer, an activation layer and an upsampling projection layer. The input of the self-constrained multi-cognitive adapter is used as the input of the downsampling projection layer, and the downsampling projection layer is respectively connected to the three cognitive convolution layers and the self-constrained multi-cognitive adapter; the output of the self-constrained multi-cognitive adapter is respectively multiplied with the output of the three cognitive convolution layers in the channel dimension to obtain a first constrained feature vector, a second constrained feature vector and a third constrained feature vector; the first constrained feature vector, the second constrained feature vector and the third constrained feature vector are added and averaged to obtain a total constrained feature vector; the total constrained feature vector is added to the output of the downsampling projection layer to obtain a residual constrained feature vector; the residual constrained feature vector is used as the input of the channel mapping layer, the output of the channel mapping layer is added to the residual constrained feature vector to obtain a residual integrated constrained feature vector, the residual integrated constrained feature vector is used as the input of the activation layer, the activation layer is connected to the upsampling projection layer, and the output of the upsampling projection layer is used as the output of the self-constrained multi-cognitive adapter.
[0011] During the training of the neural network model, the network parameters of the feature extractor are frozen.
[0012] The parameters of the three cognitive convolutional layers are different, as follows:
[0013] The convolution kernel size of the first cognitive convolution layer is 1, the stride is 1, and the padding is 0;
[0014] The convolution kernel size of the second cognitive convolution layer is 3, the stride is 1, and the padding is 1;
[0015] The convolution kernel size of the third cognitive convolution layer is 5, the stride is 1, and the padding is 2.
[0016] The self-constrained projection layer is a fully connected layer whose output dimension is three times the input dimension.
[0017] 2. A temporal action detection system based on self-constrained multi-cognitive adapter
[0018] The video segmentation unit is used to segment the acquired video into several image sequences of equal length.
[0019] An image preprocessing unit, used for preprocessing the images in each image sequence;
[0020] The time-series motion detection unit is used to store the neural network model used for time-series motion detection and use the neural network model to perform time-series motion detection on the input image sequence, thereby outputting the corresponding time-series motion detection result.
[0021] 3. A computer device
[0022] The device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method for detecting sequential actions based on a self-constrained multi-cognitive adapter are implemented.
[0023] 4. A computer-readable storage medium
[0024] The medium stores a computer program, which, when executed by a processor, implements the steps of a method for detecting sequential actions based on a self-constrained multi-cognitive adapter.
[0025] V. A computer program product
[0026] The product includes a computer program / instruction, which, when executed by a processor, implements the steps of a temporal action detection method based on a self-constrained multi-cognitive adapter.
[0027] The beneficial effects of the present invention are:
[0028] In view of the limitations of existing adapters in designing visual models, the present invention proposes a self-constrained multi-cognitive adapter, which can effectively improve detection accuracy while promoting end-to-end training of the model.
[0029] The present invention adopts a multi-cognitive convolutional layer in the self-constrained multi-cognitive adapter, which can extract time information that is sensitive to time multimodality, and is conducive to the accurate temporal segmentation of the model.
[0030] The present invention adopts a self-constrained projection layer in a self-constrained multi-cognitive adapter, obtains global information through compression, and makes the model adaptively focus on different cognitive convolutional layers through channel transformation. By adaptively allocating weights on different channels, it can achieve the effect of constraining non-essential features and strengthening essential features.
[0031] The present invention can utilize end-to-end training to jointly tune the detection head and the feature extractor under a loss function, which is conducive to better expression of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 A neural network model constructed in the method of the present invention;
[0033] Figure 2 The self-constrained multi-cognitive adapter proposed by the present invention;
[0034] Figure 3 It is the weight value output by the self-constrained projection layer in the present invention. DETAILED DESCRIPTION
[0035] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:
[0036] The specific embodiments of the present invention are as follows:
[0037] The data set of the present invention adopts the THUMOS14 sequential action detection data set. There are unsegmented videos of 20 types of actions in the data set. It includes 200 validation set videos (including 3007 behavior fragments) and 213 test set videos (including 3358 behavior fragments). These annotated unsegmented videos can be used to train and test sequential behavior detection models. There are an average of 150 action timing annotations for each category in the validation set used for training, and the average duration of each action is 4.04 seconds. There are a total of 3007 action timing annotations, and the annotated actions last for a total of 12159.8 seconds. There are an average of 167.9 action timing annotations for each category in the test set, and the average duration of each action is 4.47 seconds. There are a total of 3358 action timing annotations, and the annotated actions last for a total of 15040.3 seconds.
[0038] The present invention proposes a temporal action detection method based on a self-constrained multi-cognitive adapter, the method comprising the following steps:
[0039] 1) Divide the acquired video into several image sequences of equal length and preprocess the images in each image sequence to obtain a training set; specifically, obtain all videos from the temporal action detection dataset and mark all action intervals, then preprocess each action video to obtain a fixed-length image sequence, and then divide all image sequences into a training set and a test set;
[0040] Preprocessing includes image scaling, cropping, horizontal flipping, and adding random disturbances. The preprocessing in this embodiment is specifically as follows:
[0041] All videos in the dataset are framed and compressed to obtain a compressed image sequence with a frame rate of 15fps and a longest side of 224. The compressed video is cut with a fixed frame length of 16 frames as the input frame number to obtain an image sequence. The image sequence is scaled to a fixed aspect ratio of 182 pixels in height.
[0042] In the training phase, a random cropping with a cropping aspect ratio of 3 / 4 to 4 / 3 and a cropping range of 0.08 to 1 is applied, and then the image sequence is scaled to 160 × 160. After randomly flipping the image sequence with a probability of 0.5, the brightness, contrast, saturation and hue are randomly perturbed to obtain the augmented image sequence.
[0043] Using random pruning and random perturbations can expand the size of the data set, increase the robustness of the model to the data, reduce the risk of overfitting during training, and optimize the performance of the model on the data set.
[0044] 2) Construct a neural network model for temporal action detection, such as Figure 1 As shown, the neural network model includes a feature extractor, a self-constrained multi-cognitive adapter and a classification head connected in sequence; then the neural network model is trained using the training set to obtain a trained neural network model;
[0045] like Figure 2 As shown, the self-constrained multi-cognitive adapter includes a downsampling projection layer, a cognitive convolution layer, a self-constrained projection layer, a channel mapping layer, an activation layer and an upsampling projection layer. The input of the self-constrained multi-cognitive adapter is used as the input of the downsampling projection layer. The downsampling projection layer is connected to the three cognitive convolution layers and the self-constrained multi-cognitive adapter respectively. The output of the self-constrained multi-cognitive adapter is multiplied with the output of the three cognitive convolution layers in the channel dimension to obtain a first constrained feature vector, a second constrained feature vector and a third constrained feature vector. The first constrained feature vector, the second constrained feature vector and the third constrained feature vector are added and averaged to obtain a total constrained feature vector. The total constrained feature vector is added to the output of the downsampling projection layer to obtain a residual constrained feature vector. The residual constrained feature vector is used as the input of the channel mapping layer. The output of the channel mapping layer is added to the residual constrained feature vector to obtain a residual integrated constrained feature vector. The residual integrated constrained feature vector is used as the input of the activation layer. The activation layer is connected to the upsampling projection layer. The output of the upsampling projection layer is used as the output of the self-constrained multi-cognitive adapter. The activation function of the activation layer is GELU. The self-constrained projection layer is a fully connected layer whose output dimension is three times the input dimension.
[0046] The downsampling projection layer is a linear layer with an input dimension of 768 and an output dimension of 192.
[0047] The convolution kernel size of the first cognitive convolutional layer is 1, the stride is 1, and the padding is 0.
[0048] The convolution kernel size of the second cognitive convolutional layer is 3, the stride is 1, and the padding is 1.
[0049] The convolution kernel size of the third cognitive convolution layer is 5, the stride is 1, and the padding is 2.
[0050] The self-constrained projection layer is a linear layer with an input dimension of 192 and an output dimension of 576.
[0051] The convolution kernel size of the first channel mapping layer is 1, the stride is 1, and the padding is 0.
[0052] The upsampling projection layer is a linear layer with an input dimension of 192 and an output dimension of 768.
[0053] Specifically:
[0054] The feature vector with a shape of 96×800×768 is input to the downsampling projection layer. The compressed feature vector with an output of 96×800×192 is obtained. The compressed feature vector is transformed into a time feature vector with a shape of 200×192×384. The time feature vector is input to the first cognitive convolution layer, the second cognitive convolution layer, and the third cognitive convolution layer to obtain the first cognitive feature vector, the second cognitive feature vector, and the third cognitive feature vector, all of which have a shape of 200×192×384. The time feature vector is globally compressed to obtain a time global feature vector with a shape of 1×192×1. The time global feature vector is input to the self-constrained projection layer to obtain an adaptive weight with a shape of 1×576×1. The adaptive weight is split into three channel adaptive weights with a shape of 1×192×1, and then multiplied with the first cognitive feature vector, the second cognitive feature vector, and the third cognitive feature vector to obtain the first constrained feature vector, the second constrained feature vector, and the third constrained feature vector. The first constrained feature vector, the second constrained feature vector, and the third constrained feature vector are added and averaged to obtain the constrained feature vector. The constrained feature vector and the output of the downsampling projection layer are added to obtain the residual constrained feature vector. The residual constrained feature vector is connected as input to the first channel mapping layer to obtain the integrated constrained feature vector. The integrated constrained feature vector is transformed into an integrated constrained feature vector with a shape of 200×192×384. The residual integrated constrained feature vector is added to the constrained feature vector to obtain the GELU activation function. The output of the GELU activation function is connected to the input of the upsampling projection layer to obtain an output vector with a shape of 96×800×768.
[0055] The feature extractor in the present invention can be any backbone network for video understanding. The present invention is described using VideoMAE-S as a feature extractor.
[0056] The classification head used in the present invention may be any classification head for sequential action detection. The present invention uses ActionFormer as the classification head for illustration.
[0057] 3) The image sequence to be detected is input into the trained neural network model, and the model outputs the temporal action detection result, that is, the label of the video sequence to be detected in the input video sequence.
[0058] During the training process, the parameters in the feature extractor will be frozen to reduce the overhead generated during training. The self-constrained multi-cognitive adapter and the detection head proposed in the present invention will participate in the parameter update.
[0059] Input the test set in 1) into the trained neural network model in 2) for testing and output evaluation indicators. The evaluation indicators used are mainly the performance of mean average precision (mAP) under a specific IoU threshold and the average mAP. On the THUMOS14 dataset, the threshold values are {0.3, 0.4, 0.5, 0.6, 0.7}.
[0060] As shown in Table 1, the performance of the present invention on the THUMOS14 dataset is better than the current most advanced method, indicating that the present invention has excellent performance in temporal action detection.
[0061] Table 1 is a performance comparison table of the present invention and other methods on the THUMOS14 dataset
[0062] mAP@0.3 mAP@0.4 mAP@0.5 mAP@0.6 mAP@0.7 ave.mAP Existing methods 84.23 79.68 71.50 61.56 46.41 68.68 Method of the present invention 85.16 80.63 72.76 61.84 47.04 69.49
[0063] Figure 3 This is a visualization diagram of the weights of the self-constrained projection layer of the present invention. The vertical axis represents the depth, and the three sub-graphs are the weights of the three cognitive projection layers. It can be seen that the horizontal axis represents the channel direction. It can be seen that for the three cognitive convolutional layers, the weights obtained deepen as the depth deepens, showing a regularity. This performance of deepening the weights in sequence is conducive to improving the accuracy of sequential action detection.
[0064] The present invention proposes a temporal action detection system based on a self-constrained multi-cognitive adapter, comprising:
[0065] The video segmentation unit is used to segment the acquired video into several image sequences of equal length.
[0066] An image preprocessing unit, used for preprocessing the images in each image sequence;
[0067] The time-series motion detection unit is used to store the neural network model used for time-series motion detection and use the neural network model to perform time-series motion detection on the input image sequence, thereby outputting the corresponding time-series motion detection result.
[0068] The present invention proposes a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a sequential action detection method based on a self-constrained multi-cognitive adapter are implemented.
[0069] The present invention proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of a temporal action detection method based on a self-constrained multi-cognitive adapter are implemented.
[0070] The present invention proposes a computer program product, comprising a computer program / instruction, which implements the steps of a temporal action detection method based on a self-constrained multi-cognitive adapter when executed by a processor.
[0071] Finally, it should be noted that the above embodiments and explanations are only used to illustrate the technical solution of the present invention rather than to limit it. Those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope disclosed in the technical solution of the present invention, which should be included in the scope of protection of the claims of the present invention.
Claims
1. A temporal action detection method based on a self-constrained multi-cognitive adapter, characterized in that: The steps include: 1) Segment the acquired video into several image sequences of equal length and preprocess the images in each image sequence to obtain a training set; 2) constructing a neural network model for sequential action detection, wherein the neural network model includes a feature extractor, a self-constrained multi-cognitive adapter, and a classification head connected in sequence; then training the neural network model using the training set to obtain a trained neural network model; 3) Input the image sequence to be detected into the trained neural network model, and the model outputs the time-series action detection results.
2. According to claim 1, a method for detecting sequential actions based on a self-constrained multi-cognitive adapter is characterized in that: The preprocessing includes image scaling, cropping, horizontal flipping, and adding random perturbations.
3. The method for detecting sequential actions based on a self-constrained multi-cognitive adapter according to claim 2, characterized in that: The self-constrained multi-cognitive adapter includes a downsampling projection layer, a cognitive convolution layer, a self-constrained projection layer, a channel mapping layer, an activation layer and an upsampling projection layer. The input of the self-constrained multi-cognitive adapter is used as the input of the downsampling projection layer, and the downsampling projection layer is respectively connected to the three cognitive convolution layers and the self-constrained multi-cognitive adapter; the output of the self-constrained multi-cognitive adapter is respectively multiplied with the output of the three cognitive convolution layers in the channel dimension to obtain a first constrained feature vector, a second constrained feature vector and a third constrained feature vector; the first constrained feature vector, the second constrained feature vector and the third constrained feature vector are added and averaged to obtain a total constrained feature vector; the total constrained feature vector is added to the output of the downsampling projection layer to obtain a residual constrained feature vector; the residual constrained feature vector is used as the input of the channel mapping layer, the output of the channel mapping layer is added to the residual constrained feature vector to obtain a residual integrated constrained feature vector, the residual integrated constrained feature vector is used as the input of the activation layer, the activation layer is connected to the upsampling projection layer, and the output of the upsampling projection layer is used as the output of the self-constrained multi-cognitive adapter.
4. The method for detecting sequential actions based on a self-constrained multi-cognitive adapter according to claim 1, characterized in that: During the training of the neural network model, the network parameters of the feature extractor are frozen.
5. The method for detecting sequential actions based on a self-constrained multi-cognitive adapter according to claim 3, characterized in that: The parameters of the three cognitive convolutional layers are different, as follows: The convolution kernel size of the first cognitive convolution layer is 1, the stride is 1, and the padding is 0; The convolution kernel size of the second cognitive convolution layer is 3, the stride is 1, and the padding is 1; The convolution kernel size of the third cognitive convolution layer is 5, the stride is 1, and the padding is 2.
6. The method for detecting sequential actions based on a self-constrained multi-cognitive adapter according to claim 3, characterized in that: The self-constrained projection layer is a fully connected layer whose output dimension is three times the input dimension.
7. A temporal action detection system based on a self-constrained multi-cognitive adapter, characterized in that: include: The video segmentation unit is used to segment the acquired video into several image sequences of equal length. An image preprocessing unit, used for preprocessing the images in each image sequence; The time-series motion detection unit is used to store the neural network model used for time-series motion detection and use the neural network model to perform time-series motion detection on the input image sequence, thereby outputting the corresponding time-series motion detection result.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of a method for detecting temporal actions based on a self-constrained multi-cognitive adapter as described in any one of claims 1 to 7 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a method for detecting temporal actions based on a self-constrained multi-cognitive adapter as described in any one of claims 1 to 7 are implemented.
10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of a temporal action detection method based on a self-constrained multi-cognitive adapter as described in any one of claims 1 to 7 are implemented.