A weakly supervised action detection method based on adaptive temporal dropout mechanism

Abstract

The application relates to a weakly supervised action detection method based on an adaptive timing dropout mechanism, which obtains a candidate action instance through a classification network, and the processing process of the classification network is as follows: performing feature extraction on a video to be detected; fusing and mapping RGB features and optical flow features to an action feature space; mapping an action feature sequence to a classification space to obtain a category activation sequence; performing post-processing on the category activation sequence to obtain a candidate action instance; when the classification network is trained, an adaptive timing dropout module is used to select a significant part from the action feature sequence and remove the significant part, thereby obtaining a corresponding category activation sequence, and a loss function based on multi-instance learning is used for optimization training. Compared with the prior art, the application solves the 'local domination' problem in weakly supervised action detection in an end-to-end and data-driven manner, and has the advantages of simple process, high recognition accuracy, wide application range and the like.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of video action detection, and relates to an action detection method, in particular to a weakly supervised action detection method based on an adaptive time dropout mechanism. BACKGROUND

[0002] Human action recognition has been an important research branch in the field of computer vision for decades, and its research scope covers images, videos, three-dimensional data and many other aspects. Related technologies are widely used in information retrieval, security monitoring, entertainment and other fields. In recent years, with the continuous development of software and hardware technologies, the research goals and problems in the field of action recognition have been further refined and deepened. In the field of video, the classification problem of action has been gradually solved, and the problem of action detection has gradually been valued. For untrimmed videos, the purpose of action detection technology is not only to identify the category of each action instance, but also to locate the start and end time of each action recognition. This technology has broad social prospects and application value in the fields of video summary, public video monitoring, action evaluation and daily life safety.

[0003] At present, the labeling cost of the action detection task is too large, and it is unrealistic to obtain a large-scale data set to train a model with good generalization ability. Therefore, in order to enhance the generalization ability of the time sequence detection model, some researchers decided to explore the method of using simpler and lower-cost simple labeled data to train the model, that is, the time sequence action detection under weak supervision. In the training of weakly supervised time sequence action detection task, only coarse-grained labels are used, that is, the action category information contained in the video; and in the inference, fine-grained results need to be given, including the start and end time of each action instance in the video, the action category and the like.

[0004] In line with the trend in most computer vision tasks, action detection has experienced a transition from traditional methods to deep learning methods, and the performance of deep learning methods generally surpasses that of traditional methods. In the weakly supervised case, only the action category of the video is labeled, so most deep learning methods represented by Nguyen et al. adopt a direct classification process, that is, first perform action classification of the video, and then take the video segment that contributes most to the classification result as the detection result. This method usually only focuses on the most significant area for classification, resulting in incomplete detection results, which is referred to as the "local dominance" problem. To solve this problem, some methods use an adversarial erasing strategy, which aims to remove the most significant area in the video and make the model focus on the relatively secondary area, thereby generating a more complete detection result. Singh et al. randomly hide a part of the area from the video feature sequence to reduce model overfitting, but this approach treats all areas equally rather than focusing on the most significant area, so the effect is limited. Other works, such as Zhong et al., Zeng et al., and Su et al., use an iterative classification process with erasing for training and testing. In each stage of the iterative process, they remove a part of the segment based on the classification contribution of each segment in the video to the final result, and then splice the remaining segments into a video for a new round of classification. The classification contribution and removal annotation are both manually designed: Su et al. specify a fixed threshold as the selection criterion; Zhong et al. use a fixed threshold on a hand-designed probability value for filtering; and Zeng et al. select a fixed number of segments based on a hand-designed score. During training and testing, these models need to repeat the classification-erasing-classification process multiple times.

[0005] After analyzing the existing adversarial erasing methods in the weakly supervised action detection task, we believe that the existing methods still have some obvious shortcomings. First, training and testing are multi-stage and inefficient. Existing adversarial erasing methods use an iterative classification process, which repeatedly performs video classification and significant segment removal at each stage, resulting in a large amount of redundant computation and parameter quantity. Second, it relies on manually designed components and is not data-driven, with limited effectiveness. Current methods generally use manually designed criteria to select segments for removal, and the selection and removal are done offline, without fully utilizing the advantages of deep learning data-driven. Third, the application scope is limited to a single weakly supervised action detection method, with poor portability. Existing methods are designed for a specific base network, and their adversarial erasing methods are strongly coupled with their network structure, which cannot be migrated to the current variety of weakly supervised action detection methods based on the direct classification process.

[0006] Therefore, it is of great practical significance to develop an end-to-end, data-driven weakly supervised action detection method for adversarial erasing. SUMMARY

[0007] The present application aims to provide a weakly supervised action detection method based on an adaptive temporal dropout mechanism to overcome the defects of existing weakly supervised action detection methods that rely on multi-stage iterative classification processes and artificially designed components for adversarial erasure.

[0008] The purpose of the present application can be achieved by the following technical solutions:

[0009] A weakly supervised action detection method based on an adaptive temporal dropout mechanism, which obtains candidate action instances through a classification network, the classification network comprising:

[0010] A feature extractor for feature extraction of the video to be detected, the extracted features including RGB features and optical flow features;

[0011] A feature transformation module for fusing and mapping the RGB features and optical flow features to an action feature space to obtain an action feature sequence;

[0012] A classification module for mapping the action feature sequence to a classification space to obtain a class activation sequence; and

[0013] A post-processing module for post-processing the class activation sequence to obtain the candidate action instances;

[0014] When training the classification network using a training set, an adaptive temporal dropout module is used to select and remove significant parts from the action feature sequence corresponding to the training set, and the action feature sequence with the significant parts removed is used to obtain the corresponding class activation sequence, and a multi-instance learning-based loss function is used to simultaneously optimize and train the adaptive temporal dropout module and the classification network.

[0015] Further, the feature extractor is a pre-trained video action recognition network.

[0016] Further, the post-processing specifically includes:

[0017] For the class activation sequence obtained by the classification module, discard the classes with scores lower than a set classification threshold;

[0018] For the class activation sequences of the remaining action classes, merge the continuous segments higher than a detection threshold into candidate results, and the confidence of each candidate result is the average of the classification scores of the internal video segments;

[0019] Use non-maximum suppression on the candidate results to obtain the final detection results and generate candidate action instances.

[0020] Further, an adaptive temporal dropout module is used to select significant parts from the action feature sequence corresponding to the training set and remove specific steps including:

[0021] Probability conversion is performed on the action feature sequence to obtain a one-dimensional probability sequence;

[0022] Differentiable sampling is performed on the one-dimensional probability sequence to obtain a binary dropout mask sequence;

[0023] The generated dropout mask sequence is applied to all channels of the action feature sequence, and then normalized to obtain an action feature sequence with significant parts removed.

[0024] Further, the probability conversion is realized through a convolution layer and a Sigmoid operation.

[0025] Further, the differentiable sampling is realized through a gumbel softmax operation, and the dropout mask sequence obtained is represented as:

[0026] M = 1 - GumbelSoftmax(τP)

[0027] Where M represents the dropout mask sequence, P represents the one-dimensional probability sequence, and τ represents the probability control multiplier.

[0028] Further, the action feature sequence with significant parts removed is represented as:

[0029]

[0030] Where represents element-wise multiplication, M represents the dropout mask sequence, E represents the original action feature sequence, and L represents the sequence length.

[0031] Further, a loss function based on multi-instance learning is used to simultaneously optimize and train the adaptive temporal dropout module and the classification network, including:

[0032] Multi-instance learning is used to aggregate the classification results of each segment in the class activation sequence to obtain video-level classification results, and convert them into video-level action probabilities of each category;

[0033] Based on the action probabilities of each category, a binary cross-entropy loss is calculated;

[0034] The training target of minimizing-maximizing the binary cross-entropy loss is used for optimization to update the model parameters.

[0035] Further, the aggregation is specifically: ​

[0036] For each category, multi-instance learning is used to aggregate the top-k scores from all segments, and the average of the top-k scores is taken as the video-level classification result, denoted as:

[0037]

[0038] wherein A c represents the cth channel of the category activation sequence A, corresponding to the cth action category, represents a subset of A c , containing its top-k elements, c represents the video-level classification result.

[0039] Further, the training objective is represented as:

[0040]

[0041] wherein L Cls represents the binary cross-entropy loss, δ represents the parameter of the adaptive temporal dropout module, γ represents the parameter of the classification network, and γ * , δ * represents the optimization result.

[0042] Compared with the prior art, the present application has the following beneficial effects:

[0043] 1) The present application combines the strategy of adversarial erasing and the dropout mechanism, and designs an adaptive dropout module to complete the selection and removal of the action salient region. The module is very lightweight, has a small burden on the parameter quantity and the calculation quantity, and does not change the original training and testing process, and does not need to perform iterative classification. Compared with the traditional method based on the iterative classification process with erasing, the efficiency is obviously improved in time and space.

[0044] 2) The present application performs adversarial erasing through a parameterized adaptive dropout module. The module can be regarded as a regular term applied during training, and is trained together with the classification network in an end-to-end manner. Therefore, in the method of the present application, adversarial erasing is completed in a data-driven and online manner, which on the one hand ensures that the regions selected by the dropout module are the most salient regions at present, and on the other hand ensures that the learning of the network can benefit from the results of the erasing operation in time. In addition, the method of the present application avoids a series of artificially designed modules and artificially set hyperparameters in adversarial erasing, and can better play the advantages of deep learning. Therefore, the adversarial erasing in the present application is more accurate and effective compared with the previous adversarial erasing methods.

[0045] 3) Traditional weakly supervised action detection methods are mostly designed for a specific baseline scheme, while the present application considers the scalability of the scheme, and for the various designs derived from the current direct classification process in the field of weakly supervised action detection, only the common basic architecture is considered, and a plug-and-play dropout module is designed. The module is not strongly coupled with a specific weakly supervised action detection method, but can be integrated into various methods with a common basic architecture, so the overall scheme has good scalability. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a workflow schematic diagram of the method of the present application;

[0047] Figure 2 is a framework diagram of the present application;

[0048] Figure 3 is a structure diagram of the adaptive timing dropout module;

[0049] Figure 4 is a comparison of three processes of weakly supervised action detection, wherein (4a) is a direct classification process adopted by most weakly supervised action detection methods, (4b) is an iterative classification process with removal operation adopted by methods based on an adversarial erasure strategy, and (4c) is a direct classification process with dropout adopted by the present application. DETAILED DESCRIPTION

[0050] The present application will be described in detail below in conjunction with the drawings and specific embodiments. The present embodiment is implemented on the premise of the technical scheme of the present application, and gives a detailed implementation and specific operation process, but the protection scope of the present application is not limited to the following embodiments.

[0051] The present embodiment provides a weakly supervised action detection method based on an adaptive timing dropout mechanism. The method obtains a candidate action instance through a classification network, and refers to Figure 2 As shown, the classification network includes: a feature extractor for feature extraction of a video to be detected, the extracted features including RGB features and optical flow features; a feature transformation module for fusing and mapping the RGB features and the optical flow features to an action feature space to obtain an action feature sequence; a classification module for mapping the action feature sequence to a classification space to obtain a class activation sequence; and a post-processing module for post-processing the class activation sequence to obtain the candidate action instance. The above method realizes reliable action detection by extracting RGB and optical flow features of the video to be detected.

[0052] In the present embodiment, the classification network applied in the above method is trained and tested through the following steps, as shown in Figure 1 .

[0053] S1, extracting RGB and optical flow feature sequences from the video using a feature extractor, and then using a feature transformation module to fuse and map the input two features to an action feature space to obtain an action feature sequence;

[0054] S2, using an adaptive temporal dropout module to select significant parts from the action feature sequence and remove them;

[0055] S3, using a classification module to map the action features to a classification space to obtain a class activation sequence;

[0056] S4, during training, using multi-instance learning to aggregate the classification feature sequence into a video-level classification feature, and using a video-level weakly supervised label to train the model;

[0057] S5, during testing, post-processing based on the class activation sequence to generate candidate action instances.

[0058] In the specific embodiment, step S1 specifically comprises:

[0059] S11: divide the video sample into L non-overlapping segments, and use a feature extractor to extract a feature sequence X = {x1, x2, …, xL}, where xi is the feature of the i-th segment. L} is the feature of the i-th segment. i

[0060] The feature extractor is a video action recognition network pre-trained on other datasets. In this embodiment, the feature extractor is an I3D network pre-trained on the Kinetics dataset, which extracts features from two modalities of RGB and optical flow. The features are concatenated and used as input features for each segment.

[0061] S12: using a feature transformation module to fuse the input RGB and optical flow features and map them to an action feature space.

[0062] The feature transformation module can be adjusted according to different situations. Its simplest form is a combination of a fully connected layer and a ReLU activation layer, or it can be a more complex feature network. The module maps the input feature sequence X to a new feature space of the same dimension, and its output is represented as E.

[0063] In the specific embodiment, as shown in Figure 3 , step 2 specifically comprises:

[0064] S21: according to the video feature sequence E, through the convolutional layer and the Sigmoid operation, a one-dimensional probability sequence P with a length of L is obtained.

[0065] ​The convolutional layer consists of two layers of one-dimensional temporal convolutions. Considering that the saliency of each video segment needs to be compared with neighboring segments, we give the first convolutional layer a kernel size greater than 1. The probability sequence P can be calculated as follows:

[0066] P = Sigmoid(Conv(E))

[0067] S22: Perform differentiable sampling on the probability sequence to obtain a binary dropout mask sequence.

[0068] Differentiable sampling is implemented using a Gumbel softmax operation, which is a differentiable approximation of the non-differentiable sampling process for class distributions. Specifically, we take each element P in the probability sequence P... i Treating it as an independent Bernoulli variable, the result sampled from this distribution is 1 or 0, representing removal or non-removal. A multiplier τ is first multiplied by this probability sequence to control the dropout probability from becoming too high. The calculation of the mask sequence M can be expressed as:

[0069] M = 1 - GumbelSoftmax(τP)

[0070] S23: Apply the generated mask to all channels of the action feature sequence E, and then standardize the features so that their numerical range is consistent with that before the mask was applied.

[0071] The purpose of standardization is to avoid the need for standardization during testing. Let ⊙ represent element-wise multiplication; the calculation process using masks and standardization can be represented as:

[0072]

[0073] In a specific implementation, the classification module in step S3 can be adjusted according to different situations. Here, a simple implementation using a fully connected layer is adopted.

[0074] Furthermore, the class activation sequence represents the degree of response of each of the L segments to different classes, similar to the two-dimensional class activation map in object detection, hence the name class activation sequence.

[0075] In a specific implementation, step S4 specifically includes:

[0076] S41: Aggregate the classification results of each segment in category activation sequence A to obtain video-level classification results.

[0077] Aggregation of segment classification results, which aims to let the video class labels available in weakly supervised setting supervise the training of the model. Specifically, for each class, we employ multiple instance learning to aggregate the top k scores from all segments, and then average them to get the video-level classification result. This operation can be specifically represented as follows:

[0078]

[0079] where A c represents the c-th channel of the class activation sequence A, which corresponds to the c-th action class, represents a subset of A c , which contains its top k elements. The setting of k is:

[0080]

[0081] where r is a hyper-parameter, which is set to 8 by default in this embodiment.

[0082] S42: Use the softmax function on the aggregated video class response to convert it into the video-level action probability of each class.

[0083] The calculation of the action probability of each class on the video class response can be represented as:

[0084]

[0085] S43: Based on the video-level classification result, calculate the binary cross-entropy loss:

[0086]

[0087] S44: Use the minimax training target to optimize and update the model parameters. When maximizing the classification loss, in order to increase the loss, our dropout module learns to remove the most significant regions, because this is the simplest and most direct way to increase the loss.

[0088] Further, the optimization target involves the optimization of two parts, namely the basic classification network and the adaptive temporal dropout module we added. We denote the parameters of this module as δ, and the parameters of other parts of the network as γ. With the adaptive temporal dropout module added, we use the minimax training target, which can be represented as:

[0089]

[0090] This minimization-maximization training objective does not bring extra difficulty to implementation and has little impact on training efficiency. Specifically, we first minimize the loss to update the network parameter γ while keeping the parameter δ of the adaptive temporal dropout module unchanged, and then maximize the classification loss to update the dropout module parameter δ while keeping the network parameter γ unchanged.

[0091] If the adaptive temporal dropout module is ignored, the training of the model is consistent with the traditional direct classification model, which can be represented as:

[0092]

[0093] Further, during optimization, increasing the classification loss means that the video classification score of the correct class needs to be as low as possible. Since the other parts of the network are not updated at this time, the mask M generated by this module needs to cover as many segments as possible, especially those corresponding to the most significant regions. This gradient is passed back to the differentiable sampling of the gumbel softmax parameter, thereby updating the learnable parameters of the convolutional layer in the dropout module. Since the multiplier τ limits the proportion of regions that can be masked, not all regions can be removed. Therefore, this module will select the most significant part to make the loss as large as possible.

[0094] This online selection process is adaptive to the training process and can be as robust as possible for each sample.

[0095] In the specific implementation, step S5 specifically includes:

[0096] S51: remove the adaptive temporal dropout module from the test process to obtain the class activation sequence A;

[0097] S52: discard the classes with scores lower than a certain classification threshold in the class activation sequence;

[0098] S53: for the activation sequence of the remaining action classes, use a detection threshold to filter the segments, and merge the continuous segments higher than the threshold into candidate results, and the confidence of each candidate result is the average of the classification scores of the internal video segments;

[0099] S54: use non-maximum suppression (NMS) on the candidate results to obtain the final detection results.

[0100] Experimental results:

[0101] To verify the actual performance of the present application on the weakly supervised action detection task, experiments are carried out on two famous public datasets THUMOS14 and ActivityNet. THUMOS14 contains 20 classes of videos for the action detection task, and we train on its validation set with 200 videos and evaluate on its test set with 212 videos according to the tradition. This dataset is very challenging because a video usually contains multiple actions. The length of action instances and video samples also varies greatly, from a few seconds to a few minutes. ActivityNet has two releases, 1.2 and 1.3, of which ActivityNet1.2 has 4819 training videos, 2383 validation videos and 2480 test videos from 100 classes. ActivityNet1.3 is a superset of ActivityNet1.2, with 10024 training videos and 4926 test videos from 200 classes. On average, there are 1.5 action instances per video. According to tradition, we train using its training set and test using its validation set. In line with most methods, the present application uses mAP as the evaluation standard.

[0102] Table 1 gives the comparison of the present application and other methods on the mAP index on the THUMOS14 test set.

[0103] Table 1 mAP (%)@IoU on THUMOS14 test set

[0104]

[0105]

[0106] According to Table 1, it can be seen that the present application achieves the best accuracy on the THUMOS14 dataset, and still has a significant gain over the current best method ACM-Net. It is worth noting that in the case of using WTALC, 3C-Net, BaS-Net and ACM-Net as the baseline, the present method can still surpass them with a significant gain effect, which on the one hand fully proves the effectiveness of the present method, and on the other hand also shows that the scheme of the present application is very good in transferability and expansibility. In addition, compared with the weakly supervised action detection methods based on the adversarial erasing strategy such as hide-n-seek, step-by-step, CPMN and IWO, the present method is much better than these methods in mAP, which is due to the effect of the present method itself on the one hand, and the baseline method on the other hand. Therefore, Table 2 below further carries out a fair comparison using the same baseline to illustrate the effect of the adversarial erasing of the present method itself.

[0107] Table 2 gives the comparison of the present application and other methods in mAP index on ActivityNet1.2 validation set. Among them, avg represents the mAP after taking the average of IoU 0.5:0.05:0.95.

[0108] Table 2 mAP (%) @ IoU on ActivityNet1.2 validation set

[0109]

[0110] According to Table 2, it can be seen that the present application achieves the best accuracy on this data set. Similarly, on the three baselines of WTALC, 3C-Net and ACM-Net, the present application achieves certain improvement effect, and the increase is obvious. It is worth noting that the original ACM-Net is not the best method on ActivityNet1.2 at that time, and its effect is not as good as D2-Net and CoLA methods, but after combining the present method, it successfully surpasses all other methods.

[0111] Table 3 gives the comparison of the present application and other adversarial erasing methods in weakly supervised action detection task in mAP index on THUMOS14 test set. Among them, avg represents the mAP after taking the average of IoU 0.5:0.05:0.95. In order to compare fairly, these methods are all based on the WTALC baseline network.

[0112] Table 3 mAP (%) @ IoU of various adversarial erasing methods on THUMOS14 validation set

[0113]

[0114] From Table 3, it can be seen that the adaptive temporal dropout method proposed in the present application has obvious advantages over various adversarial erasing methods before in almost all mAP (%) @ IoU, and the effect is obviously better.

[0115] Table 4 gives the comparison of the present application and other adversarial erasing methods in weakly supervised action detection task in efficiency on THUMOS14 test set, and the flow comparison is as shown in Figure 4 In order to compare fairly, these methods are all based on the WTALC baseline network.

[0116] Table 4 Efficiency comparison of various adversarial erasing methods on THUMOS14 validation set

[0117]

[0118] As can be seen from Table 4, the adaptive timing dropout method (flow as shown in (4c)) proposed in the present application has fewer training times, uses only a single network, does not need to repeatedly perform inference during testing, and does not need to fuse detection results of multiple stages during post-processing, as compared with the previous various adversarial erasing methods (step-by-step, CPMN, IWO, flow as shown in (4b)). The only efficiency close to ours is the ordinary dropout method, but since it does not have the ability to select significant regions, there is almost no improvement in performance, which is explained in Table 3.

[0119] If the above method is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the prior art that essentially contributes or the part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0120] Those skilled in the art should understand that those skilled in the art can make changes in combination with the prior art and the above embodiments, which are not described here. Such changes do not affect the essential content of the present application, which is not described here.

[0121] The preferred embodiments of the present application are described above. It should be understood that the present application is not limited to the above specific embodiments, and the devices and structures not described in detail should be understood as being implemented in the ordinary way in the art; any person skilled in the art can make many possible changes and modifications to the disclosed methods and technical content without departing from the scope of the present application, or modify equivalent embodiments with equivalent changes, which does not affect the essential content of the present application. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application, without departing from the content of the present application, all still belong to the scope of protection of the present application.

Claims

1. A weakly supervised action detection method based on an adaptive temporal dropout mechanism, characterized in that, This method obtains candidate action instances through a classification network, wherein the classification network includes: A feature extractor used to extract features from the video to be detected. The extracted features include RGB features and optical flow features. A feature transformation module for fusing the RGB features and optical flow features and mapping them to the action feature space to obtain an action feature sequence; A classification module for mapping the action feature sequence to a classification space to obtain a category activation sequence; and A post-processing module for post-processing the category activation sequence to obtain the candidate action instance; When training the classification network using a training set, an adaptive temporal dropout module is used to select and remove significant parts from the action feature sequences corresponding to the training set, so as to obtain the corresponding category activation sequences from the action feature sequences with significant parts removed. The adaptive temporal dropout module and the classification network are optimized and trained simultaneously using a loss function based on multi-instance learning. An adaptive temporal dropout module selects significant portions of the action feature sequences corresponding to the training set and removes them, specifically including: The action feature sequence is subjected to probability transformation to obtain a one-dimensional probability sequence; Differentiable sampling is performed on the one-dimensional probability sequence to obtain a binary dropout mask sequence; The generated dropout mask sequence is applied to all channels of the action feature sequence, and then normalized to obtain the action feature sequence with significant parts removed.

2. The weakly supervised action detection method based on adaptive temporal dropout mechanism according to claim 1, characterized in that, The feature extractor is a pre-trained video action recognition network.

3. The weakly supervised action detection method based on adaptive temporal dropout mechanism according to claim 1, characterized in that, The post-processing specifically includes: For the category activation sequence obtained by the classification module, discard the categories whose scores are lower than the set classification threshold; For the category activation sequence of the remaining action categories, consecutive segments above a detection threshold are merged into candidate results, and the confidence of each candidate result is the mean of the classification scores of its internal video segments; Non-maximum suppression is applied to the candidate results to obtain the final detection results and generate candidate action instances.

4. The weakly supervised action detection method based on adaptive temporal dropout mechanism according to claim 1, characterized in that, The probability transformation is achieved through convolutional layers and a sigmoid operation.

5. The weakly supervised action detection method based on adaptive temporal dropout mechanism according to claim 1, characterized in that, The differentiable sampling is implemented through the Gumbel softmax operation, and the resulting dropout mask sequence is represented as follows: in, Represents the dropout mask sequence. Represents a one-dimensional probability sequence. This represents the probability control multiplier.

6. The weakly supervised action detection method based on adaptive temporal dropout mechanism according to claim 1, characterized in that, The action feature sequence for removing significant parts Represented as: in, This represents element-wise multiplication. Represents the dropout mask sequence. Represents the original sequence of action features. Indicates the sequence length.

7. The weakly supervised action detection method based on adaptive temporal dropout mechanism according to claim 1, characterized in that, The optimization training of both the adaptive temporal dropout module and the classification network using a loss function based on multi-instance learning includes: Multi-instance learning is used to aggregate the classification results of each segment in the category activation sequence to obtain video-level classification results, which are then converted into video-level action probabilities for each category. Calculate the binary cross-entropy loss based on the probability of each action category; The training objective of minimizing-maximizing the binary cross-entropy loss is used for optimization, and the model parameters are updated.

8. The weakly supervised action detection method based on adaptive temporal dropout mechanism according to claim 7, characterized in that, The aggregation specifically refers to: Multi-instance learning is used for each category to aggregate the maximum from all fragments. A score, with the largest The average of the scores is used as the video-level classification result, expressed as: in, Represents category activation sequence The Channel 1, with Channel 2 Each action category corresponds to... express A subset containing its largest One element, This indicates the classification results at the video level.

9. The weakly supervised action detection method based on adaptive temporal dropout mechanism according to claim 7, characterized in that, The training objective is expressed as: in, This represents the binary cross-entropy loss. These represent the parameters of the adaptive timing dropout module. The parameters of the classification network are represented. The optimization results are shown.

Images