Weakly supervised temporal action localization method based on cascade seed region growing module

Abstract

The embodiment of the application provides a weakly supervised temporal action localization method based on a seed region growing module, and relates to the technical field of video action recognition. The weakly supervised temporal action localization method comprises steps S1 to S5. S1, obtaining a video to be recognized. S2, extracting a temporal feature according to the video to be recognized. S3, inputting the temporal feature into a pre-trained weakly supervised temporal action classification model, obtaining a class activation sequence to be recognized, and obtaining a video-level action classification score according to the class activation sequence to be recognized. S4, obtaining an action classification threshold, and judging the video-level action classification score according to the action classification threshold to obtain an action category contained in the video to be recognized. S5, obtaining start and end time of the action category in the video to be recognized according to the action category and the class activation sequence to be recognized. The existing weakly supervised temporal action localization method can be prevented from falling into local optimization, and the accuracy of action recognition in the video is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of video action recognition technology, and more specifically, to a weakly supervised temporal action localization method based on cascaded seed region growth modules. Background Technology

[0002] The rapid advancements in electronic devices and the advent of the internet age have led to an explosive growth in the number of videos online, with increasingly rich content. Computer-aided video recognition, enabling automatic action localization, understanding the main content of videos, and other related tasks such as video segmentation, annotation, and video recommendation systems, is of great significance.

[0003] Temporal action localization is a technique that can automatically locate the start and end times and categories of action segments within a long input video. With the rise of deep learning, temporal action detection has developed rapidly, and many excellent algorithms have been proposed, such as S-CNN and SSAD.

[0004] While these methods can achieve good results, they require a large amount of training data, which often needs to be fully supervised and labeled, resulting in high labeling costs and limiting the amount of training data.

[0005] Therefore, some weakly supervised temporal action localization methods have emerged in the prior art. However, most of these methods are based on multi-instance learning and attention mechanisms. This approach tends to focus too much on the most discriminative regions, leading to local optima when solving for optimal parameters. Furthermore, the quality of attention weights directly affects the model's classification performance, indirectly causing poor localization results.

[0006] In view of this, the applicant hereby submits this application after studying the existing technology. Summary of the Invention

[0007] This invention provides a weakly supervised temporal action localization method based on cascaded seed region growth modules, in order to improve the high cost problem caused by the need for a large amount of fully supervised labeled training data in existing temporal action localization techniques.

[0008] First aspect

[0009] This invention provides a weakly supervised temporal action localization method based on cascaded seed region growth modules, which includes steps S1 to S5.

[0010] S1. Obtain the video to be recognized.

[0011] S2. Extract temporal features from the video to be identified.

[0012] S3. Input the temporal features into the pre-trained weakly supervised temporal action classification model to obtain the activation sequence of the class to be identified, and obtain the video-level action classification score based on the activation sequence of the class to be identified.

[0013] S4. Obtain the action classification threshold, and judge the video-level action classification score based on the action classification threshold to obtain the action categories contained in the video to be identified.

[0014] S5. Based on the action category and the activation sequence of the class to be identified, obtain the start and end times of the action category in the video to be identified.

[0015] The weakly supervised temporal action classification model includes an initial seed extraction module for extracting an initial seed based on temporal features, and at least two cascaded seed region growth modules. The seed region growth modules are used for:

[0016] Based on temporal characteristics, obtain the original class activation sequence.

[0017] Based on the original class activation sequence and the seed output by the previous module, the expanded class activation sequence is obtained through a seed growth strategy.

[0018] Based on the temporal characteristics and the expanded class activation sequence, adversarial erasure is performed to obtain the erased class activation sequence.

[0019] The original class activation sequence and the erased class activation sequence are fused to obtain the class activation sequence to be identified.

[0020] By adopting the above technical solution, the present invention can achieve the following technical effects:

[0021] This invention employs a weakly supervised temporal action localization method, which not only reduces annotation costs but also acquires more data. The weakly supervised temporal action classification model of this invention utilizes seed region growth, adversarial erasure, metric learning, and cascaded strategies, effectively addressing the problem of existing weakly supervised temporal action localization methods getting trapped in local optima, thus significantly improving the accuracy of action recognition in videos. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the weakly supervised sequential action localization method.

[0024] Figure 2 This is a schematic diagram of the weakly supervised sequential action localization method.

[0025] Figure 3 This is a schematic diagram of the feature extraction model.

[0026] Figure 4 This is a schematic diagram of the structure of a weakly supervised temporal action classification model.

[0027] Figure 5 This is a schematic diagram of the seed region growth module. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0029] Example 1

[0030] Please see Figures 1 to 5 The first embodiment of the present invention provides a weakly supervised temporal action localization method based on a cascaded seed region growth module, which can be executed by a weakly supervised temporal action localization device (hereinafter referred to as: localization device). Specifically, it is executed by one or more processors in the localization device to implement steps S1 to S5.

[0031] S1. Obtain the video to be recognized.

[0032] Specifically, the video to be identified can be either a video captured in real time by a camera or a pre-recorded video containing motion. This invention does not specifically limit the source and type of the video to be identified.

[0033] Understandably, the positioning device can be any electronic device with computing power, such as a laptop, desktop computer, server, smartphone, or tablet.

[0034] S2. Extract temporal features from the video to be identified.

[0035] Specifically, by extracting temporal features, the video to be identified can be converted into feature data that a computer can recognize and calculate.

[0036] like Figure 3 As shown, based on the above embodiments, in an optional embodiment of the present invention, step S2 specifically includes: extracting features from the video to be identified using a dual-stream feature extraction model to obtain temporal features. The temporal features include RGB features and optical flow features.

[0037] Specifically, to better describe the weakly supervised temporal action localization task, the following will define some variables related to the weakly supervised temporal action localization task from a mathematical perspective. Let N training videos be... The corresponding tag is ,in It is a C-dimensional binary vector, and each of its terms... This indicates whether the video belongs to this category. It's important to note that not every video necessarily belongs to a single category. .

[0038] The feature extraction model based on two streams in this embodiment of the invention is a two-stream I3D model. The specific method for feature extraction is to use the output of the global pooling layer in the network as the feature input for the weakly supervised temporal action localization task. The final output of the feature extraction network is... , representing the final calculation results of the RGB stream and optical flow respectively, where D represents the size of the output dimension (usually 1024). It's a video. The corresponding temporal feature length.

[0039] It's important to note that because videos vary in length, sampling is typically used to enable parallel computation by the network. The sampling strategy varies depending on the stage of the training process. During training, random sampling is generally used to increase training data and enhance the network's generalization ability. During testing, uniform sampling is used to ensure consistent test results for the same model. When performing action recognition on the video, the entire video is used as input, and no sampling is required.

[0040] S3. Input the temporal features into the pre-trained weakly supervised temporal action classification model to obtain the activation sequence of the class to be identified, and obtain the video-level action classification score based on the activation sequence of the class to be identified.

[0041] Specifically, the weakly supervised temporal action classification model provides an initial seed based on CAS and several cascaded seed region growth modules. The model employs seed region growth, adversarial erasure, metric learning, and cascading strategies to address the accuracy reduction issue caused by local optima in prior art weakly supervised temporal action localization methods.

[0042] like Figure 4 As shown, based on the above embodiments, in an optional embodiment of the present invention, step S3 specifically includes: fusing the scores of each action category according to the activation sequence of the class to be identified, and obtaining the video-level score of each action category.

[0043] Specifically, the model scores each frame in the video to be recognized, and then combines the scores to form a class activation sequence of size [size missing]. T represents the number of video frames, and C represents the number of action categories. The frame score sequences for each action category are then fused to form a video-level score for that action category, which is C-dimensional.

[0044] Preferably, in an optional embodiment, the class activation sequence obtained after feature fusion is a T x C matrix. Then, the scores of the Top-K time points corresponding to each class are weighted and averaged to obtain the final score for that class, thus obtaining the final video-level action classification score. The CAS fusion method adopts a Top-K approach, i.e.:

[0045]

[0046] S4. Obtain the action classification threshold, and determine the video-level action classification score based on the action classification threshold to obtain the action categories contained in the video to be identified. Preferably, step S4 specifically includes steps S41 and S42.

[0047] S41. Obtain the action classification threshold. The action classification threshold is determined through grid search.

[0048] S42. Based on the action classification threshold, determine the video-level score for each action category. When the video-level score exceeds the action classification threshold, the video to be identified is considered to contain that action category, and the action category contained in the video to be identified is obtained.

[0049] Specifically, each video is not necessarily single-class, and the classification score is a one-dimensional vector. Therefore, the video-level action classification score needs to be used to determine the predicted action category through a threshold. Generally, if the score is greater than a set threshold, it can be considered that an action instance of that category has appeared in the video. The threshold for action classification is often determined through grid search.

[0050] It should be noted that grid search involves the user manually specifying a parameter in the model that they want to modify, and the program automatically uses an exhaustive search to run through all available parameters and find the optimal value for that parameter. In other embodiments, the threshold can be determined using other existing technologies, or even defined manually; this invention does not specifically limit this.

[0051] S5. Based on the action category and the activation sequence of the class to be identified, obtain the start and end times of the action category in the video to be identified. Preferably, step S5 specifically includes steps S51 and S52.

[0052] S51. Based on the action category, extract the temporal activation sequence of the action category from the activation sequence of the class to be identified.

[0053] S52. Based on the action classification threshold, extract continuous sequences with scores greater than the action classification threshold from the temporal activation sequence to obtain the start and end times of the action category in the video to be identified.

[0054] Specifically, the action instances are then precisely located based on the obtained action categories. First, the temporal activation sequences corresponding to the classification results are selected. Then, continuous time segments with scores greater than the action threshold are selected based on the action threshold. That is, if the scores of a continuous time segment are all greater than the threshold, then this time segment is a predicted action instance.

[0055] This invention employs a weakly supervised temporal action localization method, which not only reduces annotation costs but also acquires more data. The weakly supervised temporal action classification model of this invention utilizes seed region growth, adversarial erasure, metric learning, and cascaded strategies, effectively addressing the problem of existing weakly supervised temporal action localization methods getting trapped in local optima, thus significantly improving the accuracy of action recognition in videos.

[0056] like Figure 4 As shown, based on the above embodiments, in an optional embodiment of the present invention, the weakly supervised temporal action classification model includes an initial seed extraction module for extracting an initial seed based on temporal features, and at least two cascaded seed region growth modules.

[0057] like Figure 4 As shown, the overall network structure mainly includes a CAS-based model that provides the initial seed and several cascaded seed region growth modules. In an optional embodiment, the initial seed extraction module includes a class activation sequence-based module and a first seed extraction module.

[0058] A module based on class activation sequences is used to obtain the initial class activation sequence based on temporal characteristics.

[0059] The first seed extraction module is used to extract the K time points with the highest scores for each action category based on the initial class activation sequence, and form the initial seed.

[0060] Specifically, the initial seed mainly considers the foreground seed, which is generated by a CAS-based network model. In this invention, the initial seed is generated using a CAS-based model called Bas-Net (i.e., the module based on class activation sequences is a Bas-Net network model). It is understood that the module based on class activation sequences in this embodiment can be any existing classification model, and this invention does not specifically limit it.

[0061] The specific method for extracting the initial seed is as follows: The top-K time-series points corresponding to the true class's CAS are selected as the initial seeds. The selection of the reliable top-K initial seeds needs to be determined through experiments such as grid search. Preferably, in this embodiment of the invention, K is 25.

[0062] like Figure 5 As shown, in an optional embodiment, the seed region growing module is used to pass the extracted features through another 1-D convolution to obtain the output as a class activation sequence. This represents the classification result in the temporal dimension. Then, the class activation sequence and the initial seed are combined using a seed growth strategy to obtain the expanded class activation sequence. Next, the expanded class activation sequence is adversarially erased to remove the most salient regions, and then fed into a 1-D convolution to obtain the erased class activation sequence. This is to further uncover potential information. Finally, to obtain all explicit features, the present invention uses the original class activation sequence. and the class activation sequence after erasure To integrate.

[0063] Specifically, the seed region growth module is used to perform steps A1 to A4.

[0064] A1. Obtain the original class activation sequence based on temporal features. Preferably, the original class activation sequence is obtained by passing a single 1-D convolutional layer based on temporal features.

[0065] Specifically, the score for each action category in each frame of the video to be identified is obtained through convolution, and then the scores of each frame are combined to form the original class activation sequence.

[0066] A2. Based on the original class activation sequence and the seed output by the previous module, obtain the expanded class activation sequence through the seed growth strategy.

[0067] Specifically, the present invention employs a seed expansion strategy in a one-dimensional time series to dynamically adjust the seed sequence. The specific expansion strategy is shown in the following formula (1):

[0068] (1)

[0069] In the formula, pos represents the neighborhood time points of all initial seeds. It is a predefined growth threshold for each action class.

[0070] According to formula (1), the seed expansion strategy yields the largest classification result at the current time point. Only time points exceeding the threshold of category c will be included in the expansion region. Since videos and images are different, their neighborhood definitions differ; this invention defines the neighborhood as a specific time point. Several surrounding time series points, i.e. .

[0071] The expanded seed is used as a form of supervisory information and introduced into the seed loss to guide the class activation sequence in the network. The generation of the seed loss is shown in equation (2):

[0072] (2)

[0073] In the formula, It is a normalized CAS, and its expression is shown in formula (3):

[0074] (3)

[0075] In the formula, This represents the score of the c-th dimension at time i in CAS.

[0076] A3. Based on the temporal features and the expanded class activation sequence, perform adversarial erasure to obtain the erased class activation sequence. Preferably, adversarial erasure is performed on the expanded class activation sequence based on the temporal features, and then the erased class activation sequence is obtained through 1-D convolution.

[0077] Specifically, from the perspective of seed region expansion, the network still focuses on the most discriminative feature regions, while secondary feature regions remain unaddressed and unused. This does not solve the problems arising from multi-instance learning. This invention employs an adversarial erasure strategy to erase the most salient features. The specific mathematical formula is described below:

[0078] (4)

[0079] In the formula, Representing the The feature input corresponding to the c-th category at each time point Normal settings , It is a hyperparameter used to control the proportion of anti-erase.

[0080] As can be seen from formula (4), the strategy against erasure is to set the input features corresponding to the regions with higher scores to zero.

[0081] A4. Fuse the original class activation sequence and the erased class activation sequence to obtain the class activation sequence to be identified. Preferably, the original class activation sequence and the erased class activation sequence are averaged and then fused to obtain the class activation sequence to be identified.

[0082] Specifically, the class activation sequences generated under the seed region growing strategy and those generated under the adversarial erasure strategy are fused. This allows the network to learn not only the most distinctive primary features but also secondary features. The specific fusion strategy involves averaging the two inputs.

[0083] In this embodiment of the invention, by employing a seed region growth strategy, an erasure strategy, and metric learning, the generated class activation sequence (CAS) results are made more accurate, thus making the classification of actions more accurate.

[0084] Based on the above embodiments, in an optional embodiment of the present invention, the weakly supervised temporal action classification model further includes a second seed extraction module configured between cascaded seed region growth modules. The second seed extraction module is used to extract the K highest-scoring temporal points for each action category based on the activation sequence of the class to be identified, forming a seed for input to the next seed region growth module.

[0085] It is understood that, as an equivalent alternative technical feature of the embodiments of the present invention, the second seed extraction module can be integrated into the seed region growth module. When performing feature fusion, the top-K features are first extracted from the class activation sequence after erasing the original class activation sequence and then formed into a seed to be input into the next seed region growth module. This technical solution also falls within the protection scope of the present invention.

[0086] Based on the above embodiments, in an optional embodiment of the present invention, the weakly supervised temporal action classification model further includes a third seed extraction module configured after the last seed region growth module. The third seed extraction module is used to extract the K temporal points with the highest scores for each action category based on the activation sequence of the class to be identified output by the last seed region growth module, forming the final output seed. Based on this, step S3 specifically includes: calculating the weighted average of the K scores for each action category according to the final output seed to obtain the video-level score for each action category.

[0087] It is understood that, in an optional embodiment, the entire class activation sequence can be used as a seed, without setting up a first seed extraction module, a second seed extraction module, and a third seed extraction module, and this solution also falls within the protection scope of the present invention.

[0088] It should be noted that, as Figure 5As shown, when training the weakly supervised temporal action classification model in this embodiment of the invention, each optimization module has three loss functions: seed loss, sphere loss, and cross-entropy loss. Finally, this invention introduces metric learning to increase inter-class distance and decrease intra-class distance, thereby making the generated CAS more similar in the same class of actions and increasing the distance between different classes. Here, Additive Angular Margin Loss is used, and its expression is shown in formula (5):

[0089] (5)

[0090] In the formula, s is the scaling factor, m is the edge spacing, and yi is its corresponding category label value. yes and The included angle, These are learnable parameters. The calculation formula is shown in formula (6):

[0091] (6)

[0092] Similar to previous work, the CAS fusion method adopts the Top-K approach, as shown in formula (7):

[0093] (7)

[0094] Therefore, the cross-entropy loss can be expressed as shown in Equation (8):

[0095] (8)

[0096] In the formula, yc is the true label value corresponding to class c, and pc is the probability that the video belongs to class c. The calculation formula is shown in formula (9):

[0097] (9)

[0098] The overall loss function of the seed region growth module This can be expressed as shown in formula (10):

[0099] (10)

[0100] In the formula, These represent the weights assigned to each loss function, and are all hyperparameters.

[0101] In the several embodiments provided in this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0102] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0103] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0104] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0105] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0106] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0107] The use of "first" and "second" in the embodiments is merely to distinguish similar objects and does not represent a specific ordering of objects. It is understood that "first" and "second" can be interchanged in a specific order or sequence where permitted. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0108] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A weakly supervised temporal action localization method based on a cascade of seed region growing modules, characterized in that, The method comprises the following steps: acquiring a video to be recognized; extracting time sequence features according to the video to be recognized; inputting the time sequence features into a pre-trained weakly supervised time sequence action classification model to acquire a to-be-recognized class activation sequence, and acquiring a video-level action classification score according to the to-be-recognized class activation sequence; acquiring an action classification threshold, and judging the video-level action classification score according to the action classification threshold to acquire an action category contained in the video to be recognized; acquiring start and end time instants of the action category in the video to be recognized according to the action category and the to-be-recognized class activation sequence; the weakly supervised time sequence action classification model comprises an initial seed extraction module used to extract initial seeds according to the time sequence features, and at least two seed region growing modules arranged in cascade; the seed region growing module is used to: acquire an original class activation sequence according to the time sequence features; acquire an expanded class activation sequence through a seed growing strategy according to the original class activation sequence and a seed output by a previous module; perform adversarial erasing according to the time sequence features and the expanded class activation sequence to acquire an erased class activation sequence; fuse the original class activation sequence and the erased class activation sequence to acquire the to-be-recognized class activation sequence; the initial seed extraction module comprises a class activation sequence-based module and a first seed extraction module; the class activation sequence-based module is used to acquire an initial class activation sequence according to the time sequence features; the first seed extraction module is used to extract K time sequence points with the highest scores of each action category according to the initial class activation sequence to form the initial seeds; the weakly supervised time sequence action classification model further comprises a second seed extraction module arranged between the seed region growing modules arranged in cascade; the second seed extraction module is used to extract K time sequence points with the highest scores of each action category according to the to-be-recognized class activation sequence to form seeds used to input a next seed region growing module; the weakly supervised time sequence action classification model further comprises a third seed extraction module arranged after the last seed region growing module; the third seed extraction module is used to extract K time sequence points with the highest scores of each action category according to a to-be-recognized class activation sequence output by the last seed region growing module to form a final output seed; acquiring a to-be-recognized class activation sequence, and acquiring a video-level action classification score according to the to-be-recognized class activation sequence, specifically comprising: respectively performing weighted mean value calculation on K scores of each action category according to the final output seed to acquire a video-level score of each action category.

2. The weakly supervised temporal action localization method based on the cascade seed region growing module according to claim 1, characterized in that, acquiring a to-be-recognized class activation sequence, and acquiring a video-level action classification score according to the to-be-recognized class activation sequence, specifically comprising: respectively performing fusion on scores of each action category according to the to-be-recognized class activation sequence to acquire a video-level score of each action classification.

3. The weakly supervised temporal action localization method based on the cascade of seed region growing modules according to claim 1, characterized in that, acquiring an action classification threshold, and judging the video-level action classification score according to the action classification threshold to acquire an action category contained in the video to be recognized, specifically comprising: obtaining an action classification threshold, wherein the action classification threshold is determined by a grid search; judging the video-level score of each action classification according to the action classification threshold; when the video-level score exceeds the action classification threshold, it is determined that the to-be-identified video contains the action category, and the action category contained in the to-be-identified video is obtained.

4. The weakly supervised temporal action localization method based on the cascade seed region growing module according to claim 1, characterized in that, According to the action category and the to-be-identified class activation sequence, the start and end time of the action category in the to-be-identified video is obtained, specifically including: According to the action category, the time sequence activation sequence of the action category is extracted from the to-be-identified class activation sequence; According to the action classification threshold, the continuous sequence with a score greater than the action classification threshold is extracted from the time sequence activation sequence, and the start and end time of the action category in the to-be-identified video is obtained.

5. The weakly supervised temporal action localization method based on the cascade seed region growing module according to any one of claims 1 to 4, characterized in that, According to the time sequence feature, an original class activation sequence is obtained, specifically including: According to the time sequence feature, a 1-D convolution is performed to obtain the original class activation sequence; According to the time sequence feature and the expanded class activation sequence, an adversarial erasing is performed to obtain an erased class activation sequence, specifically including: According to the time sequence feature, the expanded class activation sequence is subjected to adversarial erasing, and then a 1-D convolution is performed to obtain the erased class activation sequence; The original class activation sequence and the erased class activation sequence are fused to obtain the to-be-identified class activation sequence, specifically including: The original class activation sequence and the erased class activation sequence are averaged to obtain the to-be-identified class activation sequence.

6. The weakly supervised temporal action localization method based on the cascade of seed region growing modules according to any one of claims 1 to 4, characterized in that, According to the to-be-identified video, a time sequence feature is extracted, specifically including: According to the to-be-identified video, a feature extraction is performed through a double-flow-based feature extraction model to obtain the time sequence feature; the time sequence feature includes an RGB feature and an optical flow feature.

Images