Weakly supervised temporal action localization method and device based on semantic and saliency knowledge collaborative propagation

Abstract

The weakly supervised temporal action localization method and device based on semantic and saliency knowledge collaborative propagation comprises the following steps: 1) extracting temporal features and saliency foreground features from an uncropped video; 2) constructing a basic branch and a saliency perception branch to respectively process the temporal features and the saliency target features to obtain a basic class activation sequence, a motion, appearance representation score and a saliency class activation sequence, and weighting and fusing the four sequences to obtain a fused action score sequence; 3) utilizing branch distillation and branch action consistency constraint to interact semantic information and saliency information, and perfecting the fused action score sequence; 4) extracting key segments and ambiguous segments of the basic branch and the saliency perception branch, and utilizing the key segments and the ambiguous segments between branches and within branches for comparative learning to improve feature representation, combining the distillation result to perfect the fused action score sequence and obtaining an action localization result. The present application can perceive subtle human actions and accurate temporal action boundaries in uncropped videos.

Description

Technical Field

[0001] This invention relates to the field of computer video analysis technology, and in particular to a method and apparatus for locating temporal actions in weakly supervised video. Background Technology

[0002] Video temporal action localization is a key research area in computer vision, aiming to locate the start and end times of actions belonging to a specific category within uncropped videos. It has significant practical applications, such as video retrieval, video anomaly detection, intelligent surveillance, and smart transportation. Currently, training methods for video temporal action localization networks can be divided into fully supervised training and weakly supervised training. Fully supervised training requires not only action category labels within the video but also frame-by-frame manual annotation. However, in real-world applications, video data often lacks frame-level annotation, and manually annotating massive amounts of video data is extremely costly. Therefore, using fully supervised training methods in practical applications is time-consuming and labor-intensive. To overcome the limitations of frame-by-frame labeled fully supervised temporal action localization models, efficient weakly supervised temporal action localization methods have emerged, requiring only video-level classification labels for training.

[0003] Existing weakly supervised temporal action localization methods can be divided into three categories: The first category focuses on the pattern differences between classification and localization, aiming to generate more refined pseudo-label fragments to reduce these discrepancies. The second category concentrates on separating foreground and background fragments, primarily using different attention mechanisms to accurately distinguish between them. The third category emphasizes learning discriminative or fuzzy behavioral features, mitigating fuzzy features through methods such as measurement, erasure, or comparison. The first two categories often suffer from issues such as missed detections or false positives, as they focus only on the most discriminative action fragments and easily overlook fuzzy fragments.

[0004] All three methods described above can achieve some localization results by learning effective loss functions. However, they are often limited by the finite representation of appearance and motion features, making it difficult for the models to distinguish between foreground elements that blend into the background and action segments with drastic temporal changes. This results in false positives and false negatives in action detection. Therefore, exploring the collaborative relationships between semantic and saliency knowledge, and perceiving subtle motion information to extract the foreground in segments where the foreground and background are difficult to separate, is of great significance for improving the model's ability to localize temporal actions in videos. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention provides a weakly supervised temporal action localization method and apparatus based on the collaborative propagation of semantic and salient knowledge. This invention studies the propagation method between semantic and salient knowledge in videos from two perspectives: first, by distilling between branches to achieve complementarity of key information in semantic and salient knowledge; and second, by clustering feature representations between and within branches to achieve knowledge internalization. This approach aims to alleviate the problem of false positives or false negatives in models dealing with fine-grained action segments.

[0006] The technical solution adopted by this invention to solve its technical problem is:

[0007] The first aspect of this invention relates to a weakly supervised temporal action localization method based on the collaborative propagation of semantic and saliency knowledge, the method comprising the following steps:

[0008] 1) Temporal features and salient foreground features are extracted from unedited videos using a dual-stream network and a salient object detection network, respectively. Unedited videos include videos of human activities and other backgrounds.

[0009] 2) Establish a saliency perception branch, use the extracted saliency foreground features as input to train the branch to obtain saliency class activation sequences, use these as representative sequences of the branch, and sum them in the channel dimension to obtain saliency action representation scores. Use these scores to perform weighted fusion with the temporal class activation sequences, appearance action representation scores and motion action representation scores obtained from the basic branch to obtain a fused action score sequence, use this as a representative sequence of the basic branch to represent the initial fusion of semantic and saliency knowledge;

[0010] 3) Pseudo-labels are obtained by comparing the fused action score sequence and the saliency class activation sequence with the threshold, respectively. The knowledge in the local pseudo-labels is propagated to the representative sequence of another branch through the distillation method. The propagation effect is enhanced by the action consistency constraint under the prediction category dimension, and finally the complementary learning of semantic information and saliency information is achieved.

[0011] 4) Based on the pseudo-labels of each branch, corresponding key segments and ambiguous segments are mined. Key segments contain both key semantics and salient foreground information, while ambiguous segments contain contradictory semantic content and foreground salient information. Based on these segments, a contrastive learning method is used to aggregate the feature representations of key segments and guide the features of ambiguous segments to approach the features of key segments in the feature space to obtain more accurate representations. Finally, the temporal action localization results are obtained by combining the indices of the top k high-scoring actions of the fused action sequence with the temporal activation sequence.

[0012] A second aspect of the present invention relates to a weakly supervised temporal action localization device based on the collaborative propagation of semantic and saliency knowledge, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement a weakly supervised temporal action localization method based on the collaborative propagation of semantic and saliency knowledge of the present invention.

[0013] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a weakly supervised temporal action localization method based on the collaborative propagation of semantic and saliency knowledge according to the present invention.

[0014] The working principle of this invention is as follows: For the problem of temporal localization of videos containing human actions and behaviors, salient foreground information in the video is extracted, and a separate branching method is designed to combine with distillation and contrastive learning methods to fuse salient foreground information with temporal feature information, thereby improving the utilization rate of effective salient foreground information.

[0015] The advantages of this invention are: for massive videos containing human actions, a weakly supervised temporal action localization method based on the collaborative propagation of semantic and saliency knowledge is proposed, which has high accuracy in identifying and locating human actions. By effectively integrating salient foreground information and semantic information, the method greatly improves the attention to salient human action areas and enhances the ability to identify and locate human actions in videos. Attached Figure Description

[0016] Figure 1 This is a general framework diagram of the weakly supervised temporal action localization method of the present invention.

[0017] Figure 2a and Figure 2b This is a visualization example of the results of implementing the present invention, wherein, Figure 2a This is a visualization of an example of the "javelin throwing" action. Figure 2b Visualization of a cliff diving action example.

[0018] Figure 3 This is a flowchart of the method of the present invention. Specific implementation methods

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Example 1

[0021] Reference Figure 1 and Figure 2a and Figure 2b A weakly supervised temporal action localization method based on the collaborative propagation of semantic and saliency knowledge includes the following steps:

[0022] 1) Establish a saliency perception branch, using the extracted saliency foreground features as input to train this branch to obtain saliency class activation sequences. These sequences are then used as the representative sequences of this branch, and their summation along the channel dimension yields a saliency action representation score. This score is then weighted and fused with the temporal class activation sequences, appearance action representation scores, and motion action representation scores obtained from the basic branch to obtain a fused action score sequence. This fused action score sequence is used as the representative sequence of the basic branch to represent the initial fusion of semantic and saliency knowledge. The process can be represented as follows: First, appearance and optical flow features... Cascading in the channel dimension yields Then it is fed into a feature extractor layer consisting of one-dimensional convolutions and the ReLU function. In order to obtain embedded features As shown in the following formula:

[0023] (1)

[0024] Then, through Layer obtains time-class activation sequence It consists of a one-dimensional convolution and a ReLU function. C represents the total number of categories in all videos, as shown in the following formula:

[0025] (2)

[0026] Finally, for The summation is performed along the channel dimension and then fed into the sigmoid function to obtain the basic action score. As shown in the following formula:

[0027] (3)

[0028] After the saliency perception branch is established, it is used to obtain saliency knowledge of the video from the saliency foreground features. First, the saliency foreground features are... Preprocessing is performed to obtain , as in equation (5).

[0029] (4)

[0030] Use a feature extractor layer consisting of one-dimensional convolutional Conv1D and ReLU functions. activation Obtain salient feature embedding vector As shown in the following formula:

[0031] (5)

[0032] Will The input consists of a one-dimensional convolution Conv1D and a ReLU function. Obtained in the layer ,Will The representative sequence of the saliency perception branch is as follows:

[0033] (6)

[0034] On the channel dimension C Summation yields As shown in the following formula:

[0035] (7)

[0036] in Indicates that the t-th segment is in The fractions in express The activation value of the t-th segment in channel c.

[0037] To better integrate semantic and saliency knowledge, a weighted fusion of temporal activation sequences is performed. Score based on appearance and action Obtain the fusion action score sequence And this sequence is used as the representative sequence of the basic branch, as shown in equation (5).

[0038] (8)

[0039] in, This indicates the weight of the fusion.

[0040] To optimize classification results, each category of The set of instances is represented as .according to The selected metrics from the middle segments are aggregated with the corresponding category activation sequence regions to obtain video-level category predictions. Finally, the video classification loss is in the form of cross-entropy loss. The formula is as follows (6):

[0041] (9)

[0042] in, It is the actual value.

[0043] 2) Pseudo-labels are obtained by comparing the fused action score sequence and the saliency class activation sequence with the threshold, respectively. The knowledge from the local pseudo-labels is propagated to a representative branch of another branch using a distillation method. The propagation effect is enhanced by utilizing action consistency constraints under the predicted category dimension, ultimately achieving complementary learning of semantic and saliency information. This process can be represented as: Considering... and The presence of noise can affect the dissemination of key knowledge, so ternary pseudo-labels are generated as a medium for knowledge interaction and internalization. Fixed coefficients are used. and Multiply by respectively and The median is used to obtain the high and low thresholds for each branch. Using thresholds to set pseudo-labels As in equation (10):

[0044] (10)

[0045] Where t and c represent the fragment index and category, respectively. For each representative sequence The predicted score sequence in category c. for The predicted score of the t-th segment in category c.

[0046] To enhance the interaction between semantic and saliency knowledge, alleviate the differences in the meaning of activation values ​​in different sequences, and focus on the sequence portion containing specific categories, a class-specific action score sequence as shown in Equation (11) was designed. Consistency constraints will be introduced in subsequent optimization processes.

[0047] (11)

[0048] in This represents the action score of the t-th segment. Indicates pseudo-tags The pseudo-label sequence under category c.

[0049] The distillation method mainly reduces the information entropy difference between two probability distributions by designing a relative entropy (also known as information divergence) optimization function, that is, reducing the information loss or error introduced when approximating one distribution with another. Here, it mainly reduces the relative entropy when the representative sequence of the branch is approximated to the pseudo-label sequence. Therefore, in order to enable the model to achieve complementary learning of semantic and saliency information, information divergence (KL divergence) is introduced to design the branch distillation loss. The inter-branch distillation realizes the initial fusion of key information between branches in the form of distribution approximation, filters semantically irrelevant saliency knowledge, and supplements the poor semantic knowledge with semantically relevant saliency knowledge, as shown in Equation (12):

[0050] (12)

[0051] in L represents the local branch, A represents the branch opposite to the local branch, C represents the basic branch, and SA represents the saliency-aware branch. Finally, the set form is represented... Substitute the combination of values ​​into the above equation to calculate the total loss of branch distillation.

[0052] Because the interbranch distillation method may focus too much on the most discriminative fragment information while ignoring other semantically relevant information when filtering semantically irrelevant salient information, a class-specific action consistency constraint is designed, as shown in Equation (13), to achieve consistent activation of the filtered semantic and salient information.

[0053] (13)

[0054] in

[0055] 3) Based on the pseudo-labels of each branch, corresponding key segments and ambiguous segments are mined. Key segments contain both key semantics and salient foreground information, while ambiguous segments contain contradictory semantic content and foreground salient information. Based on these segments, a contrastive learning method is used to aggregate the feature representations of key segments and guide the features of ambiguous segments to approach the features of key segments in the feature space to obtain more accurate representations. Finally, the temporal action localization results are obtained by combining the indices of the top k high-scoring actions of the fused action sequence and the temporal activation sequence. The process can be represented as follows: To deepen the understanding and internalization of knowledge by each branch after knowledge interaction, consistent key segments and inconsistent ambiguous segments are mined for each branch based on the consistency and numerical differences between each pseudo-label. First, the mask is obtained. This is to obtain information about consistency and inconsistency between branches. The formula is as follows:

[0056] (14)

[0057] (15)

[0058] IC and CS represent different mask types.

[0059] Then the mask Multiply by each representative sequence S, and sum over D dimensions to obtain a descending order. The formula is as follows:

[0060] (16)

[0061] (17)

[0062] in Indicates the fragment type, These represent action sequences and background sequences, respectively. Indicates the mask type. This represents the score of the t-th segment in sequence S along dimension d. This represents the maximum score among all segments in the d-dimensional dimension representing the sequence S.

[0063] according to Select respectively and For the first k instances, obtain consistent key actions (CKA), inconsistent ambiguous actions (IAA), consistent key backgrounds (CKB), and inconsistent ambiguous backgrounds (IAB) in the two branches respectively.

[0064] InfoNCE loss is introduced as a sample contrastive learning loss both between and within branches. Contrastive learning promotes the internalization of knowledge within branches, as shown in the following formula:

[0065] (18)

[0066] The query sample is Positive samples are Negative samples are It's a temperature over-parameter. Let x represent the transpose of vector x, and M represent the number of negative sample segments. The contrastive learning sample pair settings are shown in the following equation:

[0067] (19)

[0068] (20)

[0069] Substituting the above formula into equation (18), we obtain the contrastive learning losses between branches and within branches, respectively. .

[0070] Finally, the sum of all losses is used as the optimization objective, as shown in the following formula:

[0071] (twenty one)

[0072] In the formula, That is the corresponding loss coefficient.

[0073] The temporal action localization results are used to extract fused action scores by setting a multivariate threshold. The instances in the algorithm are used as candidate action proposals for prediction, and then the nonmaximum suppression method is applied to these proposals to remove duplicate action proposals.

[0074] Experimental Results: The datasets used in this experiment are the THUMOS-14 dataset and the ActivityNet-1.2 dataset. The THUMOS-14 dataset includes 200 validation videos and 213 test videos, both containing 20 classes. This experiment uses the validation set for training and the test set for evaluation. The ActivityNet-1.2 dataset contains 4819 training videos, 2383 validation videos, and 2489 test videos, all containing 100 classes. Similar to previous mainstream methods, this experiment uses the training set for training and the validation set for evaluation. The evaluation results on the THUMOS-14 and ActivityNet-1.2 datasets are shown in Tables 1 and 2, respectively. It can be seen that the motion localization accuracy of the method in this invention surpasses that of previous mainstream methods.

[0075]

[0076] Table 1

[0077]

[0078] Table 2

[0079] Example 2

[0080] A weakly supervised temporal action localization device based on the co-propagation of semantic and saliency knowledge includes a memory and one or more processors. The memory stores executable code. When the one or more processors execute the executable code, they are used to implement the weakly supervised temporal action localization method based on the co-propagation of semantic and saliency knowledge as described in any one of Embodiments 1.

[0081] Example 3

[0082] A computer-readable storage medium stores a program that, when executed by a processor, implements the weakly supervised temporal action localization method based on the collaborative propagation of semantic and saliency knowledge as described in any one of Embodiments 1.

[0083] The embodiments described in this specification are merely examples of implementations of the inventive concept and are for illustrative purposes only. The scope of protection of this invention should not be considered limited to the specific forms described in these embodiments; rather, it extends to equivalent technical means conceived by those skilled in the art based on the inventive concept.

Claims

1. A weakly supervised temporal action localization method based on the collaborative propagation of semantic and saliency knowledge, characterized by: Includes the following steps: 1) Temporal features and salient foreground features are extracted from unedited videos using a dual-stream network and a salient object detection network, respectively. Unedited videos include videos of human activities and other backgrounds. 2) Establish a saliency perception branch, using the extracted salient foreground features as input to train this branch to obtain salient class activation sequences. These sequences are then used as the representative sequences of this branch, and their summation along the channel dimension yields a salient action representation score. This score is then weighted and fused with the temporal activation sequences, appearance action representation scores, and motion action representation scores obtained from the basic branch to obtain a fused action score sequence. This fused action score sequence is used as the representative sequence of the basic branch to represent the initial fusion of semantic and saliency knowledge. The process for obtaining the temporal activation sequences, appearance action representation scores, and motion action representation scores in the basic branch is as follows: First, appearance and optical flow features... Cascading in the channel dimension yields Then it is fed into a feature extractor layer consisting of one-dimensional convolutions and the ReLU function. In order to obtain embedded features As shown in the following formula: (19) Then, through Layer obtains time-class activation sequence It consists of a one-dimensional convolution and a ReLU function; C represents the total number of categories in all videos, as shown in the following formula: (20) Finally, for Summing along the channel dimension and inputting it into... sigmoid In the function, to obtain the basic action score. As shown in the following formula: (21) 3) Pseudo-labels are obtained by comparing the fused action score sequence and the saliency class activation sequence with the threshold. The knowledge from the local pseudo-labels is propagated to the representative sequence of another branch using a distillation method. The propagation effect is enhanced by utilizing action consistency constraints under the predicted category dimension, ultimately achieving complementary learning of semantic and saliency information. The distillation method mainly reduces the information entropy difference between two probability distributions by designing a relative entropy optimization function, i.e., reducing the information loss or error introduced when approximating one distribution with another. Here, the principle of the distillation method is used to reduce the relative entropy when the representative sequence of the branch is approximated to the pseudo-label sequence. First, to reduce the impact of noise on the propagation of key knowledge, ternary pseudo-labels are generated as a medium for knowledge interaction and internalization. Second, fixed coefficients are used... and Multiply by respectively and The median is used to obtain the high and low thresholds for each branch. And use thresholds to set pseudo-labels As shown in equation (7): (7) Where t and c represent the fragment index and category, respectively. For each representative sequence The predicted score sequence in category c. for The predicted score of the t-th segment in category c; To enhance the interaction between semantic and saliency knowledge, alleviate the differences in the meaning of activation values ​​in different sequences, and focus on the sequence portion containing specific categories, a class-specific action score sequence as shown in Equation (8) was designed. Consistency constraints will be introduced in subsequent optimization processes; (8) in This represents the action score of the t-th segment. Indicates pseudo-tags The pseudo-label sequence under category c; To facilitate complementary learning of semantic and salient information in the model, KL divergence is introduced to design branch distillation loss. The inter-branch distillation achieves the initial fusion of key information between branches in a distribution approximation manner, filters out semantically irrelevant salient knowledge, and supplements the poor semantic knowledge with semantically relevant salient knowledge, as shown in equation (9): (9) L represents the local branch, and A represents another branch that is opposite to the local branch. The basic branch is represented by SA, which represents the saliency perception branch; finally, the set form is represented... Substitute the combination into formula (9) to calculate the total loss of branch distillation; Since the interbranch distillation method may focus too much on the most discriminative fragment information and ignore other semantically relevant information when filtering semantically irrelevant salient information, class-specific action consistency constraints are designed, as shown in Equation (10), to achieve consistent activation of semantic and salient information after filtering. (10) in ; 4) Based on the pseudo-labels of each branch, the corresponding key segments and ambiguous segments are mined. The key segments contain both key semantics and salient foreground information, while the ambiguous segments contain contradictory semantic content and foreground salient information. Based on the above segments, the feature representations of the key segments are aggregated using a contrastive learning method, and the features of the ambiguous segments are guided to approach the features of the key segments in the feature space to obtain more accurate representations. Finally, the temporal action localization results are obtained by combining the indexes of the top k high-scoring actions of the fused action sequence and the temporal activation sequence. To deepen each branch's understanding and internalization of knowledge after knowledge interaction, based on the consistency and numerical differences between each pseudo-label, consistent key segments and inconsistent ambiguous segments are mined for each branch; first, a mask is obtained. This is used to obtain information on consistency and inconsistency between branches; the formula is as follows: (11) (12) Where IC and CS represent different mask types; Then the mask Multiply by each representative sequence S, and sum over D dimensions to obtain a descending order. The formula is as follows: (13) (14) in Indicates the fragment type, These represent action sequences and background sequences, respectively. Indicates the mask type. This represents the score of the t-th segment in sequence S along dimension d. This represents the maximum score among all segments in the d-dimensional dimension representing the sequence S; according to Select respectively and The front of the middle k For each instance, obtain the consistent key action CKA, inconsistent fuzzy action IAA, consistent key background CKB, and inconsistent fuzzy background IAB in the two branches respectively.

2. The weakly supervised temporal action localization method based on the collaborative propagation of semantic and saliency knowledge as described in claim 1, characterized in that: Step 2) specifically includes: after the saliency perception branch is established, it is used to obtain saliency knowledge of the video from the saliency foreground features. First, the saliency foreground features are... Preprocessing is performed to obtain As in equation (5): (1) Use a feature extractor layer consisting of one-dimensional convolutional Conv1D and ReLU functions. activation Obtain salient feature embedding vector As shown in the following formula: (2) Will The input consists of a one-dimensional convolution Conv1D and a ReLU function. Obtained in the layer ,Will The representative sequence of the saliency perception branch is as follows: (3) On the channel dimension C Summation yields As shown in the following formula: (4) in Indicates that the t-th segment is in The fractions in express The activation value of the t-th segment in channel c; To better integrate semantic and saliency knowledge, a weighted fusion of temporal activation sequences is performed. Score based on appearance and action Obtain the fusion action score sequence And this sequence is used as the representative sequence of the basic branch, as shown in equation (5): (5) in, Indicates the weights for fusion; To optimize classification results, each category of The set of instances is represented as ;according to The selected metrics from the middle segments are aggregated with the corresponding category activation sequence regions to obtain video-level category predictions. Finally, the video classification loss is in the form of cross-entropy loss. The formula is as follows (6): (6) in, It is the actual value.

3. The weakly supervised temporal action localization method based on the collaborative propagation of semantic and saliency knowledge as described in claim 1, characterized in that: Step 4) also includes: InfoNCE loss is introduced as a sample contrastive learning loss between and within branches. Contrastive learning promotes the internalization of knowledge in branches, as shown in equation (15): (15) The query sample is Positive samples are Negative samples are It's a temperature over-parameter. Let x represent the transpose of vector x, and M represent the number of negative sample segments; the contrastive learning sample pair settings are shown in equations (16) and (17): (16) (17) Substituting the above formula into equation (15), we obtain the contrastive learning losses between branches and within branches, respectively. ; Finally, the sum of all losses is used as the optimization objective, as shown in equation (18): (18) In the formula, This is the corresponding loss coefficient; The temporal action localization results are used to extract fused action scores by setting a multivariate threshold. The instances in the algorithm are used as candidate action proposals for prediction, and then the nonmaximum suppression method is applied to these proposals to remove duplicate action proposals.

4. The weakly supervised temporal action localization method based on the collaborative propagation of semantic and saliency knowledge as described in claim 3, characterized in that: The values ​​are 1.8, 2.0, 0.015, and 0.

02.

5. A weakly supervised temporal action localization device based on the collaborative propagation of semantic and saliency knowledge, characterized in that, The method includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the weakly supervised temporal action localization method based on the collaborative propagation of semantic and saliency knowledge as described in any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the weakly supervised temporal action localization method based on the collaborative propagation of semantic and saliency knowledge as described in any one of claims 1-4.

Images