Weak supervision time sequence action positioning method and device based on semantic and significance knowledge cooperative propagation

Abstract

The invention discloses a weak supervision time sequence action positioning method and device based on semantic and saliency knowledge cooperative propagation. The method comprises the following steps: 1) extracting time sequence features and saliency foreground features from an unclipped video; 2) constructing a basic branch and a saliency perception branch to respectively process the time sequence features and the saliency target features to obtain a basic class activation sequence, a motion and appearance representation score and a saliency class activation sequence, and performing weighted fusion on the four sequences to obtain a fusion action score sequence; 3) constraining interactive semantic information and significance information by using the consistency of branch distillation and branch action, and perfecting a fusion action score sequence; and 4) extracting key segments and ambiguous segments of the basic branches and the saliency perception branches, performing comparative learning by using the key segments and ambiguous segments between the branches and in the branches to improve feature representation, perfecting a fusion action score sequence in combination with a distillation result, and obtaining an action positioning result. According to the method, tiny human body actions and accurate time action boundaries can be perceived in an unedited video.

Description

Technical Field

[0001] The present invention relates to the technical field of computer video analysis, and in particular to a method and device for locating time-sequence actions in weakly supervised videos. Background Art

[0002] Video temporal action localization is one of the research focuses in the field of computer vision, which aims to locate the start and end time of the action category in the uncropped video. It has important application value and significance in real life, such as video retrieval, video abnormal content detection, intelligent monitoring, smart transportation and other fields. At present, the training methods of video temporal action localization network can be divided into fully supervised training and weakly supervised training. The training of fully supervised temporal action localization requires not only the action category labels contained in the video, but also manual annotation of each frame. However, in actual application scenarios, video data often lacks frame-level annotations, and manual annotation of massive videos requires a lot of cost. It is time-consuming and laborious to use fully supervised training methods in real-world scenarios. In order to overcome the limitations of the fully supervised temporal action localization model with frame-by-frame labeling, an efficient weakly supervised temporal action localization method has emerged, which only requires video-level classification labels for training.

[0003] Existing weakly supervised temporal action localization methods can be divided into three categories: the first type of methods focuses on the pattern difference between classification and localization. This method focuses on generating more refined pseudo-label segments to reduce the difference between classification and localization. The second type of methods focuses on the separation of foreground segments and background segments. The main means is to use different attention mechanisms to enable accurate discrimination of foreground and background; the third type of methods is a method that emphasizes learning discriminative behavioral features or fuzzy behavioral features. This method learns discriminative behavioral features through measurement, erasure, or comparison to alleviate fuzzy behavioral features. The first two types of methods often expose the problem of focusing only on the most discriminative action segments, and the fuzzy segments are easily ignored, resulting in missed detection or false detection.

[0004] The above three methods can achieve certain positioning effects by learning effective loss functions, but they are often constrained by the limited representation of appearance and motion features, making it difficult for the model to distinguish between behavior segments where part of the foreground appearance blends into the background and where the temporal picture changes dramatically, resulting in the problem of false detection and missed detection of action. Therefore, in-depth exploration of the collaborative relationship between semantic and saliency knowledge, and perceiving subtle action information in segments where the foreground and background are difficult to separate to extract the foreground, have important positive significance for improving the model's ability to locate video temporal actions. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention provides a weakly supervised temporal action localization method and device based on the collaborative propagation of semantic and saliency knowledge. The present invention studies the propagation method between semantic knowledge and saliency knowledge in videos from two perspectives: the first is to achieve the complementarity of key information in semantic and saliency knowledge through distillation between branches, and the second is to achieve the internalization of knowledge through clustering of feature representations between branches and within branches. Based on this idea, the problem of false detection or missed detection of fine-grained action clips by the model is alleviated.

[0006] The technical solution adopted by the present invention to solve its technical problem is:

[0007] The first aspect of the present invention relates to a weakly supervised temporal action localization method based on the collaborative propagation of semantic and saliency knowledge, the method comprising the following steps:

[0008] 1) Using a dual-stream network and a salient object detection network, temporal features and salient foreground features are extracted from uncut videos, where the uncut videos contain human activities and other backgrounds;

[0009] 2) Establish a saliency perception branch, use the extracted salient foreground features as input to train the branch to obtain a saliency class activation sequence, which is used as the representative sequence of the branch, and summed up in the channel dimension to obtain a salient action representation score. This score is used to perform weighted fusion with the temporal class activation sequence, appearance action representation score, and motion action representation score obtained in the basic branch to obtain a fused action score sequence, which is used as the representative sequence of the basic branch to represent the initial fusion of semantic and saliency knowledge;

[0010] 3) Pseudo labels are obtained by comparing the fused action score sequence and the saliency class activation sequence with the threshold. The knowledge in the local pseudo labels is propagated to the representative sequence of the other branch through the distillation method, and the action consistency constraint under the prediction category dimension is used to enhance the propagation effect, ultimately achieving complementary learning of semantic information and saliency information.

[0011] 4) Based on the pseudo-labels of each branch, the corresponding key segments and ambiguous segments are mined, where the key segments contain both key semantics and significant foreground information, while the ambiguous segments contain contradictory semantic content and foreground saliency information. Based on the above segments, the contrastive learning method is used to aggregate the feature representation of the key segments, and the ambiguous segment features are guided to be close to the key segment features in the feature space to obtain a more accurate representation. Finally, the temporal action localization result is obtained by combining the index of the first k high-scoring actions in the fused action sequence and the temporal class activation sequence.

[0012] The second aspect of the present invention relates to a weakly supervised temporal action localization device based on the collaborative propagation of semantics and significance knowledge, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a weakly supervised temporal action localization method based on the collaborative propagation of semantics and significance knowledge of the present invention.

[0013] A computer-readable storage medium stores a program thereon, and when the program is executed by a processor, a weakly supervised temporal action localization method based on the collaborative propagation of semantic and saliency knowledge of the present invention is implemented.

[0014] The working principle of the present invention is (analyzing the reasons for the advantages of the present invention): for the problem of temporal positioning of videos containing human action behaviors, the significant foreground information in the video is extracted, and a separate branch and distillation method and a comparative learning method are designed to combine the significant foreground information with the temporal feature information, thereby improving the utilization rate of effective significant foreground information.

[0015] The advantages of the present invention are: for massive videos containing human action behaviors, a weakly supervised temporal action localization method based on the collaborative propagation of semantic and saliency knowledge is proposed, which can identify and locate human action behaviors with high accuracy. By effectively fusing saliency foreground information and semantic information, the attention to the salient action area of ​​the human body is greatly improved, and the recognition and localization capabilities of human actions in videos are enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is an overall framework diagram of the weakly supervised temporal action localization method of the present invention.

[0017] Figure 2a and Figure 2b This is a visualization example result display diagram of the present invention, wherein: Figure 2a This is the visualization result diagram of the "Javelin Throwing" action instance. Figure 2b Visualization result diagram of the “cliff diving” action example.

[0018] Figure 3 is a flow chart of the method of the present invention. Specific implementation methods

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] Example 1

[0022] Reference Figure 1 As shown in Figure 2, the weakly supervised temporal action localization method based on the collaborative propagation of semantic and saliency knowledge includes the following steps:

[0023] 1) Establish a saliency perception branch, use the extracted salient foreground features as input to train the branch to obtain a saliency class activation sequence, use it as the representative sequence of the branch, and sum it in the channel dimension to obtain a saliency action representation score. Use this score to perform weighted fusion with the time class activation sequence, appearance action representation score, and motion action representation score obtained in the basic branch to obtain a fused action score sequence, which is used as the representative sequence of the basic branch to represent the initial fusion of semantic and saliency knowledge. The process can be expressed as follows: First, the appearance and optical flow features are combined into a fusion sequence. Concatenating in the channel dimension gives It is then fed into a feature extractor layer f consisting of a one-dimensional convolution and a ReLU function to obtain the embedding feature As follows:

[0024] E C =ReLU(Conv1D(X C )) (1)

[0025] Then, through F C Layer obtains the temporal class activation sequence It consists of a one-dimensional convolution and a ReLU function. C represents the total number of categories in all videos, as follows:

[0026] S C =ReLU(Conv1D(E C )) (2)

[0027] Finally, for S C Sum across the channel dimension and feed it into the sigmoid function to get the base action score As follows:

[0028]

[0029] After the saliency perception branch is established, it is used to obtain the saliency knowledge of the video from the salient foreground features. First, the salient foreground features After preprocessing, we get As shown in formula (5).

[0030]

[0031] Use a feature extractor layer f consisting of a one-dimensional convolution Conv1D and a ReLU function S activation Obtaining the salient feature embedding vector As follows:

[0032] E SA =ReLU(Conv1D(X SA )) (5)

[0033] E SA Input to F composed of one-dimensional convolution Conv1D and ReLU function SA Obtain the activation sequence of significant classes in the layer S SA As a representative sequence of the saliency perception branch, it is as follows:

[0034] S SA =ReLU(Conv1D(E SA )) (6)

[0035] On the channel dimension C, SA Sum up to get the significant action representation score As shown below:

[0036]

[0037] in Indicates that the tth fragment is in A SA The score in Indicates S SA The activation value of the t-th segment on channel c in .

[0038] In order to better integrate semantic and saliency knowledge, the weighted fusion time class activation sequence S C and appearance-action representation score Appearance Action Representation Score Salience Action Representation Score Get the fused action score sequence And this sequence is used as the representative sequence of the basic branch, as shown in formula (5).

[0039]

[0040] Among them, μ1, μ2, and μ3 represent the fusion weights.

[0041] In order to optimize the classification effect, the set of top-k instances of each category c is represented as T C According to T C The corresponding category activation sequence regions are aggregated to obtain the video-level category prediction p c Finally, the video classification loss in the form of cross entropy loss The formula is as formula (6):

[0042]

[0043] Among them, y c is the true value.

[0044] 2) Pseudo-labels are obtained by comparing the fused action score sequence and the saliency class activation sequence with the threshold. The knowledge in the local pseudo-labels is propagated to the representative branch of the other branch through the distillation method, and the action consistency constraint under the prediction category dimension is used to enhance the propagation effect. Finally, the complementary learning of semantic information and saliency information is achieved. The process can be expressed as follows: Considering S SA and A Score The presence of noise in will affect the dissemination of key knowledge, and the generation of ternary pseudo-labels is used as a medium for knowledge interaction and internalization. Fixed coefficients α1 and α2 are used to multiply A C and A SA The median of each branch is used to obtain the high and low thresholds σ h , σ l (σ h >σ l ), use the threshold to set the pseudo label As shown in formula (10):

[0045]

[0046] Where t, c represent the segment index and category respectively, p c For each representative sequence S∈{S SA , A Score} is the prediction score sequence on category c, p t，c is the prediction score of the t-th segment in S for category c.

[0047] In order to enhance the interaction between semantic and saliency knowledge, alleviate the differences in the meaning of activation values ​​in different sequences, and focus on the sequence parts containing specific categories, a class-specific action score sequence is designed as shown in formula (11): And consistency constraints will be introduced in the subsequent optimization process.

[0048]

[0049] Among them, ca t represents the action-like score of the t-th clip, h c Denotes a pseudo label H∈{H SA , H C} under category c.

[0050] The distillation method mainly reduces the information entropy difference between two probability distributions by designing a relative entropy (also called information divergence) optimization function, that is, reducing the information loss or error introduced when one distribution is used to approximate another distribution. Here, the main focus is on reducing the relative entropy when the branch representative sequence is approximated to the pseudo-label sequence. Therefore, in order to encourage the model to achieve complementary learning of semantic and saliency information, information divergence (KL divergence) is introduced to design branch distillation loss. Inter-branch distillation achieves the initial fusion of key information between branches in a distribution approximation manner, filters semantically irrelevant saliency knowledge, and supplements poor semantic knowledge with semantically related saliency knowledge. The formula is as follows:

[0051]

[0052] Where (L, A) ∈ {(C, SA), (SA, C)}, L represents the local branch, A represents another branch opposite to the local branch, C represents the basic branch, and SA represents the saliency-aware branch. Finally, the combination of (L, A) represented in set form is substituted into the above equation to calculate the total loss of branch distillation.

[0053] Since the inter-branch distillation method may focus too much on the most discriminative segment information and ignore other semantically relevant information when filtering semantically irrelevant salient information, we design a class-specific action consistency constraint, as shown in Equation (13), to achieve consistent activation of semantic and salient information after filtering.

[0054]

[0055] in Indicates stopping gradient backpropagation.

[0056] 3) Based on the pseudo-labels of each branch, the corresponding key fragments and ambiguous fragments are mined, where the key fragments contain both key semantics and significant foreground information, while the ambiguous fragments contain contradictory semantic content and foreground saliency information. Based on the above fragments, the contrastive learning method is used to aggregate the feature representations of key fragments, and guide the ambiguous fragment features to be close to the key fragment features in the feature space to obtain a more accurate representation. Finally, the temporal action localization results are obtained by combining the indexes of the first k high-scoring actions in the fused action sequence and the time class activation sequence. The process can be expressed as follows: In order to deepen the understanding and internalization of knowledge by each branch after knowledge interaction, based on the consistency and numerical differences between each pseudo-label, consistent key fragments and inconsistent ambiguous fragments are mined for each branch. First, obtain the mask To obtain consistent and inconsistent information between branches. The formula is as follows:

[0057]

[0058] IC and CS represent different mask types respectively.

[0059] Then the mask M M Multiply r by each representative sequence S and sum them in D dimension to obtain the descending order The formula is as follows:

[0060]

[0061] Where Q∈{Act, Bkg} represents the segment type, Act, Bkg represent action segments and background segments respectively, MT∈{CS, IC} represents the mask type, and p t，d represents the score value of the t-th segment in the sequence S in the d dimension, Represents the maximum score value among all segments in the d-dimensional representation of sequence S.

[0062] According to R MT，Q , select E C and E SA The first k instances in , the consistent key action (CKA), inconsistent ambiguous action (IAA), consistent key background (CKB) and inconsistent ambiguous background (IAB) are obtained in two branches respectively.

[0063] Between branches and within branches, InfoNCE loss is introduced as sample contrastive learning loss to promote the internalization of knowledge in branches through contrastive learning. The formula is as follows:

[0064]

[0065] The query sample is x and the positive sample is x + , the negative sample is x - , τ = 0.07 is the temperature hyperparameter, x T represents the transpose of vector x, and M represents the number of negative sample segments. The contrastive learning sample pair setting is as follows:

[0066]

[0067]

[0068] Substituting the above formula into formula (18), we can obtain the contrastive learning loss between branches and within branches respectively

[0069] Finally, the sum of all losses is used as the optimization target, and the formula is as follows:

[0070]

[0071] Where γ1, γ2, γ3, and γ4 are the corresponding loss coefficients.

[0072] The temporal action localization result is extracted by setting a multivariate threshold to obtain the fusion action score A Score The instances in are used as predicted candidate action proposals, and then the non-maximum suppression method is applied to these proposals to remove duplicate action proposals.

[0073] Experimental results: The datasets used in this experiment are the THUMOS-14 dataset and the ActivityNet-1.2 dataset. The THUMOS-14 dataset includes 200 verification videos and 213 test videos. Both video sets contain 20 classes. This experiment uses the verification set for training and the test set for evaluation. The ActivityNet-1.2 dataset contains 4819 training videos, 2383 verification videos, and 2489 test videos. All three video sets contain 100 classes. The method of dividing the dataset is the same as the previous mainstream method. This experiment uses the training set for training and the verification set for evaluation. The evaluation results on the THUMOS-14 dataset and ActivityNet-1.2 dataset are shown in Tables 1 and 2, respectively. It can be seen that the action localization accuracy of the method of the present invention exceeds that of the previous mainstream methods.

[0074]

[0075] Table 1

[0076]

[0077] Table 2

[0078] Example 2

[0079] A weakly supervised temporal action localization device based on the collaborative propagation of semantics and saliency knowledge includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the weakly supervised temporal action localization method based on the collaborative propagation of semantics and saliency knowledge described in any one of Example 1.

[0080] Example 3

[0081] A program is stored on a computer-readable storage medium. When the program is executed by a processor, the weakly supervised temporal action localization method based on the collaborative propagation of semantic and saliency knowledge described in any one of Embodiment 1 is implemented.

[0082] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A weakly supervised temporal action localization method based on the collaborative propagation of semantic and saliency knowledge, characterized by: The following steps are involved: 1) Using a dual-stream network and a salient object detection network, temporal features and salient foreground features are extracted from uncut videos, where the uncut videos contain human activities and other backgrounds; 2) Establish a saliency perception branch, use the extracted salient foreground features as input to train the branch to obtain a saliency class activation sequence, which is used as the representative sequence of the branch, and summed up in the channel dimension to obtain a salient action representation score. This score is used to perform weighted fusion with the temporal class activation sequence, appearance action representation score, and motion action representation score obtained in the basic branch to obtain a fused action score sequence, which is used as the representative sequence of the basic branch to represent the initial fusion of semantic and saliency knowledge; 3) Pseudo labels are obtained by comparing the fused action score sequence and the saliency class activation sequence with the threshold. The knowledge in the local pseudo labels is propagated to the representative sequence of the other branch through the distillation method, and the action consistency constraint under the prediction category dimension is used to enhance the propagation effect, ultimately achieving complementary learning of semantic information and saliency information. 4) Based on the pseudo-labels of each branch, the corresponding key segments and ambiguous segments are mined, where the key segments contain both key semantics and significant foreground information, while the ambiguous segments contain contradictory semantic content and foreground saliency information. Based on the above segments, the contrastive learning method is used to aggregate the feature representation of the key segments, and the ambiguous segment features are guided to be close to the key segment features in the feature space to obtain a more accurate representation. Finally, the temporal action localization result is obtained by combining the index of the first k high-scoring actions in the fused action sequence and the temporal class activation sequence.

2. The method for localizing actions in a weakly supervised manner based on the collaborative propagation of semantic and saliency knowledge as claimed in claim 1, characterized in that: Step 2) specifically includes: after the saliency perception branch is established, it is used to obtain the saliency knowledge of the video from the saliency foreground features. First, the saliency foreground features are After preprocessing, we get As shown in formula (5). Use a feature extractor layer fS consisting of a one-dimensional convolution Conv1D and a ReLU function activation Obtaining the salient feature embedding vector As follows: E SA =ReLU(Conv1D(X SA )) (2) E SA Input to F composed of one-dimensional convolution Conv1D and ReLU function SA Obtain the activation sequence of significant classes in the layer S SA As a representative sequence of the saliency perception branch, it is as follows: S SA =ReLU(Conv1D(E SA )) (3) On the channel dimension C, SA Sum up to get the significant action representation score As shown below: in Indicates that the tth fragment is in A SA The score in Indicates S SA The activation value of the t-th segment on channel c in . In order to better integrate semantic and saliency knowledge, the weighted fusion time class activation sequence S C and appearance-action representation score Appearance Action Representation Score Salience Action Representation Score Get the fused action score sequence And this sequence is used as the representative sequence of the basic branch, as shown in formula (5). Among them, μ1, μ2, and μ3 represent the fusion weights. In order to optimize the classification effect, the set of top-k instances of each category c is represented as T C According to T C The corresponding category activation sequence regions are aggregated to obtain the video-level category prediction p c Finally, the video classification loss in the form of cross entropy loss The formula is as follows (6): Among them, y c is the true value.

3. The method for localizing actions in a weakly supervised manner based on the collaborative propagation of semantic and saliency knowledge as claimed in claim 1, characterized in that: Step 3) specifically includes: wherein the distillation method mainly reduces the information entropy difference between the two probability distributions by designing a relative entropy (also called information divergence) optimization function, that is, reduces the information loss or error introduced when one distribution is used to approximate another distribution. Here, the principle of the distillation method is used to reduce the relative entropy when the branch representative sequence is approximated to the pseudo-label sequence. First, in order to reduce S SA and A Score The influence of noise on the dissemination of key knowledge is considered, and ternary pseudo-labels are generated as a medium for knowledge interaction and internalization. Secondly, fixed coefficients α1 and α2 are used to multiply A C and A SA The median of each branch is used to obtain the high and low thresholds σ h , σ l (σ h >σ l ), and use the threshold to set the pseudo label As shown in formula (7): Where t, c represent the segment index and category respectively, p c For each representative sequence S∈{S SA , A Score } is the prediction score sequence on category c, p t，c is the prediction score of the t-th segment in S for category c. In order to enhance the interaction between semantic and saliency knowledge, alleviate the differences in the meaning of activation values ​​in different sequences, and focus on the sequence parts containing specific categories, a class-specific action score sequence is designed as shown in formula (8): And consistency constraints will be introduced in the subsequent optimization process. Among them, ca t represents the action-like score of the t-th clip, h c Denotes a pseudo label H∈{H SA , H C } under category c. In order to enable the model to achieve complementary learning of semantic and salient information, information divergence (KL divergence) is introduced to design branch distillation loss. Inter-branch distillation achieves the initial fusion of key information between branches in a distribution-approximate manner, filters semantically irrelevant salient knowledge, and supplements poor semantic knowledge with semantically relevant salient knowledge. The formula is as follows: Where (L, A) ∈ {(C, SA), (SA, C)}, L represents the local branch, A represents another branch opposite to the local branch, C represents the basic branch, and SA represents the saliency-aware branch. Finally, the combination of (L, A) represented in set form is substituted into the above equation to calculate the total loss of branch distillation. Since the inter-branch distillation method may focus too much on the most discriminative segment information and ignore other semantically relevant information when filtering semantically irrelevant salient information, we design a class-specific action consistency constraint, as shown in Equation (10), to achieve consistent activation of semantic and salient information after filtering. in Indicates stopping gradient backpropagation.

4. The method for localizing actions in a weakly supervised manner based on the collaborative propagation of semantic and saliency knowledge as claimed in claim 1, characterized in that: Step 4) specifically includes: in order to deepen the understanding and internalization of knowledge by each branch after knowledge interaction, based on the consistency and numerical difference between each pseudo-label, consistent key fragments and inconsistent ambiguous fragments are mined for each branch. First, the mask is obtained. To obtain consistent and inconsistent information between branches. The formula is as follows: IC and CS represent different mask types respectively. Then the mask M M r is multiplied by each representative sequence S and summed over the D dimension to obtain the descending order The formula is as follows: Where Q∈{Act, Bkg} represents the segment type, Act, Bkg represent action segments and background segments respectively, MT∈{CS, IC} represents the mask type, and p t,d represents the score value of the t-th segment in the sequence S in the d dimension, Represents the maximum score value among all segments in the d-dimensional representation of sequence S. According to R MT,Q , select E C and E SA The first k instances in , the consistent key action (CKA), inconsistent ambiguous action (IAA), consistent key background (CKB) and inconsistent ambiguous background (IAB) are obtained in two branches respectively. Between branches and within branches, InfoNCE loss is introduced as sample contrast learning loss to promote the internalization of knowledge in branches through contrast learning. The formula is as follows: The query sample is x and the positive sample is x + , the negative sample is x - , τ = 0.07 is the temperature hyperparameter, x T represents the transpose of vector x, and M represents the number of negative sample segments. The contrastive learning sample pair settings are shown in equations (16) and (17): Substituting the above formula into formula (15), we can obtain the contrastive learning loss between branches and within branches respectively Finally, the sum of all losses is used as the optimization target, as shown in formula (18): Where γ1, γ2, γ3, and γ4 are the corresponding loss coefficients. The temporal action localization result is extracted by setting a multivariate threshold to obtain the fusion action score A Score The instances in are used as predicted candidate action proposals, and then the non-maximum suppression method is applied to these proposals to remove duplicate action proposals.

5. The method for localizing actions in a weakly supervised manner based on the collaborative propagation of semantic and saliency knowledge as claimed in claim 4, characterized in that: The values ​​of γ1, γ2, γ3, and γ4 are 1.8, 2.0, 0.015, and 0.02, respectively.

6. The method for localizing actions in a weakly supervised manner based on the collaborative propagation of semantic and saliency knowledge as claimed in claim 2, characterized in that: In step 2), the temporal class activation sequence, appearance action representation score, and motion action representation score in the basic branch are obtained as follows: first, the appearance and optical flow features are Concatenating in the channel dimension gives It is then fed into a feature extractor layer f consisting of a one-dimensional convolution and a ReLU function to obtain the embedding feature As follows: E C =ReLU(Conv1D(X C )) (19) Then, through F C Layer obtains the temporal class activation sequence It consists of a one-dimensional convolution and a ReLU function. C represents the total number of categories in all videos, as follows: S C =ReLU(Conv1D(E C )) (20) Finally, for S C Sum across the channel dimension and feed it into the sigmoid function to get the base action score As follows:

7. A weakly supervised temporal action localization device based on the collaborative propagation of semantic and saliency knowledge, characterized in that: It includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the weakly supervised temporal action localization method based on the collaborative propagation of semantic and saliency knowledge as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by a processor, the weakly supervised temporal action localization method based on the collaborative propagation of semantic and saliency knowledge described in any one of claims 1-5 is implemented.

Images