Weak supervision time sequence action positioning method and system of self-supervision pseudo label optimization network

Abstract

The invention discloses a weak supervision time sequence action positioning method and system for a self-supervision pseudo label optimization network, and the method comprises the steps: dividing all segments in a training video into a plurality of class clusters, and obtaining atomic action features representing global semantics; taking the atomic action features representing global semantics as action representation, mapping the video clip-level features extracted by the feature extractor into query vectors, and participating in attention calculation together with the atomic action features to generate enhanced fragment representation containing global context information; the method comprises the following steps: identifying an inconsistent region by utilizing a high-discrimination characteristic expressed by an enhanced fragment, executing pseudo-label correction by collecting pseudo-label results generated in two continuous training periods, judging whether to expand a label range and generate a pseudo-single-frame label yfp by adopting an affinity search strategy and utilizing characteristic similarity, and optimizing through consistency loss. And obtaining a positioning result. According to the method, the marking cost is remarkably reduced, background interference is effectively reduced, and the integrity and precision of action positioning are improved.

Description

Technical Field

[0001] The present invention belongs to the technical fields of computer vision, weakly supervised learning and temporal action detection, and particularly relates to a weakly supervised temporal action localization method and system of a self-supervised pseudo-label optimization network. Background Art

[0002] Video has become a powerful, cost-effective medium for information dissemination, with widespread applications in sports, traffic monitoring, filmmaking, and other scenarios. This has led to significant interest in intelligent video analysis within the fields of computer vision and multimedia applications. Temporal action localization in untrimmed videos is a core task in intelligent video analysis. Its goal is to detect the temporal boundaries of actions in videos and identify their categories. It plays a crucial role in tasks such as video description, video captioning, intelligent surveillance, and human-computer interaction. While some existing methods are conducted in a fully supervised setting and achieve high performance on current datasets, obtaining precise, fine-grained annotations (i.e., the start and end timestamps of each action, as well as video-level labels) during training is prohibitively expensive. Consequently, weakly supervised temporal action localization (WTAL) methods, which require only video-level annotations, have gained popularity. However, due to the lack of frame-level annotations, WTAL struggles to achieve performance approaching that of fully supervised temporal action localization.

[0003] Existing WTALs mainly adopt a classification-localization process, in which the classifier is trained using video-level annotations of action categories and used to obtain a series of category scores or predictions, namely Temporal Class Activate Maps (TCAMs). Typically, TCAMs obtain detection results through post-processing steps (e.g., threshold segmentation) or localization branches. Therefore, the quality of TCAMs determines the upper bound of the model. However, there is usually a difference between classification and localization. Since each video usually contains multiple segments, but in the case of only video-level annotations, the model is prone to focus on contextual background or the discriminative segments that contribute most to video-level classification, which limits the generation of high-quality TCAMs.

[0004] To overcome the discrepancy between classification and localization, existing methods employ pseudo-labeling. The advantage of using pseudo-labels is that segments are supervised with segment-level labels rather than video-level labels, thus bridging the gap between classification and localization. However, this approach utilizes limited information, using TCAMs or their variants generated in previous training iterations to generate pseudo-labels, which is insufficient to bridge the gap between classification and localization. Building on this, some researchers have proposed modeling characteristic segments within videos to guide pseudo-label generation. This approach captures important semantic and segment features from each video, stores them as representative segments in a memory, and propagates their useful knowledge to other segments to generate high-quality pseudo-labels. While this approach can capture the characteristics of video segments to generate and optimize pseudo-labels, it still suffers from the following drawbacks: the captured features are not necessarily action-related and therefore do not necessarily guide pseudo-label generation. Furthermore, the extracted feature segments still suffer from significant bias and error. This introduces noisy features into feature training, which impacts model learning and leads to suboptimal performance in action localization. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to address the deficiencies in the above-mentioned prior art and provide a weakly supervised temporal action localization method and system based on a self-supervised pseudo-label optimization network, which is used to solve the technical problems of inaccurate positioning caused by inconsistency between classification and positioning targets and large pseudo-label noise in effective temporal action localization. By combining clustering strategy with self-supervised learning, the model's ability to distinguish actions is improved and its robustness in complex scenes is enhanced. It also provides a new approach for video analysis tasks in a weakly supervised environment, such as fine-grained action recognition and efficient processing of long videos.

[0006] The present invention adopts the following technical solutions:

[0007] The weakly supervised temporal action localization method based on a self-supervised pseudo-label optimization network includes the following steps:

[0008] Collect segment-level feature information from all training videos and perform K-means clustering to divide all segments into several category clusters to obtain atomic action features that represent global semantics;

[0009] The atomic action features representing global semantics are used as action representations. The video segment-level features extracted by the feature extractor are mapped into query vectors, which are then used together with the atomic action features in attention calculations to generate enhanced segment representations containing global context information.

[0010] The high discrimination characteristics of enhanced segment representation are used to identify inconsistent areas. By collecting pseudo-label results generated in two consecutive training periods, pseudo-label correction operations are performed based on the differences. An affinity search strategy is used to search the front and back ranges in adjacent frames in the time dimension. Feature similarity is used to determine whether to expand the label range. Pseudo single-frame labels are generated based on the interval of confidence frames. The positioning result is obtained by optimizing through consistency loss.

[0011] Preferably, all segments are divided into several category clusters, and the atomic action features representing global semantics are obtained as follows:

[0012] Collect the features of all clips in all training data, perform K-means clustering on the entire feature pool to obtain C clusters, where C is the action category in the video; each video clip v i is assigned a label p i , each video segment v i is assigned a label p i , p i ∈{1,...,C};

[0013] Use the labels to train a visual semantic concept classifier F; the shared features xi of the visual semantic concept classifier F are mapped to the predicted cluster scores

[0014] Use the tag p i Extract visual semantic concepts from video features;

[0015] Combine the MIL classifier with the visual semantic concept v={v1,...,v N}Connect them and learn the combination relationship between action atoms;

[0016] For each action j in the video, calculate s j The distance D between all visual semantic concepts j Then select the top k visual semantic concepts with the smallest distance Denoted as the set TopK(D j ); Finally, the k visual semantic concepts are regarded as the atomic actions of action j.

[0017] Preferably, the nth visual semantic concept v n as follows:

[0018]

[0019] Where T is the time dimension of the video clip.

[0020] Preferably, the representation m of the atomic action j is generated by using the representations of these selected visual semantic concepts through a composition function C j as follows:

[0021] m j =C({v n |n∈TopK(D j )}).

[0022] Preferably, generating the enhanced segment representation containing global context information is specifically as follows:

[0023] Use the linear layer f q Project video features onto the corresponding query;

[0024] Connect the video feature X with the action atom feature m stored in the action atom memory bank j ;

[0025] Using a single linear layer f k and f v , project the connected features into K and V respectively;

[0026] Multiply the query with the adjusted key to get the global attention

[0027] The residual layer composed of ReLU and Dropout and the WeightNorm weight normalization layer are used to normalize and retain the original information.

[0028] Preferably, the global attention attn is as follows:

[0029]

[0030] K=f k (concat(X,m1,...,m c ))

[0031] V=f v (concat(X,m1,...,m c ))

[0032] Among them, d is the vector dimension, Q is the query vector, m c is the c-th action atomic feature.

[0033] Preferably, a pseudo-label correction operation is performed based on the difference, and an affinity search strategy is adopted to determine whether to expand the label range using feature similarity. The optimization is performed through consistency loss, and the positioning results are specifically as follows:

[0034] The difference between the two pseudo labels collected during the two training periods of pseudo label generation is used as the new pseudo label;

[0035] Learn an embedding space to distinguish class-independent actions and backgrounds; given the input features, use a temporal convolutional layer to learn the embedding of each frame, i.e. E = [e 1 ;e 2 ;...;e T ], e T It is a D emb dimensional vector; L2 regularization is performed on each embedding vector;

[0036] The search range of the foreground single frame pseudo label is between the previous frame and the next background single frame. Find the timestamp A(e u ,e v )>δ as the maximum small frame as the confident foreground frame C a . Find A(e in the search range u ,e v ) The minimum and maximum frames are taken as the confidence background frames C b ;

[0037] After obtaining all the confidence frames, generate pseudo single frame labels based on the interval of the confidence frames

[0038] After obtaining the foreground pseudo-label fragment p act and background pseudo-label fragment p bg Afterwards, the consistency loss is used for optimization to obtain the positioning result.

[0039] Preferably, the confident foreground frame C a and the confidence background frame C b as follows:

[0040]

[0041] Among them, s is the starting boundary, e is the ending boundary, is the starting boundary of the background frame, The ending boundary of the background frame.

[0042] Preferably, the consistency loss L con for:

[0043] L con =-logp act -log(1-p bg ).

[0044] In a second aspect, an embodiment of the present invention provides a weakly supervised temporal action localization system using a self-supervised pseudo-label optimization network, comprising:

[0045] The atomic action generation module collects segment-level feature information from all training videos and performs K-means clustering to divide all segments into several category clusters to obtain atomic action features that represent global semantics.

[0046] The self-supervised learning module uses atomic action features that represent global semantics as action representations, maps video segment-level features extracted by the feature extractor into query vectors, and jointly participates in attention calculation with the atomic action features to generate enhanced segment representations containing global context information;

[0047] The pseudo-label optimization and search module uses the high-discrimination characteristics of the enhanced segment representation to identify inconsistent areas. By collecting the pseudo-label results generated in two consecutive training periods, it performs pseudo-label correction operations based on the differences and adopts an affinity search strategy to search the front and back range in the time dimension in adjacent frames. It uses feature similarity to determine whether to expand the label range and generates pseudo single-frame labels based on the interval of confidence frames. The positioning result is obtained by optimizing through consistency loss.

[0048] In a third aspect, a computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the weakly supervised temporal action localization method of the self-supervised pseudo-label optimization network are implemented.

[0049] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, comprising a computer program, which, when executed by a processor, implements the steps of the above-mentioned weakly supervised temporal action localization method of the self-supervised pseudo-label optimization network.

[0050] In the fifth aspect, a chip includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the steps of the weakly supervised temporal action localization method of the self-supervised pseudo-label optimization network are implemented.

[0051] In a sixth aspect, an embodiment of the present invention provides an electronic device, comprising a computer program, which, when executed by the electronic device, implements the steps of the weakly supervised temporal action localization method of the self-supervised pseudo-label optimization network.

[0052] Compared with the prior art, the present invention has at least the following beneficial effects:

[0053] A weakly supervised temporal action localization method based on a self-supervised pseudo-label optimization network is proposed. In order to discover the global semantic structure in the feature space of video clips through clustering, and thus obtain atomic action features that can represent action categories, the atomic action features are combined with the extracted clip features, and the spatiotemporal dependencies between clips are captured through a self-attention mechanism, thereby generating enhanced clip features with more global context information. The pseudo-labels obtained by clustering initialization are dynamically corrected and expanded. By comparing the pseudo-labels of two consecutive training periods and combining affinity search of features from adjacent time periods, the accuracy and consistency of the pseudo-labels in the temporal dimension are improved. This method effectively solves the problem of inaccurate positioning caused by inconsistency between classification and positioning targets and high pseudo-label noise in temporal action localization. By combining clustering strategies with self-supervised learning, pseudo-label correction and affinity search, the temporal coherence of action boundaries is enhanced, reducing fragmented detection. Through consistency loss and confidence frame screening, the model's own prediction results are iteratively optimized to reduce reliance on manual labeling.

[0054] Furthermore, the segment features in all training videos are clustered using K-means to assign preliminary cluster labels to each segment, providing a basis for subsequent visual semantic concept learning. The assigned cluster labels are used to train a visual semantic concept classifier to map shared segment features to corresponding cluster scores, thereby learning the semantic features represented by different clusters. Segment features with the same pseudo-label are aggregated to construct a representation of each visual semantic concept, so that each concept can characterize its corresponding action pattern in the form of a feature center. By connecting the MIL classifier with the visual semantic concept, the combination relationship between action atoms is learned, so that segment-level features can be mapped to action prediction scores, improving the ability to identify action categories. Based on the distance between the visual semantic concept and the action prediction score, the most representative concept is selected as the atomic action representation, and the atomic action features of each action category are generated through the combination function for subsequent self-attention module calls.

[0055] Furthermore, the minimum distance selection filters out the atoms most relevant to the target action and suppresses the interference of irrelevant features; the combination function C fuses multi-atom features to simulate the semantic superposition of complex actions and enhance the representation flexibility.

[0056] Furthermore, the video clip features are mapped to a query vector, enabling cross-temporal interaction with the atomic action features during the self-attention computation. The concatenated video clip features and the action atomic features are linearly transformed to generate key and value vectors, respectively, which provide input for the self-attention computation. The query vector is multiplied by the generated key vector to calculate the global attention weight, which is then weighted and summed over the value vector to obtain a globally enhanced feature that incorporates the atomic action information.

[0057] Furthermore, linear projection and concatenation align atomic features with fragment features to avoid information loss caused by dimension mismatch, and residual normalization retains the original features (such as background texture) to prevent overfitting noise.

[0058] Furthermore, by collecting pseudo-labels generated from two consecutive training cycles and calculating the difference, the accidental fluctuations of single-period pseudo-labels can be eliminated, resulting in more stable preliminary pseudo-labels. By learning the embedding space and calculating the L2 regularization of the embedding vector for each frame, action and background features can be distinguished, thereby screening out high-confidence pseudo-label frames and providing reliable seeds for subsequent expansion. By performing a range search on adjacent frames based on feature affinity in the temporal dimension, the temporal boundaries of action pseudo-labels can be effectively expanded to ensure the continuity and integrity of action segments. After obtaining all confident frames, pseudo single-frame labels are generated based on the time interval between them, and their loss is calculated using FocalLoss to strengthen the focus on difficult-to-distinguish frames and improve the effectiveness of the pseudo-label training signal. The foreground pseudo-label segments and background pseudo-label segments are optimized using consistency loss, making the pseudo-labels smoother in time and consistent with the network prediction, thereby further improving the quality of pseudo-labels.

[0059] Furthermore, feature similarity is used to expand the label range, solve the problem of blurred action boundaries, force a smooth transition of pseudo-labels in adjacent frames, and reduce the model's sensitivity to noise.

[0060] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0061] In summary, the present invention solves the problem of inaccurate positioning caused by inconsistency between classification and positioning targets and large pseudo-label noise in weakly supervised temporal action localization. By guiding the generation of atomic action features through clustering, enhancing segment representation through self-attention, and dynamically correcting and temporally expanding pseudo-labels, the integrity and accuracy of action localization are significantly improved.

[0062] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0064] Figure 1 This is an overall flow chart of the weakly supervised temporal action localization method based on the clustering-guided self-supervised pseudo-label optimization network of the present invention;

[0065] Figure 2 Generate process flow diagrams for atomic actions;

[0066] Figure 3 Flowchart of the self-supervised learning process;

[0067] Figure 4 Flowchart of pseudo-label optimization and search process;

[0068] Figure 5 is a comparison chart of recognition performance;

[0069] Figure 6 A schematic diagram of a computer device provided in accordance with an embodiment of the present invention;

[0070] Figure 7 The present invention is a block diagram of an electronic device according to an embodiment of the present invention.

[0071] Among them, 60. Computer device; 61. Processor; 62. Memory; 63. Computer program; 600. Electronic device; 610. Processing unit; 620. Storage unit; 6201. Random access memory unit; 6202. Cache memory unit; 6203. Read-only memory unit; 6204. Program / Utility; 6205. Program module; 630. Bus; 640. Display unit; 650. Input / output interface; 660. Network adapter; 700. External device. DETAILED DESCRIPTION

[0072] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0073] In the description of the present invention, it is to be understood that the terms “include” and “comprise” indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0074] It should also be understood that the terms used in the present specification are only for the purpose of describing particular embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0075] It should be further understood that the term "and / or" as used in the present specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0076] It should be understood that although the terms "first," "second," and "third" may be used to describe preset ranges in embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are merely used to distinguish one preset range from another. For example, without departing from the scope of embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0077] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0078] The accompanying drawings illustrate various schematic diagrams of structures according to embodiments disclosed herein. These figures are not drawn to scale; for clarity, some details are exaggerated and some details may be omitted. The shapes of the various regions and layers shown in the figures, as well as their relative sizes and positional relationships, are merely exemplary and may deviate in practice due to manufacturing tolerances or technical limitations. Those skilled in the art may design regions / layers with different shapes, sizes, and relative positions as needed.

[0079] The present invention provides a weakly supervised temporal action localization method based on a self-supervised pseudo-label optimization network. All training video clip features are globally partitioned through K-means clustering to obtain atomic action features that can represent action categories. The segment-level features output by the feature extractor are then mapped to a query vector, and the atomic action features are introduced as key values. A multi-head self-attention mechanism is used to capture the spatiotemporal dependencies between segments, thereby obtaining an enhanced segment representation containing global context information. Considering the volatility and uncertainty of pseudo-labels generated by direct clustering, the pseudo-label results obtained from two consecutive training cycles are collected, their differences are calculated, and dynamic correction is performed to reduce the negative impact of single-period noise on subsequent training. Furthermore, a feature affinity search strategy is used to expand and optimize pseudo-labels in the temporal dimension based on adjacent moments. High-confidence action regions are automatically expanded to ensure temporal consistency by calculating the cosine similarity of segment features and comparing them with a threshold. A video-level cross entropy loss and a pseudo-label-assisted segment-level classification loss are integrated to construct a joint training objective. This allows the network to continuously update pseudo-labels and optimize feature representations through self-supervision, while relying solely on video-level annotations, ultimately achieving precise localization of action temporal boundaries. The present invention can complete weakly supervised temporal action detection without frame-level annotation, significantly reducing the annotation cost, and effectively reduces background interference through clustering guidance and self-attention enhancement, thereby improving the integrity and accuracy of action localization.

[0080] Example 1

[0081] See also Figure 1 The present invention provides a weakly supervised temporal action localization method based on a self-supervised pseudo-label optimization network, comprising the following steps:

[0082] See also Figure 2 , the atomic action generation method is:

[0083] S1. Collect segment-level feature information from all training videos and perform K-means clustering on the segment feature pool to divide all segments into several category clusters to obtain atomic action features that represent global semantics.

[0084] S101, collect the features of all clips in all training data, perform K-means clustering on the entire feature pool to obtain C clusters, where C is the action category in the video; in this way, each video clip v i is assigned a label p i , each video segment v i is assigned a label p i , p i ∈{1,...,C};

[0085] S102. Use the labels to train a visual semantic concept classifier F. The visual semantic concept classifier F maps the shared features xi to the predicted clustering scores

[0086] S103, use label p i Extract visual semantic concepts from video features;

[0087] In order to obtain the representation v of the nth visual semantic concept n , summarize the features of the segments whose pseudo labels are equal to n; the nth visual semantic concept v n as follows:

[0088]

[0089] S104, Multiple Instance Learning Classifier (MIL) and visual semantic concept v = {v1, ..., v N}Connect them and learn the combination relationship between action atoms;

[0090] The MIL classifier maps the segment features to the action prediction score s, and the feature distance between the visual semantic concept v and the action prediction score s in the MIL classifier is calculated as follows:

[0091]

[0092] Where D(·) is the feature distance calculation function.

[0093] S105. For each action j in the video, calculate s j The distance D between all visual semantic concepts j Then select the top k visual semantic concepts with the smallest distance Denoted as the set TopK(D j ); then these k visual semantic concepts are regarded as atomic actions of action j.

[0094] Therefore, the representation m of the atomic action j is generated by using the representations of these selected visual semantic concepts through the composition function C j .

[0095] The representation m of the atomic action j is generated by combining the representations of these selected visual semantic concepts through the composition function C j as follows:

[0096] m j =C({v n |n∈TopK(D j )}).

[0097] S2. The atomic action features obtained in step S1 are input into the self-attention module as action representations. The video segment-level features extracted by the feature extractor are mapped into query vectors, and the query vectors are jointly used in the attention calculation with the atomic action features to generate an enhanced segment feature representation with contextual dependencies.

[0098] See also Figure 3 , under the complete perspective set, the self-supervised learning method is:

[0099] S201, use linear layer f q Project video features onto the corresponding query;

[0100] The video features are projected onto the corresponding query Q as follows:

[0101] Q=f q (X)

[0102] S202, connect the video feature X with the action atom feature m stored in the action atom memory library j ;

[0103] S203, using a separate linear layer f k and f v , project the connected features into key and value respectively;

[0104] The feature projection key and value of connecting the video features with the action atom features stored in the action atom memory bank are as follows:

[0105] K=f k (concat(X,m1,...,m c ))

[0106] V=f v (concat(X,m1,...,m c )).

[0107] S204, multiply the query by the adjusted key to get the global attention

[0108] The global attention attn is as follows:

[0109]

[0110] Where d is the vector dimension.

[0111] S205: Use the residual layer composed of ReLU and Dropout and the WeightNorm weight normalization layer to normalize and retain the original information.

[0112] S3. After step S2 is completed, the pseudo-labels obtained by directly clustering the video features are usually fluctuating and inaccurate. Further, by collecting the pseudo-label results generated in two consecutive training periods, a pseudo-label correction operation is performed based on their differences, and an affinity search strategy is adopted to perform a forward and backward range search in the time dimension with adjacent frames as units. The feature similarity is used to determine whether to expand the label range, so as to improve the accuracy and temporal consistency of the pseudo-label.

[0113] See also Figure 4 , the pseudo-label optimization and search method is:

[0114] S301, false label correction;

[0115] The difference between the two pseudo labels collected during the two training periods of pseudo label generation is used as the new pseudo label, specifically:

[0116]

[0117] in, is the pseudo label generated for the current training period, Pseudo labels generated for the last training epoch.

[0118] S302, filtering pseudo-label frames using a threshold;

[0119] First, an embedding space is learned to distinguish between actions and backgrounds that are independent of the class. Given the input features, a temporal convolutional layer is used to learn the embedding of each frame, i.e., E = [e 1 ;e 2 ;...;e T ],in It is a D emb dimensional vector; L2 regularization is performed on each embedding vector.

[0120] Two embedding vectors e u and e v The affinity between them is as follows:

[0121]

[0122] Compare the affinity with the action atomic features and use Top-k to select the k highest scores as the foreground pseudo-label frames By Top-k[1-A(e u ,e v )] Also select the k highest scores as the foreground pseudo-label frames

[0123] S303, performing pseudo-label search based on affinity;

[0124] The search range of the foreground single frame pseudo label is between the previous frame and the next background single frame. Find the timestamp A(e u ,e v )>δ as the maximum small frame as the confident foreground frame C a . Find A(e in the search range u ,e v ) The minimum and maximum frames are taken as the confidence background frames C b ;

[0125] Confident foreground frame C a as follows:

[0126]

[0127] In getting C a Finally, the search range of the background marker frame pseudo-label is determined.

[0128] Confidence background frame C b as follows:

[0129]

[0130] Among them, s is the starting boundary and e is the ending boundary.

[0131] S304: After obtaining all the confidence frames, generate a pseudo single frame label according to the interval between the confidence frames.

[0132] Use FocalLoss to calculate the loss of pseudo single-frame labels;

[0133] The loss of foreground pseudo single-frame labels is as follows:

[0134]

[0135] Where I(t) is the indicator function, is the number of foreground labels in the video, θ is the Focal parameter, which is set to 2; it is 1 when there is a foreground label at time t.

[0136] The FocalLoss of the background marker frame is calculated as follows:

[0137]

[0138] Where I(t) is the indicator function, is the number of foreground labels in the video.

[0139] The total single-frame classification loss is the sum of the frame-level classification losses of foreground and background labeled frames L p , specifically:

[0140] L p =Lf +L b .

[0141] S305: After obtaining the foreground pseudo-label segment p act and background pseudo-label fragment p bg Afterwards, optimization is performed via consistency loss.

[0142] The consistency loss is optimized as follows:

[0143] L con =-logp act -log(1-p bg ).

[0144] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods, or program products. Accordingly, various aspects of the present invention may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, microcode, etc.), or in a combination of hardware and software, collectively referred to herein as "circuits," "modules," or "platforms."

[0145] Example 2

[0146] The present invention provides a weakly supervised temporal action localization system of a self-supervised pseudo-label optimization network, which can be used to implement the above-mentioned weakly supervised temporal action localization method of the self-supervised pseudo-label optimization network. Specifically, the weakly supervised temporal action localization system of the self-supervised pseudo-label optimization network includes an atomic action generation module, a self-supervised learning module, and a pseudo-label optimization and search module.

[0147] The atomic action generation module collects segment-level feature information from all training videos and performs K-means clustering to divide all segments into several category clusters to obtain atomic action features that represent global semantics.

[0148] The self-supervised learning module uses atomic action features that represent global semantics as action representations, maps video segment-level features extracted by the feature extractor into query vectors, and jointly participates in attention calculation with the atomic action features to generate enhanced segment representations containing global context information;

[0149] The pseudo-label optimization and search module uses the high-discrimination characteristics of the enhanced segment representation to identify inconsistent areas. By collecting the pseudo-label results generated in two consecutive training periods, it performs pseudo-label correction operations based on the differences and adopts an affinity search strategy to search the front and back range in the time dimension in adjacent frames. It uses feature similarity to determine whether to expand the label range and generates pseudo single-frame labels based on the interval of confidence frames. The positioning result is obtained by optimizing through consistency loss.

[0150] Example 3

[0151] The present invention provides a terminal device, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, graphics processing units (GPU), tensor processing units (TPU), digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the weakly supervised temporal action positioning method of the self-supervised pseudo-label optimization network, including:

[0152] Collect segment-level feature information from all training videos and perform K-means clustering operations to divide all segments into several category clusters to obtain atomic action features that represent global semantics; use the atomic action features that represent global semantics as action representations, map the video segment-level features extracted by the feature extractor into query vectors, and jointly participate in attention calculations with the atomic action features to generate enhanced segment representations containing global context information; use the high-discrimination characteristics of the enhanced segment representations to identify inconsistent areas, collect pseudo-label results generated over two consecutive training periods, perform pseudo-label correction operations based on differences, and adopt an affinity search strategy to perform forward and backward range searches in units of adjacent frames in the time dimension, use feature similarity to determine whether to expand the label range, and generate pseudo single-frame labels based on the interval of confidence frames. The positioning result is obtained by optimizing through consistency loss.

[0153] See also Figure 6The terminal device is a computer device. The computer device 60 of this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the computer program 63 is executed by the processor 61, the weakly supervised temporal action localization method of the self-supervised pseudo-label optimization network in the embodiment is implemented. To avoid repetition, the details are not described here. Alternatively, when the computer program 63 is executed by the processor 61, the functions of each model / unit in the weakly supervised temporal action localization system of the self-supervised pseudo-label optimization network in the embodiment are implemented. To avoid repetition, the details are not described here.

[0154] The computer device 60 may be a desktop computer, a notebook computer, a PDA, a cloud server, or other computing devices. The computer device 60 may include, but is not limited to, a processor 61 and a memory 62. It will be understood by those skilled in the art that Figure 6 This is merely an example of the computer device 60 and does not constitute a limitation of the computer device 60 . The computer device 60 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0155] The processor 61 may be a central processing unit (CPU), or other general-purpose processors, a graphics processing unit (GPU), a tensor processing unit (TPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0156] The memory 62 may be an internal storage unit of the computer device 60, such as a hard disk or memory of the computer device 60. The memory 62 may also be an external storage device of the computer device 60, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 60.

[0157] Furthermore, the memory 62 may include both an internal storage unit of the computer device 60 and an external storage device. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 may also be used to temporarily store data that has been output or is about to be output.

[0158] See also Figure 7 The terminal device is an electronic device 600, which is implemented as a general-purpose computing device. The components of the electronic device may include, but are not limited to, at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), and a display unit 640.

[0159] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0160] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0161] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0162] Bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0163] The electronic device 600 may also communicate with one or more external devices 700 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., a router, a modem). Such communication may occur via an input / output interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network, a wide area network, and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0164] Example 4

[0165] The present invention also provides a storage medium, specifically a computer-readable storage medium, which is a memory device in a terminal device for storing programs and data. It is understood that the computer-readable storage medium herein may include both built-in storage media in the terminal device and, of course, extended storage media supported by the terminal device. It may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for being loaded and executed by a processor. These instructions may be one or more computer programs (including program code). It should be noted that more specific examples of the computer-readable storage medium herein include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0166] Computer-readable storage media also include data signals propagated in baseband or as part of a carrier wave, which carry readable program code. Such propagated data signals can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium can also be any readable medium other than a readable storage medium, which can send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, radio frequency, etc., or any suitable combination of the above.

[0167] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network or a wide area network, or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0168] The processor may load and execute one or more instructions stored in a computer-readable storage medium to implement the corresponding steps of the weakly supervised temporal action localization method of the self-supervised pseudo-label optimization network in the above embodiment; the processor may load and execute the following steps:

[0169] Collect segment-level feature information from all training videos and perform K-means clustering operations to divide all segments into several category clusters to obtain atomic action features that represent global semantics; use the atomic action features that represent global semantics as action representations, map the video segment-level features extracted by the feature extractor into query vectors, and jointly participate in attention calculations with the atomic action features to generate enhanced segment representations containing global context information; use the high-discrimination characteristics of the enhanced segment representations to identify inconsistent areas, collect pseudo-label results generated over two consecutive training periods, perform pseudo-label correction operations based on differences, and adopt an affinity search strategy to perform forward and backward range searches in units of adjacent frames in the time dimension, use feature similarity to determine whether to expand the label range, and generate pseudo single-frame labels based on the interval of confidence frames. The positioning result is obtained by optimizing through consistency loss.

[0170] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0171] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0172] To verify the performance improvement of the weakly supervised temporal action localization method based on the clustering-guided self-supervised pseudo-label optimization network, this paper selected CoLA as the baseline model and conducted experimental analysis on the THUMOS14 and ActivityNet v1.2 datasets. The standard evaluation method for temporal action localization was used to evaluate the performance of the method using the mean average precision (mAP) at different temporal intersection-over-union (t-IoU) thresholds on the benchmark dataset. The higher the mAP, the better the WTAL performance. For a fair comparison, the benchmark code provided by ActivityNet was used to calculate the results. The comparison of the obtained positioning results is shown in Figure 5.

[0173] As can be seen from Figure 5(a), the average mAP of the weakly supervised temporal action localization method based on the clustering-guided self-supervised pseudo-label optimization network on the THUMOS14 dataset is 48.7%, which is 7.8% higher than CoLA's 40.9%. This is because the method proposed in this paper first generates action atoms by clustering features and stores them in the action atom library, and then uses self-attention to capture the intra-video and inter-video dependencies of the clips, thereby producing more discriminative and robust clip representations, and finally achieving accurate action localization through boundary refinement. These results confirm the effectiveness of using global learning and pseudo-label optimization in the weakly supervised setting and have achieved significant progress in the localization performance of WTAL.

[0174] On ActivityNet v1.2, the average mean average performance approximation (mAP) reached 28.8%, outperforming CoLA's 26.1%. This achievement is primarily due to the method's global action learning and pseudo-label optimization and search modules, which effectively overcome the discrepancy between localization and classification under weak supervision, achieving precise action localization.

[0175] To more intuitively demonstrate the effectiveness of the proposed method in action localization, Figure 5(b) provides a detailed qualitative visualization analysis of the detected action segments. We selected qualitative results for two action categories on THUMOS14: CleanAndJerk (left) and Diving (right). The results for Ground Truth (GT), CoLA (Baseline), and C-SPONet (Ours) are marked in green, blue, and orange, respectively. The baseline model's prediction error is indicated by a red dashed box.

[0176] Comparative visualization analysis clearly demonstrates the significant improvement in positioning performance achieved by the proposed model. The proposed model demonstrates more accurate action instance detection. Specifically, for the CleanAndJerk action, the proposed method effectively distinguishes between action and background segments, reducing CoLA's erroneous action predictions. For the Diving action, CoLA has lower activation values ​​for identifying ambiguous action segments, failing to detect and locate them. However, the proposed method detects more complete segments than CoLA.

[0177] Experimental results show that on the THUMOS14 and ActivityNet v1.2 datasets, our method improves the average mean average performance (mAP) by 7.8% and 2.7%, respectively, compared to baseline models, fully demonstrating its effectiveness and practicality. Furthermore, our method can serve as a foundational preprocessing module for subsequent temporal action recognition and high-level behavior analysis, providing new insights and technical paths for future research in the fields of few-sample, cross-domain, and multimodal weakly supervised video analysis.

[0178] In summary, the present invention provides a weakly supervised temporal action localization method and system based on a self-supervised pseudo-label optimization network, which can complete temporal action detection without large-scale frame-level annotation, avoiding the high cost of pre-defining precise timestamps and manual annotation, and improving the ability to mine global semantic information of videos by combining atomic action features generated by clustering with a self-attention mechanism. At the same time, in response to the instability of pseudo-labels generated by direct clustering, the present invention effectively reduces the pseudo-label noise and incompleteness problems through continuous training cycle pseudo-label correction and temporal affinity search strategy, thereby enhancing the prediction accuracy of action boundaries. The method also realizes self-supervised training that relies only on video-level labels through a joint loss design of video-level and pseudo-label levels, and significantly improves the positioning accuracy in weakly supervised scenarios.

[0179] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0180] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0181] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0182] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical functional division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of devices or units, and can be electrical, mechanical, or other forms.

[0183] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0184] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0185] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0186] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0187] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0188] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0189] The above content is only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A weakly supervised temporal action localization method based on a self-supervised pseudo-label optimization network, characterized by: The following steps are involved: Collect segment-level feature information from all training videos and perform K-means clustering to divide all segments into several category clusters to obtain atomic action features that represent global semantics; The atomic action features representing global semantics are used as action representations. The video segment-level features extracted by the feature extractor are mapped into query vectors, which are then used together with the atomic action features in attention calculations to generate enhanced segment representations containing global context information. The high discrimination characteristics of enhanced segment representation are used to identify inconsistent areas. By collecting pseudo-label results generated in two consecutive training periods, pseudo-label correction operations are performed based on the differences. An affinity search strategy is used to search the front and back ranges in adjacent frames in the time dimension. Feature similarity is used to determine whether to expand the label range. Pseudo single-frame labels are generated based on the interval of confidence frames. The positioning result is obtained by optimizing the consistency loss.

2. The weakly supervised temporal action localization method of the self-supervised pseudo-label optimization network according to claim 1 is characterized in that: All segments are divided into several category clusters, and the atomic action features representing global semantics are obtained as follows: Collect the features of all clips in all training data, perform K-means clustering on the entire feature pool to obtain C clusters, where C is the action category in the video; each video clip v i is assigned a label p i , each video segment v i is assigned a label p i , p i ∈{1,...,C}; Use the labels to train a visual semantic concept classifier F; the shared features xi of the visual semantic concept classifier F are mapped to the predicted cluster scores Use the tag p i Extract visual semantic concepts from video features; Combine the MIL classifier with the visual semantic concept v={v1,...,v N }Connect them and learn the combination relationship between action atoms; For each action j in the video, calculate s j The distance D between all visual semantic concepts j Then select the top k visual semantic concepts with the smallest distance Denoted as the set TopK(D j ); Finally, the k visual semantic concepts are regarded as the atomic actions of action j.

3. The weakly supervised temporal action localization method of the self-supervised pseudo-label optimization network according to claim 2 is characterized in that: The nth visual semantic concept v n as follows: Where T is the time dimension of the video clip.

4. The weakly supervised temporal action localization method of the self-supervised pseudo-label optimization network according to claim 2 is characterized in that: The representation m of the atomic action j is generated by combining the representations of these selected visual semantic concepts through the composition function C j as follows: m j =C({v n |n∈TopK(D j )})。 5. The weakly supervised temporal action localization method of the self-supervised pseudo-label optimization network according to claim 1 is characterized in that: The enhanced fragment representation containing global context information is generated as follows: Use the linear layer f q Project video features onto the corresponding query; Connect the video feature X with the action atom feature m stored in the action atom memory bank j ; Using a single linear layer f k and f v , project the connected features into K and V respectively; Multiply the query with the adjusted key to get the global attention The residual layer composed of ReLU and Dropout and the WeightNorm weight normalization layer are used to normalize and retain the original information.

6. The weakly supervised temporal action localization method of the self-supervised pseudo-label optimization network according to claim 5 is characterized in that: The global attention attn is as follows: K=f k (concat(X,m1,...,m c )) V=f v (concat(X,m1,...,m c )) Among them, d is the vector dimension, Q is the query vector, m c is the c-th action atomic feature.

7. The weakly supervised temporal action localization method of the self-supervised pseudo-label optimization network according to claim 1 is characterized in that: Based on the difference, the pseudo-label correction operation is performed, and the affinity search strategy is adopted. The feature similarity is used to determine whether to expand the label range. The consistency loss is used for optimization. The positioning results are as follows: The difference between the two pseudo labels collected during the two training periods of pseudo label generation is used as the new pseudo label; Learn an embedding space to distinguish class-independent actions and backgrounds; given the input features, use a temporal convolutional layer to learn the embedding of each frame, i.e. E = [e 1 ;e 2 ;...;e T ], e T It is a D emb dimensional vector; L2 regularization is performed on each embedding vector; The search range of the foreground single frame pseudo label is between the previous frame and the next background single frame; find the timestamp A(e u ,e v )>δ as the maximum small frame as the confident foreground frame C a ; Find A(e in the search range u ,e v ) The minimum and maximum frames are taken as the confidence background frames C b ; After obtaining all the confidence frames, generate pseudo single frame labels based on the interval of the confidence frames After obtaining the foreground pseudo-label fragment p act and background pseudo-label fragment p bg Afterwards, the consistency loss is used for optimization to obtain the positioning result.

8. The weakly supervised temporal action localization method of the self-supervised pseudo-label optimization network according to claim 7 is characterized in that: Confident foreground frame C a and the confidence background frame C b as follows: Among them, s is the starting boundary, e is the ending boundary, is the starting boundary of the background frame, The ending boundary of the background frame.

9. The weakly supervised temporal action localization method of the self-supervised pseudo-label optimization network according to claim 7 is characterized in that: Consistency loss L con for: L con =-logp act -log(1-p bg )。 10. A weakly supervised temporal action localization system based on a self-supervised pseudo-label optimization network, characterized in that: include: The atomic action generation module collects segment-level feature information from all training videos and performs K-means clustering to divide all segments into several category clusters to obtain atomic action features that represent global semantics. The self-supervised learning module uses atomic action features that represent global semantics as action representations, maps video segment-level features extracted by the feature extractor into query vectors, and jointly participates in attention calculation with the atomic action features to generate enhanced segment representations containing global context information; The pseudo-label optimization and search module uses the high-discrimination characteristics of the enhanced segment representation to identify inconsistent areas. By collecting the pseudo-label results generated in two consecutive training periods, it performs pseudo-label correction operations based on the differences and adopts an affinity search strategy to search the front and back range in the time dimension in adjacent frames. It uses feature similarity to determine whether to expand the label range and generates pseudo single-frame labels based on the interval of confidence frames. The positioning result is obtained by optimizing the consistency loss.

Images