Weak supervision time sequence action positioning method and device based on pseudo label generation

Abstract

The invention provides a weak supervision time sequence action positioning method and device based on pseudo label generation, and relates to the technical field of time sequence action positioning. The method comprises the following steps: performing clip division on an unclipped video to obtain a video clip set; performing feature extraction through a feature extractor to obtain a fragment-by-fragment feature set; performing preliminary action classification through a weak supervision branch to obtain a classification attention sequence and a multi-scale action proposal set; performing proposal fusion optimization by using a proposal fusion strategy to obtain a pseudo tag proposal set; a set of uncertainty masks is generated from the set of action proposals. Based on an attention mechanism, performing optimization training on the fully supervised branch through a dynamic optimization mechanism to obtain a second optimized fully supervised branch; and performing action positioning based on the feature extractor, the weak supervision branch and the second optimization full supervision branch. According to the weak supervision time sequence action positioning method, high-quality pseudo labels are generated, prior information is fully utilized, and a full supervision method is effectively combined.

Description

Technical Field

[0001] The present invention relates to the technical field of time sequence action positioning, and in particular to a method and device for weakly supervised time sequence action positioning based on pseudo label generation. Background Art

[0002] Temporal Action Localization (TAL) is an important task in the field of video analysis, which aims to identify all actions in a video and accurately locate their start and end times. With the rapid development of deep learning technology, especially the application of convolutional neural networks and long short-term memory networks, the field of temporal action localization has made significant progress. In traditional TAL methods based on manual features, action recognition and localization rely on manually designed features. These methods usually require a lot of manual intervention and cannot cope with complex video scenes.

[0003] In recent years, TAL methods based on deep learning have gradually become mainstream. Current TAL methods can generally be divided into two-stage methods and single-stage methods. The two-stage method locates action boundaries by generating candidate action proposals and performing fine-tuning; for example, works based on proposal generation, such as Boundary-Sensitive Network (BSN) and Boundary-Matching Network (BMN). The single-stage method directly classifies and locates actions from video sequences, simplifying the design and training of the model. For example, ActionFormer, a Transformer-based temporal action localization model, and TriDet, a single-stage framework for temporal action detection.

[0004] Temporal action localization is a complex and challenging visual task that aims to accurately classify and locate actions in untrimmed videos. However, temporal action localization has always faced the problem of inaccurate positioning, which makes it difficult to accurately predict the start and end time of an action. Although existing fully supervised and weakly supervised methods have improved to a certain extent, they still face problems such as reliance on annotation information and noisy pseudo-labels. These problems not only affect the generalization ability and accuracy of the model, but also limit the application of the method on large-scale video datasets.

[0005] Fully supervised temporal action localization methods rely on complete annotated data, including the category, start time, and end time of each action. This method can usually provide more accurate localization results because it makes full use of accurate annotation information. However, the high dependence of fully supervised methods on annotated data makes large-scale application difficult. The collection of annotated data is not only time-consuming and costly, but also almost impossible to achieve, especially when a large amount of video data is required. The generalization ability of fully supervised methods is weak. When the annotated data is limited, the training results of the model are difficult to generalize to unseen data. Therefore, the lack of generalization ability of fully supervised methods has become its main limiting factor.

[0006] Unlike the fully supervised method, the weakly supervised temporal action localization method is trained by using relatively simple global labels or a small number of segment labels. Although this method significantly reduces the annotation cost, it still faces great challenges in accuracy and stability. Due to the lack of precise segment-level labels, the model can only use global information for learning during training. This makes the positioning accuracy of the weakly supervised method relatively low in complex scenes, especially when the boundaries between actions are blurred and the actions overlap, and it cannot be compared with the fully supervised method.

[0007] The temporal action localization method based on pseudo-label generation is a bridge between fully supervised temporal action localization and weakly supervised temporal action localization. When samples are not labeled, generating pseudo-labels through confidence score thresholds is a common pseudo-label generation method, which is also used in traditional weakly supervised methods. However, this direct strategy may not accurately capture the complex temporal information in the video, resulting in low quality of generated proposals. In addition, complex proposal fusion methods are highly dependent on hyperparameters. If the hyperparameters are not set properly, it may lead to over-segmentation or omission of important proposals, affecting the final positioning accuracy. This reliance on hyperparameter adjustment increases the complexity of the model and limits its adaptability in different datasets and scenarios.

[0008] The quality of pseudo-label proposals is crucial to the final performance of the model. Existing methods have failed to effectively deal with the problem of pseudo-label noise. They directly use pseudo-labels when training models with pseudo-labels, without effectively processing low-quality boundary areas, which often contain more noise. This leads to a negative impact of pseudo-label noise on the training process, especially in the early stages of training. The early prediction results of the model may contain a lot of noise, which directly affects the stability of the model. Due to the lack of dynamic adjustment and screening mechanisms for noisy pseudo-labels, these methods are prone to learning incorrect boundary information during training, resulting in unsatisfactory model training results.

[0009] In the existing technology, there is a lack of a weakly supervised temporal action localization method for high-quality pseudo-label generation that fully utilizes prior information and effectively combines the full supervision method. Summary of the invention

[0010] In order to solve the technical problems of insufficient utilization of prior information, low quality of pseudo-label proposals and difficulty in combining with the full supervision framework in the prior art, the embodiment of the present invention provides a weakly supervised temporal action localization method and device based on pseudo-label generation. The technical solution is as follows:

[0011] On the one hand, a weakly supervised temporal action localization method based on pseudo-label generation is provided, the method is implemented by a weakly supervised temporal action localization device, and the method includes:

[0012] Obtaining an uncut video containing an action; dividing the uncut video into segments to obtain a set of video segments;

[0013] According to the video segment set, feature extraction is performed by a feature extractor to obtain a segment-by-segment feature set;

[0014] According to the segment-by-segment feature set, preliminary action classification is performed through a weak supervision branch to obtain a classified attention sequence and a multi-scale action proposal set;

[0015] According to the classified attention sequence and the action proposal set, a proposal fusion strategy is used to perform proposal fusion optimization to obtain a pseudo-label proposal set;

[0016] Based on a preset expansion ratio and a preset contraction ratio, generating an uncertainty mask set for a time boundary of each proposal in the action proposal set;

[0017] Based on the attention mechanism, according to the pseudo-label proposal set and the uncertainty mask set, the full supervision branch is optimized and trained through a dynamic optimization mechanism to obtain a second optimized full supervision branch;

[0018] Acquire a video of the action to be located; and perform action location according to the video of the action to be located based on the feature extractor, the weak supervision branch, and the second optimized full supervision branch.

[0019] On the other hand, a weakly supervised temporal action localization device based on pseudo-label generation is provided, and the device is applied to a weakly supervised temporal action localization method based on pseudo-label generation, and the device comprises:

[0020] The video segment acquisition module is used to acquire an uncut video containing actions; divide the uncut video into segments to obtain a video segment set;

[0021] A video feature extraction module, used to extract features through a feature extractor according to the video segment set to obtain a segment-by-segment feature set;

[0022] A weakly supervised branch classification module, configured to perform preliminary action classification through a weakly supervised branch according to the segment-by-segment feature set, and obtain a classified attention sequence and a multi-scale action proposal set;

[0023] A pseudo-label proposal generation module is used to perform proposal fusion optimization using a proposal fusion strategy according to the classification attention sequence and the action proposal set to obtain a pseudo-label proposal set;

[0024] An uncertainty mask generation module, configured to generate an uncertainty mask set at a time boundary of each proposal in the action proposal set based on a preset expansion ratio and a preset contraction ratio;

[0025] A fully supervised branch optimization module, used to optimize and train the fully supervised branch based on the attention mechanism, according to the pseudo-label proposal set and the uncertainty mask set, through a dynamic optimization mechanism, to obtain a second optimized fully supervised branch;

[0026] The video action localization module is used to obtain a video of the action to be located; based on the feature extractor, the weak supervision branch and the second optimized full supervision branch, the action is located according to the video of the action to be located.

[0027] On the other hand, a weakly supervised temporal action localization device is provided, which comprises: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned weakly supervised temporal action localization methods based on pseudo-label generation is implemented.

[0028] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned weakly supervised temporal action localization methods based on pseudo-label generation.

[0029] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0030] The present invention proposes a weakly supervised temporal action localization method based on pseudo-label generation. Through a dual-branch framework, the gap between weakly supervised and fully supervised temporal action localization in terms of framework and performance is bridged; by using the information in the weakly supervised branch and the fully supervised branch at the same time, the dual-branch framework can achieve more accurate temporal action localization under the condition of less labeled data; a pseudo-label proposal fusion strategy is adopted to effectively use the prior information in the pseudo-label to generate pseudo-label proposals with high confidence and accurate boundaries; this strategy significantly improves the quality of the proposal and the accuracy of the model through the fusion of multi-scale information; an uncertainty mask is introduced, and a new optimization mechanism is proposed for effective learning in the presence of noisy labels. The mechanism effectively reduces the interference of noisy labels during the training process by dynamically adjusting the quality of pseudo-labels, enhances the robustness of the regression model, and ensures that the model can converge stably during the training process. The present invention is a weakly supervised temporal action localization method for generating high-quality pseudo-labels that fully utilizes prior information and effectively combines the full supervision method. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 It is a flow chart of a weakly supervised temporal action localization method based on pseudo-label generation provided by an embodiment of the present invention;

[0033] Figure 2 It is a block diagram of a weakly supervised sequential action positioning device based on pseudo-label generation provided by an embodiment of the present invention;

[0034] Figure 3 It is a structural diagram of a weakly supervised sequential action positioning device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0036] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0037] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0038] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0039] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0040] The embodiment of the present invention provides a method for positioning a weakly supervised sequential action based on pseudo-label generation. The method can be implemented by a weakly supervised sequential action positioning device, which can be a terminal or a server. Figure 1 The flowchart of the weakly supervised temporal action localization method based on pseudo-label generation is shown in FIG. 1 . The processing flow of the method may include the following steps:

[0041] S1. Obtain an uncut video containing an action; divide the uncut video into segments to obtain a set of video segments.

[0042] In a feasible implementation, the present invention performs segmentation on the input uncut video, the main purpose of which is to reduce the amount of data that the model needs to process on the basis of retaining local temporal information. The video segments obtained in this step are regarded as the smallest processing unit that cannot be divided in the subsequent steps.

[0043] S2. Perform feature extraction through a feature extractor based on the video segment set to obtain a segment-by-segment feature set.

[0044] In a feasible implementation, in the feature extraction module, the present invention uses a deep learning model (Inflated 3D ConvNet, I3D) model for video action recognition pre-trained on the Kinetics dataset to extract RGB and optical flow features. The extracted feature dimension is 1024, and a segment feature is extracted every 16 frames.

[0045] S3. Based on the segment-by-segment feature set, preliminary action classification is performed through the weak supervision branch to obtain the classified attention sequence and multi-scale action proposal set.

[0046] Optionally, preliminary action classification is performed through a weak supervision branch based on the segment-by-segment feature set to obtain a classified attention sequence and a multi-scale action proposal set, including:

[0047] Based on the attention mechanism, adaptive action classification is performed through the basic model according to the feature set of each segment, and the attention score sequence and classification score matrix are obtained;

[0048] Perform weighted fusion based on the attention score sequence and the classification score matrix to obtain the classification attention sequence;

[0049] Based on the preset score threshold, the classified attention sequence is thresholded to obtain a multi-scale action proposal set; the action proposal set includes a time boundary set and an action category set.

[0050] In a feasible implementation, the present invention uses a baseline model based on multi-instance learning training as a basic module of a weakly supervised learning framework, and uses the basic module to obtain the features and action recognition results of each video segment.

[0051] Among them, the basic model is a baseline model based on the multi-instance learning training method.

[0052] In a feasible implementation, the weak supervision branch in the present invention adopts a weakly supervised temporal action localization framework (Dual-Evidential Learning for Uncertainty modeling, DELU). This framework gradually focuses on the entire action instance by introducing video-level and segment-level uncertainties, thereby solving the problems of background noise and action background blur.

[0053] S4. Based on the classified attention sequence and the action proposal set, the proposal fusion strategy is used to perform proposal fusion optimization to obtain a pseudo-label proposal set.

[0054] Optionally, based on the classified attention sequence and the action proposal set, a proposal fusion strategy is used to perform proposal fusion optimization to obtain a pseudo-label proposal set, including:

[0055] Based on the action proposal set, the classification attention sequence is calculated to obtain the mean internal attention score and the mean external attention score of the proposal;

[0056] Based on the mean of the proposal's internal attention score and the mean of the proposal's external attention score, the confidence is calculated according to the classified attention sequence to obtain a set of proposal confidence scores;

[0057] According to the action proposal set and the proposal confidence score set, the proposal fusion strategy is used to perform fusion optimization to obtain a pseudo-label proposal set; the proposal fusion strategy refers to an optimization strategy that fuses the action proposals mapped to the global shared space with the confidence scores corresponding to the action proposals; the proposal fusion strategy is used to filter the classification misdetection results of action proposals.

[0058] In a feasible implementation manner, the design focus of the present invention is on the generation of time-series pseudo-labels. The key to this step is how to use the constraint information existing in the video and introduce artificial prior knowledge to improve the quality and reliability of the pseudo-labels.

[0059] The output of the basic model is used as the input of the temporal pseudo-label construction module. The constraint information in the video time sequence and the artificially introduced prior information are utilized to optimize the pseudo-supervision information through the proposal fusion strategy, and the information of different scales is fused into a global shared space to reduce noise and improve the quality of pseudo-labels.

[0060] This method generates accurate and reliable pseudo labels by thresholding the proposals. The pseudo supervision information obtained has a form similar to the real temporal supervision label, that is, it can provide a confidence score similar to the intersection-over-union ratio for any action proposal. By maximizing the regression training of this score, the positioning task can be introduced into the model training, reducing the gap between the training task and the test task of the weakly supervised video action understanding model.

[0061] The proposals output by the base model of the weakly supervised branch vary in scale, and a higher threshold produces perceptions of smaller scales. Proposals are generated individually, ignoring the overall relationship between different proposals and scales. In addition, there is overlap between action proposals, which makes it impossible for action proposals to accurately describe boundaries, which in turn affects the training of the fully supervised branch.

[0062] By mapping all proposals to a unified space, global perception is obtained, spatial understanding is enhanced, and representation is simplified. The present invention uses Ricker wavelets, which emphasize the central area of ​​each proposal while suppressing the peripheral area, effectively fusing proposals of different perceptual scales. This method enhances the effect of temporal action localization by achieving balanced and robust multi-scale proposal alignment.

[0063] For each action proposal, it is mapped to the Ricker distribution space as follows (1):

[0064] (1);

[0065] in, Indicates a point in time Ricker distribution value. Action Proposal , Indicates action proposal The start time, Indicates action proposal The end time of Indicates action proposal Length, Indicates action proposal The midpoint of .

[0066] Ricker wavelet can naturally focus on the center of the proposal while suppressing the boundary area. Its suppression effect gradually increases in areas farther from the center and then gradually weakens, which helps to reduce noise and reduce uncertainty in the boundary area. This feature is highly consistent with the positioning results because it ensures that each proposal can capture the core action with a high degree of confidence, while the influence of the boundary part is effectively suppressed.

[0067] All proposals are integrated into a shared global space. The process is as follows:

[0068] (2);

[0069] in, Indicates the category The final fused wavelet distribution of the proposals is used to integrate the global representation of all proposals. represents the number of all proposals, Indicates The categories of proposals, Represents a specified action category; all action proposals are associated with corresponding confidence scores By fusing them together, misclassified predictions are filtered out, and the final pseudo labels are obtained, so that the generated pseudo proposals will have better quality and more reliable boundaries.

[0070] Compared with other pseudo-label proposal fusion methods, existing pseudo-label proposal fusion methods are highly dependent on hyperparameters. If the hyperparameters are not set properly, it may lead to over-segmentation or omission of important proposals, affecting the final positioning accuracy. This reliance on hyperparameter adjustment increases the complexity of the method and limits its adaptability in different datasets and scenarios.

[0071] In order to improve the quality of pseudo-label proposals, this paper proposes an efficient proposal fusion method, which reduces noise and enhances the correlation between proposals by fusing proposals of different scales into a unified global space. The confidence of action proposals is evaluated using the external-internal contrast score of the classified attention sequence, and it is mapped to the Ricker wavelet distribution, fusing multi-scale information to form a unified representation. And high-quality proposals are generated through threshold processing. Compared with traditional methods, the method proposed in this paper can effectively integrate predictions from different scales, reduce the impact of noise, and significantly improve the accuracy and positioning accuracy of pseudo-label proposals.

[0072] S5. Based on a preset expansion ratio and a preset contraction ratio, generate an uncertainty mask set at the time boundary of each proposal in the action proposal set.

[0073] In a feasible implementation, in order to improve the training process based on noisy pseudo-labels, the present invention uses a priori assumption based on the accuracy of the proposal: the central area of ​​the proposal is usually more reliable, while the boundary part has more uncertainty. Based on this assumption, an uncertainty mask is introduced, and the mathematical expression of the uncertainty mask is as follows (3):

[0074] (3);

[0075] in, Represented as action proposals The uncertainty mask of Expressed as a proposal The interval of the uncertain time region contains two sub-intervals and , Expressed as a proposal The uncertainty area around the start time of Expressed as a proposal The uncertainty area around the end time of and are respectively a preset expansion ratio and a preset contraction ratio. Proposals , Express proposal The start time, Express proposal The region around the boundary of each proposal is identified as the uncertainty region, which may contain more noise than other regions in the action proposal.

[0076] All masks are combined, and the process is as follows (4):

[0077] (4);

[0078] in, represents the global uncertainty mask, represents the number of anchor boxes connecting all layers, Represents the number of proposals at all levels. The uncertainty masks are generated at multiple scales after assigning proposals to different feature pyramid levels.

[0079] During subsequent training, the boundaries of each proposal are partially masked to prevent them from interfering with model training. The uncertainty mask is designed to adjust over time, and as the model's confidence in the proposal boundaries increases, the uncertainty area is gradually reduced through a simple but effective linear function. As training progresses, the accuracy of the proposal boundaries improves and the mask gradually fades, allowing the entire proposal to participate in training, ultimately reducing the impact of noisy labels and improving the overall performance of the model.

[0080] S6. Based on the attention mechanism, according to the pseudo-label proposal set and the uncertainty mask set, the full supervision branch is optimized and trained through the dynamic optimization mechanism to obtain the second optimized full supervision branch.

[0081] Optionally, based on the attention mechanism, according to the pseudo-label proposal set and the uncertainty mask set, the full supervision branch is optimized and trained through a dynamic optimization mechanism to obtain a second optimized full supervision branch, including:

[0082] Based on the morphological preprocessing method, the pseudo-label proposal set is filtered and optimized to obtain the optimized action proposal set;

[0083] The optimized action proposal set is input into the encoder for data processing to obtain the encoded pseudo-label proposal set;

[0084] According to the encoded pseudo-label proposal set, multi-scale feature extraction is performed through the feature pyramid network to obtain the pseudo-label proposal feature map;

[0085] Based on the uncertainty mask set, the pseudo-label proposal feature map is masked to obtain the first unmasked proposal feature map;

[0086] Based on the preset first training round, according to the first unoccluded proposal feature map, the first stage of training is performed through the fully supervised branch to obtain the first predicted action proposal set;

[0087] Based on the attention mechanism, the first predicted action proposal set and the uncertainty mask set are dynamically optimized according to the pseudo-label proposal set to obtain the second predicted action proposal set and update the uncertainty mask set;

[0088] The loss function is calculated based on the second predicted action proposal set and the pseudo-label proposal set to obtain the first action loss;

[0089] According to the first action loss, the parameters of the fully supervised branch are optimized to obtain the first optimized fully supervised branch;

[0090] Based on the updated uncertain mask set, the pseudo-label proposal feature map is masked to obtain a second unmasked proposal feature map;

[0091] Based on the preset second training round, according to the second unoccluded proposal feature map, the second stage training is performed through the first optimized full supervision branch to obtain a third predicted action proposal set;

[0092] The loss function is calculated based on the third predicted action proposal set and the pseudo-label proposal set to obtain the second action loss;

[0093] According to the second action loss, the parameters of the first optimized fully supervised branch are optimized to obtain the second optimized fully supervised branch.

[0094] In a feasible implementation, the pseudo-label proposals are simply preprocessed by morphological preprocessing, which relies on the basic assumption that the video action has a certain time continuity and will not start and end repeatedly in a short time.

[0095] The performance of the model is improved by fusing multiple prior information such as classification attention sequence and pseudo-label proposal. After sharing the extracted segment features with the weak supervision branch, the features are first encoded using a transformer encoder.

[0096] The feature pyramid network is used to process the encoded features at multiple scales to generate anchor boxes. All pseudo-label proposals are assigned to different scales according to the duration of each proposal. The category of each anchor box is predicted by the classification head. For each anchor box, the start time and end time of the proposal are decoded by the regression head.

[0097] If these pseudo-label proposals containing noise are directly used for training, it may mislead the model. Therefore, a denoising mechanism needs to be introduced during the training process to reduce the impact of noisy pseudo-labels. The present invention realizes dynamic denoising of pseudo-labels by designing and introducing an uncertainty mask module. During the fully supervised branch training, the mask module will be dynamically adjusted according to the confidence of the boundary of the pseudo-label proposal to eliminate the influence of those low-confidence areas. When the model calculates the loss, the loss in low-confidence areas will be ignored, and the loss in high-confidence areas will be focused on, thereby effectively avoiding the interference of noise information in pseudo-labels on model training.

[0098] Compared with a single fully-supervised or weakly-supervised framework, in order to effectively learn multiple prior knowledge in pseudo-labels and bridge the framework gap between weakly-supervised and fully-supervised temporal action localization, we simultaneously learn from proposal-level and segment-level information.

[0099] In the fully supervised branch, a regression model similar to the mainstream fully supervised temporal action localization architecture is introduced to utilize the proposal-level pseudo-labels generated in the weakly supervised branch. It is able to learn the boundaries and duration of actions like the fully supervised temporal action localization method. In addition, by learning from the classification attention sequence in the weakly supervised branch, its encoding features are enriched with segment-level classification information. Each pseudo-label contains natural priors introduced by the multi-instance learning process, as well as artificial priors added by design, which improves the overall performance of the proposed method.

[0100] The focus of this step is the design of a denoising mechanism based on uncertainty masks. The parameters in the mask can dynamically adjust the model's attention to pseudo-labels. In the first stage of training, learning mainly relies on the high-confidence part of the pseudo-labels, excluding less reliable label information. In the second stage of training, the model's confidence in the pseudo-labels gradually increases, and the scope of application of the mask gradually decreases, ultimately achieving effective use of the entire pseudo-label. Through this mechanism, the model can adaptively identify and exclude noise in pseudo-labels, improving the robustness and accuracy of the training process.

[0101] Compared with traditional noisy pseudo-label processing methods, traditional methods lack an adaptive noise label training mechanism and are often unable to dynamically identify and exclude unreliable pseudo-label boundary areas, resulting in a greater impact of noise on model performance.

[0102] In addition to the uncertainty mask and dynamic optimization mechanism, the present invention introduces an attention mechanism to achieve dynamic evaluation of pseudo-labels during training. When calculating the model training loss, the loss caused by pseudo-labels with attention below the threshold is ignored, and the loss caused by pseudo-labels with attention above the threshold is focused on. In this way, during the training process, the misleading of model training by erroneous information in pseudo-supervision can be dynamically avoided.

[0103] Optionally, based on the attention mechanism, according to the pseudo-label proposal set, the first predicted action proposal set and the uncertainty mask set are dynamically optimized to obtain the second predicted action proposal set and update the uncertainty mask set, including:

[0104] Based on the attention mechanism, the first action proposal attention sequence is obtained by calculating the pseudo-label proposal set and the first predicted action proposal set;

[0105] Based on the preset attention threshold, the first action proposal attention sequence is screened to obtain the second action proposal attention sequence;

[0106] Based on the second action proposal attention sequence, select the corresponding action set in the first predicted action proposal set to obtain the second predicted action proposal set;

[0107] Based on the preset linear function, according to the attention sequence of the second action proposal, the mask weights of the uncertainty mask set are dynamically linearly attenuated to obtain an updated uncertainty mask set; the range of the mask weights is 1 to 0.

[0108] In one possible implementation, during the training phase of the fully supervised branch, the pseudo-label proposals and uncertainty masks are optimized after the first phase of training. The pseudo-label proposals for the first phase of training are generated by the base model. After the first phase of training, the proposals generated by the regression model are considered to have more reliable boundaries and confidence scores. In each training iteration, the proposals output by the regression model in the fully supervised branch are combined and updated with the initial pseudo-label proposals. The updated proposal set is used to optimize the uncertainty mask, thereby helping the model to more accurately locate the temporal boundaries.

[0109] After the training enters the second stage of training, the parameters in the mask will decay linearly. This change shows that as the training goes on, the reliability and accuracy of the proposal boundaries gradually increase. Therefore, the model will gradually rely less on the uncertainty mask and rely more on the optimized proposals to make the final prediction.

[0110] S7, obtaining a video of the action to be located; based on the feature extractor, the weak supervision branch and the second optimized full supervision branch, performing action location according to the video of the action to be located.

[0111] Optionally, based on the feature extractor, the weak supervision branch and the second optimized full supervision branch, action positioning is performed according to the action video to be positioned, including:

[0112] Based on the feature extractor, the weak supervision branch and the second optimized full supervision branch, the temporal action positioning prediction is performed to obtain a set of predicted action proposals;

[0113] Based on the attention mechanism, according to the predicted action proposal set, the soft non-maximum suppression algorithm is used to filter redundant action proposals to obtain filtered action proposals;

[0114] Action localization is performed based on the action categories and time boundaries of the filtered action proposals.

[0115] In one feasible implementation, after two stages of training, the regression-based student model in the fully supervised branch is used for inference. To obtain video-level classification results, the attention scores predicted by the attention head are used to filter the top k scores of each category and average them. The predicted attention scores are used to filter video categories, and only proposals within the predicted categories are retained. For categories below the threshold, their proposals are filtered out. Soft non-maximum suppression is applied to filter redundant action proposals, and finally the predicted action proposal results are obtained.

[0116] In a feasible implementation, the present invention uses the DELU model as the base model of the weak supervision branch, and constructs the regression model of the full supervision branch with reference to the ActionFormer and TriDet models. In the feature extraction module, the I3D model pre-trained on the Kinetics dataset is used to extract RGB and optical flow features, and the extracted feature dimension is 1024. For the THUMOS14 dataset, according to the method of the ActionFormer model, a fragment feature is extracted every 16 frames with a stride of 4 (the stride of the basic model is 16); for the ActivityNet1.3 dataset, a fragment feature is extracted every 16 frames. For the THUMOS14 dataset, the second training round is set to 38; for the ActivityNet1.3 dataset, the second training round is set to 16, and the first rounds are 20 and 10, respectively. The learning rates of THUMOS14 and ActivityNet1.3 are 1e-4 and 1e-3, respectively. For the uncertainty mask, the THUMOS14 dataset sets the expansion ratio The shrinkage ratio is 0.1. 0; ActivityNet1.3 sets the expansion ratio The shrinkage ratio is 0.05. is 0.05.

[0117] In a feasible implementation, the performance of the description method of the present invention and the current advanced methods on the THUMOS14 dataset are compared, and the results are shown in Table 1 (Comparison table of multi-method action localization results on the THUMOS14 dataset). As can be seen from Table 1, the average precision of the method of the present invention in the interval (0.1:0.7) is 5.5% higher than that of the existing baseline model dual-evidence learning framework, and the average precision of other intervals also shows a similar improvement. Compared with the method guided by prior information, the present invention has significantly surpassed all intersection-over-union ratio thresholds (from 0.1 to 0.7), and the gap is large. It can be seen from Table 1 that the performance of the method adopted by the present invention is close to that of the early point supervision method, indicating that the gap between weakly supervised temporal action localization and fully supervised temporal action localization has narrowed.

[0118] Table 1

[0119]

[0120] The performance of the present invention and other existing methods on the ActivityNet1.3 dataset is compared. The results are shown in Table 2 (Comparison of action localization results of multiple methods on the ActivityNet1.3 dataset). The results show a similar trend: the performance of the present invention is improved by 0.9% compared with the current best method based on prior information guidance. The method proposed by the present invention exceeds some recent point supervision methods. As can be seen from Table 2, the excellent performance of the present invention method on the challenging ActivityNet1.3 dataset further confirms the strong robustness and high generalization ability.

[0121] In tests conducted on two benchmark test sets, THUMOS14 and ActivityNet1.3, the proposed method significantly outperforms existing methods in performance, has wide applicability, and bridges the gap between weakly supervised and fully supervised temporal action localization.

[0122] Table 2

[0123]

[0124] The present invention proposes a weakly supervised temporal action localization method based on pseudo-label generation. Through a dual-branch framework, the gap between weakly supervised and fully supervised temporal action localization in terms of framework and performance is bridged; by using the information in the weakly supervised branch and the fully supervised branch at the same time, the dual-branch framework can achieve more accurate temporal action localization under the condition of less labeled data; a pseudo-label proposal fusion strategy is adopted to effectively use the prior information in the pseudo-label to generate pseudo-label proposals with high confidence and accurate boundaries; this strategy significantly improves the quality of the proposal and the accuracy of the model through the fusion of multi-scale information; an uncertainty mask is introduced, and a new optimization mechanism is proposed for effective learning in the presence of noisy labels. The mechanism effectively reduces the interference of noisy labels during the training process by dynamically adjusting the quality of pseudo-labels, enhances the robustness of the regression model, and ensures that the model can converge stably during the training process. The present invention is a weakly supervised temporal action localization method for generating high-quality pseudo-labels that fully utilizes prior information and effectively combines the full supervision method.

[0125] Figure 2 1 is a block diagram of a weakly supervised temporal action positioning device based on pseudo-label generation according to an exemplary embodiment, wherein the device is used in a weakly supervised temporal action positioning method based on pseudo-label generation. Figure 2 The device includes a video clip acquisition module 210, a video feature extraction module 220, a weakly supervised branch classification module 230, a pseudo-label proposal generation module 240, an uncertainty mask generation module 250, a fully supervised branch optimization module 260 and a video action positioning module 270. Among them:

[0126] The video segment acquisition module 210 is used to acquire an uncut video containing actions; divide the uncut video into segments to obtain a video segment set;

[0127] The video feature extraction module 220 is used to extract features through a feature extractor according to the video segment set to obtain a segment-by-segment feature set;

[0128] A weakly supervised branch classification module 230 is used to perform preliminary action classification through a weakly supervised branch according to a segment-by-segment feature set to obtain a classified attention sequence and a multi-scale action proposal set;

[0129] A pseudo-label proposal generation module 240 is used to perform proposal fusion optimization using a proposal fusion strategy according to the classified attention sequence and the action proposal set to obtain a pseudo-label proposal set;

[0130] An uncertainty mask generating module 250, configured to generate an uncertainty mask set at a time boundary of each proposal in the action proposal set based on a preset expansion ratio and a preset contraction ratio;

[0131] A fully supervised branch optimization module 260 is used to optimize the fully supervised branch through a dynamic optimization mechanism based on an attention mechanism and according to a pseudo-label proposal set and an uncertainty mask set, to obtain a second optimized fully supervised branch;

[0132] The video action localization module 270 is used to obtain a video of an action to be located; based on a feature extractor, a weak supervision branch and a second optimized full supervision branch, the action is located according to the video of the action to be located.

[0133] Optionally, the weakly supervised branch classification module 230 is further used to:

[0134] Based on the attention mechanism, adaptive action classification is performed through the basic model according to the feature set of each segment, and the attention score sequence and classification score matrix are obtained;

[0135] Perform weighted fusion based on the attention score sequence and the classification score matrix to obtain the classification attention sequence;

[0136] Based on the preset action threshold, the classified attention sequence is thresholded to obtain a multi-scale action proposal set; the action proposal set includes a time boundary set and an action category set.

[0137] Among them, the basic model is a baseline model based on the multi-instance learning training method.

[0138] Optionally, the pseudo-label proposal generating module 240 is further configured to:

[0139] Based on the action proposal set, the classification attention sequence is calculated to obtain the mean internal attention score and the mean external attention score of the proposal;

[0140] Based on the mean of the proposal's internal attention score and the mean of the proposal's external attention score, the confidence is calculated according to the classified attention sequence to obtain a set of proposal confidence scores;

[0141] According to the action proposal set and the proposal confidence score set, the proposal fusion strategy is used to perform fusion optimization to obtain a pseudo-label proposal set; the proposal fusion strategy refers to an optimization strategy that fuses the action proposals mapped to the global shared space with the confidence scores corresponding to the action proposals; the proposal fusion strategy is used to filter the classification misdetection results of action proposals.

[0142] Optionally, the fully supervised branch optimization module 260 is further used to:

[0143] Based on the morphological preprocessing method, the pseudo-label proposal set is filtered and optimized to obtain the optimized action proposal set;

[0144] The optimized action proposal set is input into the encoder for data processing to obtain the encoded pseudo-label proposal set;

[0145] According to the encoded pseudo-label proposal set, multi-scale feature extraction is performed through the feature pyramid network to obtain the pseudo-label proposal feature map;

[0146] Based on the uncertainty mask set, the pseudo-label proposal feature map is masked to obtain the first unmasked proposal feature map;

[0147] Based on the preset first training round, according to the first unoccluded proposal feature map, the first stage of training is performed through the fully supervised branch to obtain the first predicted action proposal set;

[0148] Based on the attention mechanism, the first predicted action proposal set and the uncertainty mask set are dynamically optimized according to the pseudo-label proposal set to obtain the second predicted action proposal set and update the uncertainty mask set;

[0149] The loss function is calculated based on the second predicted action proposal set and the pseudo-label proposal set to obtain the first action loss;

[0150] According to the first action loss, the parameters of the fully supervised branch are optimized to obtain the first optimized fully supervised branch;

[0151] Based on the updated uncertain mask set, the pseudo-label proposal feature map is masked to obtain a second unmasked proposal feature map;

[0152] Based on the preset second training round, according to the second unoccluded proposal feature map, the second stage training is performed through the first optimized full supervision branch to obtain a third predicted action proposal set;

[0153] The loss function is calculated based on the third predicted action proposal set and the pseudo-label proposal set to obtain the second action loss;

[0154] According to the second action loss, the parameters of the first optimized fully supervised branch are optimized to obtain the second optimized fully supervised branch.

[0155] Optionally, the fully supervised branch optimization module 260 is further used to:

[0156] Based on the attention mechanism, the first action proposal attention sequence is obtained by calculating the pseudo-label proposal set and the first predicted action proposal set;

[0157] Based on the preset attention threshold, the first action proposal attention sequence is screened to obtain the second action proposal attention sequence;

[0158] Based on the second action proposal attention sequence, select the corresponding action set in the first predicted action proposal set to obtain the second predicted action proposal set;

[0159] Based on the preset linear function, according to the attention sequence of the second action proposal, the mask weights of the uncertainty mask set are dynamically linearly attenuated to obtain an updated uncertainty mask set; the range of the mask weights is 1 to 0.

[0160] Optionally, the video action positioning module 270 is further configured to:

[0161] Based on the feature extractor, the weak supervision branch and the second optimized full supervision branch, the temporal action positioning prediction is performed to obtain a set of predicted action proposals;

[0162] Based on the attention mechanism, according to the predicted action proposal set, the soft non-maximum suppression algorithm is used to filter redundant action proposals to obtain filtered action proposals;

[0163] Action localization is performed based on the action categories and time boundaries of the filtered action proposals.

[0164] The present invention proposes a weakly supervised temporal action localization method based on pseudo-label generation. Through a dual-branch framework, the gap between weakly supervised and fully supervised temporal action localization in terms of framework and performance is bridged; by using the information in the weakly supervised branch and the fully supervised branch at the same time, the dual-branch framework can achieve more accurate temporal action localization under the condition of less labeled data; a pseudo-label proposal fusion strategy is adopted to effectively use the prior information in the pseudo-label to generate pseudo-label proposals with high confidence and accurate boundaries; this strategy significantly improves the quality of the proposal and the accuracy of the model through the fusion of multi-scale information; an uncertainty mask is introduced, and a new optimization mechanism is proposed for effective learning in the presence of noisy labels. The mechanism effectively reduces the interference of noisy labels during the training process by dynamically adjusting the quality of pseudo-labels, enhances the robustness of the regression model, and ensures that the model can converge stably during the training process. The present invention is a weakly supervised temporal action localization method for generating high-quality pseudo-labels that fully utilizes prior information and effectively combines the full supervision method.

[0165] Figure 3 is a structural diagram of a weakly supervised sequential action positioning device provided by an embodiment of the present invention, such as Figure 3 As shown, the weakly supervised temporal action positioning device may include the above Figure 2 The weakly supervised temporal action localization device based on pseudo label generation is shown. Optionally, the weakly supervised temporal action localization device 310 may include a first processor 2001 .

[0166] Optionally, the weakly supervised sequential action localization device 310 may further include a memory 2002 and a transceiver 2003 .

[0167] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0168] Combine the following Figure 3 The components of the weakly supervised sequential action positioning device 310 are specifically introduced as follows:

[0169] The first processor 2001 is the control center of the weakly supervised sequential motion positioning device 310, and may be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or may be application specific integrated circuits (ASICs), or may be configured to implement one or more integrated circuits of the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0170] Optionally, the first processor 2001 can perform various functions of the weakly supervised sequential motion positioning device 310 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0171] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 3 CPU0 and CPU1 are shown in FIG.

[0172] In a specific implementation, as an embodiment, the weakly supervised sequential action localization device 310 may also include multiple processors, such as Figure 3 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0173] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0174] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0175] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0176] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 3 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0177] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and locate the device 310 through the interface circuit ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0178] It should be noted that Figure 3 The structure of the weakly supervised sequential action positioning device 310 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0179] In addition, the technical effects of the weakly supervised temporal action localization device 310 can refer to the technical effects of the weakly supervised temporal action localization method based on pseudo-label generation described in the above method embodiment, which will not be repeated here.

[0180] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0181] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0182] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0183] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0184] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0185] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0186] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0187] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0188] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0189] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0191] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0192] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A weakly supervised temporal action localization method based on pseudo-label generation, characterized in that: The method comprises: Obtaining an uncut video containing an action; dividing the uncut video into segments to obtain a set of video segments; According to the video segment set, feature extraction is performed by a feature extractor to obtain a segment-by-segment feature set; According to the segment-by-segment feature set, preliminary action classification is performed through a weak supervision branch to obtain a classified attention sequence and a multi-scale action proposal set; According to the classified attention sequence and the action proposal set, a proposal fusion strategy is used to perform proposal fusion optimization to obtain a pseudo-label proposal set; Based on a preset expansion ratio and a preset contraction ratio, generating an uncertainty mask set for a time boundary of each proposal in the action proposal set; Based on the attention mechanism, according to the pseudo-label proposal set and the uncertainty mask set, the full supervision branch is optimized and trained through a dynamic optimization mechanism to obtain a second optimized full supervision branch; Acquire a video of the action to be located; and perform action location according to the video of the action to be located based on the feature extractor, the weak supervision branch, and the second optimized full supervision branch.

2. The weakly supervised temporal action localization method based on pseudo-label generation according to claim 1, characterized in that: According to the segment-by-segment feature set, preliminary action classification is performed through a weak supervision branch to obtain a classification attention sequence and a multi-scale action proposal set, including: Based on the attention mechanism, adaptive action classification is performed through the basic model according to the segment-by-segment feature set to obtain an attention score sequence and a classification score matrix; Performing weighted fusion according to the attention score sequence and the classification score matrix to obtain a classification attention sequence; Based on a preset action threshold, the classified attention sequence is thresholded to obtain a multi-scale action proposal set; the action proposal set includes a time boundary set and an action category set.

3. The weakly supervised temporal action localization method based on pseudo-label generation according to claim 2 is characterized in that: The basic model is a baseline model based on a multiple instance learning training method.

4. The weakly supervised temporal action localization method based on pseudo-label generation according to claim 1, characterized in that: According to the classified attention sequence and the action proposal set, a proposal fusion strategy is used to perform proposal fusion optimization to obtain a pseudo-label proposal set, including: Based on the action proposal set, calculate according to the classified attention sequence to obtain the mean internal attention score of the proposal and the mean external attention score of the proposal; Based on the mean of the proposal internal attention score and the mean of the proposal external attention score, confidence calculation is performed according to the classified attention sequence to obtain a proposal confidence score set; According to the action proposal set and the proposal confidence score set, a proposal fusion strategy is used to perform fusion optimization to obtain a pseudo-label proposal set; the proposal fusion strategy refers to an optimization strategy that fuses the action proposals mapped to the global shared space with the confidence scores corresponding to the action proposals; the proposal fusion strategy is used to filter out the classification misdetection results of action proposals.

5. The weakly supervised temporal action localization method based on pseudo-label generation according to claim 1, characterized in that: The method of optimizing and training the full supervision branch based on the attention mechanism and the pseudo-label proposal set and the uncertainty mask set through a dynamic optimization mechanism to obtain a second optimized full supervision branch includes: Based on the morphological preprocessing method, the pseudo-label proposal set is filtered and optimized to obtain an optimized action proposal set; Inputting the optimized action proposal set into the encoder for data processing to obtain an encoded pseudo-label proposal set; According to the encoded pseudo-label proposal set, multi-scale feature extraction is performed through a feature pyramid network to obtain a pseudo-label proposal feature map; Based on the uncertainty mask set, masking the pseudo-label proposal feature map to obtain a first unmasked proposal feature map; Based on the preset first training round, according to the first unoccluded proposal feature map, a first stage of training is performed through a fully supervised branch to obtain a first predicted action proposal set; Based on the attention mechanism, dynamically optimize the first predicted action proposal set and the uncertainty mask set according to the pseudo-label proposal set to obtain a second predicted action proposal set and update the uncertainty mask set; Calculate the loss function according to the second predicted action proposal set and the pseudo label proposal set to obtain a first action loss; According to the first action loss, optimizing the parameters of the fully supervised branch to obtain a first optimized fully supervised branch; Based on the updated uncertain mask set, the pseudo-label proposal feature map is masked to obtain a second unmasked proposal feature map; Based on the preset second training round, according to the second unoccluded proposal feature map, the second stage training is performed through the first optimized full supervision branch to obtain a third predicted action proposal set; Calculate the loss function according to the third predicted action proposal set and the pseudo label proposal set to obtain a second action loss; According to the second action loss, parameters of the first optimized fully supervised branch are optimized to obtain a second optimized fully supervised branch.

6. The weakly supervised temporal action localization method based on pseudo-label generation according to claim 5, characterized in that: The method of dynamically optimizing the first predicted action proposal set and the uncertainty mask set based on the attention mechanism and according to the pseudo-label proposal set to obtain a second predicted action proposal set and update the uncertainty mask set includes: Based on the attention mechanism, a first action proposal attention sequence is obtained by performing calculation according to the pseudo-label proposal set and the first predicted action proposal set; Based on a preset attention threshold, the first action proposal attention sequence is screened to obtain a second action proposal attention sequence; Based on the second action proposal attention sequence, select a corresponding action set in the first predicted action proposal set to obtain a second predicted action proposal set; Based on a preset linear function, according to the second action proposal attention sequence, the mask weights of the uncertainty mask set are dynamically linearly attenuated to obtain an updated uncertainty mask set; the range of the mask weights is 1 to 0.

7. The weakly supervised temporal action localization method based on pseudo-label generation according to claim 1, characterized in that: The method of performing action positioning according to the action video to be positioned based on the feature extractor, the weak supervision branch and the second optimized full supervision branch includes: Perform temporal action location prediction based on the feature extractor, the weak supervision branch, and the second optimized full supervision branch to obtain a set of predicted action proposals; Based on the attention mechanism, according to the predicted action proposal set, redundant action proposals are filtered through a soft non-maximum suppression algorithm to obtain filtered action proposals; Action localization is performed according to the action category and time boundary of the filtered action proposals.

8. A weakly supervised temporal action positioning device based on pseudo-label generation, the weakly supervised temporal action positioning device based on pseudo-label generation is used to implement the weakly supervised temporal action positioning method based on pseudo-label generation as claimed in any one of claims 1 to 7, characterized in that: The device comprises: The video segment acquisition module is used to acquire an uncut video containing actions; divide the uncut video into segments to obtain a video segment set; A video feature extraction module, used to extract features through a feature extractor according to the video segment set to obtain a segment-by-segment feature set; A weakly supervised branch classification module, used to perform preliminary action classification through a weakly supervised branch according to the segment-by-segment feature set, to obtain a classified attention sequence and a multi-scale action proposal set; A pseudo-label proposal generation module is used to perform proposal fusion optimization using a proposal fusion strategy according to the classification attention sequence and the action proposal set to obtain a pseudo-label proposal set; An uncertainty mask generation module, configured to generate an uncertainty mask set at a time boundary of each proposal in the action proposal set based on a preset expansion ratio and a preset contraction ratio; A fully supervised branch optimization module, used to optimize and train the fully supervised branch based on the attention mechanism, according to the pseudo-label proposal set and the uncertainty mask set, through a dynamic optimization mechanism, to obtain a second optimized fully supervised branch; The video action localization module is used to obtain a video of the action to be located; based on the feature extractor, the weak supervision branch and the second optimized full supervision branch, the action is located according to the video of the action to be located.

9. A weakly supervised sequential action positioning device, characterized in that: The weakly supervised sequential action positioning device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images