Semi-supervised temporal action positioning method, system, device and medium

Abstract

The present invention discloses a semi-supervised temporal action localization method, system, device, and medium. The semi-supervised temporal action localization method includes the following steps: obtaining an encoded video feature sequence of an original video sequence for temporal action localization; performing temporal action detection using a pre-trained temporal action detection model based on the obtained encoded video feature sequence to obtain a temporal action localization result; wherein the temporal action localization result includes the start time, end time, and category of the action. The method provided by the present invention is specifically a semi-supervised temporal action localization method based on noisy pseudo-label learning, which can effectively utilize unlabeled data to improve the performance of semi-supervised temporal action localization.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and pattern recognition technology, relates to the field of temporal action positioning, and particularly relates to a semi-supervised temporal action positioning method, system, device and medium. Background Art

[0002] Temporal action localization aims to automatically retrieve the start and end times of actions in videos and predict the action category.

[0003] Most existing temporal action localization methods rely on dense action labels in training videos; however, meticulously annotating every action instance is tedious and time-consuming. Semi-supervised temporal action localization is a remedy that requires only a small amount of labeled videos and a large amount of unlabeled videos, attracting increasing attention from both academia and industry.

[0004] Existing semi-supervised temporal action localization methods can be roughly divided into two categories: one based on consistency regularization and the other based on self-training strategies. Consistency regularization methods aim to generate consistent action proposals for videos with the same data but different data augmentation methods, such as temporal warping or temporal feature shifting. In contrast, self-training methods achieve new state-of-the-art performance by alternating between pseudo-labeling of unlabeled videos and retraining with pseudo-labels.

[0005] Currently, self-training-based methods focus on designing a proposal-free framework to address the propagation of action proposal errors that exist in previous methods, but ignore the important role of pseudo-labels in the self-training process. Specifically, pseudo-label noise remains a core challenge for semi-supervised temporal action localization, hindering effective learning from large amounts of unlabeled videos. Generally speaking, label noise often leads to two thorny problems: boundary position deviation and action category errors, which worsen as the number of labeled videos decreases. Therefore, noisy pseudo-labels significantly degrade the performance of semi-supervised temporal action localization. Summary of the Invention

[0006] The present invention aims to provide a method, system, device, and medium for semi-supervised temporal action localization to address one or more of the above-mentioned technical problems. Specifically, the method provided by the present invention is a semi-supervised temporal action localization method based on noisy pseudo-label learning, which can effectively utilize unlabeled data to improve the performance of semi-supervised temporal action localization.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A first aspect of the present invention provides a semi-supervised temporal action localization method, comprising the following steps:

[0009] Obtaining an encoded video feature sequence of an original video sequence to be temporally positioned;

[0010] Based on the obtained encoded video feature sequence, a pre-trained temporal action detection model is used to perform temporal action detection to obtain a temporal action positioning result; wherein the temporal action positioning result includes the start time, end time and category of the action;

[0011] The training and acquisition steps of the pre-trained temporal action detection model include:

[0012] Based on a training video set for temporal action localization, a labeled video set and an unlabeled video set are obtained; wherein the labeled video set includes instance-level data annotations for each action;

[0013] Training a first temporal action detector based on the labeled video set to obtain a trained first temporal action detector;

[0014] Based on the trained first temporal action detector, predict each unlabeled video in the unlabeled video set to obtain an initial pseudo label for each unlabeled video;

[0015] For each unlabeled video, sort and filter the noise labels based on the initial pseudo labels to obtain the sorted and filtered pseudo labels;

[0016] Based on the pseudo labels after sorting and filtering of each unlabeled video, a second temporal action detector is trained to obtain the pre-trained temporal action detection model.

[0017] A further improvement of the method of the present invention is that, for each unlabeled video, in the step of sorting and filtering the noise labels based on the initial pseudo labels to obtain the sorted and filtered pseudo labels,

[0018] When performing noisy label ranking, for each unlabeled video in the unlabeled video set, all pseudo labels are ranked by evaluating the action category confidence and action boundary localization confidence of each pseudo label.

[0019] A further improvement of the method of the present invention is that, when performing noise label sorting, for each unlabeled video in the unlabeled video set, in the step of sorting all pseudo labels by evaluating the action category confidence and action boundary location confidence of each pseudo label,

[0020] For the jth pseudo label, its confidence score s j Expressed as Based on s j To sort and select pseudo labels with high confidence; among them, represents the action classification score of the pseudo label; represents the variance of the set of pseudo labels and the predicted pseudo labels of its adjacent video segments;

[0021]

[0022] Where, and denote the variance of the start and end boundaries respectively, and They are respectively j Boundary variance after scale normalization;

[0023] in, The smaller the pseudo label p j The smaller the boundary ambiguity, the higher the boundary positioning reliability.

[0024] A further improvement of the method of the present invention is that, for each unlabeled video, in the step of sorting and filtering the noise labels based on the initial pseudo labels to obtain the sorted and filtered pseudo labels,

[0025] When filtering, redundant pseudo-labels are adaptively filtered according to the number of action categories and pseudo-labels contained in each unlabeled video.

[0026] A further improvement of the method of the present invention is that, when filtering, in the step of adaptively filtering redundant pseudo labels according to the action categories and the number of pseudo labels contained in each unlabeled video,

[0027] The adaptive filtering strategy expression is:

[0028]

[0029] Where K adp The total number of pseudo labels is selected adaptively according to the number of pseudo labels and action categories of each video; N represents the total number of pseudo labels in the video; N pos is a predefined hyperparameter that controls the number of positive samples in each video.

[0030] A further improvement of the method of the present invention is that, in the step of training the second temporal action detector based on the pseudo labels after sorting and filtering of each unlabeled video to obtain the pre-trained temporal action detection model,

[0031] The second temporal action detector is trained by penalizing inconsistent boundary positions of adjacent video clips to predict reliable action boundaries, and finally the pre-trained temporal action detection model is obtained.

[0032] A further improvement of the method of the present invention is that in the step of training the second temporal action detector by penalizing inconsistent boundary positions of adjacent video segments to predict reliable action boundaries,

[0033] Training loss function L u The expression is,

[0034]

[0035] Among them, U i represents the i-th video in the unlabeled video set, N u Indicates the number of unlabeled videos, and They are cross entropy loss function and temporal consistency loss function respectively;

[0036]

[0037] Where M u Indicates the number of pseudo labels involved in training in the video, M p represents the number of video segments contained in each pseudo label, b j，t represents the predicted start or end boundary position of the t-th video segment within the j-th pseudo label.

[0038] A second aspect of the present invention provides a semi-supervised temporal action localization system, comprising:

[0039] A data acquisition module, used to obtain the encoded video feature sequence of the original video sequence to be temporally positioned;

[0040] A result acquisition module is configured to perform temporal action detection based on the obtained encoded video feature sequence using a pre-trained temporal action detection model to obtain a temporal action positioning result; wherein the temporal action positioning result includes the start time, end time and category of the action;

[0041] The training and acquisition steps of the pre-trained temporal action detection model include:

[0042] Based on a training video set for temporal action localization, a labeled video set and an unlabeled video set are obtained; wherein the labeled video set includes instance-level data annotations for each action;

[0043] Training a first temporal action detector based on the labeled video set to obtain a trained first temporal action detector;

[0044] Based on the trained first temporal action detector, predict each unlabeled video in the unlabeled video set to obtain an initial pseudo label for each unlabeled video;

[0045] For each unlabeled video, sort and filter the noise labels based on the initial pseudo labels to obtain the sorted and filtered pseudo labels;

[0046] Based on the pseudo labels after sorting and filtering of each unlabeled video, a second temporal action detector is trained to obtain the pre-trained temporal action detection model.

[0047] A third aspect of the present invention provides an electronic device, comprising:

[0048] at least one processor; and,

[0049] a memory communicatively connected to the at least one processor; wherein,

[0050] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the semi-supervised temporal action localization method as described above in any one of the present inventions.

[0051] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-mentioned semi-supervised temporal action localization methods of the present invention.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] The method provided by the present invention is specifically a semi-supervised temporal action localization method based on noisy pseudo-label learning. It proposes noise label sorting, noise label filtering and noise label learning to address the problems of action classification errors and boundary positioning deviations of pseudo-labels, and can effectively utilize unlabeled data to improve the performance of semi-supervised temporal action localization.

[0054] To further explain, the present invention solves a key problem that has not been explored in the field of semi-supervised temporal action localization, namely how to solve pseudo-label noise. Specifically, a unified noisy pseudo-label learning framework is proposed to simultaneously solve the pseudo-label action classification noise and boundary localization noise, which includes noisy label sorting, noisy label filtering and noisy label learning. The noisy label sorting in the present invention is based on the action semantic confidence and the novel boundary localization reliability comprehensive metric pseudo-label; the noisy label filtering in the present invention adopts a novel adaptive filtering strategy to solve the noisy pseudo-label class imbalance problem ignored by the prior art; the noisy label learning in the present invention adopts a novel unsupervised temporal consistency loss to penalize inconsistent predictions from adjacent frames to perform pseudo-label noise-tolerant learning.

[0055] In the comparative data of the subsequent embodiments of the present invention, based on the comparison of two public data sets, the technical solution disclosed in the present invention achieves a significant improvement in the performance of semi-supervised time series positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art; obviously, the drawings described below are some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1 1 is a flow chart of a semi-supervised temporal action localization method provided by an embodiment of the present invention;

[0058] Figure 2 2 is a schematic diagram of the overall architecture of a semi-supervised temporal action localization system based on noise pseudo-label learning in an embodiment of the present invention;

[0059] Figure 3 is a sample diagram of pseudo-label noise in an embodiment of the present invention;

[0060] Figure 4 is a schematic diagram of the relationship between the boundary variance and the boundary positioning quality proposed by the noise label sorting in an embodiment of the present invention; wherein, Figure 4 (a) is a scatter diagram of the relationship between action classification scores and IoU; Figure 4 (b) is a scatter diagram of the relationship between action boundary variance and IoU;

[0061] Figure 5 is a schematic diagram of experimental results of solving pseudo-label class imbalance by noise label filtering in an embodiment of the present invention; wherein, Figure 5 (a) is a schematic diagram showing the ratio of pseudo labels to true labels for each action category before noise label filtering; Figure 5 (b) is a schematic diagram showing the ratio of pseudo labels to true labels for each action category after noise labels are filtered;

[0062] Figure 6 1 is a schematic diagram of the training convergence rate of noise label learning in an embodiment of the present invention;

[0063] Figure 7 4 is a schematic diagram of a semi-supervised temporal action localization system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0065] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0066] The present invention is described in further detail below with reference to the accompanying drawings:

[0067] See also Figure 1 , an embodiment of the present invention provides a semi-supervised temporal action localization method, comprising the following steps:

[0068] Step 1: Obtain an encoded video feature sequence of an original video sequence for temporal action location. Exemplarily, the original video sequence for temporal action location may be decomposed into a plurality of video segments and encoded to obtain an encoded video feature sequence.

[0069] Step 2: Based on the encoded video feature sequence obtained in step 1, a pre-trained temporal action detection model is used to perform temporal action positioning to obtain a temporal action positioning result; wherein the temporal action positioning result includes the start time, end time and action category of the action;

[0070] The training and acquisition steps of the pre-trained temporal action detection model include:

[0071] The training video set for temporal action localization is randomly divided into a labeled video set and an unlabeled video set. The labeled video set contains instance-level data annotations for each action, namely the start time, end time, and action category label of each action instance in each video. The unlabeled video set does not contain any annotation information. Based on the labeled video set, a temporal action detector is pre-trained. The pre-trained temporal action detector is used to predict action instances in the unlabeled video set, and the predicted results are used as pseudo labels for the unlabeled video set.

[0072] For each unlabeled video in the unlabeled video set, all pseudo labels are sorted by evaluating the action category confidence and action boundary location confidence of each pseudo label; a noisy label filtering method is used to adaptively filter redundant pseudo labels according to the number of action categories and pseudo labels contained in each unlabeled video; a noisy label learning method is used to train a temporal action detector based on the pseudo labels obtained after sorting and filtering by penalizing inconsistent boundary positions of adjacent video clips to predict reliable action boundaries, thereby obtaining the pre-trained temporal action detection model.

[0073] In a specific exemplary embodiment of the present invention, the steps of decomposing an original video sequence for temporal action location into a plurality of video segments and encoding the segments, and obtaining the encoded video feature sequence may specifically include:

[0074] Decompose the original video sequence to be temporally positioned into several video segments, each of which contains consecutive frames of a fixed length of the original video sequence;

[0075] A pre-trained feature extractor is used to encode each video segment to obtain an encoded video feature sequence; wherein the encoded video feature sequence includes RGB information of each video segment and optical flow information between adjacent segments.

[0076] Further exemplary, the encoded video feature sequence is expressed as Among them, f t represents the feature vector of the t-th video clip, and T represents the total number of video clips.

[0077] In an exemplary embodiment of the present invention, the step of obtaining pseudo labels for an unlabeled video set specifically includes:

[0078] The training set videos are randomly sampled with sampling ratios set to 10%, 20%, 40%, and 60% respectively to obtain a labeled video set. Each labeled video contains a complete label for each action, namely the action start time, end time, and action category; the remaining videos after sampling constitute the unlabeled video set;

[0079] Pre-train a temporal action detection model based on a set of labeled videos;

[0080] The pre-trained temporal action detection model is applied to the unlabeled video set to obtain pseudo labels for the unlabeled video set.

[0081] For further example, the specific steps of pre-training a temporal action detector include:

[0082] The training set of temporal action localization is randomly divided into a set of labeled videos and an unlabeled video set N l and N u Represents the number of videos in the labeled video set and the unlabeled video set respectively;

[0083] Given a video in a labeled video set, it contains a set of labeled action instances, each of which can be represented as (t s , t e , c), where t s , t e and c represent the start time, end time and action category of the action instance respectively;

[0084] Based on annotated video collection Pre-train a temporal action detector;

[0085] Predicting Unlabeled Videos Using Pre-trained Temporal Action Detectors Each video contains actions, and each prediction result can be expressed as in and are the predicted start time, end time and action classification score of the action respectively; the prediction results are finally defined as pseudo labels of the unlabeled video set.

[0086] In the embodiment of the present invention, a new pseudo-label confidence evaluation method is proposed in the noisy label sorting, which can reflect the action semantic confidence and boundary positioning reliability of the pseudo-label. The steps include:

[0087] Given a pseudo label That is, it represents the action prediction result of the jth video clip of any unlabeled video. and Represents the start time, end time and action classification score of pseudo label respectively; explanatory, It can only reflect the semantic confidence of the action but cannot guarantee the positioning reliability of the action boundary.

[0088] In the technical solution of the embodiment of the present invention, by evaluating p j The set of predicted pseudo labels of its adjacent video segments The variance of is used to express the reliability of boundary positioning, and the expression is,

[0089]

[0090]

[0091]

[0092] in, and denote the variance of the start and end boundaries respectively, and Through p j Boundary variance after scale normalization.

[0093] therefore, The smaller the pseudo label p j The smaller the boundary ambiguity, the higher the reliability of its boundary positioning; the final confidence score of each pseudo label is expressed as:

[0094]

[0095] The noise label sorting proposed in the embodiment of the present invention is based on s j to sort and select pseudo labels with high confidence.

[0096] In the embodiment of the present invention, the pseudo labels of the unlabeled video set are usually of the wrong category, which leads to the category imbalance problem in the semi-supervised temporal action localization task. To solve this imbalance problem, the embodiment of the present invention proposes an adaptive filtering strategy to adjust the distribution of pseudo labels for each action category. The steps of noise label filtering specifically include:

[0097] Given any unlabeled video, the adaptive filtering strategy proposed in this invention adaptively selects K adp pseudo labels participate in the training model; the adaptive filtering strategy can be expressed as,

[0098]

[0099] That is, K is adaptively selected according to the number of pseudo labels and action categories of each video. adp pseudo labels, where N represents the total number of pseudo labels in the video, N pos is a predefined hyperparameter that controls the number of positive samples in each video.

[0100] In an embodiment of the present invention, a noise-tolerant training algorithm is proposed for pseudo labels with boundary noise, which is used to penalize inconsistent boundary predictions of adjacent video segments. The noise label learning step specifically includes:

[0101] Given the sorted and filtered pseudo-labels, the present invention proposes a noise-tolerant training algorithm for penalizing inconsistent boundary predictions of adjacent video segments. The noise-tolerant training algorithm can be expressed as:

[0102]

[0103] Among them, M u Indicates the number of pseudo labels involved in training in the video, M p represents the number of video segments contained in each pseudo label, b j，t represents the predicted start or end boundary position of the t-th video segment within the j-th pseudo label.

[0104] Finally, the training loss function of the unlabeled dataset is,

[0105]

[0106] Among them, U i represents the i-th video in the unlabeled video set, N u Indicates the number of unlabeled videos, and They are respectively the common cross entropy loss function and the temporal consistency loss function proposed in this invention;

[0107] The temporal action detector trained based on the labeled video set and the unlabeled video set is used to predict the action instances of the test set videos and obtain the detection results of temporal action localization.

[0108] See also Figure 2 A semi-supervised temporal action localization method based on noise pseudo-label learning according to an embodiment of the present invention comprises the following steps:

[0109] Step 1: Given an unedited video sequence, to reduce computational complexity, the unedited video sequence is decomposed into several video segments, each consisting of consecutive frames of a fixed length. Specifically, in this embodiment of the present invention, a pretrained two-stream network is used as a feature extractor to encode each video segment, thereby obtaining an encoded video feature sequence. Ultimately, a training video set and a test video set are obtained after feature encoding.

[0110] In step 2, given the video feature sequence obtained in step 1, a random sampling ratio is set to divide the training video set into a labeled video set and an unlabeled video set. A temporal action detection model is pre-trained based on the labeled video set, and the pre-trained temporal action detection model is used to predict action instances in the unlabeled video set, obtaining pseudo labels for the unlabeled video set.

[0111] Step 3: Given the pseudo-labels of the unlabeled video set obtained in step 2, sort all pseudo-labels by evaluating the action category confidence and action boundary localization confidence of each pseudo-label; adaptively filter redundant pseudo-labels; retrain the action detector based on the sorted and filtered pseudo-labels to obtain accurate temporal action localization results on the test set.

[0112] In the embodiment of the present invention, the specific steps of encoding the video sequence using the dual-stream network in step 1 include:

[0113] Step 1.1: Given an uncut video sequence, decompose the video sequence into T non-overlapping segments as the input of the two-stream network I3D network;

[0114] In step 1.2, in order to explore the appearance and motion information of the video sequence respectively, the original video sequence is encoded to generate a video representation. Specifically, the embodiment of the present invention uses an I3D network to process each video clip and extract a D-dimensional RGB feature vector and an optical flow feature vector.

[0115] In step 2 of the embodiment of the present invention, given a known encoded training set, the specific steps of dividing the training set and obtaining pseudo labels include:

[0116] Step 2.1: Randomly sample the training set videos. The sampling ratio can be set to 10%, 20%, 40% and 60% to obtain the labeled video set. and unlabeled video sets N l and N u Represents the number of videos in the labeled video set and the unlabeled video set. Given a labeled video, it contains the complete label of each action (t s , t e , c), the action start time t s , end time t e and action category c.

[0117] Step 2.2, based on the labeled video set Pre-train an anchor-free temporal action detector. During the training phase, each action video segment is supervised by the corresponding action category label and the distance from the segment to the action start boundary and end boundary. The loss function of the pre-trained anchor-free temporal action detector is expressed as:

[0118]

[0119] in, As the classification loss function, the commonly used cross entropy loss function is adopted. As the regression loss function, the commonly used DIoU loss function is adopted; based on the trained temporal action detection model, it is applied to the unlabeled video set. Obtain pseudo labels for a set of unlabeled videos.

[0120] In step 3 of the embodiment of the present invention, the pseudo labels of the unlabeled video set obtained in step 2 are known. Pseudo labels usually have label noise, that is, position deviation and category error, which will worsen as the number of labeled videos decreases. Figure 3 .

[0121] The noise pseudo-label learning framework proposed in this paper includes the following specific steps:

[0122] Step 3.1, given a pseudo label That is, it represents the prediction of the jth video segment of any unlabeled video. and Represent the start time and end time of the pseudo label and the action classification score respectively. Sorting is based on the above formula, The smaller the pseudo label p j The smaller the boundary ambiguity, the higher the boundary positioning reliability. Figure 4 .

[0123] In step 3.2, the pseudo labels of the unlabeled video set are usually of the wrong category, which leads to the category imbalance problem in the semi-supervised temporal action localization task. Figure 5 In order to solve this type of imbalance problem, an adaptive filtering strategy is proposed to adjust the distribution of pseudo labels for each action category. The adaptive filtering strategy can refer to the above formula; the noise label filtering proposed in the embodiment of the present invention can effectively solve the problem of pseudo label class imbalance.

[0124] In step 3.3, a noise-tolerant training algorithm is proposed for pseudo labels with boundary position noise to penalize inconsistent boundary predictions of adjacent video segments. Finally, the training loss function of the unlabeled dataset is:

[0125]

[0126] Among them, U i represents the i-th video in the unlabeled video set, N u Indicates the number of unlabeled videos, and They are respectively the common cross entropy loss function and the temporal consistency loss function proposed in this invention.

[0127] The final noise label learning module can improve the convergence speed and stability of the model, see Figure 6 .

[0128] In summary, the embodiments of the present invention disclose a semi-supervised temporal action localization method based on noisy pseudo-label learning, which belongs to the field of computer vision and pattern recognition. The steps include: first, using a two-stream network to encode the feature sequence of the input video, which contains the appearance feature information and motion optical flow information within the video; then, randomly dividing the training set into a labeled video set and an unlabeled video set; pre-training a temporal action detector based on the labeled video set; using the pre-trained temporal action detector to predict action instances in the unlabeled video set, and using the obtained prediction results as pseudo-labels for the unlabeled video set; based on the noisy pseudo-labels of the unlabeled video set, using the noisy label ranking method proposed in the present invention, all pseudo-labels are ranked by evaluating the action category confidence and action boundary localization confidence of each pseudo-label; then, redundant pseudo-labels are adaptively filtered to address the pseudo-label class imbalance problem; finally, using the noisy label learning method proposed in the present invention, the action detector is retrained based on the pseudo-labels. The semi-supervised temporal action localization method based on noisy label learning disclosed in the embodiments of the present invention addresses the main challenge of the semi-supervised temporal action localization task, namely, pseudo-label noise. The noisy label ranking proposed in this paper can comprehensively measure the quality of pseudo-labels from two aspects: semantic confidence and boundary positioning reliability. In addition, the noisy label filtering proposed in this paper can effectively solve the problem of pseudo-label class imbalance. Finally, the noisy label learning proposed in this paper provides a novel noise-tolerant training algorithm, which can improve the stability and robustness of the semi-supervised temporal action localization model.

[0129] Compared with existing methods, the method disclosed in the embodiment of the present invention achieves significant performance improvement in the evaluation indicator mAP@tIoU of the semi-supervised temporal action localization task, indicating that the method of the present invention can accurately locate and classify specific actions. The embodiment of the present invention uses the mean average precision (mAP) to evaluate the proposed method, and uses the official evaluation code provided by ActivityNet to evaluate the performance of the present invention on these two datasets. In particular, for THUMOS14 and ActivityNet v1.3, the tIoU thresholds are from {0.3, 0.4, 0.5, 0.5, 0.6, 0.7} and {0.5, 0.75, 0.95, 0.95}, respectively. On ActivityNet v1.3, the present invention reports mAP for 10 different tIoU thresholds [0.5: 0.05: 0.95]. Table 1 shows the experimental results of different methods on the public datasets THUMOS14 and ActivityNet v1.3 used to evaluate the semi-supervised temporal action localization task. The sampling ratios of the annotated video sets are set to 10%, 20%, 40%, and 60%, respectively. As can be seen from Table 1, the semi-supervised temporal action localization method based on noisy label learning proposed in this embodiment of the present invention has obvious advantages in positioning performance.

[0130] Table 1. Comparison results with different semi-supervised temporal action localization methods on the THUMOS14 dataset and ActivityNet v1.3 dataset

[0131]

[0132] The following are device embodiments of the present invention, which can be used to perform the method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0133] See also Figure 7 , an embodiment of the present invention provides a semi-supervised temporal action localization system, comprising:

[0134] A data acquisition module, used to obtain the encoded video feature sequence of the original video sequence to be temporally positioned;

[0135] A result acquisition module is configured to perform temporal action detection based on the obtained encoded video feature sequence using a pre-trained temporal action detection model to obtain a temporal action positioning result; wherein the temporal action positioning result includes the start time, end time and category of the action;

[0136] The training and acquisition steps of the pre-trained temporal action detection model include:

[0137] Based on a training video set for temporal action localization, a labeled video set and an unlabeled video set are obtained; wherein the labeled video set includes instance-level data annotations for each action;

[0138] Training a first temporal action detector based on the labeled video set to obtain a trained first temporal action detector;

[0139] Based on the trained first temporal action detector, predict each unlabeled video in the unlabeled video set to obtain an initial pseudo label for each unlabeled video;

[0140] For each unlabeled video, sort and filter the noise labels based on the initial pseudo labels to obtain the sorted and filtered pseudo labels;

[0141] Based on the pseudo labels after sorting and filtering of each unlabeled video, a second temporal action detector is trained to obtain the pre-trained temporal action detection model.

[0142] In summary, the embodiments of the present invention disclose a method, system, device, and medium for semi-supervised temporal action localization based on noisy pseudo-label learning. The method comprises the following steps: after obtaining an encoded video feature sequence, pre-training a temporal action detector based on a labeled video set, and then using the pre-trained temporal action detector to obtain pseudo-labels for an unlabeled video set; based on the pseudo-labels, the present invention proposes a noisy label sorting module, the purpose of which is to sort all pseudo-labels based on action semantic confidence and boundary localization reliability; based on the sorted pseudo-labels, the present invention proposes a noisy label filtering module, the purpose of which is to adaptively select appropriate pseudo-labels for temporal action localization model training based on the action category and number of pseudo-labels in each video; based on the filtered pseudo-labels, the present invention proposes a noisy label learning module, the purpose of which is to penalize inconsistent boundary predictions of adjacent video segments when training the temporal action localization model. The technical solution of the embodiments of the present invention is the first to deeply study the main challenge of the semi-supervised temporal action localization task, namely noisy pseudo-labels, and introduces three novel techniques to enable the temporal action localization model to effectively learn knowledge from the unlabeled video set, thereby improving the performance of temporal action localization.

[0143] In another embodiment of the present invention, a computer device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the semi-supervised temporal action positioning method.

[0144] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the semi-supervised temporal action localization method in the above embodiment.

[0145] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0146] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0147] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A semi-supervised temporal action localization method, characterized in that: The following steps are involved: Obtaining an encoded video feature sequence of an original video sequence to be temporally positioned; Based on the obtained encoded video feature sequence, a pre-trained temporal action detection model is used to perform temporal action detection to obtain a temporal action positioning result; wherein the temporal action positioning result includes the start time, end time and category of the action; The training and acquisition steps of the pre-trained temporal action detection model include: Based on a training video set for temporal action localization, a labeled video set and an unlabeled video set are obtained; wherein the labeled video set includes instance-level data annotations for each action; Training a first temporal action detector based on the labeled video set to obtain a trained first temporal action detector; Based on the trained first temporal action detector, predict each unlabeled video in the unlabeled video set to obtain an initial pseudo label for each unlabeled video; For each unlabeled video, sort and filter the noise labels based on the initial pseudo labels to obtain the sorted and filtered pseudo labels; Training a second temporal action detector based on the sorted and filtered pseudo labels of each unlabeled video to obtain the pre-trained temporal action detection model; In the step of sorting and filtering the noise labels based on the initial pseudo labels for each unlabeled video to obtain the sorted and filtered pseudo labels, when sorting the noise labels, for each unlabeled video in the unlabeled video set, all pseudo labels are sorted by evaluating the action category confidence and action boundary location confidence of each pseudo label; Among them, when performing the noise label sorting, for each unlabeled video in the unlabeled video set, by evaluating the action category confidence and action boundary location confidence of each pseudo label, in the step of sorting all pseudo labels, for the jth pseudo label, its confidence score s j Expressed as Based on s j To sort and select pseudo labels with high confidence; among them, represents the action classification score of the pseudo label; represents the variance of the set of pseudo labels and the predicted pseudo labels of its adjacent video segments; Where, and denote the variance of the start and end boundaries respectively, and They are respectively j Boundary variance after scale normalization; in, The smaller the pseudo label p j The smaller the boundary ambiguity, the higher the boundary positioning reliability.

2. A semi-supervised temporal action localization method according to claim 1, characterized in that: For each unlabeled video, in the step of sorting and filtering the noise labels based on the initial pseudo labels to obtain the sorted and filtered pseudo labels, When filtering, redundant pseudo-labels are adaptively filtered according to the number of action categories and pseudo-labels contained in each unlabeled video.

3. A semi-supervised temporal action localization method according to claim 2, characterized in that: When filtering, in the step of adaptively filtering redundant pseudo labels according to the number of action categories and pseudo labels contained in each unlabeled video, The adaptive filtering strategy expression is: Where K adp The total number of pseudo labels is selected adaptively according to the number of pseudo labels and action categories of each video; N represents the total number of pseudo labels in the video; e pos is a predefined hyperparameter that controls the number of positive samples in each video.

4. A semi-supervised temporal action localization method according to claim 1, characterized in that: In the step of training the second temporal action detector based on the pseudo labels after sorting and filtering of each unlabeled video to obtain the pre-trained temporal action detection model, The second temporal action detector is trained by penalizing inconsistent boundary positions of adjacent video clips to predict reliable action boundaries, and finally the pre-trained temporal action detection model is obtained.

5. A semi-supervised temporal action localization method according to claim 4, characterized in that: In the step of training a second temporal action detector by penalizing inconsistent boundary positions of adjacent video segments to predict reliable action boundaries, Training loss function L u The expression is, Among them, U i represents the i-th video in the unlabeled video set, N u Indicates the number of unlabeled videos, and They are cross entropy loss function and temporal consistency loss function respectively; Where M u Indicates the number of pseudo labels involved in training in the video, M p represents the number of video segments contained in each pseudo label, b j,t represents the predicted start or end boundary position of the t-th video segment within the j-th pseudo label.

6. A semi-supervised temporal action localization system, characterized in that: The method for implementing the semi-supervised temporal action localization method according to claim 1 comprises: A data acquisition module, used to obtain the encoded video feature sequence of the original video sequence to be temporally positioned; A result acquisition module is configured to perform temporal action detection based on the obtained encoded video feature sequence using a pre-trained temporal action detection model to obtain a temporal action positioning result; wherein the temporal action positioning result includes the start time, end time and category of the action; The training and acquisition steps of the pre-trained temporal action detection model include: Based on a training video set for temporal action localization, a labeled video set and an unlabeled video set are obtained; wherein the labeled video set includes instance-level data annotations for each action; Training a first temporal action detector based on the labeled video set to obtain a trained first temporal action detector; Based on the trained first temporal action detector, predict each unlabeled video in the unlabeled video set to obtain an initial pseudo label for each unlabeled video; For each unlabeled video, sort and filter the noise labels based on the initial pseudo labels to obtain the sorted and filtered pseudo labels; Based on the pseudo labels after sorting and filtering of each unlabeled video, a second temporal action detector is trained to obtain the pre-trained temporal action detection model.

7. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the semi-supervised temporal action localization method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the semi-supervised temporal action localization method according to any one of claims 1 to 5 is implemented.

Images