Point supervision time sequence action positioning method and related device

Abstract

The invention belongs to the technical field of computer vision and mode recognition, and discloses a point supervision time sequence action positioning method and a related device. The point supervision time sequence action positioning method comprises the following steps: acquiring a video sequence to be subjected to time sequence action positioning; based on the obtained video sequence to be subjected to time sequence action positioning, performing time sequence action positioning by utilizing the trained point supervision time sequence action positioning model to obtain a time sequence action positioning result; wherein the time sequence action positioning result comprises the starting time, the ending time and the type of the action; and the point supervision time sequence action positioning model adopts a DETR structure. According to the specific technical scheme disclosed by the invention, a DETR structure is introduced, and a solution is provided for the problem that the DETR structure cannot directly utilize sparse label training, so that the accuracy of a point supervision time sequence action positioning result is ensured while the whole process is optimized.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and pattern recognition, and in particular relates to a point-supervised temporal action positioning method and related devices. Background Art

[0002] With the rapid growth of video content, how to efficiently understand and analyze long, unedited videos has become an important research topic in the field of video understanding. Among them, temporal action localization aims to identify the start and end times and categories of actions of interest in the video, which is one of the basic tasks in video understanding.

[0003] Existing fully supervised temporal action localization (TAL) methods rely on dense and precise frame-level annotations. While these methods offer excellent performance, the annotation process is time-consuming and highly dependent on manual judgment, making it extremely costly. To reduce the annotation cost, various weakly supervised strategies have been proposed. Among them, weakly supervised methods that use only video-level category labels reduce this dependency to some extent, but due to the lack of action boundary information, the model struggles to accurately distinguish action segments from background noise. To strike a balance between labeling cost and detection performance, SF-Net proposes a point-supervised temporal action localization task. This task requires only a single time point to be annotated for each action instance, significantly reducing the annotation burden. However, due to the scarcity of supervised information, existing point-supervised temporal action localization methods generally adopt a multi-instance learning paradigm, predicting action frames using frame-level category activation sequences (CAS). While these methods have achieved some progress, they suffer from two common issues: first, they struggle to model the global temporal context of the video; second, they rely heavily on heuristic post-processing (such as thresholding and score merging), resulting in hyperparameter sensitivity and insufficient robustness.

[0004] In recent years, the DETR (end-to-end object detection model based on the Transformer architecture) architecture has demonstrated strong modeling capabilities in multiple tasks. It can effectively capture long-range dependencies and naturally supports end-to-end training, avoiding tedious post-processing operations. Introducing it into the point-supervised temporal action localization task can effectively optimize the overall process. However, in the point-supervised scenario, the DETR structure cannot directly utilize sparse label training. It still faces two key challenges: how to effectively integrate point annotation information; and how to select the optimal training proposal for the decoder in the absence of complete boundary annotations. Summary of the Invention

[0005] The purpose of the present invention is to provide a method and related device for point-supervised temporal action localization to solve one or more of the above-mentioned technical problems. The technical solution disclosed in the present invention is specifically a point-supervised temporal action localization solution that integrates query reconstruction and optimal transmission. It introduces the DETR structure and proposes a solution to the problem that the DETR structure cannot directly use sparse label training. While optimizing the overall process, it ensures the accuracy of the point-supervised temporal action localization results.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a method for positioning a point-supervised temporal action, comprising: Obtain a video sequence for temporal action positioning; Based on the acquired video sequence to be temporally-localized action, temporally-localize the action using the trained point-supervised temporal action localization model to obtain a temporal action localization result; wherein the temporal action localization result includes the start and end time and type of the action; The point-supervised temporal action localization model adopts a DETR structure, and the training steps of the point-supervised temporal action localization model specifically include: Obtain a training sample data set; wherein each training sample includes a labeled video, and each action in the labeled video is provided with a point annotation label; Based on the acquired training sample data set, the point-supervised temporal action localization model is trained using a point-supervised training method. The loss is calculated according to the set loss function and the model parameters are updated. After reaching the preset convergence condition, the trained point-supervised temporal action localization model is obtained. When the point-supervised training method is used to train the point-supervised temporal action localization model, the features of each training sample are first extracted. The Transformer encoder in the DETR structure inputs the extracted features and outputs the encoded features. Based on the encoded features, the pseudo-label generator is used to obtain the optimal pseudo-label. Based on the extracted features, the query reconstruction module is used to obtain the labeled features, and clustering is performed based on the acquired labeled features to obtain the feature centers of the clusters. The feature centers of the clusters are summed with the randomly initialized queries to obtain the reconstructed queries. The Transformer decoder in the DETR structure inputs the reconstructed queries and outputs prediction proposals. The optimal pseudo-label is used as a substitute for the true label for matching training with the prediction proposals, and the parameters of the Transformer encoder and Transformer decoder are updated.

[0007] A further improvement of the technical solution of the present invention is that the step of obtaining the optimal pseudo label using the pseudo label generator based on the encoding feature specifically includes: Perform supplier-consumer modeling; where, for a given input video feature sequence ,Include annotated segments and unlabeled segments, D represents the dimension; each labeled segment is considered as a supplier, with labels; consider each unlabeled segment as a consumer, and each consumer needs a label; the conditions are ; Define the transmission cost function , indicating that the The labeled fragments are assigned to The cost of unlabeled segments; Based on the results of supplier-consumer modeling, the optimal pseudo-label is generated according to the minimum transmission cost; Among them, the transmission cost function The expression is: ; Where, Indicates the total length of the video. is the balance coefficient, represents the semantic cost, represents the physical cost; ; Where, Indicates the Clips belong to category The classification score of Indicates the The fragment belongs to snippet tag categories probability; ; Where, Indicates the The position of the clip in the video, Indicates the The position of the segment in the video.

[0008] A further improvement of the technical solution of the present invention is that, based on the results of supplier-consumer modeling, the step of generating an optimal pseudo-label according to the minimum transmission cost includes: For each unlabeled segment, the pairwise transmission cost on the shortest path between it and all labeled segments is calculated; then, each unlabeled segment is preliminarily assigned a label; the preliminarily assigned label comes from the labeled segment with the minimum transmission cost in the entire video range; After the initial assignment is completed, the corresponding pseudo labels are generated; the center of each pseudo action segment is calculated based on the pseudo labels initially assigned , this center Replace the original labeled fragment as the new reference and recalculate each fragment and its center The shortest path transmission cost between The value of is determined by counting the number of fragments with the minimum cost for each center. The calculation formula is as follows: ; ; Where, represents the number of segments assigned as the i-th segment representative category; is the indicator function. When the transmission cost is the minimum, j The categories assigned to the fragments are the same as those of i 1 if the fragments are the same, 0 otherwise.

[0009] A further improvement of the technical solution of the present invention is that in the step of calculating the loss and updating the model parameters according to the set loss function, The expression of the loss function is: ; ; ; Where, 、 、 、 is the balance coefficient of the loss function; is the loss function of the detr model part, is the number of queries, is the classification loss, is the IoU loss, for loss, To find the best prediction that matches the pseudo-label; is the total number of action clips in the video, Indicates the iii Pseudo-label categories for segments, Indicates the iii The classification scores of the segments, is the focusing factor.

[0010] A second aspect of the present invention provides a point-supervised sequential motion positioning system, comprising: A data acquisition module, used to acquire the video sequence to be temporally positioned; A positioning result acquisition module is used to perform temporal action positioning based on the acquired video sequence to be temporal action positioning using the trained point-supervised temporal action positioning model to obtain a temporal action positioning result; wherein the temporal action positioning result includes the start and end time and type of the action; The point-supervised temporal action localization model adopts a DETR structure, and the training steps of the point-supervised temporal action localization model specifically include: Obtain a training sample data set; wherein each training sample includes a labeled video, and each action in the labeled video is provided with a point annotation label; Based on the acquired training sample data set, the point-supervised temporal action localization model is trained using a point-supervised training method. The loss is calculated according to the set loss function and the model parameters are updated. After reaching the preset convergence condition, the trained point-supervised temporal action localization model is obtained. When the point-supervised training method is used to train the point-supervised temporal action localization model, the features of each training sample are first extracted. The Transformer encoder in the DETR structure inputs the extracted features and outputs the encoded features. Based on the encoded features, the pseudo-label generator is used to obtain the optimal pseudo-label. Based on the extracted features, the query reconstruction module is used to obtain the labeled features, and clustering is performed based on the acquired labeled features to obtain the feature centers of the clusters. The feature centers of the clusters are summed with the randomly initialized queries to obtain the reconstructed queries. The Transformer decoder in the DETR structure inputs the reconstructed queries and outputs prediction proposals. The optimal pseudo-label is used as a substitute for the true label for matching training with the prediction proposals, and the parameters of the Transformer encoder and Transformer decoder are updated.

[0011] A further improvement of the technical solution of the present invention is that the step of obtaining the optimal pseudo label using the pseudo label generator based on the encoding feature specifically includes: Perform supplier-consumer modeling; where, for a given input video feature sequence ,Include annotated segments and unlabeled segments, D represents the dimension; each labeled segment is considered as a supplier, with labels; each unlabeled segment is considered a consumer, and each consumer requires a label; the conditions are ;Define the transmission cost function , indicating that the The labeled fragments are assigned to The cost of unlabeled segments; Based on the results of supplier-consumer modeling, the optimal pseudo-label is generated according to the minimum transmission cost; Among them, the transmission cost function The expression is: ; Where, Indicates the total length of the video. is the balance coefficient, represents the semantic cost, represents the physical cost; ; Where, Indicates the Clips belong to category The classification score of Indicates the The fragment belongs to snippet tag categories probability; ; Where, Indicates the The position of the clip in the video, Indicates the The position of the segment in the video.

[0012] A further improvement of the technical solution of the present invention is that, based on the results of supplier-consumer modeling, the step of generating an optimal pseudo-label according to the minimum transmission cost includes: For each unlabeled segment, the pairwise transmission cost on the shortest path between it and all labeled segments is calculated; then, each unlabeled segment is preliminarily assigned a label; the preliminarily assigned label comes from the labeled segment with the minimum transmission cost in the entire video range; After the initial assignment is completed, the corresponding pseudo labels are generated; the center of each pseudo action segment is calculated based on the pseudo labels initially assigned , this center Replace the original labeled fragment as the new reference and recalculate each fragment and its center The shortest path transmission cost between them; Finally, each action category is assigned the number of fragments The value of is determined by counting the number of fragments with the minimum cost for each center. The calculation formula is as follows: ; ; Where, represents the number of segments assigned as the i-th segment representative category; is the indicator function. When the transmission cost is the minimum, j The categories assigned to the fragments are the same as those of i1 if the fragments are the same, 0 otherwise.

[0013] A further improvement of the technical solution of the present invention is that in the step of calculating the loss and updating the model parameters according to the set loss function, The expression of the loss function is: ; ; ; Where, 、 、 、 is the balance coefficient of the loss function; is the loss function of the detr model, The loss function for the pseudo-label generation part, is the number of queries, is the classification loss, is the IoU loss, for loss, To find the best prediction that matches the pseudo-label; is the total number of action clips in the video, Indicates the iii Pseudo-label categories for segments, Indicates the iii The classification scores of the segments, is the focusing factor.

[0014] In a third aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the point-supervised temporal action positioning method as described in any one of the first aspects of the present invention is implemented.

[0015] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the point-supervised temporal action positioning method as described in any one of the first aspects of the present invention.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The present invention discloses a method for point-supervised temporal action localization, specifically a method based on the DETR structure. This method is the first to introduce the DETR structure into point-supervised tasks to address the challenge of sparse supervision. The present invention's technical solution specifically uses a trained point-supervised temporal action localization model to obtain action localization results. The point-supervised temporal action localization model adopts the DETR structure and introduces query reconstruction and optimal transmission mechanisms during training to improve the model's localization capabilities under sparse supervision. Specifically, the present invention designs a query reconstruction mechanism that encodes single-frame annotation information into a query with semantic consensus, guiding the model to focus on key action-related segments. Furthermore, the present invention proposes a pseudo-label generation technique that expands single-frame annotations into segment-level pseudo-labels through an optimal transmission strategy. The method then selects the optimal decoder proposal for action prediction based on a selected algorithm (e.g., the Hungarian algorithm). Furthermore, experimental results demonstrate that the present invention significantly outperforms existing methods on multiple standard datasets, demonstrating enhanced robustness and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below; obviously, the drawings described below are some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 1 is a flow chart of a method for positioning a point-supervised sequential action in an embodiment of the present invention; Figure 2 1 is a schematic diagram of the training process of a point-supervised temporal action localization model in an embodiment of the present invention; Figure 3 Schematic diagram of the overall architecture of the point-supervised temporal action localization model in an embodiment of the present invention; Figure 4 Schematic diagram of a point-supervised sequential motion positioning system in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention; it is obvious that the described embodiments and technical solutions are only part of the embodiments of the present invention, not all of the embodiments.

[0020] All other embodiments obtained by persons of ordinary skill in the art based on the technical solutions disclosed in the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatuses.

[0021] See also Figure 1 The embodiment of the present invention provides a method for positioning a point-supervised temporal action, which specifically includes the following steps: Step 1: Obtain the video sequence for temporal action positioning; Step 2: Based on the video sequence to be temporally positioned obtained in step 1, the trained point-supervised temporal action positioning model is used to perform temporal action positioning to obtain a temporal action positioning result; wherein the temporal action positioning result includes the start and end time and type of the action; Among them, the point-supervised temporal action localization model adopts a DETR structure, which includes a backbone network, a Transformer encoder, a Transformer decoder, and a prediction head. For further explanation, the backbone network can use a convolutional neural network (such as ResNet), which is responsible for extracting features from video frames; the Transformer encoder is used to input the features extracted by the backbone network, learn position embedding, capture global contextual relationships, and output encoded features; the Transformer decoder is used to input encoded features and queries, and generate target predictions through self-attention mechanisms and mutual attention mechanisms. In temporal action localization, these queries are used to predict the position and category of action clips; the output of the decoder is processed by the prediction head to generate the category and bounding box of the action clip (in temporal action localization, the bounding box represents the start and end timestamps of the action); In a specific solution of an embodiment of the present invention, the training step of the point-supervised temporal action localization model specifically includes: Obtain a training sample dataset suitable for point-supervised training; wherein each training sample includes a labeled video, and each action in the labeled video is set with a point annotation label; Based on the acquired training sample data set, the point-supervised temporal action localization model is trained using a point-supervised training method. The loss is calculated according to the set loss function and the model parameters are updated. After reaching the preset convergence condition, a trained point-supervised temporal action localization model is obtained. When the point-supervised training method is used to train the point-supervised temporal action localization model, the features of each training sample are first extracted. The Transformer encoder inputs the extracted features and outputs the encoded features. The pseudo-label generator inputs the encoded features to obtain the classification score, and calculates the optimal transmission strategy based on the classification score to obtain the optimal pseudo-label. The query reconstruction module inputs the extracted features and obtains labeled features based on the point annotation labels of each action in each training sample. Clustering is performed based on the acquired labeled features to obtain the feature centers of the clusters. The feature centers of the clusters are summed with the randomly initialized query to obtain the reconstructed query. The Transformer decoder inputs the reconstructed query and the encoded features and outputs a prediction proposal. The optimal pseudo-label is used as a substitute for the true label and matched with the prediction proposal for training to update the parameters of the Transformer encoder and Transformer decoder.

[0022] In the technical solution disclosed in the embodiment of the present invention, the reconstruction query strategy integrates point annotation information into the query by constructing a semantic consensus query reinforcement model, increases the focus on action-related areas, thereby overcoming the semantic bias problem caused by the random distribution of point labels and improving the overall recognition effect. In addition, in the pseudo-label generation strategy, label propagation is modeled as an optimal transmission problem, which effectively realizes the label extension from point to segment, can provide a more accurate supervision signal for the decoder, and optimizes the end-to-end DETR architecture performance. In summary, the embodiment of the present invention provides a point-supervised temporal action localization solution that integrates query reconstruction and optimal transmission. It introduces the DETR structure and proposes a solution to the problem that the DETR structure cannot be directly trained with sparse labels. While optimizing the overall process, it ensures the accuracy of the point-supervised temporal action localization results.

[0023] See also Figures 1 to 3 In a specific embodiment of the present invention, a point-supervised temporal action positioning method based on a DETR structure is provided, comprising the following steps: Step 1: Extract the video using the pre-trained model. The specific steps include: Step 1.1, given an uncut video sequence , define the video clip containing the action as , Represents the video frames contained in the video sequence, and Represent the start and end time of the video segment respectively; the located video segment can also be expressed as a time span ,in represents the center of the video clip, Indicates the duration of the video clip.

[0024] It needs further explanation that in the point supervision setting, there is a random frame of annotation for each action, which is then used as the input of the video two-stream network.

[0025] Step 1.2: In order to obtain the video sequence features respectively, the original video sequence is encoded to generate a video representation; specifically, the embodiment of the present invention uses the I3D network to process each video segment and extract Dimensional video features .

[0026] Step 2: Given a video feature sequence (i.e., training sample) containing single-frame annotated data, the point-supervised temporal action localization model uses the DETR framework to implement the point-supervised video temporal localization process. The point annotation information is used to effectively initialize the moment query in the DETR structure to improve the model's ability to focus on the action area. For example, the specific steps of the method are as follows: In step 2.1, all video features with point annotations are extracted from the training set. Since actions of the same category usually have semantic similarity, these features are first grouped by category.

[0027] In step 2.2, based on the grouping results, within each category, the features are clustered using the K-means clustering algorithm to obtain more representative semantic feature centers. These cluster centers are used to capture the diversity and commonality within the category. The specific implementation process is as follows: ; in, Indicates the first The characteristics of the marker, represents the number of clusters, Represents a partition of the cluster.

[0028] In step 2.3, the feature centers obtained by clustering are used as semantic consensus queries to initialize the moment query in the DETR decoder.

[0029] Step 3: To alleviate the problem that the DETR structure under point supervision cannot be directly trained using sparse labels, the present invention uses an optimal transmission strategy to expand segment-level pseudo labels from point labels for training decoder proposals. For example, the specific method steps are as follows: Step 3.1, given the feature sequence of the input video , which contains annotated segments and unlabeled segments; then perform supplier-consumer modeling, treating each labeled segment as a supplier, which has labels; consider each unlabeled segment as a consumer, and each consumer needs a label; satisfy Supplier-consumer transmission cost calculation: define the transmission cost function , indicating that the The labeled fragments are assigned to Based on the cost function, the optimal pseudo-label is generated according to the minimum transmission cost, and the Sinkhorn-Knopp iterative algorithm can be used to solve the optimal transmission plan. , to achieve soft allocation.

[0030] In the pseudo-label generation of the embodiment of the present invention, point labels are assigned to unlabeled segments based on the minimum transmission cost, and the final pseudo-labels are generated for training decoder proposals. The specific implementation steps are as follows: In step 3.1.1, in order to characterize the possibility of different segments belonging to the same action, the present invention defines two types of cost graphs: (1) Semantic cost map Indicates the confidence that the segment and the true label belong to the same category. The present invention introduces a classification layer in the encoder to generate Semantics on categories , No. The labeled fragments and The semantic cost between unlabeled segments is defined as: ; Where, Indicates that the i-th segment belongs to the label category The probability of Indicates the The fragment belongs to snippet tag categories probability.

[0031] (2) Physical cost map reflects the time interval between segments, The tagged segments and The physical cost of an unlabeled segment is defined as: ; Where, Indicates the The position of the clip in the video, Indicates the Since action clips have a limited duration, when the distance between clips increases, the probability that they belong to the same category decreases, resulting in an increase in the cost.

[0032] Step 3.1.2, combine the semantic cost and physical cost graphs, build the transmission graph model and calculate the total cost, and construct the fragment into a 2-connected planar graph , where the vertex Represents all segments, edge sets Contains the connection relationship between adjacent segments on the time axis, and the length of each edge is composed of the weighted semantic cost and physical cost. The final transmission cost Defined as Point labels to The total cost of the path of unlabeled segments on the graph is: ; Where, Indicates the total length of the video. is the balance coefficient.

[0033] Step 3.2: Dynamically determine the number of pseudo labels that should be generated for each labeled segment , specifically including the following two key steps: Step 3.2.1: For each unlabeled segment, calculate the shortest path between it and all labeled segments. Then, each unlabeled segment is initially assigned a label, which comes from the labeled segment with the minimum transmission cost in the entire video. Since the positions of these labeled segments within the action are random and may be distributed at different stages of the segment, there is a certain degree of randomness in the initial assigned positions.

[0034] Step 3.2.2: Once the initial assignment is completed, the system can generate the corresponding pseudo-action labels. Then, the center point of each pseudo-action segment is calculated based on the pseudo-labels assigned initially. , this center will replace the original labeled segment as the new reference; then recalculate each fragment relative to its center The shortest path transmission cost between them; Finally, each action category is assigned the number of fragments The value of is determined by counting the number of fragments with the minimum cost for each center. The calculation formula is as follows: ; Where, represents the number of segments assigned to the i-th segment representative category, is the indicator function, which is 1 when the transmission cost is minimized and the category assigned to the j-th segment is the same as that of the i-th segment, and 0 otherwise.

[0035] This iterative calculation process continuously optimizes the fragment number estimate , thereby achieving a more accurate assignment of pseudo labels. Through continuous updating, this strategy can more accurately match unlabeled segments with corresponding point labels.

[0036] Step 4: Network training optimization, including: Step 4.1: First, assign the generated pseudo labels and optimal segments, and the predicted action set is recorded as , , Respectively represent the start and end time of the predicted action segment, represents the corresponding classification confidence, Represents the predicted action category. Pseudo label set Contains a filled "no action" placeholder, denoted as The matching cost is defined as follows: ; ; ; in, is the predicted probability of the ii-th predicted segment, is the category of the pseudo label of the ii-th predicted segment.

[0037] Classification loss Using Focal Loss, and They are temporal IoU loss and L1 distance loss respectively. The goal of bipartite graph matching between two sets is to find a permutation with minimum cost : ; in, 、 and are the weights of the three losses, with values ​​of 2, 2, and 5 respectively.

[0038] Step 4.2, get the best match Finally, the optimization objectives of DETR training are as follows: ; in, is an indicator function, which is 1 when the category is not “no action”. 、 and are the weights of the corresponding loss items, set to 2, 2, and 5 respectively.

[0039] In the process of pseudo-label generation, to facilitate model training, we also use Focal Loss: ; in, represents the total number of action clips in the video, is the focusing factor, set to 2. and Respectively represent Semantic scores and pseudo labels for video clips.

[0040] In step 4.3, the final training goal of the model is: ; in and are the weights of the two losses, which are set to 1 and 0.5 respectively.

[0041] In the specific embodiment of the present invention, the proposed method is evaluated by calculating the mean average precision under multiple different IoU (Intersection over Union) thresholds. For the THUMOS14 dataset, the IoU threshold is set to ; On the ActivityNet-v1.3 dataset, the results are at the IoU threshold Report to the The average mAP is calculated within the range.

[0042] Table 1. Comparison results of different methods on the THUMOS14 dataset

[0043] Table 1 shows the experimental results of different methods on the public dataset THUMOS14 used to evaluate video temporal action localization. It can be seen from the results that the point-supervised temporal action localization method based on integrated query reconstruction and optimal transmission proposed in the embodiment of the present invention has obvious advantages.

[0044] Table 2. Comparison results of different methods on the ActivityNet-v1.3 dataset

[0045] Table 2 shows the results on the ActivityNet-v1.3 dataset, which also proves the effectiveness of the present invention and has good generalization performance on different datasets.

[0046] Table 3. Results of ablation experiments on different modules on the THUMOS14 dataset

[0047] Table 3 is an ablation experiment on the THUMOS14 dataset for the impact of different modules proposed by the method of the present invention on the results. The first row shows the results of converting the fully supervised DETR model to a point-supervised DETR model, where the segment used for Hungarian matching is a segment that expands the point labels to both ends in a certain proportion. The second row shows the effect of combining the optimal transmission method. Generate pseudo labels and use the pseudo labels generated by optimal transmission as substitutes for the true labels. The third row shows the results obtained by adding the query reconstruction module. The experiments show the effectiveness of our proposed method for point-supervised temporal action localization.

[0048] In summary, point-supervised temporal action localization aims to accurately identify complete action intervals when each action is only annotated with a single frame; similar to most weakly supervised learning tasks, its main challenge is that the scarcity of annotation information makes it difficult to obtain the global temporal context, and the pseudo-label generation process is unstable, which limits the performance of the model. Existing methods generally rely on multi-instance learning mechanisms, but such methods rely on heuristic post-processing, which makes it difficult to achieve end-to-end optimization, and are easily affected by the quality of pseudo-labels and unstable performance. The embodiment of the present invention specifically provides a point-supervised temporal action localization method that integrates query reconstruction and optimal transmission. It is an end-to-end point-supervised positioning framework that integrates a query reconstruction mechanism and an optimal transmission algorithm. Specifically, the present invention designs a set of semantic consensus queries for effectively encoding point label information and guiding the model to focus on key segments related to the action; on this basis, an optimal transmission mechanism is introduced to generate high-quality pseudo-labels, and the optimal candidate intervals are screened for training based on the Hungarian algorithm, which significantly improves the positioning accuracy and training stability. A large number of experiments have shown that the method of the present invention significantly outperforms the existing MIL method on both THUMOS14 and ActivityNet-v1.3 benchmark datasets, and achieves more stable and accurate temporal action localization performance in point-supervised scenarios. In summary, compared with other methods, the method of the embodiment of the present invention has achieved a significant improvement in the task of temporal action localization in point-supervised videos.

[0049] The following are device embodiments of the present invention, which can be used to perform the method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0050] See also Figure 4 In an embodiment of the present invention, a point supervision sequential action positioning system is provided, comprising: A data acquisition module, used to acquire the video sequence to be temporally positioned; A positioning result acquisition module is used to perform temporal action positioning based on the acquired video sequence to be temporal action positioning using the trained point-supervised temporal action positioning model to obtain a temporal action positioning result; wherein the temporal action positioning result includes the start and end time and type of the action; The point-supervised temporal action localization model adopts a DETR structure, and the training steps of the point-supervised temporal action localization model specifically include: Obtain a training sample data set; wherein each training sample includes a labeled video, and each action in the labeled video is provided with a point annotation label; Based on the acquired training sample data set, the point-supervised temporal action localization model is trained using a point-supervised training method. The loss is calculated according to the set loss function and the model parameters are updated. After reaching the preset convergence condition, the trained point-supervised temporal action localization model is obtained. When the point-supervised training method is used to train the point-supervised temporal action localization model, the features of each training sample are first extracted. The Transformer encoder in the DETR structure inputs the extracted features and outputs the encoded features. Based on the encoded features, the pseudo-label generator is used to obtain the optimal pseudo-label. Based on the extracted features, the query reconstruction module is used to obtain the labeled features, and clustering is performed based on the acquired labeled features to obtain the feature centers of the clusters. The feature centers of the clusters are summed with the randomly initialized queries to obtain the reconstructed queries. The Transformer decoder in the DETR structure inputs the reconstructed queries and outputs prediction proposals. The optimal pseudo-label is used as a substitute for the true label for matching training with the prediction proposals, and the parameters of the Transformer encoder and Transformer decoder are updated.

[0051] In one embodiment of the present invention, a computer device is provided, comprising a processor and a memory, wherein the memory is configured to store a computer program, the computer program including program instructions, and the processor is configured to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or may be another general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal and is suitable for implementing one or more instructions, specifically loading and executing one or more instructions in a computer storage medium to implement a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used to perform operations in a point supervision sequential action positioning method.

[0052] In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory). The computer-readable storage medium is a memory device in a computer device, used to store programs and data. It is understood that the computer-readable storage medium herein may include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for being loaded and executed by a processor. These instructions may be one or more computer programs (including program code). It should be noted that the computer-readable storage medium herein may be a high-speed random access memory (RAM) or a non-volatile memory, such as at least one disk drive. The processor may load and execute the one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the point-supervisory time-series action location method described in the above-mentioned embodiment.

[0053] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, optical storage, etc.) containing computer-usable program code.

[0054] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0055] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0056] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0057] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A point-supervised temporal action localization method, characterized in that: include: Obtain a video sequence for temporal action positioning; Based on the acquired video sequence to be temporally-localized action, temporally-localize the action using the trained point-supervised temporal action localization model to obtain a temporal action localization result; wherein the temporal action localization result includes the start and end time and type of the action; The point-supervised temporal action localization model adopts a DETR structure, and the training steps of the point-supervised temporal action localization model specifically include: Obtain a training sample data set; wherein each training sample includes a labeled video, and each action in the labeled video is provided with a point annotation label; Based on the acquired training sample data set, the point-supervised temporal action localization model is trained using a point-supervised training method. The loss is calculated according to the set loss function and the model parameters are updated. After reaching the preset convergence condition, the trained point-supervised temporal action localization model is obtained. When the point-supervised training method is used to train the point-supervised temporal action localization model, the features of each training sample are first extracted. The Transformer encoder in the DETR structure inputs the extracted features and outputs the encoded features. Based on the encoded features, the pseudo-label generator is used to obtain the optimal pseudo-label. Based on the extracted features, the query reconstruction module is used to obtain the labeled features, and clustering is performed based on the acquired labeled features to obtain the feature centers of the clusters. The feature centers of the clusters are summed with the randomly initialized queries to obtain the reconstructed queries. The Transformer decoder in the DETR structure inputs the reconstructed queries and outputs prediction proposals. The optimal pseudo-label is used as a substitute for the true label for matching training with the prediction proposals, and the parameters of the Transformer encoder and Transformer decoder are updated.

2. A point-supervised temporal action positioning method according to claim 1, characterized in that: Based on the encoding features, the steps of using the pseudo-label generator to obtain the optimal pseudo-label include: Perform supplier-consumer modeling; where, for a given input video feature sequence ,Include annotated segments and unlabeled segments, D represents the dimension; each labeled segment is considered as a supplier, with labels; each unlabeled segment is considered a consumer, and each consumer requires a label; the conditions are ;Define the transmission cost function , indicating that the The labeled fragments are assigned to The cost of unlabeled segments; Based on the results of supplier-consumer modeling, the optimal pseudo-label is generated according to the minimum transmission cost; Among them, the transmission cost function The expression is: ； Where, Indicates the total length of the video. is the balance coefficient, represents the semantic cost, Indicates physical cost; ； Where, Indicates the Clips belong to category The classification score of Indicates the The fragment belongs to snippet tag categories probability; ； Where, Indicates the The position of the clip in the video, Indicates the The position of the segment in the video.

3. A point-supervised temporal action positioning method according to claim 2, characterized in that: Based on the results of supplier-consumer modeling, the steps to generate the optimal pseudo-label according to the minimum transmission cost include: For each unlabeled segment, the pairwise transmission cost on the shortest path between it and all labeled segments is calculated; then, each unlabeled segment is preliminarily assigned a label; the preliminarily assigned label comes from the labeled segment with the minimum transmission cost in the entire video range; After the initial assignment is completed, the corresponding pseudo labels are generated; the center of each pseudo action segment is calculated based on the pseudo labels initially assigned , this center Replace the original labeled fragment as the new reference and recalculate each fragment and its center The shortest path transmission cost between them; Finally, each action category is assigned the number of fragments The value of is determined by counting the number of fragments with the minimum cost for each center. The calculation formula is as follows: ； ； Where, represents the number of segments assigned as the i-th segment representative category; is the indicator function. When the transmission cost is the minimum, j The categories assigned to the fragments are the same as those of i 1 if the fragments are the same, 0 otherwise.

4. A point-supervised temporal action positioning method according to claim 1, characterized in that: In the step of calculating the loss and updating the model parameters according to the set loss function, The expression of the loss function is: ； ； ； Where, 、 、 、 is the balance coefficient of the loss function; is the loss function of the detr model part, is the number of queries, is the classification loss, is the IoU loss, for loss, To find the best prediction that matches the pseudo-label; is the total number of action clips in the video, Indicates the iii Pseudo-label categories for segments, Indicates the iii The classification scores of the segments, is the focusing factor.

5. A point-supervised sequential action positioning system, characterized in that: include: A data acquisition module, used to acquire the video sequence to be temporally positioned; A positioning result acquisition module is used to perform temporal action positioning based on the acquired video sequence to be temporal action positioning using the trained point-supervised temporal action positioning model to obtain a temporal action positioning result; wherein the temporal action positioning result includes the start and end time and type of the action; The point-supervised temporal action localization model adopts a DETR structure, and the training steps of the point-supervised temporal action localization model specifically include: Obtain a training sample data set; wherein each training sample includes a labeled video, and each action in the labeled video is provided with a point annotation label; Based on the acquired training sample data set, the point-supervised temporal action localization model is trained using a point-supervised training method. The loss is calculated according to the set loss function and the model parameters are updated. After reaching the preset convergence condition, the trained point-supervised temporal action localization model is obtained. When the point-supervised training method is used to train the point-supervised temporal action localization model, the features of each training sample are first extracted. The Transformer encoder in the DETR structure inputs the extracted features and outputs the encoded features. Based on the encoded features, the pseudo-label generator is used to obtain the optimal pseudo-label. Based on the extracted features, the query reconstruction module is used to obtain the labeled features, and clustering is performed based on the acquired labeled features to obtain the feature centers of the clusters. The feature centers of the clusters are summed with the randomly initialized queries to obtain the reconstructed queries. The Transformer decoder in the DETR structure inputs the reconstructed queries and outputs prediction proposals. The optimal pseudo-label is used as a substitute for the true label for matching training with the prediction proposals, and the parameters of the Transformer encoder and Transformer decoder are updated.

6. A point-supervised sequential motion positioning system according to claim 5, characterized in that: Based on the encoding features, the steps of using the pseudo-label generator to obtain the optimal pseudo-label include: Perform supplier-consumer modeling; where, for a given input video feature sequence ,Include annotated segments and unlabeled segments, D represents the dimension; each labeled segment is considered as a supplier, with labels; consider each unlabeled segment as a consumer, and each consumer needs a label; the conditions are ; Define the transmission cost function , indicating that the The labeled fragments are assigned to The cost of unlabeled segments; Based on the results of supplier-consumer modeling, the optimal pseudo-label is generated according to the minimum transmission cost; Among them, the transmission cost function The expression is: ； Where, Indicates the total length of the video. is the balance coefficient, represents the semantic cost, Indicates physical cost; ； Where, Indicates the Clips belong to category The classification score of Indicates the The fragment belongs to snippet tag categories probability; ； Where, Indicates the The position of the clip in the video, Indicates the The position of the segment in the video.

7. A point-supervised sequential motion positioning system according to claim 6, characterized in that: Based on the results of supplier-consumer modeling, the steps to generate the optimal pseudo-label according to the minimum transmission cost include: For each unlabeled segment, the pairwise transmission cost on the shortest path between it and all labeled segments is calculated; then, each unlabeled segment is preliminarily assigned a label; the preliminarily assigned label comes from the labeled segment with the minimum transmission cost in the entire video range; After the initial assignment is completed, the corresponding pseudo labels are generated; the center of each pseudo action segment is calculated based on the pseudo labels initially assigned , this center Replace the original labeled fragment as the new reference and recalculate each fragment and its center The shortest path transmission cost between them; Finally, each action category is assigned the number of fragments The value of is determined by counting the number of fragments with the minimum cost for each center. The calculation formula is as follows: ； ； Where, represents the number of segments assigned as the i-th segment representative category; is the indicator function. When the transmission cost is the minimum, j The categories assigned to the fragments are the same as those of i 1 if the fragments are the same, 0 otherwise.

8. A point-supervised sequential motion positioning system according to claim 5, characterized in that: In the step of calculating the loss and updating the model parameters according to the set loss function, The expression of the loss function is: ； ； ； Where, 、 、 、 is the balance coefficient of the loss function; is the loss function of the detr model part, The loss function for the pseudo-label generation part, is the number of queries, is the classification loss, is the IoU loss, for loss, To find the best prediction that matches the pseudo-label; is the total number of action clips in the video, Indicates the iii Pseudo-label categories for segments, Indicates the iii The classification scores of the segments, is the focusing factor.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the point-supervised temporal action positioning method according to any one of claims 1 to 4 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the point-supervised temporal action positioning method according to any one of claims 1 to 4 is implemented.

Images