A fully supervised temporal action localization method, system, device and medium

Abstract

This invention discloses a fully supervised temporal action localization method, system, device, and medium, belonging to the field of computer vision and pattern recognition technology. The fully supervised temporal action localization method includes the following steps: acquiring the encoded video feature sequence of the action to be localized, and extracting features from the encoded video feature sequence using a backbone network; based on the obtained features of the encoded video feature sequence, using a trained temporal action localization model for prediction to obtain classification and localization results. Specifically, the fully supervised temporal action localization method disclosed in this invention is a joint optimization-based method. It proposes an IoU-aware objective function and an alignment weight allocation strategy to align the classification head and the localization head, effectively improving the performance of the fully supervised temporal action localization detector.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and pattern recognition technology, and specifically relates to a fully supervised temporal action localization method, system, device and medium. Background Technology

[0002] Temporal action localization aims to automatically retrieve the start time, end time, and predicted category of actions in a video; most existing temporal action localization methods rely on dense action labels in the training set videos.

[0003] Currently, existing TAL (Temporal Action Localization) methods typically generate a set of redundant anchor points or proposed scores with classification, and then deploy NMS (Non-Maximum Suppression) to remove duplicate results. The localization and classification heads are usually trained independently, but are used together during testing. In this existing approach, the classification score cannot always reflect the localization quality of the temporal action, which greatly impairs the detection performance and reduces the performance of the temporal action localization detector.

[0004] In summary, establishing the correct relationship between the classification head and the positioning head is crucial, and there is an urgent need for a new fully supervised temporal action positioning method, system, device, and medium based on joint optimization. Summary of the Invention

[0005] The purpose of this invention is to provide a fully supervised temporal action localization method, system, device, and medium to solve one or more of the aforementioned technical problems. Specifically, the fully supervised temporal action localization method disclosed in this invention is a joint optimization-based method. It utilizes a joint optimization approach to propose an IoU-aware objective function and an alignment weight allocation strategy to align the classification head and the localization head, effectively improving the performance of the fully supervised temporal action localization detector.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] The first aspect of this invention provides a fully supervised temporal action localization method, comprising the following steps:

[0008] Obtain the encoded video feature sequence for the action to be located in time sequence, and use the backbone network to extract the features of the encoded video feature sequence;

[0009] Based on the features of the obtained encoded video feature sequence, a trained temporal action localization model is used to make predictions and obtain classification and localization results.

[0010] During training, the trained temporal action localization model performs gradient backpropagation based on a weighted loss function.

[0011] The expression for the loss function is,

[0012]

[0013] In the formula, p∈[0,1] is the predicted classification score, which is obtained by passing the output result through a sigmoid function; t is the one-hot format corresponding to the category label, t=1 means that the current feature is the main feature, t=0 means that the current feature is the background feature; q represents the one-hot label after the IoU score is replaced;

[0014] Weighting is achieved by multiplying the sample weights by the corresponding loss function to align the weights. The sample weights ω... pos The expression is,

[0015] ω pos = βω ;ω=IoU*s;

[0016] In the formula, β is a hyperparameter used to adjust the weight allocation magnitude; IoU is the intersection-union ratio of the predicted start and end boundary values ​​generated by the localizer during the current training process to the true values; and s represents the classification score generated by the classifier.

[0017] A further improvement of the method of the present invention is that the encoded video feature sequence includes the RGB information of each video segment and the optical flow information between adjacent segments.

[0018] A further improvement of the method of the present invention is that the backbone network is an I3D, TSP or SLOWFAST network.

[0019] A further improvement to the method of the present invention is that the temporal action localization model includes:

[0020] The feature extraction module is used to extract features from the input encoded video feature sequence and output the extracted features.

[0021] The sample selection module is used to input the features output by the feature input module and assign labels to them to obtain labeled features.

[0022] The classification and localization fusion module includes a classification head and a localization head, which are used to classify and locate labeled features as input, and output classification results and localization results.

[0023] A further improvement of the method of the present invention is that the feature extraction module is a CNN network or a Tranformer model.

[0024] A further improvement of the method of the present invention is that the training steps of the trained temporal action localization model include:

[0025] Obtain the training sample dataset; each training sample includes features of the encoded video feature sequence and instance-level data annotations for each action;

[0026] During training, for selected training samples, the features of the encoded video feature sequence in the selected training samples are input into the temporal action localization model for prediction, and the start time, end time and action type label prediction value of each action are obtained; gradient backpropagation is performed based on the weighted loss function, and the trained temporal action localization model is obtained after the preset convergence condition is reached.

[0027] A second aspect of the present invention provides a fully supervised temporal action localization system, comprising:

[0028] The feature acquisition module is used to acquire the encoded video feature sequence for the timing action localization and to extract the features of the encoded video feature sequence using the backbone network.

[0029] The result acquisition module is used to predict the classification and localization results based on the features of the obtained encoded video feature sequence and the trained temporal action localization model.

[0030] During training, the trained temporal action localization model performs gradient backpropagation based on a weighted loss function.

[0031] The expression for the loss function is,

[0032]

[0033] In the formula, p∈[0,1] is the predicted classification score, which is obtained by passing the output result through a sigmoid function; t is the one-hot format corresponding to the category label, t=1 means that the current feature is the main feature, t=0 means that the current feature is the background feature; q represents the one-hot label after the IoU score is replaced;

[0034] Weighting is achieved by multiplying the sample weights by the corresponding loss function to align the weights. The sample weights ω... pos The expression is,

[0035] ω pos = βω ;ω=IoU*s;

[0036] In the formula, β is a hyperparameter used to adjust the weight allocation magnitude; IoU is the intersection-union ratio of the predicted start and end boundary values ​​generated by the localizer during the current training process to the true values; and s represents the classification score generated by the classifier.

[0037] A third aspect of the present invention provides an electronic device comprising:

[0038] At least one processor; and,

[0039] A memory communicatively connected to the at least one processor; wherein,

[0040] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the fully supervised timing action localization method as described in any of the first aspects of the present invention.

[0041] The fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the fully supervised timing action localization method according to any one of the first aspects of the present invention.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] To address an unexplored problem in temporal action localization tasks—namely, how to resolve the performance degradation caused by the mismatch between the two subtasks of temporal action localization—this invention provides a fully supervised temporal action localization method, specifically a joint optimization-based method. This method utilizes an IoU-aware objective function and an alignment weight allocation strategy based on joint optimization to align the classification head and the localization head, effectively improving the performance of the fully supervised temporal action localization detector. Specifically, the loss function of this invention establishes a close relationship between the classification and localization tasks. To merge the detection heads of the two subtasks, this invention uses the IoU scores of positive samples instead of traditional one-hot labels for learning. Simultaneously, during training, more important anchor points are learned, and a new weighting paradigm (i.e., alignment weighting) is explored to specify suggestions with higher classification and localization scores. This joint training approach effectively mitigates the mismatch problem between the two subtasks in temporal action localization and amplifies the ability to utilize key samples, ultimately effectively improving the performance of the fully supervised temporal action localization detector. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art are briefly introduced below; obviously, the drawings described below are some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0045] Figure 1This is a flowchart illustrating a fully supervised temporal action localization method provided in an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of the architecture of the temporal action localization model in an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram illustrating the research motivation in an embodiment of the present invention;

[0048] Figure 4 This is a schematic diagram comparing the three-dimensional functions of the classification head objective function in an embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram of some detection results on the Thumos14 dataset in an embodiment of the present invention;

[0050] Figure 6 This is a schematic diagram of the overall results statistically analyzed on the Thumo14 dataset in an embodiment of the present invention; wherein, Figure 6 (a) is a schematic diagram illustrating the error rate statistics of the baseline. Figure 6 (b) is a schematic diagram illustrating the error rate statistics of the method in the embodiment of the present invention;

[0051] Figure 7 This is a schematic diagram of a fully supervised timing action positioning system provided in an embodiment of the present invention. Detailed Implementation

[0052] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0053] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0054] The present invention will now be described in further detail with reference to the accompanying drawings:

[0055] Please see Figure 1 The present invention provides a fully supervised temporal action localization method, comprising the following steps:

[0056] Step 1: Obtain the encoded video feature sequence for the action to be located in time sequence; Based on the obtained encoded video feature sequence, use a backbone network to extract the features of the encoded video feature sequence.

[0057] Step 2: Based on the features obtained in Step 1, use the trained temporal action localization model to make predictions and obtain classification and localization results.

[0058] During training, the trained temporal action localization model performs gradient backpropagation based on a weighted loss function.

[0059] The expression for the loss function is,

[0060]

[0061] In the formula, p∈[0,1] is the predicted classification score, which is obtained by passing the output result through a sigmoid function; t is the one-hot format corresponding to the category label, t=1 means that the current feature is the main feature, t=0 means that the current feature is the background feature; q represents the one-hot label after the IoU score is replaced;

[0062] Weighting is achieved by multiplying the sample weights by the corresponding loss function to align the weights. The sample weights ω... pos The expression is,

[0063] ω pos = βω ;ω=IoU*s;

[0064] In the formula, β is a hyperparameter used to adjust the weight allocation magnitude; IoU is the intersection-union ratio of the predicted start and end boundary values ​​generated by the localizer during the current training process to the true values; and s represents the classification score generated by the classifier.

[0065] In this invention, a novel joint optimization method is proposed to establish a correct connection between the classification subtask and the localization subtask in temporal action localization. This results in a classification score obtained from the localization information perception, which is then used in the final non-maximum suppression stage. This eliminates errors during the training and testing phases, ultimately leading to correct test results. Due to the different nature of the classification and localization heads during training, the feature distributions of localization and classification are not entirely the same. Therefore, an alignment weight is designed to filter the overall samples, removing those samples that would have a significant negative impact on the model.

[0066] In step 1 of this embodiment of the invention, the step of obtaining the encoded video feature sequence for temporal action localization specifically includes:

[0067] The original, unedited video sequence for timing action positioning is decomposed into several video segments, each containing consecutive frames of a fixed length from the original video sequence.

[0068] Each video segment is encoded using a pre-trained feature extractor to obtain an encoded video feature sequence; wherein the encoded video feature sequence contains the RGB information of each video segment and the optical flow information between adjacent segments. For example, the encoded video feature sequence is represented as follows: Among them, f t Let t represent the feature vector of the t-th video segment, and T represent the total number of video segments.

[0069] In step 1 of this embodiment of the invention, the backbone network can be I3D, TSP or SLOWFAST network.

[0070] In step 2 of this embodiment of the invention, the architecture of the temporal action localization model includes:

[0071] The feature extraction module, which can be a CNN network or a Tranformer model, is used to extract features from the input encoded video feature sequence and output the extracted features.

[0072] The sample selection module is used to input the features output by the feature input module and assign labels to them to obtain labeled features.

[0073] The classification and localization fusion module includes a classification head and a localization head, which are used to classify and locate labeled features as input, and output classification results and localization results.

[0074] In step 2 of this embodiment of the invention, the training steps of the trained temporal action localization model include:

[0075] Obtain the training sample dataset; each training sample includes: features of the encoded video feature sequence, and instance-level data annotations for each action (i.e., start time, end time, and action type label for each action); specifically, for example, the annotation format of the temporal action localization dataset is as follows: Where N represents the number of videos in the dataset; given a set of action instances in a labeled dataset, the annotation for each action can be represented as (t s , t e c), where t s t ec and represent the start time, end time, and action category of the action instance, respectively;

[0076] During training, for selected training samples, the features of the encoded video feature sequence in the selected training samples are input into the temporal action localization model for prediction, and the start time, end time and action type label prediction value of each action are obtained; gradient backpropagation is performed based on the weighted loss function, and the trained temporal action localization model is obtained when the preset convergence condition is met.

[0077] In this embodiment of the invention, based on the IoU score-aware classification objective function, the regression information of the localization head, namely the IoU score, is added to the objective function of the classification head to establish the correct connection between the classification head and the localization head; based on the alignment weight classification strategy, the classification and localization results obtained in training are used to allocate the sample weights for the next round of screening, and the final generated weights are allocated to each objective function in the form of coefficients.

[0078] Please see Figures 1 to 4 The present invention provides a fully supervised temporal action localization method based on joint optimization, comprising the following steps:

[0079] Step 1: Given an unedited video sequence, to reduce computational load, the unedited video sequence is decomposed into several video segments, each containing consecutive frames of a fixed length. Specifically, in this embodiment of the invention, a pre-trained two-stream network is used as a feature extractor to encode each video segment, thereby obtaining an encoded video feature sequence; finally, a training video set and a test video set with encoded features are obtained.

[0080] Step 2: Given the video feature sequence obtained in Step 1, input it into the temporal action localization model designed in this embodiment of the invention, and finally output the corresponding detection results; the model architecture is exemplarily shown below. Figure 2 As shown;

[0081] In step 1 of this embodiment of the invention, the specific steps of using a pre-trained two-stream network as a feature extractor to encode each video segment, thereby obtaining an encoded video feature sequence, include:

[0082] Step 1.1: Given an unedited video sequence, decompose the video sequence into T non-overlapping segments, which will serve as the input to the two-stream network I3D network;

[0083] Step 1.2: In order to explore the appearance and motion information of the video sequence respectively, the original video sequence is encoded to generate a video representation; specifically, in this embodiment of the invention, the I3D network is used to process each video segment and extract D-dimensional RGB feature vectors and optical flow feature vectors.

[0084] In step 2 of this embodiment of the invention, a new detector model for temporal action localization is designed based on the existing one-stage method, which may specifically include the following steps:

[0085] Step 2.1 is based on the video features obtained in Step 1. t represents the number of segments in the video features, and the label for the temporal action localization task can be represented as ψ. X ={Φ m ,c m}, Φ m ={s m ,e m} represents the start and end times of the current video segment, c m This represents the type of action represented by the current video segment; after obtaining the features of the current video segment, the detector model can predict the temporal boundary of the current segment. in These represent the distances from the current video feature to the start and end of the label, respectively. Therefore, the prediction result for each video segment can be represented as...

[0086] Step 2.2, based on the above detector, this embodiment of the invention proposes a new localization-aware classification loss function to establish a correct connection between the classifier and the localizer; specifically, the IoU score given by the localizer during training is used to replace the objective function in the classification loss function, and the loss function is based on the binary cross-entropy loss function, specifically:

[0087]

[0088] Where p∈[0,1] is the predicted classification score, which is obtained by passing the output result through a sigmoid function; t is the one-hot format of the corresponding category label, t=1 represents the current feature as the primary feature, and t=0 represents the current feature as the background feature; q represents the corresponding IoU score replacing the subsequent one-hot label. This form initially establishes the connection between the classification head and the localization head; for example... Figure 4 As shown, compared with previous methods, the present invention establishes a correct connection between the classification head and the positioning head;

[0089] Step 2.3: After establishing the correct connection between the classification head and the localization head, another problem to be solved is that existing temporal action localization detection methods do not effectively filter samples. During training, in order to better match the task, positive samples should be dynamically optimized. Since some sample features cannot obtain true label prediction results during training, and the prediction results of these sample features are also harmful to the final model, it is necessary to reduce their weights in training. In this embodiment of the invention, the alignment weight strategy is specifically expressed as: constructing a function of IoU and classification score, and using the set of this function values ​​to assign weights to existing samples to reduce those samples that have a greater negative impact on the model, specifically including:

[0090] ω = IoU * s; where IoU is generated by the localization result of the regression head during the current training process, and s represents the classification score generated by the classification head;

[0091] After obtaining this score, a function is used to design the sample weights.

[0092] ω pos = βω ;

[0093] Where β is a hyperparameter used to adjust the magnitude of weight allocation;

[0094] Finally, the weight is multiplied by the loss function corresponding to the sample to achieve the effect of aligning the weight allocation.

[0095] Please see Figure 5 and Figure 6 In this embodiment of the invention, the proposed method was evaluated using mean accuracy (mAP), and the performance of the technical solution of this embodiment on these two datasets was evaluated using the official evaluation code provided by ActivityNet. For THUMOS14 and ActivityNet v1.3, the tIoU thresholds were {0.3, 0.4, 0.5, 0.5, 0.6, 0.7} and {0.5, 0.75, 0.95, 0.95}, respectively. On ActivityNet v1.3, this embodiment reports the mAP for 10 different tIoU thresholds [0.5: 0.05: 0.95].

[0096] Table 1 shows the experimental results of different methods on the public datasets THUMOS14 and ActivityNet v1.3 used to evaluate semi-supervised temporal action localization tasks. As can be seen from Table 1, the fully supervised temporal action localization method based on joint optimization proposed in this embodiment of the invention has a significant advantage in localization performance.

[0097] Table 1. Experimental results comparing different temporal action localization tasks on the Thumos14 and ActivityNet v1.3 datasets.

[0098]

[0099] In summary, this invention discloses a fully supervised temporal action localization method based on joint optimization, belonging to the field of computer vision and pattern recognition. The steps include: firstly, encoding the feature sequence of the input video using a two-stream network, which includes appearance feature information and motion optical flow information within the video; then, inputting the features into a detector designed in this invention to obtain the desired output result. This invention designs two joint optimization methods: one is to establish a localization-aware classification loss function, which establishes a correct connection between the classification sub-task and the localization sub-task of temporal action localization by changing the one-hot labels in the traditional classification loss function; the other is to design a new weight allocation scheme, based on a designed joint index function, which aims to reduce the negative impact of samples that are harmful to the model during training. This invention addresses the problem of sub-task misalignment in temporal action localization for the first time, introducing two new joint optimization techniques to solve this problem, significantly improving the performance of existing methods.

[0100] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0101] Please see Figure 7 The present invention provides a fully supervised temporal action localization system, comprising:

[0102] The feature acquisition module is used to acquire the encoded video feature sequence for the timing action localization and to extract the features of the encoded video feature sequence using the backbone network.

[0103] The result acquisition module is used to predict the classification and localization results based on the features of the obtained encoded video feature sequence and the trained temporal action localization model.

[0104] During training, the trained temporal action localization model performs gradient backpropagation based on a weighted loss function.

[0105] The expression for the loss function is,

[0106]

[0107] In the formula, p∈[0,1] is the predicted classification score, which is obtained by passing the output result through a sigmoid function; t is the one-hot format corresponding to the category label, t=1 means that the current feature is the main feature, t=0 means that the current feature is the background feature; q represents the one-hot label after the IoU score is replaced;

[0108] Weighting is achieved by multiplying the sample weights by the corresponding loss function to align the weights. The sample weights ω... pos The expression is,

[0109] ω pos = βω ;ω=IoU*s;

[0110] In the formula, β is a hyperparameter used to adjust the weight allocation magnitude; IoU is the intersection-union ratio of the predicted start and end boundary values ​​generated by the localizer during the current training process to the true values; and s represents the classification score generated by the classifier.

[0111] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions from the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a fully supervised timing action positioning method.

[0112] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the fully supervised timing action localization method in the above embodiments.

[0113] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0115] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0116] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A fully supervised temporal action localization method, characterized in that, Includes the following steps: Obtain the encoded video feature sequence for the action to be located in time sequence, and use the backbone network to extract the features of the encoded video feature sequence; Based on the features of the obtained encoded video feature sequence, a trained temporal action localization model is used to make predictions and obtain classification and localization results. During training, the trained temporal action localization model performs gradient backpropagation based on a weighted loss function. The expression for the loss function is, In the formula, p∈[0,1] is the predicted classification score, which is obtained by passing the output result through a sigmoid function; t is the one-hot format corresponding to the category label, t=1 means that the current feature is the main feature, t=0 means that the current feature is the background feature; q represents the one-hot label after the IoU score is replaced; Weighting is achieved by multiplying the sample weights by the corresponding loss function to align the weights. The sample weights ω... pos The expression is, oh pos = βω ;ω=IoU*s; In the formula, β is a hyperparameter used to adjust the weight allocation magnitude; IoU is the intersection-union ratio of the predicted start and end boundary values ​​generated by the localizer during the current training process to the true values; and s represents the classification score generated by the classifier.

2. The fully supervised temporal action localization method according to claim 1, characterized in that, The encoded video feature sequence contains RGB information for each video segment and optical flow information between adjacent segments.

3. The fully supervised temporal action localization method according to claim 1, characterized in that, The backbone network is an I3D, TSP, or SLOWFAST network.

4. The fully supervised temporal action localization method according to claim 1, characterized in that, The temporal action localization model includes: The feature extraction module is used to extract features from the input encoded video feature sequence and output the extracted features. The sample selection module is used to input the features output by the feature input module and assign labels to them to obtain labeled features. The classification and localization fusion module includes a classification head and a localization head, which are used to classify and locate labeled features as input, and output classification results and localization results.

5. The fully supervised temporal action localization method according to claim 4, characterized in that, The feature extraction module is a CNN network or a Tranformer model.

6. The fully supervised temporal action localization method according to claim 1, characterized in that, The training steps of the trained temporal action localization model include: Obtain the training sample dataset; each training sample includes features of the encoded video feature sequence and instance-level data annotations for each action; During training, for selected training samples, the features of the encoded video feature sequence in the selected training samples are input into the temporal action localization model for prediction, and the start time, end time and action type label prediction value of each action are obtained; gradient backpropagation is performed based on the weighted loss function, and the trained temporal action localization model is obtained after the preset convergence condition is reached.

7. A fully supervised temporal motion positioning system, characterized in that, include: The feature acquisition module is used to acquire the encoded video feature sequence for the timing action localization and to extract the features of the encoded video feature sequence using the backbone network. The result acquisition module is used to predict the classification and localization results based on the features of the obtained encoded video feature sequence and the trained temporal action localization model. During training, the trained temporal action localization model performs gradient backpropagation based on a weighted loss function. The expression for the loss function is, In the formula, p∈[0,1] is the predicted classification score, which is obtained by passing the output result through a sigmoid function; t is the one-hot format corresponding to the category label, t=1 means that the current feature is the main feature, t=0 means that the current feature is the background feature; q represents the one-hot label after the IoU score is replaced; Weighting is achieved by multiplying the sample weights by the corresponding loss function to align the weights. The sample weights ω... pos The expression is, oh pos = βω ;ω=IoU*s; In the formula, β is a hyperparameter used to adjust the weight allocation magnitude; IoU is the intersection-union ratio of the predicted start and end boundary values ​​generated by the localizer during the current training process to the true values; and s represents the classification score generated by the classifier.

8. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the fully supervised timing action localization method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the fully supervised timing action localization method as described in any one of claims 1 to 6.

Images