Video timing action nomination generation method and system

Abstract

Embodiments of the present application provide a video timing action nomination generation system and method, which processes video features extracted from an input video via an encoder to obtain video coding features with global information, and provides corresponding interested segment features from the video coding features to a decoder by a plurality of pre-trained nomination segments, the decoder generates segment features based on the corresponding interested segment features of each nomination segment and the pre-trained nomination features of each nomination segment, and provides the segment features to a prediction module; the prediction module generates a timing action nomination result based on the segment features from the decoder. The scheme of the embodiments of the present application can effectively capture the global context information of the video, obtain video coding features with stronger representation ability; and by introducing a plurality of learnable nomination segments to extract feature sequences at corresponding positions from the video coding features for subsequent prediction, the training convergence speed is improved and the computational burden is greatly reduced.

Description

Technical Field

[0001] This invention relates to video processing, and more particularly to a system and method for generating video temporal action nominations. Background Technology

[0002] Generating video temporal action nominations is a crucial step in video temporal action detection. Its purpose is to detect action segments containing human behavior from an uncropped long video, i.e., to determine the start and end times of the action. High-quality video temporal action nominations should possess the following two key characteristics: (1) accurate temporal boundaries, meaning the generated action nominations should completely cover the area where the action occurs; and (2) reliable confidence scores, used to accurately evaluate the quality of the generated nominations for subsequent retrieval and ranking. Combining video temporal action nominations with specific action categories can further complete subsequent video temporal action detection tasks. Efficiently and efficiently generating high-quality video temporal action nominations is beneficial for improving and enhancing the accuracy of video action recognition. Summary of the Invention

[0003] The purpose of this invention is to provide a novel method and system for generating video temporal action nominations to quickly and efficiently generate high-quality video temporal action nominations. This objective is achieved through the following technical solutions:

[0004] According to a first aspect of the present invention, a video temporal action nomination generation system is provided, comprising a feature extraction module, a feature processing module, and a prediction module. The feature extraction module is used to extract video features related to the input video. The feature processing module includes a pre-trained encoder and a decoder. The encoder obtains video encoded features with global information based on the video features from the feature extraction module, and extracts segment features of interest corresponding to each nominated segment from the video encoded features using several pre-trained nominated segments, providing these features to the decoder. The decoder generates segment features based on the segment features of interest corresponding to each nominated segment and the pre-trained nominated features corresponding to the nominated segments, and provides these features to the prediction module. The prediction module generates temporal action nomination results based on the segment features from the decoder, including nomination boundaries and confidence scores.

[0005] In some embodiments of the present invention, the encoder includes a graph attention layer, a multi-head self-attention layer, and a feedforward layer, wherein the encoder takes the result of adding video features and position encoding as the value vector input of the multi-head self-attention layer, and simultaneously provides the result as input to the graph attention layer for processing, and its output is linearly transformed to obtain the query vector and key vector of the multi-head self-attention layer.

[0006] In some embodiments of the present invention, the decoder includes a multi-head self-attention layer, a sparse interaction module, and a feedforward layer. The decoder processes the nomination features corresponding to the nomination fragment through the multi-head self-attention layer and provides them to the sparse interaction module to perform sparse interaction with the interest fragment features corresponding to the nomination fragment. The output of the sparse interaction module is processed by the feedforward layer to obtain the fragment features.

[0007] In some embodiments of the present invention, the feature processing module may be constructed based on a transformer model.

[0008] In some embodiments of the present invention, the prediction module may perform boundary regression and binary classification prediction based on fragment features from the decoder.

[0009] According to a second aspect of the present invention, a method for generating temporal action nominations using a system according to a first aspect of the present invention is also provided, comprising: step S1) extracting video features from an input video via a feature extraction module; step S2) processing the extracted video features via an encoder to obtain video coding features having global context information of the input video; step S3) extracting corresponding segment of interest features from the video coding features using each of a plurality of pre-trained nomination segments; step S4) generating segment features via a decoder based on the segment of interest features corresponding to each nomination segment and pre-trained nomination features corresponding to the nomination segments; and step S5) performing boundary regression and binary classification prediction via a prediction module based on the segment features from the decoder, and outputting corresponding temporal action nomination results.

[0010] In some embodiments of the present invention, the encoder may include a graph attention layer, a multi-head self-attention layer and a feedforward layer, wherein step S2) includes adding the video features and the position encoding as the value vector input of the multi-head self-attention layer, and providing the result as input to the graph attention layer for processing, and the output of the graph attention layer is linearly transformed to obtain the query vector and key vector of the multi-head self-attention layer.

[0011] According to a third aspect of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, wherein the program, when executed, implements the method described in the second aspect of the above embodiments.

[0012] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0013] This scheme can effectively capture the global contextual information of the video and obtain video coding features with stronger representation capabilities. Moreover, by introducing several learnable nomination segments to extract feature sequences at corresponding positions from the video coding features for subsequent prediction, it greatly improves the training convergence speed and significantly reduces the computational burden.

[0014] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0016] Figure 1 A schematic diagram of the operation flow of a video timing action nomination generation system according to an embodiment of the present invention is shown.

[0017] Figure 2 A schematic diagram of the sparse interaction process of a sparse interaction module according to an embodiment of the present invention is shown.

[0018] Figure 3 A flowchart illustrating a video timing action nomination generation method according to an embodiment of the present invention is shown. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of the invention. However, those skilled in the art will recognize that the technical solutions of the invention can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of the invention.

[0021] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0022] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0023] Existing video temporal action nomination generation methods can be divided into anchor-box-based methods and boundary-based methods. Anchor-box-based methods perform boundary regression on predefined, uniformly distributed anchor boxes of specific sizes and proportions, and employ a binary classifier to evaluate the confidence score of the nominations. Specifically, anchor boxes of predefined sizes and proportions are placed at each position in the one-dimensional feature sequence of the video. If the length of the one-dimensional feature sequence is T, and K anchor boxes are placed at each position, a total of TK anchor box results need to be predicted. During the training phase, positive and negative samples are selected based on the intersection-union (IOU) ratio with the ground truth bounding boxes. Temporal boundary regression and binary classification prediction of anchor box confidence are performed on these TK anchor boxes. During the model inference phase, since the predicted anchor box results have a large amount of overlap, a non-maximum suppression method is needed to remove redundant prediction results to obtain the final nomination generation result. Common methods include Prop-SSAD (Lin,T.,Zhao,X.,&Shou,Z.,Temporal convolution basedaction proposal:Submission to activitynet 2017.arXiv preprint arXiv:1707.06750.) and RapNet (Gao,J.,Shi,Z.,Wang,G.,Li,J.,Yuan,Y.,Ge,S.,&Zhou,X..Accurate temporal action proposal generation with relation-aware pyramidnetwork.In Proceedings of the AAAI Conference on Artificial Intelligence,Vol.34,No.07,pp.10810-10817,April 2020). The performance of these methods is highly dependent on the manual design of the anchor boxes, making them difficult to scale and cumbersome to apply to different scenarios. In contrast, boundary-based methods generate candidate nominations of arbitrary length by listing all candidate start and end points, and then perform boundary probability prediction on each candidate nomination to obtain a two-dimensional confidence map. The basic module of this type of method is the convolutional layer, which can only capture information in local areas and cannot capture the long-term semantic information of the video.BMN (Lin, T., Liu, X., Li, Vision, pp.3889-3898, 2019), DBG(Lin,C.,Li,J.,Wang,Y.,Tai,Y.,Luo,D.,Cui,Z.,...&Ji,R..Fast learning of temporal actionproposal via dense boundary generator.In Proceedings of the AAAI Conference on Artificial Intelligence, Vol.34, No.07, pp.11499-11506, April 2020), BSN++ (Su, H., Gan, W., Wu, W., Qiao, Y., & Yan, J. (2020). Bsn++: Complementary boundary regressor with scale-balanced relation modeling for temporal action proposal generation. arXiv preprint arXiv:2009.07641.) belongs to this class of methods.

[0024] In addition, both methods have the following two drawbacks. First, as the video length increases, the number of predefined anchor boxes and the size of the generated confidence map will increase significantly, consuming huge amounts of computational resources and making it difficult to apply to real-world scenarios. Second, both methods generate a large number of redundant nominations, requiring post-processing methods such as non-maximum suppression to eliminate redundant prediction results. Post-processing not only requires meticulous parameter selection but also significantly reduces the inference speed of the model.

[0025] This invention provides a video temporal action nomination generation system, comprising a feature extraction module, a feature processing module, and a prediction module. The feature extraction module extracts video features related to the input video. The feature processing module is built based on a Transformer model and includes an encoder and a decoder. The encoder obtains video encoded features with global information based on the video features from the feature extraction module, and extracts the segment of interest (IOI) features corresponding to each of the pre-defined nomination segments from the video encoded features, providing these to the decoder. The decoder generates segment features based on the IIO features and the nomination features corresponding to each nomination segment, and provides these to the prediction module. The prediction module generates temporal action nomination results based on the segment features from the decoder, including nomination boundaries and confidence scores.

[0026] In this embodiment, the feature processing module and prediction module of the system are first trained using a training set consisting of a large number of video clips labeled with temporal action nominations (this can be called the offline training phase). Then, the video clips to be processed are provided as input to the trained system, and the output is the temporal action nomination of the input video, including each nomination boundary and corresponding confidence score (this can be called the online prediction phase). Initially, several preset nomination clips and their corresponding nomination features, as well as the parameters involved in the encoder, decoder, and prediction module, are randomly set. During training, these parameters are continuously adjusted until training ends, and these trained parameters are used in the subsequent online prediction phase. It should be noted that the feature extraction module and prediction module can employ any type of machine learning model suitable for video feature extraction and for predicting nomination boundaries and confidence scores based on input features, including but not limited to neural network models; this document does not impose any restrictions. Considering that the extraction and processing of video features are essentially the same in the training phase and the online processing phase, the following mainly combines... Figure 1 The process of processing video features during the training phase is introduced.

[0027] First, for the input video, a feature extraction module extracts video features related to the video, such as image features (e.g., RGB features) and optical flow features. In one example, a neural network such as a Temporal Segment Network (TSN) can be used to extract video features. For the extracted video features of different dimensions, they are transformed into a series of feature sequences with the same feature dimension. The feature dimension of the feature sequence can be set according to actual needs and is not limited here. For ease of description, the video features are denoted as follows in the example below. Where R represents a real number, M represents the length of the video (which can be understood as the number of frames), and C represents the dimension of the feature vector, i.e., the dimension of the feature vector extracted from each video frame. It can be seen that the video feature f can also be considered as a sequence of video features composed of feature vectors from M video frames, with each video frame having its own specific position in the sequence. The video features extracted by the feature extraction module are provided to the feature processing module for processing. It should be understood that the above video features can be appropriately transformed to adapt to or match the feature dimension set in the feature processing module. For example, the extracted features can be aligned in terms of feature dimension by passing them through a one-dimensional convolutional layer with a kernel size of 1. The transformed video feature sequence can then be used as the input to the encoder in subsequent processes.

[0028] refer to Figure 1 The encoder mainly consists of a multi-head self-attention layer and a feedforward layer. The multi-head self-attention layer comprises multiple independent self-attention layers. The self-attention layer adopts an attention-based structure, the core of which is to examine other sequence information in the input sequence while encoding the corresponding sequence, concatenating sequences pairwise to effectively capture the global context information of the input sequence and construct long-distance dependencies between sequences. Therefore, it can enhance relevant features and suppress irrelevant features. The input of the multi-head self-attention layer is a triple, consisting of a query vector Q, a key vector K, and a value vector V. The calculation process of each self-attention layer is as follows:

[0029]

[0030] Where d k Here, T represents the transpose, and softmax() represents the activation function. As shown in the formula above, the score between each pair of features in the sequence is calculated by performing a dot product operation between the query vector Q and the key vector K. This score represents the correlation between the two features. To maintain gradient stability, a scaling factor d is used. k The scores are normalized, then the softmax function is used to standardize the values ​​to between 0 and 1. Finally, the scores are weighted with the value vector V to enhance relevant features and suppress irrelevant features. Based on this, the multi-head self-attention layer contains multiple independent self-attention layers, each focusing on a specific portion of the contextual information. The outputs of these self-attention layers (each output can be denoted as head, head = Attention(Q,K,V)) are concatenated and further aggregated through a linear layer to obtain the more robust output MultiHead(Q,K,V) of the multi-head self-attention layer, calculated as follows:

[0031] MultiHead(Q,K,V)=Concat(head1,...,head h W O Where h represents the total number of self-attention layers contained in the multi-head self-attention layer, W o This is the parameter matrix of the linear layer used for feature aggregation. For example... Figure 1 As shown, the output of the multi-head self-attention layer is further processed through addition and normalization operations before being input to the feedforward layer. The feedforward layer can consist of two linear transform layers and a nonlinear activation function, ReLU. The output of the feedforward layer, after addition and normalization operations, yields the output of the entire encoder, i.e., the video coding features.

[0032] In some embodiments, as in a typical Transformer model, the inputs Q, K, V of the multi-head self-attention layer in the encoder are obtained by passing the input feature sequence through three matrices (W, K, V) with different parameters. Q W K W V The values ​​are obtained by mapping through a linear transformation layer. For example, assuming the input sequence is T0, Q, K, and V are calculated using the following formula:

[0033] Q = T0W Q K = T0W k V = T0W V .

[0034] And in Figure 1 In the illustrated embodiment, a graph attention layer is introduced into the encoder on top of the multi-head self-attention layer to preprocess the input sequence. This allows the encoder to better focus on segments in the video where actions occur and to construct relationships between these segments, thereby obtaining encoded features with stronger representational capabilities. Furthermore, to address the relative positional relationships in the video features, such as the relative positions and temporal relationships between video frames, positional encoding is employed in the encoder. The input video feature sequence is used as the sum of the positional encoding results. Where d represents the dimension of the features applicable in the encoder. The dimension of the positional encoding is the same as the dimension of the input video features, meaning that the feature vector of each video frame in the input video feature sequence has its own corresponding positional encoding. As mentioned above, the positional encoding, which is one of the parameters of the encoder, is randomly set during system initialization and is continuously adjusted during subsequent training.

[0035] like Figure 1As shown, the input x, obtained by adding the input video feature sequence to the positional encoding, is directly used as the value vector V of the multi-head self-attention layer. Simultaneously, this input x is fed into a graph attention layer for transformation. The output of the graph attention layer undergoes further linear transformation to obtain the query vector Q and key vector K of the multi-head self-attention layer. The graph attention layer is used to further strengthen the connection between features at different time points in the video, using the i-th input vector x... i For example, after passing through the graph attention layer, it transforms into:

[0036]

[0037] Where || represents the concatenation operation, K is the number of multi-head attention layers in the graph, i = 1, 2, ..., M, where M, as mentioned above, represents the length of the video, which can be understood as the number of frames in the video. W k Let σ be the learnable weight matrix of the k-th graph self-attention layer, and σ be a non-linear activation function, such as the Leaky ReLU function. For the feature vector x in the k-th graph attention layer i For x j The weighting coefficient, which characterizes the degree of correlation between the two, is calculated as follows:

[0038]

[0039] Where, α k Here, T represents the transpose operation, which is a learnable weight vector. By introducing the graph attention mechanism described above into the encoder, the connections between different frames of the video feature sequence can be dynamically constructed, thereby capturing global contextual information more accurately and helping the encoder obtain video coding features with stronger representational capabilities.

[0040] Continue to refer to Figure 1In embodiments of this invention, N learnable nomination segments and their corresponding nomination features are introduced to further process the video coding features output by the encoder. Each nomination segment is used to extract a feature sequence at the corresponding position from the video coding features to obtain the segment of interest (IOI) feature, which, along with the nomination feature corresponding to that nomination segment, is provided as input to the decoder. Each nomination segment is a normalized two-dimensional coordinate (value between 0 and 1), representing a segment on the video timeline; each nomination feature is a vector of dimension d. Here, the lengths of the nomination segments can be different, and therefore the dimensions of the extracted feature sequences may also be different. Therefore, in one example, after extracting the feature sequence at the corresponding position from the video coding features using the nomination segments, bilinear interpolation can be used to adjust all extracted features to the same length M′, i.e., the dimension of each IIO feature is M′×d. As mentioned above, similar to the positional encoding of the encoder, these N nomination segments and their corresponding nomination features are parameters obtained during training, randomly set during system initialization, and continuously adjusted during subsequent training.

[0041] In the decoder, these N nominated features are first input to a multi-head self-attention layer. This layer obtains information about the long-range dependencies between the nominated features. After the output of the multi-head self-attention layer is summed and normalized, the nominated feature corresponding to each nominated segment and the feature of interest corresponding to that nominated segment interact one-to-one in a sparse interaction module. The output of this sparse interaction module is further summed and normalized before being fed to a feedforward layer. The output of the feedforward layer, after summing and normalizing, outputs N segment features, which are the decoder's output. Figure 2 Taking the k-th nominated feature as an example, the sparse interaction process between it and the corresponding feature of the segment of interest in the sparse interaction module is demonstrated. Specifically, the nominated feature vector of dimension d is passed through a linear layer and scaled to obtain a size of d×d. h and d h The two parameters of ×d (here, d) h (This can be configured according to specific decoder requirements). The features of interest segments are sequentially multiplied by these two parameters to obtain segment features of size M′×d. This process can be viewed as the feature segments of interest passing through two one-dimensional convolutional layers, and therefore can also be called dynamic convolution operation. In the decoder described above, the nominated features interact with the corresponding features of interest segments, without needing to interact with the global video coding features, thus greatly improving the training convergence speed.

[0042] Continue to refer to Figure 1The prediction module receives N segment features from the decoder, performs boundary regression and binary classification prediction, and outputs N nomination prediction results, including nomination boundaries and corresponding confidence scores. In each training iteration, the N nomination prediction results obtained through the above process are matched one-to-one with the corresponding ground truth nomination labels using optimal binary matching. For example, using the Focal loss function as the binary classification loss function and the L1 and GIOU loss functions as the regression loss functions, for a video, the sum of the classification cost and regression cost of the N nomination prediction results for each nomination label is calculated. Finally, for each ground truth nomination label, the unique nomination prediction result with the lowest total cost is selected as the positive sample, while nomination prediction results that do not match the ground truth nomination label are considered negative samples. In this embodiment, the prediction module consists of two independent feedforward layers: one feedforward layer consists of a single linear layer used to evaluate the confidence score of the generated nomination results, and the other feedforward layer consists of three linear layers used to regress the boundary coordinates of the nominations. The training process described above is iteratively repeated on the training set, with the nomination boundaries output by the prediction module in each round of training serving as the N nomination segments for the next round. After training, the N nomination segments involved in the system, their corresponding nomination features, and the parameters involved in the encoder, decoder, and prediction module are determined and can then be used in the subsequent online prediction stage. In this paper, the value of N can be set based on the length of the video segment to be processed, actual needs, and system performance. For example, a 1-minute video segment typically has 2 to 3 nominations, so N can be set to be at least greater than the number of possible nominations on that video segment, such as setting N to any integer greater than 3. However, it should be understood that the larger N is, the greater the computational cost. Therefore, N is usually no more than a multiple of the number of possible nominations on the video segment to be processed. For example, for a 1-minute video segment to be processed, N can be set to an integer between 3 and 30.

[0043] In the online prediction phase, the video segment to be processed is provided to the system. The system first extracts video features from these features, and an encoder transforms these features into video-coded features with global context information of the input video. These features are then combined with each of the N pre-trained nomination segments to extract the corresponding segment of interest (IOI) features from the video-coded features. Next, a decoder performs a one-to-one interaction between the IIO features and the corresponding nomination features for each nomination segment to obtain segment features, which are then provided to the prediction module. Finally, the prediction module performs boundary regression and binary classification prediction on the segment features from the decoder and outputs N nomination generation results corresponding to the video segment to be processed. Unlike existing technologies, this system directly obtains N action nomination results by introducing N learnable nomination segments and their corresponding nomination features, without the need for non-maximum suppression post-processing. Furthermore, the number of generated action nominations is independent of the video length, thus significantly reducing the computational burden and greatly improving the generation speed of temporal action nominations.

[0044] As can be seen, the system according to the above embodiments can effectively capture the global context information of the video and obtain video coding features with stronger representation capabilities; moreover, by introducing several learnable nomination segments to extract feature sequences at corresponding positions from the video coding features for subsequent prediction, the training convergence speed is greatly improved and the computational burden is significantly reduced.

[0045] Figure 3 A flowchart illustrating a method for generating temporal action nominations using the video temporal action nomination generation system according to an embodiment of the present invention is shown. The method includes: step S1) extracting video features from the input video via a feature extraction module; step S2) processing the extracted video features via an encoder to obtain video encoded features with global context information of the input video; step S3) extracting corresponding segment of interest features from the video encoded features using each of a plurality of preset nomination segments; step S4) interacting the nomination features corresponding to each nomination segment with the segment of interest features corresponding to that nomination segment via a decoder to obtain segment features; and step S5) performing boundary regression and binary classification prediction based on the segment features from the decoder via a prediction module, and outputting the corresponding temporal action nomination result.

[0046] To better illustrate the performance of the present invention, the inventors also compared the performance of the temporal action nomination generation method of the present invention with that of commonly used existing temporal action nomination generation methods based on the THUMOS14 dataset and the ActivityNet-1.3 dataset.

[0047] During the training process, utilize Figure 1The system architecture shown is trained iteratively on the training set for 20 epochs. After each epoch, the loss on the validation set is calculated to evaluate the performance of the system, and the system architecture with the smallest loss on the validation set is selected as the system that has been trained.

[0048] In the prediction phase, video features are input into the trained system, and the output of the prediction module is used as the final N nominations. The generated nominations are compared with the true nomination labels, and the recall on the validation set is calculated to verify the performance of the trained model structure. Table 1 compares the performance of the method of this invention with current mainstream methods on the THUMOS14 dataset, using the nomination recall as the evaluation metric. The results show that the method of this invention outperforms other methods. Table 2 compares the inference speed of the method of this invention with other mainstream algorithms on the ActivityNet-1.3 dataset. For a fair comparison, the average inference time for each video is calculated, and the results show that the method of this invention is at least 8 times faster than existing methods.

[0049] Table 1

[0050] method AR@50 AR@100 AR@200 AR@500 BSN 37.46 46.06 53.21 60.64 BMN 39.36 47.72 54.70 62.07 RapNet 40.35 48.23 54.92 61.41 DBG 37.32 46.67 54.50 62.21 This invention 40.40 48.70 55.51 62.20

[0051] Table 2

[0052] method BSN BMN GTAD DBG This invention <![CDATA[T pro (sec)]]> 0.671 0.118 0.103 0.219 0.056 <![CDATA[T all (sec)]]> 0.815 0.745 0.862 0.596 0.074

[0053] In another embodiment of the present invention, a computer-readable storage medium is also provided, on which a computer program or executable instructions are stored. When the computer program or executable instructions are executed by a processor or other computing unit, the technical solution described in the foregoing embodiments is implemented. The implementation principle is similar and will not be repeated here. In the embodiments of the present invention, the computer-readable storage medium can be any tangible medium capable of storing data and readable by a computing device. Examples of computer-readable storage media include hard disk drives, network attached storage (NAS), read-only memory, random access memory, CD-ROM, CD-R, CD-RW, magnetic tape, and other optical or non-optical data storage devices. The computer-readable storage medium may also include computer-readable media distributed across a network-coupled computer system so that computer programs or instructions can be stored and executed in a distributed manner.

[0054] References to "various embodiments," "some embodiments," "one embodiment," or "embodiment," etc., in this specification refer to a specific feature, structure, or property described in connection with the said embodiment, included in at least one embodiment. Therefore, the appearance of the phrases "in various embodiments," "in some embodiments," "in one embodiment," or "in an embodiment," etc., throughout this specification does not necessarily refer to the same embodiment. Furthermore, specific features, structures, or properties can be combined in any suitable manner in one or more embodiments. Therefore, a specific feature, structure, or property shown or described in connection with one embodiment can be combined, in whole or in part, with features, structures, or properties of one or more other embodiments without limitation, provided that the combination is not illogical or inoperable.

[0055] The terms "comprising," "having," and similar expressions used in this specification are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus. "A" or "an" does not exclude multiple instances. Furthermore, the elements in the accompanying drawings are for illustrative purposes only and are not drawn to scale.

[0056] Although the present invention has been described through the above embodiments, the present invention is not limited to the embodiments described herein, and includes various changes and variations without departing from the scope of the present invention.

Claims

1. A video temporal action nomination generation system comprising feature extraction a module, a feature processing module, and a prediction module, wherein: a feature extraction module configured to extract video features related to the input video from the input video; the feature processing module comprises a pre-trained encoder and a decoder, wherein the encoder is configured to obtain video encoding features with global information based on the video features from the feature extraction module, and extract, from the video encoding features, a respective segment of interest feature corresponding to each of a plurality of pre-trained nomination segments, and provide the respective segment of interest feature to the decoder, and the decoder is configured to generate, based on the respective segment of interest feature corresponding to each of the plurality of pre-trained nomination segments and a pre-trained nomination feature corresponding to each of the plurality of pre-trained nomination segments, a segment feature, and provide the segment feature to the prediction module; the prediction module is configured to generate, based on the segment feature from the decoder, a temporal action nomination result comprising a nomination boundary and a confidence score; the encoder comprises a graph attention layer, a multi-head self-attention layer, and a feed-forward layer, wherein the encoder is configured to input, as a value vector of the multi-head self-attention layer, a result of adding the video features and position encodings, and provide the result as an input to the graph attention layer, and output, after linear transformation, a query vector and a key vector of the multi-head self-attention layer; the decoder comprises a multi-head self-attention layer, a sparse interaction module, and a feed-forward layer, wherein the decoder is configured to provide, after processing by the multi-head self-attention layer, the pre-trained nomination feature corresponding to each of the plurality of pre-trained nomination segments to the sparse interaction module for sparse interaction with the respective segment of interest feature corresponding to each of the plurality of pre-trained nomination segments, and output, after processing by the feed-forward layer, the segment feature.

2. The system of claim 1, wherein the feature processing module is constructed based on a transformer model.

3. The system of claim 1, wherein the prediction module is configured to perform boundary regression and binary classification prediction based on the segment feature from the decoder.

4. A method of generating a temporal action nomination result using the system of any one of claims 1-3, comprising: Step S1) extracting, via the feature extraction module, video features from an input video; Step S2) processing, via the encoder, the extracted video features to obtain video encoding features with global context information of the input video; Step S3) extracting, from the video encoding features, a respective segment of interest feature using each of a plurality of pre-trained nomination segments; Step S4) generating, via the decoder, a segment feature based on the respective segment of interest feature corresponding to each of the plurality of pre-trained nomination segments and a pre-trained nomination feature corresponding to each of the plurality of pre-trained nomination segments; Step S5) performing, via the prediction module, boundary regression and binary classification prediction based on the segment feature from the decoder, and outputting a respective temporal action nomination result.

5. The method of claim 4, wherein the encoder comprises a graph attention layer, a multi-head self-attention layer, and a feed-forward layer, and wherein Step S2) comprises inputting, as a value vector of the multi-head self-attention layer, a result of adding the video features and position encodings, and providing the result as an input to the graph attention layer, and outputting, after linear transformation, a query vector and a key vector of the multi-head self-attention layer.

6. The method of claim 4, wherein the decoder comprises a multi-head self-attention layer, a sparse interaction module and a feed-forward layer, and wherein step S4) comprises processing the nomination feature corresponding to the nomination segment through the multi-head self-attention layer, providing the processed nomination feature to the sparse interaction module to perform sparse interaction with the segment feature corresponding to the segment of interest corresponding to the nomination segment, and processing an output of the sparse interaction module through the feed-forward layer to obtain the segment feature.

7. A computer readable storage medium characterized in that, A computer readable storage medium stores a computer program, which, when executed, implements the method of any one of claims 4-6.

Images