A method and system for generating time-series action nominations in combination with internal positions of actions

Abstract

The present invention provides a method and system for generating temporal action nominations in combination with the internal position of an action, and relates to the technical field of computer vision. The method first uses a fixed time interval to extract an uncropped video frame sequence to obtain a video frame segment; then the video frame segment is sent to a dual-stream network to obtain the corresponding dual-stream features; then the dual-stream features are input to the backbone network, and the offset window Transformer module predicts the start and end probability sequence of the action in the video frame segment; the nomination evaluation module uniformly samples the nomination-level features and obtains the nomination confidence map; the multi-scale hole convolution attention module predicts multiple internal positions of the action in the video frame segment, and generates multiple internal position probability sequences of the action; finally, candidate nominations are generated and score fusion is performed to output the final temporal action nomination set. The method makes full use of the feature information of the internal position of the action and improves the accuracy of the temporal action nomination.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and system for generating temporal action nominations in combination with internal positions of actions. Background Art

[0002] Action recognition is an important research direction in video analysis and is gradually attracting widespread attention. Video action recognition aims to accurately identify and understand the movements and behaviors of sports from video sequences, and has a wide range of applications, such as intelligent monitoring, somatosensory games, medical diagnosis, etc. However, video action recognition faces a series of challenges. Factors such as background interference, illumination changes, and posture diversity in video sequences make action recognition complex and challenging. These problems bring about another challenging task: temporal action detection. Although existing action recognition methods have achieved good results, the detection accuracy of temporal action detection tasks is still low. Therefore, the task of temporal action nomination generation in uncropped videos has attracted much attention in recent years. This task only needs to detect the start and end time of the action, without considering the action category in the video, and has a wide range of applications, such as video recommendation, intelligent monitoring, and wonderful content editing.

[0003] At present, there are two main types of methods for generating time series action nominations: one is the sliding window method based on the anchor mechanism; the other is the boundary probability prediction method. The anchor mechanism-based method requires a lot of prior knowledge for manual setting, has poor flexibility, and is difficult to cover all action instances. Therefore, most current methods use the boundary-based probability prediction method.

[0004] Although boundary-based methods have brought remarkable performance improvements, they still ignore another important information in the action, namely the internal location of the action. The internal location of the action is of great significance for understanding and describing the execution process and morphological characteristics of the action.

[0005] The Chinese invention patent with patent publication number CN117197725A discloses a method and system for generating time-series action nominations based on multi-position collaboration. The patent uses the left and right 1 / 4 positions of the action to generate action center area nominations, and in the subsequent stage, the candidate nominations composed of the start and end time points are corrected through the action center area nominations. But there are still many shortcomings. First of all, this method only uses the 1 / 8 position point, and does not use other internal positions such as 1 / 16. In addition, the method uses the boundary prediction module BPM and the quarter prediction module QLP to predict the action boundary and internal position respectively. The BPM module only uses multi-layer convolution modules to predict the boundary position, and the prediction accuracy is low. The QLP module divides the features into multiple windows, performs Transformer operations within the window, and does not establish the relationship between different windows. Summary of the invention

[0006] Purpose of the invention: To propose a method for generating temporal action nominations in combination with the internal position of the action, and further propose a temporal action nomination generation system that can implement the above method, so that the prediction of boundaries and internal positions is more accurate, thereby effectively solving the above problems existing in the prior art.

[0007] The present invention proposes a method for generating a time sequence action nomination in combination with the internal position of the action, and the steps are as follows:

[0008] Get the uncropped video frame sequence;

[0009] Extracting an uncut video frame sequence at a fixed time interval to obtain a video frame segment; sending the video frame segment to a dual-stream network consisting of a spatial stream network and a temporal stream network to obtain dual-stream features corresponding to the video frame segment;

[0010] The dual-stream features are input into a backbone network consisting of an offset window Transformer module, a nomination evaluation module, and a multi-scale dilated convolution attention module; the offset window Transformer module predicts the start and end probability sequences of actions in the video frame segment; the nomination evaluation module uniformly samples the nomination-level features and obtains a nomination confidence map; the multi-scale dilated convolution attention module predicts multiple internal positions of actions in the video frame segment and generates multiple internal position probability sequences of actions;

[0011] Based on the start and end probability sequences, the nomination confidence map, and the internal position probability sequence, candidate nominations are generated and score fusion is performed to output a final temporal action nomination set.

[0012] In a further embodiment, the uncropped video frame sequence is represented as ;in represents the tth frame in the video; Indicates the length of the video;

[0013] The temporal annotation set of the uncropped video frame sequence U consists of a series of temporal action instances, represented as ;in is the number of actual action instances; and Respectively represent action instances The start and end time of

[0014] Given an uncropped video frame sequence U, the task of temporal action nomination generation aims to generate a comprehensive coverage The Timed Action Nominations , where a nomination is defined as a video segment that may contain an action.

[0015] In a further embodiment, a fixed time interval is used to extract an uncut video frame sequence to obtain a video frame segment, which is expressed as follows:

[0016]

[0017] In the formula, represents a video frame segment; S represents a video frame segment sequence consisting of multiple video frame segments; ,in Indicates the length of the video. For fixed time intervals.

[0018] In a further embodiment, the spatial stream network uses a single frame image as input; the temporal stream network uses stacked optical flows As input;

[0019] Segment the video frame Send it to the dual-stream network for feature extraction to obtain the dual-stream features corresponding to the current video frame segment , in represents the output features of the spatial stream network, is the output feature of the temporal stream network;

[0020] The output features of the spatial stream network And the output features of the temporal stream network Concatenate as the input of the backbone network; the input feature size is , where L is the time series length of the feature and C is the feature dimension.

[0021] In a further embodiment, the offset window Transformer module includes a general window and an offset window, and the input of the offset window is connected to the output calculated by the general window, and the calculation process is as follows:

[0022]

[0023]

[0024]

[0025]

[0026] Where LN(*) represents the layer normalization layer, W-MSA(*) represents the window multi-head self-attention layer, SW-MSA(*) represents the offset window multi-head self-attention layer, MLP(*) represents the fully connected layer, represents the output of the first fully connected layer, represents the output of the previous offset window Transformer module, represents the output of the next offset window Transformer module, Represents the output after the general window calculates the multi-head attention, Represents the output after calculating the multi-head attention with the offset window.

[0027] In a further embodiment, the input feature F is input to the offset window Transformer module, and the offset window Transformer module projects the dual stream feature onto a linear transformation to obtain a feature map Q F , K F , the calculation process is as follows:

[0028]

[0029] Where A is the generated attention map, the input feature F is the dual-stream feature corresponding to the video frame segment; d is the feature map Q F and K F Dimensions; K F The transpose of the attention map represents the feature map Q F And feature map K F The similarity between

[0030] The output of the offset window Transformer module will finally pass through two network layers consisting of two layers of 1D convolution and one Sigmoid layer to obtain the start and end probability sequence of the action in the video frame segment.

[0031] In a further embodiment, the process of uniformly sampling nomination-level features and obtaining a nomination confidence map by the nomination evaluation module specifically includes:

[0032] Set the size to A matrix where each position in the matrix represents a nomination; T is the number of time positions, and D represents the maximum duration of a nomination that is preset;

[0033] For each position in the matrix, construct a The feature is uniformly sampled from the two-stream feature, and finally we get feature map; where C represents the feature dimension and N represents the number of sampling points;

[0034] A 3D convolution is used to fuse the information of all sampling points of a single nomination, and three 2D convolutions are used to fuse the features of adjacent nominations. Sigmoid is Each nomination generates a confidence level in the range [0, 1];

[0035] Get two sizes of Classification confidence map of And regression confidence map ;

[0036] Classification confidence map And regression confidence map Each position (i, j) in represents a nomination with a start time of i and an end time of i+j; in the classification confidence graph The confidence value corresponding to position (i, j) is , in the regression confidence graph The confidence value corresponding to position (i, j) is .

[0037] In a further embodiment, the multi-scale dilated convolutional attention module comprises stacked multi-scale dilated convolutions and temporal attention;

[0038] The multi-scale dilated convolution consists of three parallel one-dimensional dilated convolution layers. The dilation rates d of the first two layers are set to 1 and 3 respectively, and the dilation rate of the third layer is It gradually increases with the increase of network depth, and r represents the depth value;

[0039] The outputs of the three parallel one-dimensional dilated convolutional layers are connected along the channel dimension and fused using a one-dimensional convolution with a kernel size of 1. The mathematical expression is as follows:

[0040]

[0041]

[0042] In the formula, is a nonlinear activation function; represents a dilated convolution with a dilation rate of 1; represents a dilated convolution with a dilation rate of 3; The void rate is dilated convolution; Indicates splicing by channel dimension; Represents the output of the concatenation of the results of multiple layers of dilated convolution; Represents the output features of multi-scale dilated convolution;

[0043] The temporal attention uses the ReLU activation function to convert the output features of the multi-scale dilated convolution After two convolutional layers, the final output contains seven paths of two one-dimensional convolutional layers and one Sigmoid layer, thereby generating action probability sequences at left 1 / 4, right 1 / 4, left 1 / 8, right 1 / 8, left 1 / 16, right 1 / 16 and center positions.

[0044] In a further embodiment, the process of generating candidate nominations based on the start and end probability sequences, the nomination confidence map, and the internal position probability sequence and performing score fusion specifically includes:

[0045] The start and end probability sequences output by the offset window Transformer module are screened, and positions with a probability greater than 50% of the maximum probability or at the peak are retained; the start probability value and the end probability value of each position are judged respectively to obtain two sets of positions that meet the conditions;

[0046] Pair the positions in the two sets one by one. If the duration is less than the maximum duration D set by the data set, a candidate nomination is generated. ;

[0047] , , , , , Respectively represent the start time, end time, start probability value, end probability value, confidence value corresponding to the classification confidence, and confidence value corresponding to the regression confidence;

[0048] For each candidate nomination , the starting probability value, the ending probability value, the confidence value corresponding to the classification confidence, and the confidence value corresponding to the regression confidence are integrated to obtain the total confidence score :

[0049]

[0050] Based on the overall confidence score , calculate the final nomination confidence :

[0051]

[0052] In the formula, Indicates the number of internal region nominations searched within the start and end time interval of each candidate nomination; Indicates the duration of the nomination; is the enhancement coefficient, which is a value greater than 1;

[0053] Output the final set of sequential action nominations .

[0054] The present invention also provides a time sequence action nomination generation system that can execute the above time sequence action nomination generation method combined with the internal position of the action. The time sequence action nomination generation system includes an acquisition unit, a first processing unit, a second processing unit, and an output unit.

[0055] The acquisition unit is used to acquire an uncropped video frame sequence.

[0056] The first processing unit extracts an uncropped video frame sequence at a fixed time interval to obtain a video frame segment; the video frame segment is sent to a dual-stream network consisting of a spatial stream network and a temporal stream network to obtain dual-stream features corresponding to the video frame segment.

[0057] The second processing unit is used to input the dual-stream features into a backbone network composed of an offset window Transformer module, a nomination evaluation module, and a multi-scale dilated convolution attention module; the offset window Transformer module predicts the start and end probability sequences of actions in video frame segments; the nomination evaluation module uniformly samples nomination-level features and obtains a nomination confidence map; the multi-scale dilated convolution attention module predicts multiple internal positions of actions in video frame segments and generates multiple internal position probability sequences of actions.

[0058] The output unit generates candidate nominations based on the start and end probability sequences, the nomination confidence map, and the internal position probability sequence, performs score fusion, and outputs a final temporal action nomination set.

[0059] Beneficial effects: The present invention proposes a method and system for generating temporal action nominations in combination with the internal position of the action, which can improve the shortcoming of the traditional boundary-based temporal action nomination generation method that ignores the internal position of the action. The method makes full use of the feature information of the internal position of the action and improves the accuracy of the temporal action nomination. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a diagram of the overall network structure involved in the method for generating sequential action nominations in an embodiment of the present invention.

[0061] Figure 2 4 is a structural diagram of an offset window Transformer module in an embodiment of the present invention.

[0062] Figure 3 Schematic diagram of multi-scale dilated convolutional attention in an embodiment of the present invention.

[0063] Figure 4 This is a diagram showing the effect of the nomination suppression algorithm in an embodiment of the present invention.

[0064] Figure 5 Schematic diagram of the final detection result in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features known in the art are not described.

[0066] Embodiment 1:

[0067] This embodiment discloses a method for generating a time sequence action nomination in combination with the internal position of the action. The overall network structure diagram involved in this method is as follows: Figure 1 The method mainly includes the following five steps:

[0068] Get an uncropped video frame sequence. Use a fixed time interval to extract the uncropped video frame sequence to obtain video frame segments. Send the video frame segments to the two-stream network, which consists of a temporal stream network and a spatial stream network. The spatial stream network takes a single frame image as input, and the temporal stream network takes stacked optical flow as input. Finally, the spatial features and temporal features are spliced ​​as the input features of the backbone network.

[0069] The backbone network consists of an offset window Transformer module, a nomination evaluation module, and a multi-scale dilated convolution attention module.

[0070] The offset window Transformer module predicts the start and end probability sequence of the action in the video frame segment. When predicting the action boundary position, the offset window Transformer is introduced to increase the attention to the local information in the video features and weaken the noise impact brought by the global information.

[0071] The nomination evaluation module exhaustively enumerates possible nominations and uniformly samples nomination-level features on the input features. Finally, the nomination confidence score is obtained through processing in the convolutional layer.

[0072] The multi-scale atrous convolutional attention module predicts the 1 / 16, 1 / 8 and 1 / 4 position probability sequences of the action in the video frame segment. This process adds the prediction of multiple internal positions of the action and designs a multi-scale atrous convolutional attention module to select high-probability internal positions to generate action internal area nominations, thereby enhancing high-quality action nominations.

[0073] Based on the start and end probability sequences, nomination confidence maps, and internal position probability sequences, candidate nominations are generated and score fusion is performed to output the final temporal action nomination set.

[0074] A new nomination suppression method DS-NMS is used. DS-NMS considers both the intersection-over-union ratio and the confidence between nominations, and has a better effect on retaining high-quality nominations.

[0075] Embodiment 2:

[0076] Based on Example 1, Example 2 discloses the specific details of the method for generating temporal action nominations combined with the internal position of the action from the perspectives of task definition, feature extraction, backbone network construction (offset window Transformer module, nomination evaluation module, multi-scale hole convolution attention module), score fusion, and nomination suppression.

[0077] 1. Task definition: Represent an uncut video sequence as ,in represents the tth frame in the video, Represents the video length. The temporal annotation set of U consists of a series of temporal action instances, expressed as ,in is the number of actual action instances, and Respectively represent action instances The start time and end time of a video. Given an untrimmed video , the task of temporal action nomination generation aims to generate an accurate and comprehensive The Timed Action Nominations , where a nomination is defined as a video segment that may contain an action.

[0078] 2. Feature extraction: The present invention adopts a dual-stream network for feature encoding, which achieves high action recognition accuracy and is widely used in many video analysis methods.

[0079] For an uncropped sequence of video frames , first use a fixed time interval To extract video frames (in ). Then the video frame segment Send it to the dual-stream network for feature extraction to obtain the dual-stream features corresponding to the current video frame segment , here represents the output features of the spatial stream network, is the output feature of the temporal stream network. The spatial stream network uses Resnet as the backbone and uses a single frame image as input. The temporal stream network uses BN-Inception as the backbone to stack the optical flow As input. Finally, the spatial features and temporal features are concatenated as the input of the backbone network, and the input feature size is .

[0080] 3. Backbone network (including offset window Transformer module, nomination evaluation module, multi-scale hole convolution attention module).

[0081] 3.1. Offset Window Transformer Module (SWTM):

[0082] The present invention adopts an offset window Transformer module to locally model the temporal features, which can better capture the local correlation and detail information within the window, thereby reducing the impact of global noise.

[0083] The offset window Transformer module contains two structures: general Window Transformer and Shifted Window Transformer, see Figure 2 , Figure 2 In the figure, (a) is the general window and (b) is the offset window.

[0084] The general structure of Window Transformer mainly includes LayerNorm, W-MSA and MLP. W-MSA is the window multi-head self-attention. The input feature shape is , that is, divide the window on T. For each attention head in MSA, project the input feature F onto the linear transformation to obtain Q F , K F . Then its calculation method is shown in Formula 1:

[0085]

[0086] A is the generated attention map, d is the feature map Q F and K F The attention map represents the feature map Q F And feature map K F The similarity between .

[0087] For Shifted Window Transformer, it is used to connect the output calculated by the general Window Transformer. The offset distance of the shifted window is half of the original window, and the internal structure is the same as the general Window Transformer.

[0088] The calculation of general Window Transformer and Shifted Window Transformer is shown in Formula 2.

[0089]

[0090]

[0091]

[0092]

[0093] Among them, LN represents the layer normalization layer, W-MSA represents the window multi-head self-attention layer, SW-MSA represents the offset window multi-head self-attention layer, and MLP represents the fully connected layer.

[0094] The output of the offset window Transformer will finally pass through two network layers consisting of two layers of 1D convolution and one Sigmoid layer to obtain the start and end probability sequences of the action.

[0095] 3.2. Nomination Evaluation Module (PEM):

[0096] PEM follows the method in the classic network BMN (Boundary-Matching Network for Temporal ActionProposal Generation). Define a size of , where T is the number of time positions and D is the maximum duration of a nomination that is set in advance. In this way, each position in the matrix represents a nomination, and the whole covers all possible nomination situations. With such a matrix, we then need to construct a size of The feature is uniformly sampled from the dual-stream features, and finally we can get Finally, a 3D convolution is used to fuse the information of all sampling points of a single nomination, three 2D convolutions fuse the features of adjacent nominations, and Sigmoid is The confidence level of each nomination is in the range [0, 1]. Classification confidence map of And regression confidence map Used for binary classification loss and regression loss. and Each position (i, j) in represents a nomination with a start time of i and an end time of i+j. The confidence value corresponding to this position in ,exist The confidence value corresponding to this position in .

[0097] 3.3. Multi-scale dilated convolutional attention module (MDA):

[0098] The multi-scale dilated convolutional attention module also takes the dual-stream features as input. Its structure is shown in Figure 3As shown. Multi-scale dilated convolutional attention (MDA) allows operation with different dilation rates, so that local information can be extracted at different scales. This flexibility enables the module to better adapt to the feature expression requirements at different scales, thereby improving the model's ability to capture local information at different scales of the image. Since multi-scale dilated convolution can process information at multiple scales at the same time, local features in the image can be extracted more efficiently. This parallel processing method can accelerate the feature extraction process and improve the operating efficiency of the model. In addition, the addition of attention can better fuse the extracted information at multiple scales, allowing the model to more comprehensively understand the local and global information in the action sequence, thereby improving the ability to accurately predict the internal position of the action.

[0099] The multi-scale dilated convolution attention module is used to predict the internal position of the action, including 1 / 4, 1 / 8, 1 / 16 and center positions. The MDA module consists of stacked multi-scale dilated convolutions and temporal attention. The multi-scale dilated convolution part consists of three parallel one-dimensional dilated convolution layers with a kernel size of 3. The dilation rates d of the first two layers are fixed at 1 and 3 respectively, while the dilation rate d of the third layer is fixed at 1. It will gradually increase with the increase of network depth, and r represents the depth value. Finally, the outputs of these three layers are connected along the channel dimension and then fused using a one-dimensional convolution with a kernel size of 1. As shown in Formula 3.

[0100]

[0101]

[0102] in is a non-linear activation function.

[0103] The temporal attention part adopts a mechanism similar to self-attention. First, the output features of the multi-scale dilated convolution are converted using the ReLU activation function. After two convolutional layers, the shape is The feature maps A and B are then matrix multiplied by the transpose of B and A, and Softmax is used to calculate the time-aware attention, thus obtaining a T × T time attention map G, as shown in Formula 4.

[0104]

[0105] In the formula, represents the mth row of feature map A, represents the nth column of feature map B, Represents the element of the temporal attention map G at the mth row and nth column.

[0106] At the same time, the characteristics Send it to the convolution layer to generate a new feature map C, which is matrix multiplied by the transpose of C and G. Finally, the result is multiplied by the scaling parameter , and perform element-wise summation with feature F' to obtain the output .

[0107] The number of multi-scale atrous convolutions and temporal attention layers is The final output is processed through seven paths consisting of two one-dimensional convolutional layers and one sigmoid layer to generate action probability sequences at left 1 / 4, right 1 / 4, left 1 / 8, right 1 / 8, left 1 / 16, right 1 / 16, and center positions.

[0108] 4. Score Fusion:

[0109] First, filter the boundary probabilities output by the offset window Transformer module and retain the positions with a probability greater than 50% of the maximum probability or at the peak. and the end probability The judgment is made to obtain a set of locations that meet the conditions. Then the locations in the two sets are matched one by one. If the duration is less than the maximum duration D set by the data set, a candidate nomination is generated. , respectively represent the start time, end time, the probability of the action occurring at the start position and the probability of the action ending at the end position, the classification confidence of the nomination and the regression confidence of the nomination. The latter two parameters can be obtained from and obtained in the matrix.

[0110] Using a similar method to filter boundary positions, we take the 1 / 16 position of the action as an example for the internal probability sequence of the action output by the multi-scale dilated convolutional attention module. We filter out the left 1 / 16 position and the right 1 / 16 position of the action based on the 1 / 16 probability sequence of the action, and generate the action internal region nomination after pairing. , and They represent 1 / 16 time and 1 / 16 time of the action respectively. Here, we no longer select the position greater than 50% of the maximum probability like we did when filtering the boundary position, but the position greater than 70% of the maximum probability, because the quality of the selected 1 / 16 position must be high enough. Similarly, we select 1 / 8 and 1 / 4 of the action and put them into the internal nomination set of the action.

[0111] For each candidate nomination , the boundary probability score is combined with the nomination confidence score to get the total confidence score , as shown in Formula 5.

[0112]

[0113] In order to obtain a more accurate confidence score, the number of internal region nominations searched within the start and end time interval of each candidate nomination is recorded as . The larger the value, the more credible the candidate nomination is. However, generally speaking, the longer the nomination duration is, the more nominations the central area that meets the conditions will have, so the nomination duration should also be considered. Use Formula 6 to calculate the final nomination confidence.

[0114]

[0115] The enhancement factor is a value greater than 1, is the duration of the nomination. Finally, the final temporal action nomination set is obtained as shown in Formula 7.

[0116]

[0117] 5. Nomination suppression:

[0118] For the final timed action nominations , which contains a large number of overlapping nominations, is suppressed using Soft-NMS. However, when calculating the Gaussian decay function, it only uses the intersection over union (IoU), ignoring the impact of nomination confidence, and high-quality candidate nominations are still easily suppressed. In this application, a more flexible nomination suppression algorithm DS-NMS (Dynamic Soft-NMS) is proposed to reduce the degree of confidence decay of high-confidence nominations. See the effect diagram. Figure 4 .

[0119] For Soft-NMS, all nominations are sorted from high to low according to their confidence. Then the nomination with the highest confidence is selected. and compare it with the rest of the nominations Perform IoU calculation, if the overlap is greater than the overlap threshold , then the confidence scores of the remaining nominations are reduced by the Gaussian decay function. As for DS-NMS, the present application redesigns the parameter form of the Gaussian decay function in Soft-NMS, as shown in Equations 8 and 9.

[0120]

[0121]

[0122] represents the attenuation function, k is the parameter of the attenuation function, Representative Nomination and The intersection ratio between Nomination The confidence level, Nomination The confidence level, Set to 0.6, Set to 0.5, Set to 0.2. The bigger The larger the value, the better the nominations with high quality are retained. For nominations with better quality but relatively low confidence, the confidence becomes lower after Soft-NMS, but this attenuation effect is weakened after DS-NMS, making the nomination more likely to be retained in subsequent indicator calculations.

[0123] Figure 5 The final detection result obtained by using the process of this embodiment is shown.

[0124] In this embodiment, NVIDIA GPU graphics card is used as the computing platform, and Pytorch is used as the backbone network model framework.

[0125] Download the THUMOS-14 dataset to your local computer. The THUMOS-14 dataset is a subset of the UCF101 dataset. The validation set and test set contain 1,010 videos and 1,574 videos, respectively, with a total of 20 categories. For the action nomination generation and detection tasks, there are 200 and 212 videos with time annotations in the validation set and test set, respectively. Our model is trained on the validation set and evaluated on the test set.

[0126] For the THUMOS-14 dataset, the sampling interval during feature extraction They are set to 5 respectively. Because the length of each video is different, the video feature sequence is cropped, and an overlapping window with a size of L = 256 and a step size of 128 is used as the input of the network. The maximum duration D is set to 64, which can cover nearly 98% of the action instances.

[0127] During the training phase, the training batch size was set to 8, and a total of 10 rounds of training were performed. The learning rate for the first 7 rounds was set to 0.001, and the learning rate for the last 3 rounds was reduced to 0.0001, and the weight decay was 0.0001.

[0128] Enhancement coefficient in the fractional fusion stage The value of is set to 10.

[0129] Embodiment 3:

[0130] This embodiment also proposes a temporal action nomination generation system, which includes an acquisition unit, a first processing unit, a second processing unit, and an output unit. The acquisition unit is used to acquire an uncropped video frame sequence. The first processing unit extracts the uncropped video frame sequence at a fixed time interval to obtain a video frame segment; the video frame segment is sent to a dual-stream network consisting of a spatial stream network and a temporal stream network to obtain dual-stream features corresponding to the video frame segment. The second processing unit is used to input the dual-stream features into a backbone network consisting of an offset window Transformer module, a nomination evaluation module, and a multi-scale hole convolution attention module; the offset window Transformer module predicts the start and end probability sequence of the action in the video frame segment; the nomination evaluation module uniformly samples the nomination-level features and obtains the nomination confidence map; the multi-scale hole convolution attention module predicts multiple internal positions of the action in the video frame segment to generate multiple internal position probability sequences of the action. The output unit generates candidate nominations based on the start and end probability sequence, the nomination confidence map, and the internal position probability sequence, performs score fusion, and outputs the final temporal action nomination set.

[0131] When the time sequence action nomination generation system is running, the time sequence action nomination generation method combined with the internal position of the action can be executed. The specific process is described in Embodiment 1 and Embodiment 2, which will not be described here.

[0132] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes may be made to it in form and detail without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A method for generating temporal action nominations in combination with the internal position of the action, characterized in that: The steps include: Get the uncropped video frame sequence; Extracting an uncut video frame sequence at a fixed time interval to obtain a video frame segment; sending the video frame segment to a dual-stream network consisting of a spatial stream network and a temporal stream network to obtain dual-stream features corresponding to the video frame segment; The dual-stream features are input into a backbone network consisting of an offset window Transformer module, a nomination evaluation module, and a multi-scale dilated convolution attention module; the offset window Transformer module predicts the start and end probability sequences of actions in the video frame segment; The nomination evaluation module uniformly samples nomination-level features and obtains a nomination confidence map, which includes: Set the size to A matrix where each position in the matrix represents a nomination; T is the number of time positions, and D represents the maximum duration of a nomination that is preset; For each position in the matrix, construct a The feature is uniformly sampled from the two-stream feature, and finally we get feature map; where C represents the feature dimension and N represents the number of sampling points; A 3D convolution is used to fuse the information of all sampling points of a single nomination, and three 2D convolutions are used to fuse the features of adjacent nominations. Sigmoid is Each nomination generates a confidence level in the range [0, 1]; Get two sizes of Classification confidence map of And regression confidence map ; Classification confidence map And regression confidence map Each position (i, j) in represents a nomination with a start time of i and an end time of i+j; in the classification confidence graph The confidence value corresponding to the position (i, j) is , in the regression confidence graph The confidence value corresponding to the position (i, j) is ; The multi-scale dilated convolution attention module includes stacked multi-scale dilated convolutions and temporal attention; The multi-scale dilated convolution consists of three parallel one-dimensional dilated convolution layers. The dilation rates d of the first two layers are set to 1 and 3 respectively, and the dilation rate of the third layer is It gradually increases with the increase of network depth, and r represents the depth value; The outputs of three parallel one-dimensional dilated convolutional layers are connected along the channel dimension and fused using one-dimensional convolution with a kernel size of 1. The mathematical expression is as follows: ； ； In the formula, is a nonlinear activation function; represents a dilated convolution with a dilation rate of 1; represents a dilated convolution with a dilation rate of 3; The void rate is dilated convolution; Indicates splicing by channel dimension; Represents the output of the concatenation of the results of multiple layers of dilated convolution; Represents the output features of multi-scale dilated convolution; The temporal attention uses the ReLU activation function to convert the output features of the multi-scale dilated convolution After two convolutional layers, the final output consists of seven paths including two one-dimensional convolutional layers and one sigmoid layer, thus generating action probability sequences of left 1 / 4, right 1 / 4, left 1 / 8, right 1 / 8, left 1 / 16, right 1 / 16 and center positions; The multi-scale dilated convolutional attention module predicts multiple internal positions of actions in the video frame segment and generates a probability sequence of internal positions of multiple actions; Based on the start and end probability sequences, the nomination confidence map, and the internal position probability sequence, candidate nominations are generated and score fusion is performed to output a final temporal action nomination set.

2. The method for generating time-series action nominations in combination with the internal position of an action according to claim 1, characterized in that: The uncropped video frame sequence is represented as ;in represents the tth frame in the video; Indicates the length of the video; The temporal annotation set of the uncropped video frame sequence U consists of a series of temporal action instances, represented as ;in is the number of actual action instances; and Respectively represent action instances The start and end time of Given an uncropped video frame sequence U, the task of temporal action nomination generation aims to generate a comprehensive coverage The Timed Action Nominations , where a nomination is defined as a video clip that may contain an action.

3. The method for generating time sequence action nominations in combination with the internal position of the action according to claim 2, characterized in that: The fixed time interval is used to extract the uncut video frame sequence to obtain the video frame segment, which is expressed as follows: ； In the formula, represents a video frame segment; S represents a video frame segment sequence consisting of multiple video frame segments; ,in Indicates the length of the video. For fixed time intervals.

4. The method for generating time sequence action nominations in combination with the internal position of the action according to claim 3, characterized in that: The spatial stream network uses a single frame image as input; the temporal stream network uses stacked optical flows As input; Segment the video frame Send it to the dual-stream network for feature extraction to obtain the dual-stream features corresponding to the current video frame segment , in represents the output features of the spatial stream network, is the output feature of the temporal stream network; The output features of the spatial stream network And the output features of the temporal stream network Concatenate as the input of the backbone network; the input feature size is , where L is the time series length of the feature and C is the feature dimension.

5. The method for generating time sequence action nominations in combination with the internal position of the action according to claim 1, characterized in that: The offset window Transformer module includes a general window and an offset window. The input of the offset window is connected to the output of the general window after calculation. The calculation process is as follows: ； ； ； ； Where LN(*) represents the layer normalization layer, W-MSA(*) represents the window multi-head self-attention layer, SW-MSA(*) represents the offset window multi-head self-attention layer, MLP(*) represents the fully connected layer, represents the output of the first fully connected layer, represents the output of the previous offset window Transformer module, represents the output of the next offset window Transformer module, Represents the output after the general window calculates the multi-head attention, Represents the output after calculating the multi-head attention with the offset window.

6. The method for generating time sequence action nominations in combination with internal positions of actions according to claim 5, characterized in that: The input feature F is input to the offset window Transformer module, and the offset window Transformer module projects the dual-stream features onto the linear transformation to obtain the feature map Q F , K F , the calculation process is as follows: ； Where A is the generated attention map, the input feature F is the dual-stream feature corresponding to the video frame segment; d is the feature map Q F and K F Dimensions; Represents the feature map K F The transpose of the attention map represents the feature map Q F And feature map K F The similarity between The output of the offset window Transformer module will finally pass through two network layers consisting of two layers of 1D convolution and one Sigmoid layer to obtain the start and end probability sequence of the action in the video frame segment.

7. The method for generating time sequence action nominations in combination with the internal position of the action according to claim 1, characterized in that: The generating candidate nominations based on the start and end probability sequences, the nomination confidence map, and the internal position probability sequence and performing score fusion specifically includes: The start and end probability sequences output by the offset window Transformer module are screened, and positions with a probability greater than 50% of the maximum probability or at the peak are retained; the start probability value and the end probability value of each position are judged respectively to obtain two sets of positions that meet the conditions; Pair the positions in the two sets one by one. If the duration is less than the maximum duration D set by the data set, a candidate nomination is generated. ; , , , , , Respectively represent the start time, end time, start probability value, end probability value, confidence value corresponding to the classification confidence, and confidence value corresponding to the regression confidence; For each candidate nomination , the starting probability value, the ending probability value, the confidence value corresponding to the classification confidence, and the confidence value corresponding to the regression confidence are integrated to obtain the total confidence score : ； Based on the overall confidence score , calculate the final nomination confidence : ； In the formula, Indicates the number of internal region nominations searched within the start and end time interval of each candidate nomination; Indicates the duration of the nomination; is the enhancement coefficient, which is a value greater than 1; Output the final set of sequential action nominations .

8. A time-series action nomination generation system, characterized in that: Used to implement the method for generating a time sequence action nomination in combination with an internal position of an action according to any one of claims 1 to 7; the time sequence action nomination generation system comprises: An acquisition unit; the acquisition unit is used to acquire an uncut video frame sequence; A first processing unit; the first processing unit extracts an uncut video frame sequence at a fixed time interval to obtain a video frame segment; the video frame segment is sent to a dual-stream network consisting of a spatial stream network and a temporal stream network to obtain dual-stream features corresponding to the video frame segment; The second processing unit is used to input the dual-stream features into a backbone network consisting of an offset window Transformer module, a nomination evaluation module, and a multi-scale hole convolution attention module; the offset window Transformer module predicts the start and end probability sequences of actions in the video frame segment; the nomination evaluation module uniformly samples the nomination-level features and obtains the nomination confidence map; the multi-scale hole convolution attention module predicts multiple internal positions of the action in the video frame segment and generates multiple internal position probability sequences of the action; Output unit: The output unit generates candidate nominations based on the start and end probability sequences, the nomination confidence map, and the internal position probability sequence, performs score fusion, and outputs a final temporal action nomination set.

Images