Temporal action localization method based on convolution and multi-head self-attention

Abstract

The present invention discloses a temporal action localization method based on convolution and multi-head self-attention, which belongs to the field of deep learning video understanding. The method of the present invention first extracts each video frame in the video to be identified as a video representation sequence, and uses it as the input of a multi-time span context model, and then stacks the feature sequences output by each layer of the backbone network of the encoder to form a feature pyramid, and finally the prediction module outputs the final action classification result and the corresponding action start time point and action end time point to achieve accurate positioning of the time point, thereby reducing the error caused by a single regression head. The backbone network integrates an instantaneous average convolution module with nonlinear branches and an instantaneous average void convolution module, and combines it with a multi-head self-attention module to enhance the ability to learn local features and the distinction between adjacent frame features. In addition, the instantaneous average void convolution module integrates void convolution to enhance the ability to learn deep features.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning video understanding, and in particular relates to a temporal action localization method based on convolution and multi-head self-attention. Background Art

[0002] Temporal action detection (TAD) is a daunting challenge in the field of video understanding. With the development of the modern Internet, the video content transmitted and stored online has become increasingly diverse, bringing great benefits and development to social progress. However, the surge in this diverse content has also brought new challenges to research in areas such as Internet video data analysis and video data security. For example, scenarios such as intelligent monitoring systems, sports video analysis, film and television content management, medical health monitoring, and human-computer interaction combined with virtual reality applications. Factors such as complex action scenarios and varying video quality affect the accuracy of temporal action localization in these videos. In addition, the pursuit of high-quality temporal action localization requires a lot of computing resources, time, and energy, which further increases the difficulty.

[0003] In the TAD task, it is crucial to choose a suitable backbone network to extract appropriate features. Convolutional neural networks (CNNs) are widely used and have achieved remarkable results in many fields of deep learning, especially in the field of vision. Given that video data contains spatial information and local correlations, CNNs can capture high-quality local information at a low computational cost, and thus show excellent performance in the video field, such as applications in human-computer interaction and sports video analysis. Since its introduction in the field of natural language processing (NLP), the Transformer model has attracted much attention for its ability to capture and learn long-range contextual information. It not only performs well in natural language processing, but also has achieved good results in the field of video recognition, such as applications in sports video analysis and health management.

[0004] Most existing models use pure convolution or pure Transformer to extract features from video sequences. However, convolution and Transformer are good at extracting local and global features, respectively, which may lead to insufficient utilization of the other type of features. In addition, most models do not address the issue of similarity between adjacent video frames. Finally, using a single regression detection head to predict the boundaries of action instances may lead to inaccuracies.

[0005] In order to address the challenge of utilizing both local and global features, a new approach is proposed in the prior art, inspired by the Vision Transformer (ViT), which integrates a CNN module in front of the Transformer module to form a CNN-Transformer hybrid architecture. The advantages of CNN and Transformer in capturing local and global features, respectively, are used to build a more powerful video sequence coding model. To alleviate the problem of similarity between adjacent frames, nonlinear branches are introduced into the model to enhance the distinction between adjacent frames. In addition, in order to enhance the learning ability of deep features, dilated convolutional layers are added to the convolutional modules of the deep network. Finally, in order to overcome the potential errors that may be caused by a single detection head in Actionformer, the Tridet detection head is improved to enhance the detection ability of video action classification boundaries. Summary of the invention

[0006] The purpose of the present invention is to solve the above-mentioned technical problems existing in the prior art and to provide a temporal action localization method based on convolution and multi-head self-attention, which can improve the accuracy of temporal action localization in action videos.

[0007] In order to achieve the above-mentioned invention object, the present invention specifically adopts the following technical solutions:

[0008] A temporal action localization method based on convolution and multi-head self-attention includes the following steps:

[0009] S1. Each video frame in the video to be identified is extracted into a video representation sequence by the trained convolutional projection layer, and each video representation sequence is used as the input of the trained multi-time span context model. It is first processed by the encoder, and the feature sequences output by each layer of the backbone network of the encoder are stacked to form a feature pyramid.

[0010] The encoder comprises a 2N-layer backbone network, each backbone network layer belonging to the first N layers comprises an instantaneous average convolution module, a multi-head self-attention module and a multi-layer perceptron, and each backbone network layer belonging to the last N layers comprises an instantaneous average hole convolution module, a multi-head self-attention module and a multi-layer perceptron; wherein the instantaneous average convolution module comprises an instantaneous feature capture module, a nonlinear instantaneous feature capture module, a time period average feature capture module and a nonlinear time period average feature capture module; the instantaneous average hole convolution module comprises an instantaneous feature capture module, a nonlinear instantaneous feature capture module, a time period average feature capture module, a nonlinear time period average feature capture module and a hole convolution time period average feature capture module;

[0011] S2. Input the feature pyramid into a prediction module, and output the final action classification result and the corresponding action start time point and action end time point; wherein the prediction module includes a class regression head, an action start regression head, and an action end regression head.

[0012] Based on the above scheme, each step can be implemented in the following preferred specific manner.

[0013] Preferably, in step S1, the specific processing process in the nth layer backbone network belonging to the first N layers is as follows:

[0014] The input feature sequence of the nth layer is input into the instantaneous average convolution module to obtain the output feature of the nth layer instantaneous average convolution module. The output feature of the nth layer instantaneous average convolution module is input into the multi-head self-attention module after layer normalization to obtain the first output feature of the nth layer. The first output feature of the nth layer is added to the output feature of the nth layer instantaneous average convolution module and then input into the multi-layer perceptron after layer normalization to obtain the second output feature of the nth layer. The output feature of the nth layer instantaneous average convolution module and the first and second output features of the nth layer are added to obtain the output feature sequence of the nth layer. After the output feature sequence of the nth layer passes through the downsampling layer, the input feature sequence of the n+1th layer is obtained.

[0015] Preferably, the specific processing process in the instantaneous average convolution module is as follows:

[0016] The input feature sequence of the nth layer is first layer-normalized to obtain a first input feature sequence, and the first input feature sequence is respectively input into the instantaneous feature capture module, the nonlinear instantaneous feature capture module, the time period average feature capture module and the nonlinear time period average feature capture module, to obtain the instantaneous features, nonlinear instantaneous features, time period average features and nonlinear time period average features of the instantaneous average convolution module of the nth layer, and the instantaneous features, nonlinear instantaneous features, time period average features and nonlinear time period average features of the instantaneous average convolution module of the nth layer are added to the first input feature sequence as the output features of the instantaneous average convolution module of the nth layer.

[0017] As a preferred embodiment, the specific processing process in the nth layer backbone network belonging to the latter N layers is as follows:

[0018] The input feature sequence of the nth layer is input into the instantaneous average dilated convolution module to obtain the output features of the nth layer instantaneous average dilated convolution module. The output features of the nth layer instantaneous average dilated convolution module are input into the multi-head self-attention module after layer normalization to obtain the third output features of the nth layer. The third output features of the nth layer are added to the output features of the nth layer instantaneous average dilated convolution module and then input into the multi-layer perceptron after layer normalization to obtain the fourth output features of the nth layer. The output features of the nth layer instantaneous average dilated convolution module and the third and fourth output features of the nth layer are added to obtain the output feature sequence of the nth layer. After the output feature sequence of the nth layer passes through the downsampling layer, the input feature sequence of the n+1th layer is obtained.

[0019] Preferably, the specific processing process in the instantaneous average dilated convolution module is as follows:

[0020] The input feature sequence of the nth layer is first layer-normalized to obtain a second input feature sequence, and the second input feature sequence is respectively input into the instantaneous feature capture module, the nonlinear instantaneous feature capture module, the time period average feature capture module, the nonlinear time period average feature capture module and the void convolution time period average feature capture module, so as to obtain the instantaneous feature, nonlinear instantaneous feature, time period average feature, nonlinear time period average feature and void convolution time period average feature of the nth layer instantaneous average void convolution module accordingly, and the instantaneous feature, nonlinear instantaneous feature, time period average feature, nonlinear time period average feature and void convolution time period average feature of the nth layer instantaneous average void convolution module are added with the second input feature sequence as the output feature of the nth layer instantaneous average void convolution module.

[0021] Preferably, in the instantaneous average hole convolution module, the second input feature sequence is subjected to a first instantaneous convolution process to obtain a first feature map, the second input feature sequence is subjected to a first long-term instantaneous convolution process to obtain a second feature map, and the first feature map and the second feature map are multiplied to obtain an instantaneous feature output by the instantaneous average hole convolution module;

[0022] In the nonlinear instantaneous feature capture module, the second input feature sequence is subjected to a second instantaneous convolution process to obtain a third feature map, the second input feature sequence is subjected to a second long-term instantaneous convolution process to obtain a fourth feature map, the third feature map and the fourth feature map are multiplied to obtain a fifth feature map, and the fifth feature map is subjected to a ReLu activation function process to obtain a nonlinear instantaneous feature output by the nonlinear instantaneous feature capture module;

[0023] In the time period average value feature capture module, the second input feature sequence is subjected to a first average time period convolution process to obtain a sixth feature map, the second input feature sequence is subjected to a first large-scale average time period convolution process to obtain a seventh feature map, the sixth feature map and the seventh feature map are added to obtain an eighth feature map, and the sixth feature map and the eighth feature map are multiplied to obtain the time period average value feature output by the time period average value feature capture module;

[0024] In the nonlinear time period average value feature capture module, the second input feature sequence is subjected to a second average time period convolution process to obtain a ninth feature map, the second input feature sequence is subjected to a second largest range average time period convolution process to obtain a tenth feature map, the ninth feature map is added to the tenth feature map to obtain an eleventh feature map, the ninth feature map is subjected to a ReLu activation function process and then multiplied with the eleventh feature map to obtain a twelfth feature map, and the twelfth feature map is subjected to a ReLu activation function process to obtain a nonlinear time period average value feature output by the nonlinear time period average value feature capture module;

[0025] In the hole convolution time period average value feature capture module, the second input feature sequence is subjected to a third average time period convolution process to obtain a thirteenth feature map, the second input feature sequence is subjected to a large range average time period hole convolution process to obtain a fourteenth feature map, the thirteenth feature map and the fourteenth feature map are added to obtain a fifteenth feature map, the thirteenth feature map and the fifteenth feature map are multiplied to obtain the hole convolution time period average value feature output by the hole convolution time period average value feature capture module.

[0026] Preferably, the first long time period instantaneous convolution and the second long time period instantaneous convolution are both convolution layers with a convolution kernel of 3; the first instantaneous convolution and the second instantaneous convolution are both convolution layers with a convolution kernel of 1; the first average time period convolution, the second average time period convolution and the third average time period convolution are all convolution layers with a convolution kernel of 1; the first large-range average time period convolution and the second large-range average time period convolution are both convolution layers with a convolution kernel of 11; the large-range average time period void convolution is a convolution layer with a convolution kernel of 11 and an expansion rate of 2.

[0027] Preferably, in the encoder, N is a positive integer greater than or equal to 2.

[0028] Preferably, in the multi-head self-attention module, the number of self-attention heads is 2.

[0029] Preferably, when training the multi-time span context model, the batch size is set to 8, the dropout rate is set to 0.1, and the activation function is set to ReLU.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The method of the present invention first pre-processes the video to be identified to obtain a video representation sequence, which is used as the input of the multi-time span context model. It is first processed by an encoder. In the backbone network of the encoder, the instantaneous average convolution module and the instantaneous average hole convolution module are used to obtain local context information, and the multi-head self-attention module is used to obtain global context information. Finally, the feature sequence output by each layer of the backbone network of the encoder is stacked to form a feature pyramid. The prediction module composed of the Tridet detection head outputs the final action classification result and the corresponding action start time point and action end time point. The present invention has been tested on three public data sets, proving that the multi-time span context model has excellent temporal action localization performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flow chart of the steps of the method of the present invention;

[0033] Figure 2 It is a schematic diagram of the framework of the method of the present invention;

[0034] Figure 3 Schematic diagram of the architecture of the instantaneous average convolution module IACM in the method of the present invention;

[0035] Figure 4 Schematic diagram of the architecture of the instantaneous average dilated convolution module IADCM in the method of the present invention;

[0036] Figure 5 Detailed structural diagram of the instantaneous average convolution module in the method of the present invention;

[0037] Figure 6 Detailed structural diagram of the instantaneous average dilated convolution module in the method of the present invention;

[0038] Figure 7 It is a schematic diagram of the backbone network and neck structure of the method of the present invention;

[0039] Figure 8 It is a schematic diagram of the prediction module structure in the method of the present invention. DETAILED DESCRIPTION

[0040] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0041] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0042] like Figure 1 As shown, in a preferred implementation of the present invention, the above-mentioned temporal action localization method based on convolution and multi-head self-attention includes the following steps S1 to S2. The specific implementation process is described below.

[0043] S1. Each video frame in the video to be identified is extracted into a video representation sequence by the trained convolutional projection layer. Each video representation sequence is used as the input of the trained multi-time span context model. It is first processed by the encoder, and the feature sequences output by each layer of the encoder backbone network are stacked to form a feature pyramid.

[0044] like Figure 2 As shown, the encoder includes a 2N-layer backbone network, each backbone network layer belonging to the first N layers includes an instantaneous average convolution module (IACM), a multi-head self-attention module and a multi-layer perceptron, and each backbone network layer belonging to the last N layers includes an instantaneous average hole convolution module (IADCM), a multi-head self-attention module and a multi-layer perceptron; wherein the instantaneous average convolution module (IACM) includes an instantaneous feature capture module, a nonlinear instantaneous feature capture module, a time period average feature capture module and a nonlinear time period average feature capture module; the instantaneous average hole convolution module (IADCM) includes an instantaneous feature capture module, a nonlinear instantaneous feature capture module, a time period average feature capture module, a nonlinear time period average feature capture module and a hole convolution time period average feature capture module.

[0045] It should be noted that in the present invention, a multi-time span context model is constructed based on a multi-head self-attention mechanism, which includes a multi-layer backbone network composed of an instantaneous average convolution module (IACM), an instantaneous average hole convolution module (IADCM), a multi-head self-attention module, and a multi-layer sensor. Among them, the first N layers of the backbone network include an instantaneous average convolution module (IACM) and a multi-head self-attention module, and the last N layers of the backbone network include an instantaneous average hole convolution module (IADCM) and a multi-head self-attention module. The combination of the instantaneous average convolution module (IACM) or the instantaneous average hole convolution module (IADCM) and the multi-head self-attention module balances the backbone network's ability to capture global features and local features, and improves the robustness of the backbone network. Subsequently, the feature sequence output by each layer of the backbone network is stacked to form a feature pyramid and used as the input of the prediction module composed of the improved Tridet detection head, which helps to improve the accuracy of the prediction module in classifying actions at different time scales.

[0046] The following introduces the specific processing process of the first N-layer backbone network and the last N-layer backbone network of the multi-time span context model.

[0047] like Figure 2 As shown in FIG. 1 , the specific processing process in the nth layer backbone network belonging to the first N layers is as follows: the input feature sequence of the nth layer is input into the instantaneous average convolution module to obtain the output feature of the instantaneous average convolution module of the nth layer; the output feature of the instantaneous average convolution module of the nth layer is input into the multi-head self-attention module after layer normalization to obtain the first output feature of the nth layer; the first output feature of the nth layer is added to the output feature of the instantaneous average convolution module of the nth layer and then input into the multi-layer perceptron after layer normalization to obtain the second output feature of the nth layer; the output feature of the instantaneous average convolution module of the nth layer and the first output feature and the second output feature of the nth layer are added to obtain the output feature sequence of the nth layer; after the output feature sequence of the nth layer passes through the downsampling layer, the input feature sequence of the n+1th layer is obtained.

[0048] like Figure 2As shown in the figure, the specific processing process in the n-th layer backbone network belonging to the last N layers is as follows: the input feature sequence of the n-th layer is input into the instantaneous average dilated convolution module to obtain the output feature of the n-th layer instantaneous average dilated convolution module; the output feature of the n-th layer instantaneous average dilated convolution module is input into the multi-head self-attention module after layer normalization to obtain the third output feature of the n-th layer; the third output feature of the n-th layer is added to the output feature of the n-th layer instantaneous average dilated convolution module and then input into the multi-layer perceptron after layer normalization to obtain the fourth output feature of the n-th layer; the output feature of the n-th layer instantaneous average dilated convolution module and the third and fourth output features of the n-th layer are added to obtain the output feature sequence of the n-th layer; after the output feature sequence of the n-th layer passes through the downsampling layer, the input feature sequence of the n+1-th layer is obtained.

[0049] In the present invention, the output features of the instantaneous average dilated convolution module and the output features of the instantaneous average dilated convolution module can be converted into the output features of the backbone network according to the following formula:

[0050]

[0051] in, Represents the output features of the instantaneous average convolution module of the nth layer Or the output feature of the nth layer instantaneous average dilated convolution module ; Represents the output feature sequence of the nth layer of the backbone network; Representation layer normalization; represents a multi-head self-attention module; Represents a multi-layer perceptron.

[0052] Furthermore, the output feature sequence of the nth layer of the backbone network is passed through the downsampling layer to obtain the input feature sequence of the n+1th layer ,in, Represents a downsampling layer with a sampling rate of 2.

[0053] In the encoder of the multi-time span context model of the present invention, N can be a positive integer greater than or equal to 2. In this embodiment, Figure 7 As shown in the figure, the encoder contains 6 layers of backbone network (backbone network), that is, taking N=3 as an example, the backbone networks of the first 3 layers contain instantaneous average convolution modules, and the backbone networks of the last 3 layers contain instantaneous average hole convolution modules. The output features of each layer of the backbone network are used to construct a feature pyramid, and the output features of each layer of the backbone network are used as the input of the next layer of the backbone network after the downsampling layer.

[0054] It should be noted that the specific processing process in the instantaneous average convolution module is as follows: Figure 3As shown, the input feature sequence of the nth layer is first layer-normalized to obtain a first input feature sequence, and the first input feature sequence is respectively input into the instantaneous feature capture module, the nonlinear instantaneous feature capture module, the time period average feature capture module and the nonlinear time period average feature capture module, and the instantaneous feature, nonlinear instantaneous feature, time period average feature and nonlinear time period average feature of the instantaneous average convolution module of the nth layer are correspondingly obtained, and the instantaneous feature, nonlinear instantaneous feature, time period average feature and nonlinear time period average feature of the instantaneous average convolution module of the nth layer are added to the first input feature sequence as the output feature of the instantaneous average convolution module of the nth layer.

[0055] It should be noted that the specific processing process in the instantaneous average hole convolution module is as follows: Figure 4 As shown, the input feature sequence of the nth layer is first layer-normalized to obtain a second input feature sequence, and the second input feature sequence is respectively input into the instantaneous feature capture module, the nonlinear instantaneous feature capture module, the time period average feature capture module, the nonlinear time period average feature capture module and the void convolution time period average feature capture module, so as to obtain the instantaneous feature, nonlinear instantaneous feature, time period average feature, nonlinear time period average feature and void convolution time period average feature of the nth layer instantaneous average void convolution module, and add the instantaneous feature, nonlinear instantaneous feature, time period average feature, nonlinear time period average feature and void convolution time period average feature of the nth layer instantaneous average void convolution module to the second input feature sequence as the output feature of the nth layer instantaneous average void convolution module.

[0056] In this embodiment, for the backbone network of the first N layers, the input feature sequence of the nth layer is first layer-normalized to obtain the first input feature sequence and input it into the instantaneous average convolution module (IACM). The instantaneous average convolution module (IACM) includes four parallel feature capture modules and residual connections, wherein the four feature capture modules are: an instantaneous feature capture module, a nonlinear instantaneous feature capture module, a time period average feature capture module, and a nonlinear time period average feature capture module. Finally, the output features of the four feature capture modules and the input feature sequence of the nth layer after layer normalization are residually connected to obtain the output features of the instantaneous average convolution module IACM. For the backbone network of the last N layers, the input feature sequence of the nth layer is first layer-normalized to obtain the second input feature sequence and input it into the instantaneous average dilated convolution module (IADCM). The instantaneous average atrous convolution module (IADCM) contains five parallel feature capture modules and residual connections, where the five feature capture modules are: instantaneous feature capture module, nonlinear instantaneous feature capture module, time period average feature capture module, nonlinear time period average feature capture module and atrous convolution time period average feature capture module. The output features of the five feature capture modules and the input feature sequence of the nth layer after layer normalization are residually connected to obtain the output features of the instantaneous average atrous convolution module IADCM.

[0057] The specific processing procedures of the instantaneous feature capture module, nonlinear instantaneous feature capture module, time period average feature capture module, nonlinear time period average feature capture module and void convolution time period average feature capture module are introduced below.

[0058] 1) Instantaneous feature capture module

[0059] As for the instantaneous feature capture module, it is a module shared by the instantaneous average convolution module (IACM) and the instantaneous average dilated convolution module (IADCM), which is used to capture the instantaneous level features in the video representation sequence.

[0060] Specifically, in the instantaneous average dilated convolution module of the present invention, the second input feature sequence After the first instantaneous convolution processing, the first feature map is obtained. The second input feature sequence is subjected to the first long-term instantaneous convolution processing to obtain the second feature map. The first feature map and the second feature map are multiplied to obtain the instantaneous feature map output by the instantaneous average hole convolution module. :

[0061]

[0062] in, It is the instantaneous convolution of the first long time period; is the first instantaneous convolution.

[0063] 2) Nonlinear instantaneous feature capture module

[0064] The nonlinear instantaneous feature capture module is also a module shared by the instantaneous average convolution module (IACM) and the instantaneous average atrous convolution module (IADCM). This module is used to capture instantaneous level features in the video representation sequence, while enhancing the nonlinear components to prevent the learned features from being a linear combination of the video representation sequence and to enhance the differences in feature sequences of adjacent frames.

[0065] Specifically, in the nonlinear instantaneous feature capture module of the present invention, the second input feature sequence After the second instantaneous convolution processing, the third feature map is obtained. The second input feature sequence is subjected to the second long-term instantaneous convolution processing to obtain the fourth feature map. The third feature map and the fourth feature map are multiplied to obtain the fifth feature map. After the fifth feature map is processed by the ReLu activation function, the nonlinear instantaneous feature captured by the nonlinear instantaneous feature capture module is obtained. :

[0066]

[0067] in, is the second longest time period instantaneous convolution; is the second instantaneous convolution.

[0068] 3) Time period average feature capture module

[0069] The time period average feature capture module is also a module shared by the instantaneous average convolution module (IACM) and the instantaneous average dilated convolution module (IADCM). This module is used to capture the time interval level features in the video representation sequence.

[0070] Specifically, in the time period average feature capture module of the present invention, the second input feature sequence is subjected to the first average time period convolution processing to obtain the sixth feature map, the second input feature sequence is subjected to the first large range average time period convolution processing to obtain the seventh feature map, the sixth feature map and the seventh feature map are added to obtain the eighth feature map, the sixth feature map and the eighth feature map are multiplied to obtain the time period average feature map output by the time period average feature capture module :

[0071]

[0072] in, is the first average time period convolution; It is the first large-scale average time period convolution.

[0073] 4) Nonlinear time period average feature capture module

[0074] The nonlinear time period average feature capture module is also a module shared by the instantaneous average convolution module (IACM) and the instantaneous average atrous convolution module (IADCM). This module is used to capture the time interval level features in the video representation sequence, while enhancing the nonlinear components to prevent the learned features from being a linear combination of the video representation sequence and enhance the differences between the feature sequences of adjacent frames.

[0075] Specifically, in the nonlinear time period average value feature capture module of the present invention, the second input feature sequence is subjected to a second average time period convolution process to obtain a ninth feature map, the second input feature sequence is subjected to a second largest range average time period convolution process to obtain a tenth feature map, the ninth feature map is added to the tenth feature map to obtain an eleventh feature map, the ninth feature map is subjected to a ReLu activation function process and then multiplied with the eleventh feature map to obtain a twelfth feature map, the twelfth feature map is subjected to a ReLu activation function process to obtain a nonlinear time period average value feature map output by the nonlinear time period average value feature capture module :

[0076]

[0077] in, is the second average time period convolution, It is the second largest range of average time period convolution.

[0078] 5) Dilated convolution time period average feature capture module

[0079] The dilated convolution time segment average feature capture module is a module unique to the instantaneous average dilated convolution module (IADCM). This module is used to capture time interval level features in video representation sequences and is also used to learn deeper feature sequences, which requires a larger receptive field. Therefore, this module uses dilated convolution.

[0080] Specifically, in the hole convolution time period average feature capture module of the present invention, the second input feature sequence After the third average time period convolution processing, the thirteenth feature map is obtained. The second input feature sequence is subjected to a large range average time period void convolution processing to obtain a fourteenth feature map. The thirteenth feature map and the fourteenth feature map are added to obtain a fifteenth feature map. The thirteenth feature map and the fifteenth feature map are multiplied to obtain the void convolution time period average feature. The void convolution time period average feature capture module outputs the void convolution time period average feature. :

[0081]

[0082] in, is the third average time period convolution, It is a large-scale average time period dilated convolution.

[0083] Therefore, if Figure 5 As shown, for the instantaneous average convolution module (IACM), its output can be synthesized by adding the output features of the four modules, namely, the instantaneous feature capture module, the nonlinear instantaneous feature capture module, the time period average feature capture module, and the nonlinear time period average feature capture module, to the generated first input feature sequence to form the output feature sequence of the instantaneous average convolution module (IACM), capturing the local contextual relationship in the video representation sequence, namely:

[0084]

[0085] like Figure 6 As shown in the figure, for the instantaneous average dilated convolution module (IADCM), its output can be synthesized by adding the output features of the five modules, namely, the instantaneous feature capture module, the nonlinear instantaneous feature capture module, the time period average feature capture module, the nonlinear time period average feature capture module, and the dilated convolution time period average feature capture module, to the output feature sequence of the instantaneous average dilated convolution module (IADCM), capturing the local contextual relationship in the video representation sequence, namely:

[0086]

[0087] In this embodiment, the long-time period instantaneous convolutions in the instantaneous feature capture module and the nonlinear instantaneous feature capture module are all convolution layers with a convolution kernel of 3, and the instantaneous convolutions are all convolution layers with a convolution kernel of 1; the average time period convolutions in the time period average feature capture module, the nonlinear time period average feature capture module and the void convolution time period average feature capture module are all convolution layers with a convolution kernel of 1, and the first and second largest-range average time period convolutions are both convolution layers with a convolution kernel of 11; the large-range average time period void convolution of the void convolution time period average feature capture module is a convolution layer with a convolution kernel of 11 and an expansion rate of 2.

[0088] It should also be noted that the first N-layer backbone network or the last N-layer backbone network of the present invention contains a multi-head self-attention module, which is used to capture the global contextual relationship in the video representation sequence, and its implementation belongs to the prior art. In order to facilitate those skilled in the art to better understand the present invention, the implementation of the multi-head self-attention module is now described.

[0089] The multi-head self-attention module of the present invention specifically adopts a two-head global attention mechanism to learn the feature correlation relationship in the global range, and its calculation process can be described as follows: The input feature sequences at different times constitute the input sequence of the multi-head self-attention module ,in, represents the input feature sequence at the i-th moment, Indicates the number of moments; Represents the dimension size of the input feature sequence at each moment. A head, Projection , and , Indicates The self-attention head uses the weight matrix right The query parameters obtained by projection, Indicates The self-attention head uses the weight matrix right The key parameters obtained by projection, Indicates The self-attention head uses the weight matrix right The projected value parameter,

[0090] represents the feature dimension of each attention head, Self-attention output of self-attention heads for:

[0091]

[0092] Then, the output of each self-attention head is concatenated and passed through a linear layer. The result of multi-head self-attention is It can be expressed as:

[0093]

[0094] in, represents the weights of the linear layer, Respectively represent The self-attention output of each head; Represents the input of the multi-head self-attention module.

[0095] From the above formula, we can see that the time complexity of single-head self-attention is In order to reduce the computational complexity, the present invention replaces the single-head self-attention with multi-head self-attention, which can reduce the computational complexity to , since the window size W is larger than Much smaller, which can reduce most of the computational overhead. For the multi-head self-attention module MSA of the present invention, the effect is best when the number of self-attention heads is 2.

[0096] S2. Then, the feature pyramid is input into a prediction module, and the final action classification result and the corresponding action start time point and action end time point are output; wherein, the prediction module includes a class regression head, an action start regression head, and an action end regression head.

[0097] It should be noted that, in step S2 of the present invention, Figure 8 As shown in the figure, the input of the prediction module is the feature pyramid generated by the 2N-layer backbone network. The prediction module specifically uses the Tridet detection head, and its implementation method belongs to the prior art. The module specifically includes a class regression head, an action start regression head, and an action end regression head. The features extracted by the class regression head are affected by both classification and boundary deviation, while the features extracted by the action start regression head and the action end regression head are only affected by boundary deviation. The convolution methods of the class regression head, the action start regression head, and the action end regression head are also different.

[0098] The specific processes in the class return header, the action start return header, and the action end return header are described in detail below to facilitate understanding by those skilled in the art.

[0099] In the class regression head, the specific process is as follows: For the feature sequence in the k-th layer feature pyramid , the probability distribution of the feature sequence of each frame becoming the boundary is obtained through the convolution layer and softmax. At the same time, in order to prevent similar features from being indistinguishable, the present invention converts the feature sequence First, it is processed by the normal convolution branch, then by layer normalization, one-dimensional convolution, layer normalization, one-dimensional convolution, and then by ReLu output decoding feature sequence. Finally, the action classification result of this video frame can be obtained based on this decoding feature sequence and the back propagation algorithm. ,Since different categories have different predicted probabilities, the video frame with the highest probability among ,the video frames that continuously predict the same classification will be selected as the ,classification center.

[0100] According to the above description, the class regression head output process can be expressed as:

[0101]

[0102] In the action start regression head and the action end regression head, for the predicted central moment t of each output category C, when the central moment t is within the duration of an action category, the length of the selected time period is 2×r+1, and a period of time frames[tr:t] and frames[t:t+r] to the left of the central moment t is selected, so that each time period contains the feature vectors of r+1 moments.

[0103] Then the frames[tr:t] time period is input into the action start regression head, which is implemented through a 1D convolutional network. It combines the action boundary offset and the offset of the action classification boundary relative to other time periods to model the action, and outputs the probability distribution from tr to t as the start action boundary. :

[0104]

[0105] in, Represents the offset of each moment in the tr:t time period relative to the action boundary; The features extracted by the class regression head for the video frames at different times tr:t in the area to the left of the classification center representing category C.

[0106] Then, by multiplying the predicted probability of all moments in each segment by the integral of the expected distance between the feature and the boundary at time t, the predicted offset of the start boundary of the action at time t can be obtained. :

[0107]

[0108] in, is the time offset relative to the cluster center moment; It represents the probability that the time tR is predicted to be the start time of action classification (R is a value between 1 and r).

[0109] Similarly, the time period frames[t:t+r] is input into the action end regression head, and the probability distribution from time t to time t+r is output as the end action boundary. :

[0110]

[0111] in, Represents the offset of each moment in the time period t:t+r relative to the action boundary; The features extracted by the class regression head for the video frames at different times t: t+r on the left side of the classification center representing category C.

[0112] Then, by multiplying the predicted probability of all moments in each segment by the integral of the expected distance between the feature and the boundary at time t, the predicted offset of the end boundary of the action at time t can be obtained. :

[0113]

[0114] in, is the time offset relative to the cluster center moment; It represents the probability that the time t+R is predicted to be the end time of the action classification (R is a value between 1 and r).

[0115] Then, we can get the start and end time of the action to which the feature sequence at the tth moment belongs, which is output by the kth layer of the feature pyramid. That is, the positions located by the class regression head, the action start regression head, and the action end regression head at different feature pyramid layers are averaged. Since each layer is compressed in the time dimension after downsampling, that is, each layer of the feature pyramid must be downsampled to shorten the time axis of the feature sequence to half of the original, it is necessary to Restore the original time multiplier and decompress to get the original position:

[0116]

[0117]

[0118] in, is the starting time point of the action (i.e., the time boundary when the action starts); It is the end time point of the action (i.e. the time boundary when the action ends).

[0119] The present invention will now use a specific example to demonstrate the application effect of the temporal action localization method based on convolution and multi-head self-attention described in S1 to S2 in the above embodiments on a specific data set, so as to facilitate understanding of the essence of the present invention.

[0120] Example

[0121] The specific implementation process of the temporal action localization method based on convolution and multi-head self-attention used in this embodiment is as described above, and will not be repeated here. The specific data set, some specific parameter settings and implementation results of this embodiment are mainly demonstrated. And for the convenience of description, the method shown in steps S1~S2 is referred to as the method of the present invention, and the multi-time span context model used is recorded as IACFormer.

[0122] This embodiment uses three widely used real-scene datasets, namely: THUMOS14, ActivityNet1.3, and EPIC-Kitchens100. Among them, the THUMOS14 dataset contains 413 complete original videos and time boundary annotations of 20 action categories. The embodiment of the present invention uses 200 videos from the validation set for training and 213 videos from the test set for evaluation; the ActivityNet1.3 dataset is a large dataset, which includes more than 10,000 training videos, more than 4,000 validation videos, and more than 5,000 test videos. The ActivityNet1.3 dataset contains 200 actions, specifically covering sports, home and work actions. ActivityNet1.3 contains only 1.5 actions per video on average, and most videos contain a single action category, with an average of 36% of the content being background; the EPIC-Kitchens100 dataset is a comprehensive collection of action videos from a first-person perspective, which contains 100 hours of footage from 700 sessions, recording various kitchen cooking activities. In addition, compared with the THUMOS14 dataset, the total video time of the EPIC-Kitchens100 dataset is three times that of the THUMOS14 dataset, and the video time of action instances is more than ten times that of the THUMOS14 dataset.

[0123] In this embodiment, the video frames in all data sets are first encoded using a video encoder, such as using I3D networks (Inflated 3D ConvNets) and TSN networks (Temporal Segment Networks) as video encoders to obtain encoded video feature sequences (i.e., video representation sequences). First, this embodiment limits the maximum input length of data processing to 2304. For sequences whose length is much smaller or much larger than the length limit, this embodiment will perform corresponding cropping or padding to adapt. In the multi-time span context model, there are three instantaneous average convolution modules (IACM) without adding hole convolutions, and there are also three instantaneous average hole convolution modules (IADCM) with adding hole convolutions. All convolution layer parameters in the instantaneous average convolution module and the instantaneous average hole convolution module (IADCM) have been Gaussian initialized, there are six multi-head self-attention modules, and the number of multi-Attention heads in each multi-head self-attention module is 2.

[0124] In this embodiment, the multi-time span context model is trained on the Python platform, using an Intel (R) Xeon (R) Platinum 8358P CPU with an operating frequency of 2.60 GHz, and equipped with 8.00 GB memory and Nvidia Tesla A800 graphics card. Referring to the parameter settings of the prior art TriDet, the batch size is 8, the dropout rate is 0.1, the activation function is RELU, and the learning rate is set to 10^(-4) for the THUMOS14 data set and the EPIC-Kitchens 100 data set. For the ActivityNet1.3 data set, the learning rate is set to 10^(-3). Preheat training for 20 rounds and train for 20 rounds on the THUMOS14 data set, preheat training for 5 rounds and train for 14 rounds on the EPIC-Kitchens 100 noun data set, and preheat training for 5 rounds and train for 22 rounds on the EPIC-Kitchens 100 verb data set.

[0125] In addition, this embodiment also compares the method of the present invention with several traditional prediction methods. The traditional prediction methods used as comparisons are: (1) BSN: a temporal action localization model based on proposal generation; (2) BMN: a temporal action localization model based on end-to-end proposal generation based on boundary matching mechanism. (3) AFSD: an anchor-free temporal action localization model. (4) BasicTad: an end-to-end temporal action localization model that combines the existing TAD method and only uses the RGB dataset to approximate the two-stream method. (5) BCNet+PGCN: a temporal action localization model based on proposal generation using a background-assisted method. (6) ReAct: a temporal action localization model based on a relational query mechanism. (7) TALLFormer: a temporal action localization model based on a long-term memory mechanism. (8) STPT: a temporal action localization model based on a spatiotemporal pyramid transformer. (9) TCANet: a temporal action localization model that generates action proposals based on the complementarity of local context and global context. (10) TadTR: an end-to-end temporal action localization model based on Transformer. (11) ActionFormer: A temporal action localization model that combines multi-scale feature representation and local self-attention mechanism to use a lightweight decoder to classify and estimate action boundaries. (12) G-TAD: A temporal action localization model based on graph convolutional neural network. (13) P-GCN: A temporal action localization model based on graph convolutional network formed by proposal relations. (14) VSGN: A temporal action localization model based on video self-stitching and cross-scale graph pyramid network.

[0126] In order to demonstrate the effectiveness of the multi-time span context model IACFormer, the embodiment of the present invention adopts a widely used TAD evaluation indicator, namely the mAP score calculated at different time intersections on the joint tloU, and further displays the average mAP value of all tloU thresholds. For the THUMOS14 dataset and the EPIC-Kitchens100 dataset, the thresholds displayed by the embodiment of the present invention are [0.3, 0.7, 0.1] and [0.1, 0.5, 0.1] respectively. For the ActivityNet1.3 dataset, the embodiment of the present invention displays the result of the tloU threshold of [0.5, 0.75, 0.05].

[0127] In this embodiment, the CNN-based method, the Transformer-based method and the multi-time span context model IACFormer of the present invention are first compared on the THUMOS14 dataset, and the pre-trained features used by each method are shown in Table 1. As can be seen from Table 1, the multi-time span context model IACFormer of the present invention reaches the SOTA level without using features extracted from a large model, and performs well, indicating that the accuracy of IACFormer using the convolution + Transformer architecture is higher than that of the model using only CNN or only Transformer. Compared with Actionformer, the model of the present invention improves the average mAP value by 1.68%.

[0128] For the ActivityNet1.3 dataset, this embodiment uses the pre-training method from TSP and compares it with other methods, and the results are shown in Table 2. The multi-time span context model IACFormer of the present invention is superior to the Transformer-based method.

[0129] In the EPIC-Kitchens 100 dataset, the multi-time span context model IACFormer of the present invention is compared with other methods, and the results are shown in Table 3. It can be seen that the average mAP of the verbs and nouns of the multi-time span context model IACFormer of the present invention reaches 24.58% and 22.81% respectively. Since this dataset is a first-person dataset, the detection includes not only the action itself, but also the direct relationship between the action and the target object of the action. The average mAP of the method of the present invention in the noun subtask is 0.91% higher than that of ActionFormer, and the average mAP in the verb subtask is 1.08% higher than that of ActionFormer.

[0130] Table 1. Experimental results on the THUMOS14 dataset

[0131]

[0132] Table 2. Experimental results of ActivityNet1.3 dataset

[0133]

[0134] Table 3. Experimental results on the EPIC-Kitchens 100 dataset

[0135]

[0136] In summary, the method provided by the present invention has the following advantages compared with the prior art:

[0137] The method of the present invention first extracts each video frame in the video to be identified as a video representation sequence by the trained convolution projection layer, and then uses each video representation sequence as the input of the trained multi-time span context model, and first processes it through an encoder, and stacks the feature sequences output by each layer of the encoder backbone network to form a feature pyramid. The first N layers of the encoder backbone network integrate the instantaneous average convolution module and the multi-head self-attention module, and the back N layers of the backbone network integrate the instantaneous average hole convolution module and the multi-head self-attention module, which can simultaneously capture the local features and global features of the video representation sequence. Since the instantaneous average convolution module and the instantaneous average hole convolution module have strong local feature capture capabilities, the multi-head self-attention module has strong global feature capture capabilities, and the instantaneous average convolution module and the instantaneous average hole convolution module require less calculation, the accuracy of action recognition can be significantly improved when the amount of calculation is not increased. In addition, both the instantaneous average convolution module and the instantaneous average hole convolution module contain nonlinear branches, which enhances the ability to distinguish similar feature sequences. The instantaneous average dilated convolution module additionally includes a dilated convolution time period average feature capture module, which improves the ability to capture deep features through a larger receptive field. The above features enable the method of the present invention to better model the video representation sequence. Finally, the prediction module can predict the feature pyramid generated by the encoder and output the final action classification result and the corresponding action start time point and action end time point.

[0138] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A temporal action localization method based on convolution and multi-head self-attention, characterized in that: The following steps are involved: S1. Each video frame in the video to be identified is extracted into a video representation sequence by the trained convolutional projection layer, and each video representation sequence is used as the input of the trained multi-time span context model. It is first processed by the encoder, and the feature sequences output by each layer of the backbone network of the encoder are stacked to form a feature pyramid. The encoder comprises a 2N-layer backbone network, each backbone network layer belonging to the first N layers comprises an instantaneous average convolution module, a multi-head self-attention module and a multi-layer perceptron, and each backbone network layer belonging to the last N layers comprises an instantaneous average hole convolution module, a multi-head self-attention module and a multi-layer perceptron; wherein the instantaneous average convolution module comprises an instantaneous feature capture module, a nonlinear instantaneous feature capture module, a time period average feature capture module and a nonlinear time period average feature capture module; the instantaneous average hole convolution module comprises an instantaneous feature capture module, a nonlinear instantaneous feature capture module, a time period average feature capture module, a nonlinear time period average feature capture module and a hole convolution time period average feature capture module; S2. Input the feature pyramid into a prediction module, and output the final action classification result and the corresponding action start time point and action end time point; wherein the prediction module includes a class regression head, an action start regression head, and an action end regression head.

2. The temporal action localization method based on convolution and multi-head self-attention as claimed in claim 1, characterized in that: In step S1, the specific processing process in the nth layer backbone network belonging to the first N layers is as follows: The input feature sequence of the nth layer is input into the instantaneous average convolution module to obtain the output feature of the nth layer instantaneous average convolution module. The output feature of the nth layer instantaneous average convolution module is input into the multi-head self-attention module after layer normalization to obtain the first output feature of the nth layer. The first output feature of the nth layer is added to the output feature of the nth layer instantaneous average convolution module and then input into the multi-layer perceptron after layer normalization to obtain the second output feature of the nth layer. The output feature of the nth layer instantaneous average convolution module and the first and second output features of the nth layer are added to obtain the output feature sequence of the nth layer. After the output feature sequence of the nth layer passes through the downsampling layer, the input feature sequence of the n+1th layer is obtained.

3. The temporal action localization method based on convolution and multi-head self-attention as claimed in claim 2, characterized in that: The specific processing process in the instantaneous average convolution module is as follows: The input feature sequence of the nth layer is first layer-normalized to obtain a first input feature sequence, and the first input feature sequence is respectively input into the instantaneous feature capture module, the nonlinear instantaneous feature capture module, the time period average feature capture module and the nonlinear time period average feature capture module, to obtain the instantaneous features, nonlinear instantaneous features, time period average features and nonlinear time period average features of the instantaneous average convolution module of the nth layer, and the instantaneous features, nonlinear instantaneous features, time period average features and nonlinear time period average features of the instantaneous average convolution module of the nth layer are added to the first input feature sequence as the output features of the instantaneous average convolution module of the nth layer.

4. The temporal action localization method based on convolution and multi-head self-attention as claimed in claim 1, characterized in that: The specific processing process in the nth layer backbone network belonging to the last N layers is as follows: The input feature sequence of the nth layer is input into the instantaneous average dilated convolution module to obtain the output features of the nth layer instantaneous average dilated convolution module. The output features of the nth layer instantaneous average dilated convolution module are input into the multi-head self-attention module after layer normalization to obtain the third output features of the nth layer. The third output features of the nth layer are added to the output features of the nth layer instantaneous average dilated convolution module and then input into the multi-layer perceptron after layer normalization to obtain the fourth output features of the nth layer. The output features of the nth layer instantaneous average dilated convolution module and the third and fourth output features of the nth layer are added to obtain the output feature sequence of the nth layer. After the output feature sequence of the nth layer passes through the downsampling layer, the input feature sequence of the n+1th layer is obtained.

5. The temporal action localization method based on convolution and multi-head self-attention as claimed in claim 4, characterized in that: The specific processing process in the instantaneous average dilated convolution module is as follows: The input feature sequence of the nth layer is first layer-normalized to obtain a second input feature sequence, and the second input feature sequence is respectively input into the instantaneous feature capture module, the nonlinear instantaneous feature capture module, the time period average feature capture module, the nonlinear time period average feature capture module and the void convolution time period average feature capture module, so as to obtain the instantaneous feature, nonlinear instantaneous feature, time period average feature, nonlinear time period average feature and void convolution time period average feature of the nth layer instantaneous average void convolution module accordingly, and the instantaneous feature, nonlinear instantaneous feature, time period average feature, nonlinear time period average feature and void convolution time period average feature of the nth layer instantaneous average void convolution module are added with the second input feature sequence as the output feature of the nth layer instantaneous average void convolution module.

6. The temporal action localization method based on convolution and multi-head self-attention as claimed in claim 5, characterized in that: In the instantaneous average hole convolution module, the second input feature sequence is subjected to a first instantaneous convolution process to obtain a first feature map, the second input feature sequence is subjected to a first long-term instantaneous convolution process to obtain a second feature map, and the first feature map and the second feature map are multiplied to obtain an instantaneous feature output by the instantaneous average hole convolution module; In the nonlinear instantaneous feature capture module, the second input feature sequence is subjected to a second instantaneous convolution process to obtain a third feature map, the second input feature sequence is subjected to a second long-term instantaneous convolution process to obtain a fourth feature map, the third feature map and the fourth feature map are multiplied to obtain a fifth feature map, and the fifth feature map is subjected to a ReLu activation function process to obtain a nonlinear instantaneous feature output by the nonlinear instantaneous feature capture module; In the time period average value feature capture module, the second input feature sequence is subjected to a first average time period convolution process to obtain a sixth feature map, the second input feature sequence is subjected to a first large-scale average time period convolution process to obtain a seventh feature map, the sixth feature map and the seventh feature map are added to obtain an eighth feature map, and the sixth feature map and the eighth feature map are multiplied to obtain the time period average value feature output by the time period average value feature capture module; In the nonlinear time period average value feature capture module, the second input feature sequence is subjected to a second average time period convolution process to obtain a ninth feature map, the second input feature sequence is subjected to a second largest range average time period convolution process to obtain a tenth feature map, the ninth feature map is added to the tenth feature map to obtain an eleventh feature map, the ninth feature map is subjected to a ReLu activation function process and then multiplied with the eleventh feature map to obtain a twelfth feature map, and the twelfth feature map is subjected to a ReLu activation function process to obtain a nonlinear time period average value feature output by the nonlinear time period average value feature capture module; In the hole convolution time period average value feature capture module, the second input feature sequence is subjected to a third average time period convolution process to obtain a thirteenth feature map, the second input feature sequence is subjected to a large range average time period hole convolution process to obtain a fourteenth feature map, the thirteenth feature map and the fourteenth feature map are added to obtain a fifteenth feature map, the thirteenth feature map and the fifteenth feature map are multiplied to obtain the hole convolution time period average value feature output by the hole convolution time period average value feature capture module.

7. The temporal action localization method based on convolution and multi-head self-attention as claimed in claim 6, characterized in that: The first long time period instantaneous convolution and the second long time period instantaneous convolution are both convolution layers with a convolution kernel of 3; the first instantaneous convolution and the second instantaneous convolution are both convolution layers with a convolution kernel of 1; the first average time period convolution, the second average time period convolution and the third average time period convolution are all convolution layers with a convolution kernel of 1; the first large-range average time period convolution and the second large-range average time period convolution are both convolution layers with a convolution kernel of 11; the large-range average time period void convolution is a convolution layer with a convolution kernel of 11 and an expansion rate of 2.

8. The method for temporal action localization based on convolution and multi-head self-attention as claimed in claim 1, characterized in that: In the encoder, N is a positive integer greater than or equal to 2.

9. The temporal action localization method based on convolution and multi-head self-attention as claimed in claim 1, characterized in that: In the multi-head self-attention module, the number of self-attention heads is 2.

10. The temporal action localization method based on convolution and multi-head self-attention as claimed in claim 1, characterized in that: When training the multi-time span context model, the batch size is set to 8, the dropout rate is set to 0.1, and the activation function is set to ReLU.

Images