Video space-time action positioning method and system based on end-to-end converter model

Abstract

The invention discloses a video space-time action positioning method and system based on an end-to-end converter model, and relates to the field of computer vision, and the method comprises the steps: building a video space-time action positioning model, obtaining a video data set used for training the video space-time action positioning model, and defining an algorithm target; extracting spatio-temporal features in the video data set; feeding back, querying and generating an action detection frame on a decoder network based on the spatio-temporal characteristics; defining a loss function based on the generated action detection frame, training a video space-time action positioning model based on the loss function, and performing space-time action positioning on the input video by using the model; according to the end-to-end processing method provided by the invention, an encoder and decoder structure is introduced to fully mine the spatio-temporal information of the video, so that the method has a good application value in an intelligent video analysis system, and the accuracy of downstream tasks can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a video spatiotemporal action localization method and system based on an end-to-end converter model. Background Art

[0002] In today's society, video has become an indispensable part of human society. Such an environment has greatly developed people's research on video semantic content. Video spatiotemporal action localization is a subtask of video semantic understanding. Its main purpose is to detect action clips in the video and predict the corresponding action category. The nature of the task is similar to object detection, but video action detection considers more the fusion modeling of temporal and spatial features, while object detection only considers spatial feature modeling.

[0003] Most existing methods for localizing spatiotemporal action in videos adopt the method of proposing candidate locations and then screening them, using encoders to extract video features, and then performing subsequent prediction and localization tasks. Such methods often rely on the accuracy of the candidate action locations proposed in the first step. If the accuracy is low, it is easy to cause cascading errors. In general, existing methods for localizing spatiotemporal action in videos are divided into two stages: first extracting video features offline, and then performing subsequent tasks. This may cause semantic differences between the feature encoder and the data in the target dataset. Summary of the invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by the present invention is: how to meet the requirement of obtaining detection results from video in one step.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present invention provides a video spatiotemporal action localization method based on an end-to-end transformer model, comprising:

[0008] Establish a video spatiotemporal action localization model including an encoder network and a decoder network, obtain a video dataset for training the video spatiotemporal action localization model, and define the algorithm objectives;

[0009] Based on the algorithm goal, using the video spatiotemporal action localization model to extract spatiotemporal features in the video dataset;

[0010] Feedback query on decoder network to generate action detection boxes based on spatiotemporal features;

[0011] A loss function is defined based on the generated action detection frame, a video spatiotemporal action localization model is trained based on the loss function, and the model is used to perform spatiotemporal action localization on the input video.

[0012] As a preferred solution of the video spatiotemporal action localization method based on the end-to-end transformer model, wherein:

[0013] The video dataset includes a sample video set X train And the manually annotated video action label set Y train ;

[0014] The algorithm goal is defined as follows: given a sample video x = {x1, x2, ..., x L}, predict the action position and corresponding category y = {y l =(b l ,a l )|l∈[1,L]}; where x l represents the lth frame, L represents the total number of video frames, y l Represents the label of the lth frame, including the detection box b of the lth frame l And the corresponding category a l .

[0015] As a preferred solution of the video spatiotemporal action localization method based on the end-to-end transformer model, wherein:

[0016] The extracting the spatiotemporal features in the video data set using the video spatiotemporal action localization model comprises:

[0017] The sample video x is input into the transformer encoding network, the spatiotemporal features x′ of the video block are extracted, and the extracted spatiotemporal features of the video are input into the transformer decoding network.

[0018] As a preferred solution of the video spatiotemporal action localization method based on the end-to-end transformer model, wherein:

[0019] The generating action detection frame by querying feedback on the decoder network based on the spatiotemporal features includes:

[0020] In the decoding phase, initialize the query parameters to be trained and The total query parameters obtained by calculation are:

[0021]

[0022] Where q represents the total query parameters, Represents the query parameters of the i-th detection box in each frame, represents the query parameters corresponding to the lth frame;

[0023] In the decoding stage, the associated encoded feature x is expressed as:

[0024] u=MHSA(q)+q

[0025] v=CA(u,x)+u

[0026] z=MLP(v)+v

[0027] Among them, MHSA stands for multi-head attention mechanism, CA stands for cross attention mechanism, MLP stands for multi-layer perception network, u and v are the intermediate products in the decoding process of query parameter q, and z is the feature mid-level representation.

[0028] As a preferred solution of the video spatiotemporal action localization method based on the end-to-end transformer model, wherein:

[0029] The generating action detection frame by querying feedback on the decoder network based on the spatiotemporal features also includes:

[0030] According to the calculated result z, the spatiotemporal action frame in the video is predicted, which is expressed as:

[0031] b′=w b z

[0032] a′=w a z

[0033] where w b and w a are all training parameters, b′, a′=y′={(b′ l ,a′ l )|l∈[1,L]} is the set of detection box locations and corresponding classification probabilities on all frames.

[0034] As a preferred solution of the video spatiotemporal action localization method based on the end-to-end transformer model, wherein:

[0035] The loss function defined based on the generated action detection frame includes:

[0036] According to the calculated b′, ​​a′, compare the real label and define the loss function L(y, y′):

[0037]

[0038] Where L frame Represents the loss function of each frame, ∑ represents the summation symbol, y′ l Represents the prediction result of the lth frame, L box Represents the position prediction loss function of each action detection box, L iou represents the intersection-over-union loss function of each action detection box, L class Represents the category prediction loss function for each detection box, b l,i represents the position of the manually marked i-th detection box in the l-th frame, b′ l,i is the corresponding network prediction result, al,i represents the category of the manually labeled i-th detection box in the l-th frame, a′ l,i is the corresponding network prediction result.

[0039] As a preferred solution of the video spatiotemporal action localization method based on the end-to-end transformer model, wherein:

[0040] The loss function-based training of the video spatiotemporal action localization model includes:

[0041] The entire description generation model is trained using the Adam optimization algorithm and the back-propagation algorithm under the loss function L(y,y′).

[0042] In a second aspect, an embodiment of the present invention provides a video spatiotemporal action localization system based on an end-to-end converter model, comprising:

[0043] An initialization module is used to establish a video spatiotemporal action localization model including an encoder network and a decoder network, obtain a video dataset for training the video spatiotemporal action localization model, and define the algorithm objectives;

[0044] An extraction module, for extracting spatiotemporal features from the video data set using a video spatiotemporal action localization model based on an algorithm objective;

[0045] An action detection module, which is used to generate action detection boxes by querying feedback on the decoder network based on spatiotemporal features;

[0046] A positioning module is used to define a loss function based on the generated action detection frame, train a video spatiotemporal action positioning model based on the loss function, and use the model to perform spatiotemporal action positioning on an input video.

[0047] In a third aspect, an embodiment of the present invention provides a computing device, including:

[0048] Memory and processor;

[0049] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the one or more programs are executed by the one or more processors, the one or more processors implement the video spatiotemporal action localization method based on the end-to-end converter model as described in any embodiment of the present invention.

[0050] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, can implement the video spatiotemporal action localization method based on the end-to-end converter model.

[0051] Beneficial effects of the present invention: The present invention takes into account that the two-stage spatiotemporal action localization method may introduce cascade errors, and proposes an end-to-end processing method; an encoder-decoder structure is introduced to fully mine the spatiotemporal information of the video; the video spatiotemporal action localization method provided by the present invention has good application value in intelligent video analysis systems, and can effectively improve the accuracy of downstream tasks; corresponding to video surveillance, higher quality video images can be obtained, and corresponding to safety supervision, it can help engineers make more accurate judgments. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0053] Figure 1 It is an overall flow chart of the video spatiotemporal action localization method based on the end-to-end converter model described in the first embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0055] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0056] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0057] Example 1

[0058] Reference Figure 1 , which is the first embodiment of the present invention, provides a video spatiotemporal action localization method based on an end-to-end transformer model, comprising:

[0059] S1: Establish a video spatiotemporal action localization model including an encoder network and a decoder network, obtain a video dataset for training the video spatiotemporal action localization model, and define the algorithm objectives;

[0060] In the embodiment of the present application, the video data set includes a sample video set X train And the manually annotated video action label set Y train ;

[0061] Defining the algorithm objectives includes: given a sample video x = {x1, x2, ..., x L}, predict the action position and corresponding category y = {y l =(b l ,a l )|l∈[1,L]}; where x l represents the lth frame, L represents the total number of video frames, y l Represents the label of the lth frame, including the detection box b of the lth frame l And the corresponding category a l .

[0062] Specifically, the video is preprocessed as necessary, such as adjusting the resolution, cropping, normalization, etc., to meet the input requirements of the model. Continuous video frames are converted into a format that the model can process, such as cutting the video into fixed-length segments or extracting key frames.

[0063] S2: Based on the algorithm goal, the video spatiotemporal action localization model is used to extract the spatiotemporal features in the video dataset;

[0064] In an embodiment of the present application, extracting spatiotemporal features from the video data set using a video spatiotemporal action localization model includes:

[0065] The sample video x is input into the transformer encoding network, the spatiotemporal features x′ of the video block are extracted, and the extracted spatiotemporal features of the video are input into the transformer decoding network.

[0066] Specifically, a suitable convolutional neural network (CNN) or transformer structure is selected or designed as an encoder to effectively extract spatiotemporal features in video frames and ensure that the encoder can capture dynamic information and spatial relationships in the video.

[0067] S3: Feedback query on decoder network to generate action detection boxes based on spatiotemporal features;

[0068] In an embodiment of the present application, generating an action detection frame by querying feedback on a decoder network based on spatiotemporal features includes:

[0069] In the decoding phase, initialize the query parameters to be trained and The total query parameters obtained by calculation are:

[0070]

[0071] Where q represents the total query parameters, Represents the query parameters of the i-th detection box in each frame, represents the query parameters corresponding to the lth frame;

[0072] In the decoding stage, the associated encoded feature x is expressed as:

[0073] u=MHSA(q)+q

[0074] v=CA(u,x)+u

[0075] z=MLP(v)+v

[0076] Among them, MHSA stands for multi-head attention mechanism, CA stands for cross attention mechanism, MLP stands for multi-layer perception network, u and v are the intermediate products in the decoding process of query parameter q, and z is the feature mid-level representation.

[0077] According to the calculated result z, the spatiotemporal action frame in the video is predicted, which is expressed as:

[0078] b′=w b z

[0079] a′=w a z

[0080] where w b and w a are all training parameters, b′, a′=y′={(b′ l ,a′ l )|l∈[1,L]} is the set of detection box locations and corresponding classification probabilities on all frames.

[0081] It should be noted that the use of the multi-head attention mechanism (MHSA) can make the model more flexible and comprehensive in processing spatiotemporal features, thereby improving the ability to understand complex action patterns; the introduction of the cross attention mechanism (CA) helps the decoder to effectively capture the correlation information between them when processing features at different times, thereby improving the accuracy of the action detection frame; through the multi-layer perception network (MLP) to further nonlinearly map and integrate the features, the expressiveness and robustness of the final prediction results can be improved.

[0082] S4: Define a loss function based on the generated action detection box, train a video spatiotemporal action localization model based on the loss function, and use the model to perform spatiotemporal action localization on the input video.

[0083] In an embodiment of the present application, defining a loss function based on the generated action detection frame includes:

[0084] According to the calculated b′, ​​a′, compare the real label and define the loss function L(y, y′):

[0085]

[0086]

[0087] Where L frame Represents the loss function of each frame, ∑ represents the summation symbol, y′ l Represents the prediction result of the lth frame, L box Represents the position prediction loss function of each action detection box, L iou represents the intersection-over-union loss function of each action detection box, L class Represents the category prediction loss function for each detection box, b l,i represents the position of the manually marked i-th detection box in the l-th frame, b′ l,i is the corresponding network prediction result, a l,i represents the category of the manually labeled i-th detection box in the l-th frame, a′ l,i is the corresponding network prediction result.

[0088] Training the video spatiotemporal action localization model based on the loss function includes:

[0089] The entire description generation model is trained using the Adam optimization algorithm and the back-propagation algorithm under the loss function L(y,y′).

[0090] It should be noted that the present invention introduces an end-to-end converter model to directly construct a path from input video to output detection result, reducing the impact of cascade errors, thereby making video spatiotemporal action positioning more detailed and accurate.

[0091] The above is a schematic scheme of the video spatiotemporal action localization method based on the end-to-end converter model of this embodiment. It should be noted that the technical scheme of the video spatiotemporal action localization system based on the end-to-end converter model and the technical scheme of the video spatiotemporal action localization method based on the end-to-end converter model belong to the same concept. For the details not described in detail in the technical scheme of the video spatiotemporal action localization system based on the end-to-end converter model in this embodiment, please refer to the description of the technical scheme of the video spatiotemporal action localization method based on the end-to-end converter model.

[0092] The video spatiotemporal action localization system based on the end-to-end converter model in this embodiment includes:

[0093] An initialization module is used to establish a video spatiotemporal action localization model including an encoder network and a decoder network, obtain a video dataset for training the video spatiotemporal action localization model, and define the algorithm objectives;

[0094] An extraction module, for extracting spatiotemporal features from the video data set using a video spatiotemporal action localization model based on an algorithm objective;

[0095] An action detection module, which is used to generate action detection boxes by querying feedback on the decoder network based on spatiotemporal features;

[0096] A positioning module is used to define a loss function based on the generated action detection frame, train a video spatiotemporal action positioning model based on the loss function, and use the model to perform spatiotemporal action positioning on an input video.

[0097] This embodiment further provides a computing device, applicable to the case of a video spatiotemporal action localization method based on an end-to-end transformer model, comprising:

[0098] A memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the video spatiotemporal action localization method based on the end-to-end converter model proposed in the above embodiment.

[0099] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the video spatiotemporal action localization method based on the end-to-end converter model proposed in the above embodiment is implemented.

[0100] The storage medium proposed in this embodiment and the video spatiotemporal action localization method based on the end-to-end converter model proposed in the above embodiment belong to the same inventive concept. The technical details not described in detail in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0101] Example 2

[0102] Referring to Table 1, an embodiment of the present invention provides a video spatiotemporal action localization method based on an end-to-end converter model. In order to verify the beneficial effects of the present invention, a scientific demonstration is carried out through simulation experiments.

[0103] The present invention conducts experiments on the video action understanding dataset AVA, which contains 430 video clips of 15 minutes in length, 210,000 labeled video frames in the training set, 57,000 labeled video frames in the validation set, and a total of 80 action categories.

[0104] The present invention uses the Adam optimizer to train the model, and applies a learning rate of 0.0001 and a batch size of 4.

[0105] The present invention uses a quantitative method to evaluate the performance of the method. Compared with the two-stage spatiotemporal action localization method, the indicator used is frame average accuracy (fAP). The specific results are shown in Table 1.

[0106] Table 1 Comparison of frame average accuracy

[0107] method fAP Two-stage 25.2 The present invention (end-to-end) 26.7

[0108] The results show that the end-to-end method proposed in this method has a significant improvement over the two-stage basic method, indicating that the video spatiotemporal action localization method based on the end-to-end transformer network can reduce the impact of the cascade error caused by the two-stage detection to a certain extent and has better application value.

[0109] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A video spatiotemporal action localization method based on an end-to-end transformer model, characterized in that: include: Establish a video spatiotemporal action localization model including an encoder network and a decoder network, obtain a video dataset for training the video spatiotemporal action localization model, and define the algorithm objectives; Based on the algorithm goal, using the video spatiotemporal action localization model to extract spatiotemporal features in the video dataset; Feedback query on decoder network to generate action detection boxes based on spatiotemporal features; A loss function is defined based on the generated action detection frame, a video spatiotemporal action localization model is trained based on the loss function, and the model is used to perform spatiotemporal action localization on the input video.

2. The video spatiotemporal action localization method based on the end-to-end transformer model as claimed in claim 1, characterized in that: include: The video dataset includes a sample video set X train And the manually annotated video action label set Y train ; The algorithm goal is defined as follows: given a sample video x = {x1, x2, ..., x L }, predict the action position and corresponding category y = {y l =(b l ,a l )|l∈[1,L]}; where x l represents the lth frame, L represents the total number of video frames, y l Represents the label of the lth frame, including the detection box b of the lth frame l And the corresponding category a l .

3. The video spatiotemporal action localization method based on the end-to-end transformer model as claimed in claim 2, characterized in that: The extracting the spatiotemporal features in the video data set using the video spatiotemporal action localization model comprises: The sample video x is input into the transformer encoding network to extract the spatiotemporal features x of the video block ′ , the extracted video spatiotemporal features are input into the transformer decoding network.

4. The video spatiotemporal action localization method based on the end-to-end transformer model as claimed in claim 3, characterized in that: The generating action detection frame by querying feedback on the decoder network based on the spatiotemporal features includes: In the decoding phase, initialize the query parameters to be trained and The total query parameters obtained by calculation are: Where q represents the total query parameters, Represents the query parameters of the i-th detection box in each frame, represents the query parameters corresponding to the lth frame; In the decoding stage, the associated encoded feature x is expressed as: u=MHSA(q)+q v=CA(u,x)+u z=MLP(v)+v Among them, MHSA stands for multi-head attention mechanism, CA stands for cross attention mechanism, MLP stands for multi-layer perception network, u and v are the intermediate products in the decoding process of query parameter q, and z is the feature mid-level representation.

5. The video spatiotemporal action localization method based on the end-to-end transformer model as claimed in claim 4, characterized in that: The generating action detection frame by querying feedback on the decoder network based on the spatiotemporal features also includes: According to the calculated result z, the spatiotemporal action frame in the video is predicted, which is expressed as: b′=w b z a′=w a With where w b and w a are all training parameters, b′, a′=y′={(b′ l ,a′ l )|l∈[1,L]} is the set of detection box locations and corresponding classification probabilities on all frames.

6. The video spatiotemporal action localization method based on the end-to-end transformer model as claimed in claim 5, characterized in that: The loss function defined based on the generated action detection frame includes: According to the calculated b ′ ,a ′ , compare the true label and define the loss function L(y,y ′ ): Where L frame Represents the loss function of each frame, ∑ represents the summation symbol, y′ l Represents the prediction result of the lth frame, L box Represents the position prediction loss function of each action detection box, L iou represents the intersection-over-union loss function of each action detection box, L class Represents the category prediction loss function for each detection box, b l,i represents the position of the manually marked i-th detection box in the l-th frame, b′ l,i is the corresponding network prediction result, a l,i represents the category of the manually labeled i-th detection box in the l-th frame, a′ l,i is the corresponding network prediction result.

7. The video spatiotemporal action localization method based on the end-to-end transformer model as claimed in claim 6, characterized in that: The loss function-based training of the video spatiotemporal action localization model includes: Using the Adam optimization algorithm and the back propagation algorithm to optimize the loss function L(y,y ′ ) to train the entire description generation model.

8. A system using the video spatiotemporal action localization method based on the end-to-end transformer model as claimed in any one of claims 1 to 7, characterized in that: include: An initialization module is used to establish a video spatiotemporal action localization model including an encoder network and a decoder network, obtain a video dataset for training the video spatiotemporal action localization model, and define the algorithm objectives; An extraction module, for extracting spatiotemporal features from the video data set using a video spatiotemporal action localization model based on an algorithm objective; An action detection module, which is used to generate action detection boxes by querying feedback on the decoder network based on spatiotemporal features; A positioning module is used to define a loss function based on the generated action detection frame, train a video spatiotemporal action positioning model based on the loss function, and use the model to perform spatiotemporal action positioning on an input video.

9. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the video spatiotemporal action localization method based on the end-to-end converter model as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the video spatiotemporal action localization method based on the end-to-end transformer model as described in any one of claims 1 to 7.

Images