A method, device, storage medium and terminal for detecting sequential motion

By using local self-attention module and global self-attention module to construct multiple local-global fusion features in the timing action detection network, the problem of the timing action detection network in the prior art is solved. Achieve more efficient information extraction and computing performance.

CN114429641BActive Publication Date: 2024-11-01GUANGTEHAIZHI MARINE TECHNOLOGY (QINGDAO) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111574171.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-21
Publication Date
2024-11-01
Estimated Expiration
2041-12-21

AI Technical Summary

Technical Problem

The existing timing action detection network will cause information loss when processing changes in the shape size of the input source data. The training method is phased independent training, resulting in too long training time and low computing efficiency, and it is difficult to handle extreme scale changes in action instances, resulting in Feel the problem of misalignment of domain and time boundary ranges.

Method used

Using multiple local-global fusion features built on the local self-attention module and the global self-attention module, the self-attention module retains long-term feature information, captures global features, and contexts the timing video through the local-global fusion feature. Information fusion improves GPU utilization and computing efficiency.

Benefits of technology

It reduces information loss, improves GPU utilization and computing efficiency, improves the performance of timing action detection, and can effectively process timing action information in action video.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114429641B_ABST
    Figure CN114429641B_ABST
Patent Text Reader

Abstract

The present invention discloses a method, device, storage medium and terminal for detecting sequential actions, the method comprising: obtaining an action video; inputting the action video into a pre-trained action detection network; wherein the pre-trained action detection network is generated based on a plurality of local-global fusion feature training, and the plurality of local-global fusion features are constructed based on a local self-attention module and a global self-attention module; and outputting the sequential action information in the action video. The present application adopts a self-attention module, which can not only retain the information of the long-series features extracted by the convolution layer in the video, but also capture the global features on the long-series features, thereby reducing information loss, and at the same time, constructing a plurality of local-global fusion features through the local self-attention module and the global self-attention module, which can fuse and utilize the context information of the sequential video, so that the utilization rate and computing efficiency of the GPU in the process of training the network based on the plurality of local-global fusion features are greatly improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, storage medium and terminal for detecting a timing action. Background Art

[0002] With the massive increase in online and personal media archives, people are generating, storing, and consuming a large amount of videos. Under this trend, the development of efficient algorithms is encouraged to intelligently parse video data. One of the fundamental challenges for the success of these improvements is action detection in videos in both temporal and spatial aspects, that is, temporal action detection. The current mainstream temporal action detection network mainly consists of CNN network layers and fully connected layers, where the CNN network layer is mainly responsible for feature extraction and the fully connected layer is mainly responsible for behavior classification. In general, the input shape size and output shape size of the CNN network layer and the fully connected layer are fixed in the model definition stage, so the shape size of the input source data is also fixed. If the shape size of the input source data changes, the action detection network will be unable to process the data.

[0003] In the prior art, in order to solve the problem that the action detection network cannot process data if the shape and size of the input source data changes, the current mainstream algorithm needs to first downsample the time series features to unify them to the time series length supported by the action detection network, but the downsampling operation will cause information loss. In addition, CNN has locality and spatial inductive bias, that is, CNN can connect spatially adjacent features, so it has better performance on data with strong correlation in local space (such as images), but it cannot establish connections between long-distance features, so its performance is weaker on sequence problems.

[0004] At the same time, the current mainstream multi-stage temporal action detection network training method is usually divided into two stages of independent training: one stage trains the nomination network, and the other stage trains the detection network, which leads to a long training time and the detection results are subject to the error accumulation of the two stages. The reason is that temporal action detection is different from traditional 2D image object detection: for traditional 2D image object detection, the anchor box generated at each position on its spatial feature map (i.e., 2D feature map) uses the same size of receptive field (because it is on the same layer of feature map), and there is no situation where the receptive field is different, so there is rarely a problem of misalignment between the anchor box and the receptive field; on the contrary, one challenge of temporal action detection is the extreme scale (duration) variation of action instances, ranging from a few seconds to a few minutes. Although the self-attention module can focus on features in different ranges, the extreme scale (duration) variation causes the network model to have a high probability of misalignment between the receptive field range and the corresponding behavior time boundary range, that is, there is a deviation between the content within the model perception range and the content within the time boundary range. In order to reduce the deviation, the current mainstream multi-stage temporal action detection network first selects high-confidence behavior time boundaries (starting time points and ending time points, usually in seconds) through the nomination network, and then extracts the content within the boundary range at the video level based on the selected behavior time boundaries, and then inputs it into the detection network for boundary refinement and behavior classification. However, this method requires a large amount of memory access, data exchange between video memory and internal memory, and even hard disk reading and writing, which leads to reduced GPU utilization and low computing efficiency. Summary of the invention

[0005] The embodiments of the present application provide a method, device, storage medium and terminal for detecting sequential actions. In order to have a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary is not a general review, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.

[0006] In a first aspect, an embodiment of the present application provides a method for detecting a time sequence action, the method comprising:

[0007] Get action video;

[0008] Inputting the action video into a pre-trained action detection network; wherein the pre-trained action detection network is generated based on a plurality of local-global fusion features, and the plurality of local-global fusion features are constructed based on a local self-attention module and a global self-attention module;

[0009] Output the timing motion information in the motion video.

[0010] Optionally, generate a pre-trained action detection network by following these steps, including:

[0011] Constructing an action detection network; wherein the action detection network includes at least an I3D network, a convolution module, a local self-attention module, a global self-attention module, a regression model, and a classification model;

[0012] Obtain an uncut source video, and generate reduced-dimensional temporal features based on the I3D network, the convolution module, and the uncut source video;

[0013] According to the local self-attention module, the global self-attention module and the reduced-dimensional time series features, multiple local fusion features and global fusion features are constructed;

[0014] Fuse each local fusion feature with the global fusion feature to generate multiple local-global fusion features;

[0015] Generate multiple offset parameters and action parameters according to multiple local-global fusion features, regression models, and classification models;

[0016] A target loss value of the action detection network is calculated according to a plurality of offset parameters and action parameters, and it is determined whether to generate a pre-trained action detection network based on the target loss value.

[0017] Optionally, generate reduced-dimensional temporal features based on the I3D network, the convolution module, and the uncut source video, including:

[0018] Sample RGB images and optical flow images of preset time sequence length in the uncut source video;

[0019] Input the RGB image and the optical flow image into the I3D network respectively, and output the RGB image sequence features and the optical flow image sequence features;

[0020] The optical flow map sequence features are concatenated with the RGB map sequence features to obtain the concatenated temporal features;

[0021] The spliced ​​time series features are convolved multiple times through the convolution module to reduce the feature channels of the spliced ​​time series features to the preset feature channels to obtain the reduced-dimensional time series features.

[0022] Optionally, multiple local fusion features and global fusion features are constructed according to the local self-attention module, the global self-attention module and the reduced-dimensional time series features, including:

[0023] A plurality of sliding window features are obtained by sliding a plurality of sliding windows of different window sizes in advance on the reduced-dimensional time series features along the time series direction;

[0024] Input each sliding window feature into the local self-attention module in turn, and output each sliding window local feature set;

[0025] Each local feature in each sliding window local feature set is spliced ​​along the time series direction to obtain multiple local fusion features;

[0026] The reduced-dimensional time series features are input into the global self-attention module and the global fusion features are output.

[0027] Optionally, multiple offset parameters and action parameters are generated according to multiple local-global fusion features, regression models, and classification models, including:

[0028] Multiple target fusion features are input into the pre-trained BMN network in parallel for nomination, and the starting boundary point and the ending boundary point are output;

[0029] According to the starting boundary point and the ending boundary point, the starting local area feature, the ending local area feature and the main local area feature are obtained;

[0030] Input the starting local area features and the ending local area features into the regression model respectively, and output the starting point offset and the ending point offset;

[0031] After splicing the starting local area features, the ending local area features and the main local area features, a local area fusion feature is generated;

[0032] The local region fusion features are input into the regression model to output the segment center point offset, segment length offset and the probability of containing an action;

[0033] Input the local area fusion features into the classification model and output the action category;

[0034] A starting point offset, an ending point offset, a segment center point offset, and a segment length offset are determined as a plurality of offset parameters, and a probability of including an action and an action category are determined as action parameters.

[0035] Optionally, the starting local area feature, the ending local area feature, and the main local area feature are obtained according to the starting boundary point and the ending boundary point, including:

[0036] Taking the starting boundary point and the ending boundary point as the center point, the starting local interest region and the ending local interest region are expanded;

[0037] Determine the area between the start boundary point and the end boundary point as the main local interest area;

[0038] The starting local region of interest, the ending local region of interest, and the main local region of interest are respectively aligned with the corresponding temporal features to obtain the starting local region feature, the ending local region feature, and the main local region feature.

[0039] Optionally, calculating a target loss value of the action detection network according to the plurality of offset parameters and the action parameters, and determining whether to generate a pre-trained action detection network based on the target loss value, includes:

[0040] Input multiple offset parameters into a preset offset regression loss function and output an offset regression loss value;

[0041] Input the action parameters into the preset classification loss function and output the classification loss value;

[0042] Add the offset regression loss value and the classification loss value to get the target loss value of the action detection network;

[0043] When the target loss value reaches the minimum, a pre-trained action detection network is generated;

[0044] or,

[0045] When the target loss value does not reach the minimum, the network parameters of the action detection network are adjusted, and the step of obtaining the uncut source video is continued.

[0046] In a second aspect, an embodiment of the present application provides a sequential motion detection device, the device comprising:

[0047] A video acquisition module, used to acquire action videos;

[0048] A video input module, used for inputting an action video into a pre-trained action detection network; wherein the pre-trained action detection network is generated based on a plurality of local-global fusion features, and the plurality of local-global fusion features are constructed based on a local self-attention module and a global self-attention module;

[0049] The temporal action information output module is used to output the temporal action information in the action video.

[0050] In a third aspect, an embodiment of the present application provides a computer storage medium, which stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the above-mentioned method steps.

[0051] In a fourth aspect, an embodiment of the present application provides a terminal, which may include: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.

[0052] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0053] In an embodiment of the present application, a temporal action detection device first obtains an action video, and then inputs the action video into a pre-trained action detection network; wherein the pre-trained action detection network is generated based on a plurality of local-global fusion feature training, and the plurality of local-global fusion features are constructed based on a local self-attention module and a global self-attention module, and finally outputs the temporal action information in the action video. Since the present application adopts a self-attention module, it can not only retain the information of the long temporal features extracted by the convolution layer in the video, but also capture global features on the long temporal features, thereby reducing information loss, and at the same time, multiple local-global fusion features are constructed by the local self-attention module and the global self-attention module, and the contextual information of the temporal video can be fused and utilized, so that the utilization rate and computing efficiency of the GPU in the process of training the network based on multiple local-global fusion features are greatly improved.

[0054] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0056] Figure 1 It is a flowchart of a method for detecting a sequential action provided by an embodiment of the present application;

[0057] Figure 2 It is a flowchart of a training method for a temporal action detection network provided in an embodiment of the present application;

[0058] Figure 3 It is a structural schematic diagram of a timing action detection device provided in an embodiment of the present application;

[0059] Figure 4 It is a structural diagram of a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The following description and the drawings sufficiently illustrate specific embodiments of the invention to enable those skilled in the art to practice them.

[0061] It should be clear that the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are only examples of devices and methods consistent with some aspects of the present invention as detailed in the attached claims.

[0063] In the description of the present invention, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. In addition, in the description of the present invention, unless otherwise specified, "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previously associated objects are in an "or" relationship.

[0064] The present application provides a method, device, storage medium and terminal for detecting time-series actions to solve the problems existing in the above-mentioned related technical problems. In the technical solution provided by the present application, since the present application adopts a self-attention module, it can not only retain the information of the long-term features extracted by the convolution layer in the video, but also capture the global features on the long-term features, thereby reducing information loss. At the same time, multiple local-global fusion features are constructed through the local self-attention module and the global self-attention module, and the context information of the time-series video can be fused and utilized, so that the utilization rate and computing efficiency of the GPU in the process of training the network based on multiple local-global fusion features are greatly improved. The following is a detailed description using an exemplary embodiment.

[0065] The following will be combined with the attached Figure 1 -Attached Figure 2 , a method for detecting sequential motion provided by an embodiment of the present application is introduced in detail. The method can be implemented by a computer program and can be run on a sequential motion detection device based on a von Neumann system. The computer program can be integrated into an application or run as an independent tool application.

[0066] See also Figure 1 , which is a flow chart of a method for detecting sequential actions according to an embodiment of the present application. Figure 1 As shown, the method of the embodiment of the present application may include the following steps:

[0067] S101, obtaining an action video;

[0068] In this embodiment, the execution subject (e.g., user terminal) of the method for detecting sequential actions can obtain the action video collected by the video acquisition device (e.g., smart camera). The action video can be a video obtained by shooting the movement process of the target. The target can include but is not limited to people, animals, vehicles, etc.

[0069] S102, inputting the action video into a pre-trained action detection network;

[0070] Among them, the pre-trained action detection network is generated based on multiple local-global fusion feature training, and the multiple local-global fusion features are constructed based on local self-attention modules and global self-attention modules; the action video usually includes a set of temporal actions.

[0071] In this embodiment, when generating a pre-trained action detection network, the action detection network is first constructed; wherein the action detection network includes at least an I3D network, a convolution module, a local self-attention module, a global self-attention module, a regression model and a classification model, and then an uncut source video is obtained, and a reduced-dimensional time series feature is generated according to the I3D network, the convolution module and the uncut source video, and then a plurality of local fusion features and a global fusion feature are constructed according to the local self-attention module, the global self-attention module and the reduced-dimensional time series feature respectively, and then each local fusion feature is fused with the global fusion feature respectively to generate a plurality of local-global fusion features, and then a plurality of offset parameters and action parameters are generated according to the plurality of local-global fusion features, the regression model and the classification model, and finally a target loss value of the action detection network is calculated according to the plurality of offset parameters and the action parameters, and it is determined whether to generate a pre-trained action detection network based on the target loss value.

[0072] In a possible implementation, when the user terminal obtains the action video captured by the smart camera, the action video is input into a pre-trained action detection network for processing.

[0073] S103: Outputting the temporal action information in the action video.

[0074] The time sequence action information may include the start time, end time and action category of each time sequence action.

[0075] In a possible implementation, after the processing based on step S102 is completed, information such as the start time, end time, and action category of each sequential action in the action video may be output.

[0076] In an embodiment of the present application, a temporal action detection device first obtains an action video, and then inputs the action video into a pre-trained action detection network; wherein the pre-trained action detection network is generated based on a plurality of local-global fusion feature training, and the plurality of local-global fusion features are constructed based on a local self-attention module and a global self-attention module, and finally outputs the temporal action information in the action video. Since the present application adopts a self-attention module, it can not only retain the information of the long temporal features extracted by the convolution layer in the video, but also capture global features on the long temporal features, thereby reducing information loss, and at the same time, multiple local-global fusion features are constructed by the local self-attention module and the global self-attention module, and the contextual information of the temporal video can be fused and utilized, so that the utilization rate and computing efficiency of the GPU in the process of training the network based on multiple local-global fusion features are greatly improved.

[0077] See also Figure 2 , which is a flow chart of a method for training a motion detection network according to an embodiment of the present application. Figure 2 As shown, the method of the embodiment of the present application may include the following steps:

[0078] S201, constructing an action detection network;

[0079] The action detection network includes at least an I3D network, a convolution module, a local self-attention module, a global self-attention module, a regression model, and a classification model;

[0080] Generally, an I3D network is a two-stream expanded 3D convolutional network for extracting time-domain video features. A convolutional module is a convolutional layer in a convolutional neural network. A self-attention module is a network that can assign corresponding weights to features according to the importance of information. A local self-attention module and a global self-attention module are two branch submodules of the self-attention module. A regression model is used to implement regression operations, and a classification model is used to implement classification operations.

[0081] In this embodiment, when performing network training, an initial action detection network is first constructed based on the I3D network, convolution module, local self-attention module, global self-attention module, regression model and classification model.

[0082] S202, obtaining an uncut source video, and generating a reduced-dimensional temporal feature according to the I3D network, the convolution module, and the uncut source video;

[0083] In an embodiment of the present application, when generating reduced-dimensionality temporal features, firstly, an RGB image and an optical flow image of a preset temporal length are sampled in an unedited source video, and then the RGB image and the optical flow image are respectively input into an I3D network, and the RGB image sequence features and the optical flow image sequence features are output, and then the optical flow image sequence features and the RGB image sequence features are spliced ​​with the RGB image sequence features to obtain spliced ​​temporal features, and finally, the spliced ​​temporal features are convolved multiple times through a convolution module to obtain reduced-dimensionality temporal features after the feature channels of the spliced ​​temporal features are reduced to preset feature channels.

[0084] In a possible implementation, a dual-stream I3D network is used to extract features from uncut videos. v Uncut video of Sampling is performed in small segments with σ = 16 frames, and 16 frames of RGB image I rgb ∈R 3×16 and optical flow graph I opt ∈R 1×16 As input to the I3D network, we get the RGB-based sequence features. And sequence features based on optical flow graph Where C = 2048 is the feature channel dimension, l f is the length of the sequence feature, i.e. l f = l v / σ, and finally F opt In the feature channel direction, splice in F rgb Get later The above steps can be expressed by the following mathematical formula: rgb =I3D(I rgb ), F opt =I3D(I opt ), F seq =Concat(F rgb ,F opt ) C .

[0085] Among them, Concat(x,y) C Indicates that the splicing operation is performed in the feature channel direction.

[0086] It should be noted that the length of the temporal features obtained after the I3D network is different for different video lengths. Since the mini-batch strategy is used during training, usually b temporal features F seq Stacked into a batch as input F batch , each time series feature length must have the same shape to be stacked, otherwise it is impossible to form a valid batch, so the longest time series length l in the batch is used maxAs the time series length of the batch l batch , and then fill in the samples shorter than this length with 0. That is:

[0087]

[0088]

[0089]

[0090] Among them, max means finding the maximum element; ZeroPad is the "fill with 0 operation", the first parameter is the time series feature that needs to be filled with 0, and the second parameter is the length to be filled; finally, F batch The shape should be

[0091] Specifically, when performing the dimensionality reduction operation, three 1D convolution operations are performed on the time series features (the convolution kernel size is 3*3, using Conv1D(x) output_c Indicates that output_c is the number of output channels), reducing the feature channels from 4096 to 512, the formula is:

[0092]

[0093] S203, constructing a plurality of local fusion features and a global fusion feature according to the local self-attention module, the global self-attention module and the reduced-dimensional time series features respectively;

[0094] Usually, the local self-attention module and the global self-attention module are two branch submodules of the self-attention module. One branch mainly slides 1D sliding windows of three different window sizes on the reduced-dimensional time series features, and inputs the features in the sliding window into the self-attention module to fuse three different local fusion features of the local time domain receptive domain; the other branch directly uses the self-attention module on the reduced-dimensional time series features to obtain the global fusion features; finally, the global features are fused with the three local features respectively to obtain three "local-global" features. Among them, the 1D sliding window is a convolution layer with a convolution kernel.

[0095] In an embodiment of the present application, when constructing multiple local fusion features and global fusion features, first, a plurality of sliding window features are obtained by sliding a plurality of pre-set sliding windows of different window sizes on the reduced-dimensional time series features along the time series direction, and then each sliding window feature is input into the local self-attention module in turn, and each sliding window local feature set is output. Secondly, each local feature in each sliding window local feature set is spliced ​​along the time series direction to obtain multiple local fusion features. Finally, the reduced-dimensional time series features are input into the global self-attention module, and the global fusion features are output.

[0096] In one possible implementation, when generating multiple local fusion features, 1D sliding windows of three different window sizes are mainly used to reduce the dimension of the time series features. On the time direction (ie T = l f direction) and adjust the features in the sliding window (wt s is the timing position where the sliding window starts, wt e is the timing position where the sliding window ends) and is input to the local self-attention module. Since the output length of the self-attention module at this location is defined to be consistent with the input length, the output The time series length and sliding window length l w are the same. Then the current sliding window local feature Splice to the previous sliding window local feature along the time sequence direction After that, after the sliding window operation is completed, Where n = l f / l w According to the above operations, three local fusion features based on different local time domain receptive field sizes (due to the use of 1D sliding windows with three different window sizes) can be obtained: in The receptive field of is the smallest. The receptive domain is medium. The receptive field is the largest.

[0097] In a possible implementation, when generating the global fusion feature, another branch global self-attention module directly processes the reduced-dimensional time series feature to obtain the global fusion feature F global .

[0098] S204, fusing each local fusion feature with the global fusion feature to generate multiple local-global fusion features;

[0099] In one possible implementation, F global Respectively and Fusion, get The fusion method is as follows:

[0100]

[0101]

[0102]

[0103] in All are learnable weights.

[0104] The calculation method of the self-attention module mentioned above is as follows:

[0105]

[0106]

[0107]

[0108] where γ, ρ, and η represent three linear changes using different parameters; f refers to any feature input, and f * Refers to feature output; S i Equivalent to the similarity between the Q query and the K key in the attention mechanism, W i is the score after similarity is mapped to the interval [0,1] by Softmax, and C represents the channel dimension of the feature. All subscripts refer to a certain time sequence position.

[0109] S205, generating multiple offset parameters and action parameters according to multiple local-global fusion features, regression models, and classification models;

[0110] In an embodiment of the present application, when generating multiple offset parameters and action parameters, first, multiple target fusion features are input in parallel into a pre-trained BMN network for nomination, and the starting boundary point and the ending boundary point are output. Then, the starting local area feature, the ending local area feature and the main local area feature are obtained according to the starting boundary point and the ending boundary point. Then, the starting local area feature and the ending local area feature are respectively input into the regression model, and the starting point offset and the ending point offset are output. Then, the starting local area feature, the ending local area feature and the main local area feature are spliced ​​to generate a local area fusion feature. Secondly, the local area fusion feature is input into the regression model, and the fragment center point offset, the fragment length offset and the probability of containing an action are output. Then, the local area fusion feature is input into the classification model, and the action category is output. Finally, the starting point offset, the ending point offset, the fragment center point offset and the fragment length offset are determined as multiple offset parameters, and the probability of containing an action and the action category are determined as action parameters.

[0111] Specifically, when obtaining the starting local area features, the ending local area features and the main local area features according to the starting boundary point and the ending boundary point, the starting local interest region and the ending local interest region are first expanded with the starting boundary point and the ending boundary point as the center points, and then the area between the starting boundary point and the ending boundary point is determined as the main local interest region. Finally, the starting local interest region, the ending local interest region and the main local interest region are aligned with the corresponding time series features to obtain the starting local area features, the ending local area features and the main local area features.

[0112] In one possible implementation, The pre-trained BMN network is input in parallel to nominate the temporal segments where the action may occur, that is, the pre-trained BMN network will output the predicted starting boundary point P start and the predicted end boundary point P end .

[0113] The starting boundary point P start and the end boundary point P end The starting local interest region and the ending local interest region are expanded as the center, and the area between the two points is the main local interest region; the three local areas and the corresponding temporal features are aligned using temporal interest regions. The principle of temporal interest region alignment and spatial interest region alignment (RoIAlign) is the same: the bilinear interpolation method is used to obtain the feature values ​​of the feature points whose temporal coordinates are floating-point values ​​(also called sub-temporal coordinates), thereby converting the alignment process into an operation on continuous values. Finally, the corresponding aligned local features F are obtained. s ,F c ,F e .

[0114] The starting local area features and the ending local area features are input into two regression models (startingregression and end regression) respectively to achieve the boundary optimization of action timing nomination (i.e., predict the offset of the starting point and the ending point, respectively). and It can be expressed by the following formula:

[0115]

[0116] The starting local area features, the ending local area features and the main local area features are spliced ​​to obtain F a =Concat(F s ,F c ,F e ), respectively input into a regression model and a classification model, the former mainly predicts the offset of the center point of the fragment Fragment length offset And the probability S of predicting the action act , which mainly predicts the corresponding action category appearing in the nomination. It can be expressed by the following formula:

[0117]

[0118]

[0119] in is the probability of the behavior category, k is the number of behavior categories, ReLU and Sigmoid are activation functions.

[0120] It should be noted that nomination is to use the BMN network to predict the temporal boundary points where a certain behavior may exist, including the predicted starting point and ending point, and the probability of the behavior existing within the range of all boundary point combinations (all predicted starting points and ending points are combined one by one). The probability is represented by a probability matrix, with columns representing predicted starting points and rows representing predicted ending points.

[0121] S206, calculating a target loss value of the action detection network according to the plurality of offset parameters and the action parameters, and determining whether to generate a pre-trained action detection network based on the target loss value.

[0122] In the embodiment of the present application, after obtaining a plurality of offset parameters and action parameters, the plurality of offset parameters are first input into a preset offset regression loss function, and the offset regression loss value is output, and then the action parameter is input into a preset classification loss function, and the classification loss value is output, and then the offset regression loss value is summed with the classification loss value to obtain the target loss value of the action detection network, and finally, when the target loss value reaches the minimum, a pre-trained action detection network is generated;

[0123] Alternatively, when the target loss value does not reach the minimum, the network parameters of the action detection network are adjusted, and the step of obtaining the uncut source video is continued.

[0124] In one possible implementation, the result is and The offset regression loss value and classification loss value are respectively input into the corresponding loss function to calculate the offset regression loss value and the classification loss value, and the offset regression loss value and the classification loss value are added to get the total loss value, and then the entire network parameters are updated based on the total loss value.

[0125] It should be noted that the current mainstream multi-stage temporal action detection network training method is usually divided into two stages for independent training, which brings about problems such as long training time, low computational efficiency, and error accumulation; when analyzing why it is necessary to divide it into two stages for processing, part of the reason is that there are extreme scale (duration) changes in the behavior fragments, which causes the network model to have a high probability of not being able to align the receptive domain range with the corresponding behavior time boundary range, that is, there is a deviation between the content within the model perception range and the content within the time boundary range; therefore, the key point to solve this problem is how to align the model perception range with the time boundary range. In response to this problem, the present invention adopts a method of expanding the boundary range and refining the boundary at the same time. First, according to the starting boundary point and the ending boundary point of the temporal action nomination (i.e., a segment that may contain behavior) predicted by BMN for three "local-global" features, the starting local interest region and the ending local interest region are expanded with the two points as the center, and then the area between the two points is the main local interest region; then, the three local regions and the corresponding temporal features are aligned using the temporal interest region to obtain the corresponding aligned local features. Next, the starting local region of interest features and the ending local region of interest features are input into two regression models (starting regression and end regression) respectively to achieve boundary refinement of action timing nomination (i.e., predicting the offset of the starting and ending points), thereby reducing the training time and improving the computational efficiency.

[0126] In an embodiment of the present application, a temporal action detection device first obtains an action video, and then inputs the action video into a pre-trained action detection network; wherein the pre-trained action detection network is generated based on a plurality of local-global fusion feature training, and the plurality of local-global fusion features are constructed based on a local self-attention module and a global self-attention module, and finally outputs the temporal action information in the action video. Since the present application adopts a self-attention module, it can not only retain the information of the long temporal features extracted by the convolution layer in the video, but also capture global features on the long temporal features, thereby reducing information loss, and at the same time, multiple local-global fusion features are constructed by the local self-attention module and the global self-attention module, and the contextual information of the temporal video can be fused and utilized, so that the utilization rate and computing efficiency of the GPU in the process of training the network based on multiple local-global fusion features are greatly improved.

[0127] The following are embodiments of the device of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments of the present invention, please refer to the method embodiments of the present invention.

[0128] See also Figure 3, which shows a schematic diagram of the structure of a sequential action detection device provided by an exemplary embodiment of the present invention. The sequential action detection device can be implemented as all or part of a terminal through software, hardware or a combination of both. The device 1 includes a video acquisition module 10, a video input module 20, and a sequential action information output module 30.

[0129] The video acquisition module 10 is used to acquire the action video;

[0130] A video input module 20 is used to input the action video into a pre-trained action detection network; wherein the pre-trained action detection network is generated based on a plurality of local-global fusion features, and the plurality of local-global fusion features are constructed based on a local self-attention module and a global self-attention module;

[0131] The time sequence action information output module 30 is used to output the time sequence action information in the action video.

[0132] It should be noted that the sequential action detection device provided in the above embodiment only uses the division of the above functional modules as an example when executing the sequential action detection method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the sequential action detection device provided in the above embodiment and the sequential action detection method embodiment belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.

[0133] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0134] In an embodiment of the present application, a temporal action detection device first obtains an action video, and then inputs the action video into a pre-trained action detection network; wherein the pre-trained action detection network is generated based on a plurality of local-global fusion feature training, and the plurality of local-global fusion features are constructed based on a local self-attention module and a global self-attention module, and finally outputs the temporal action information in the action video. Since the present application adopts a self-attention module, it can not only retain the information of the long temporal features extracted by the convolution layer in the video, but also capture global features on the long temporal features, thereby reducing information loss, and at the same time, multiple local-global fusion features are constructed by the local self-attention module and the global self-attention module, and the contextual information of the temporal video can be fused and utilized, so that the utilization rate and computing efficiency of the GPU in the process of training the network based on multiple local-global fusion features are greatly improved.

[0135] The present invention also provides a computer-readable medium having program instructions stored thereon, and when the program instructions are executed by a processor, the sequential action detection methods provided by the above-mentioned various method embodiments are implemented.

[0136] The present invention also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the sequential motion detection method of each of the above method embodiments.

[0137] See also Figure 4 , which is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. Figure 4 As shown, the terminal 1000 may include: at least one processor 1001 , at least one network interface 1004 , a user interface 1003 , a memory 1005 , and at least one communication bus 1002 .

[0138] The communication bus 1002 is used to realize the connection and communication between these components.

[0139] The user interface 1003 may include a display screen (Display) and a camera (Camera), and the optional user interface 1003 may also include a standard wired interface and a wireless interface.

[0140] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0141] Among them, the processor 1001 may include one or more processing cores. The processor 1001 uses various interfaces and lines to connect various parts within the entire electronic device 1000, and executes various functions and processes data of the electronic device 1000 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1005, and calling data stored in the memory 1005. Optionally, the processor 1001 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 1001 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1001, and it can be implemented separately through a chip.

[0142] Among them, the memory 1005 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1005 may optionally be at least one storage device located away from the aforementioned processor 1001. As Figure 4 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a sequential motion detection application.

[0143] exist Figure 4In the terminal 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 1001 can be used to call the sequential motion detection application stored in the memory 1005, and specifically perform the following operations:

[0144] Get action video;

[0145] Inputting the action video into a pre-trained action detection network; wherein the pre-trained action detection network is generated based on a plurality of local-global fusion features, and the plurality of local-global fusion features are constructed based on a local self-attention module and a global self-attention module;

[0146] Outputs the timing motion information in the motion video.

[0147] In one embodiment, when the processor 1001 generates a pre-trained action detection network, the processor 1001 specifically performs the following operations:

[0148] Constructing an action detection network; wherein the action detection network includes at least an I3D network, a convolution module, a local self-attention module, a global self-attention module, a regression model, and a classification model;

[0149] Obtain an uncut source video, and generate reduced-dimensional temporal features based on the I3D network, the convolution module, and the uncut source video;

[0150] According to the local self-attention module, the global self-attention module and the reduced-dimensional time series features, multiple local fusion features and global fusion features are constructed;

[0151] Fuse each local fusion feature with the global fusion feature to generate multiple local-global fusion features;

[0152] Generate multiple offset parameters and action parameters according to multiple local-global fusion features, regression models, and classification models;

[0153] A target loss value of the action detection network is calculated according to a plurality of offset parameters and action parameters, and it is determined whether to generate a pre-trained action detection network based on the target loss value.

[0154] In one embodiment, when the processor 1001 generates the reduced-dimensional temporal features according to the I3D network, the convolution module, and the uncut source video, the following operations are specifically performed:

[0155] Sample RGB images and optical flow images of preset time sequence length in the uncut source video;

[0156] Input the RGB image and the optical flow image into the I3D network respectively, and output the RGB image sequence features and the optical flow image sequence features;

[0157] The optical flow map sequence features are concatenated with the RGB map sequence features to obtain the concatenated temporal features;

[0158] The spliced ​​time series features are convolved multiple times through the convolution module to reduce the feature channels of the spliced ​​time series features to the preset feature channels to obtain the reduced-dimensional time series features.

[0159] In one embodiment, when the processor 1001 constructs multiple local fusion features and global fusion features according to the local self-attention module, the global self-attention module and the reduced-dimensional temporal features, the following operations are specifically performed:

[0160] A plurality of sliding window features are obtained by sliding a plurality of sliding windows of different window sizes in advance on the reduced-dimensional time series features along the time series direction;

[0161] Input each sliding window feature into the local self-attention module in turn, and output each sliding window local feature set;

[0162] Each local feature in each sliding window local feature set is spliced ​​along the time series direction to obtain multiple local fusion features;

[0163] The reduced-dimensional time series features are input into the global self-attention module and the global fusion features are output.

[0164] In one embodiment, when the processor 1001 generates multiple offset parameters and action parameters according to multiple local-global fusion features, regression models, and classification models, the following operations are specifically performed:

[0165] Multiple target fusion features are input into the pre-trained BMN network in parallel for nomination, and the starting boundary point and the ending boundary point are output;

[0166] According to the starting boundary point and the ending boundary point, the starting local area feature, the ending local area feature and the main local area feature are obtained;

[0167] Input the starting local area features and the ending local area features into the regression model respectively, and output the starting point offset and the ending point offset;

[0168] After splicing the starting local area features, the ending local area features and the main local area features, a local area fusion feature is generated;

[0169] The local region fusion features are input into the regression model to output the segment center point offset, segment length offset and the probability of containing an action;

[0170] Input the local area fusion features into the classification model and output the action category;

[0171] A starting point offset, an ending point offset, a segment center point offset, and a segment length offset are determined as a plurality of offset parameters, and a probability of including an action and an action category are determined as action parameters.

[0172] In one embodiment, when the processor 1001 obtains the starting local area feature, the ending local area feature, and the main local area feature according to the starting boundary point and the ending boundary point, the processor 1001 specifically performs the following operations:

[0173] Taking the starting boundary point and the ending boundary point as the center point, the starting local interest region and the ending local interest region are expanded;

[0174] Determine the area between the start boundary point and the end boundary point as the main local interest area;

[0175] The starting local region of interest, the ending local region of interest, and the main local region of interest are respectively aligned with the corresponding temporal features to obtain the starting local region feature, the ending local region feature, and the main local region feature.

[0176] In one embodiment, when the processor 1001 calculates the target loss value of the action detection network according to the multiple offset parameters and the action parameters, and determines whether to generate a pre-trained action detection network based on the target loss value, the processor 1001 specifically performs the following operations:

[0177] Input multiple offset parameters into a preset offset regression loss function and output an offset regression loss value;

[0178] Input the action parameters into the preset classification loss function and output the classification loss value;

[0179] Add the offset regression loss value and the classification loss value to get the target loss value of the action detection network;

[0180] When the target loss value reaches the minimum, a pre-trained action detection network is generated;

[0181] or,

[0182] When the target loss value does not reach the minimum, the network parameters of the action detection network are adjusted, and the step of obtaining the uncut source video is continued.

[0183] In an embodiment of the present application, a temporal action detection device first obtains an action video, and then inputs the action video into a pre-trained action detection network; wherein the pre-trained action detection network is generated based on a plurality of local-global fusion feature training, and the plurality of local-global fusion features are constructed based on a local self-attention module and a global self-attention module, and finally outputs the temporal action information in the action video. Since the present application adopts a self-attention module, it can not only retain the information of the long temporal features extracted by the convolution layer in the video, but also capture global features on the long temporal features, thereby reducing information loss, and at the same time, multiple local-global fusion features are constructed by the local self-attention module and the global self-attention module, and the contextual information of the temporal video can be fused and utilized, so that the utilization rate and computing efficiency of the GPU in the process of training the network based on multiple local-global fusion features are greatly improved.

[0184] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program for sequential motion detection can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0185] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A method for detecting sequential motions, It is characterized in that The method comprises: Get action video; Inputting the action video into a pre-trained action detection network; wherein the pre-trained action detection network is generated based on a plurality of local-global fusion feature training, and the plurality of local-global fusion features are constructed according to a local self-attention module and a global self-attention module; Outputting the temporal action information in the action video; wherein, Follow these steps to generate a pre-trained action detection network, including: Constructing an action detection network; wherein the action detection network includes at least an I3D network, a convolution module, a local self-attention module, a global self-attention module, a regression model, and a classification model; Obtaining an uncut source video, and generating a reduced-dimensional temporal feature according to the I3D network, the convolution module, and the uncut source video; Constructing a plurality of local fusion features and a global fusion feature according to the local self-attention module, the global self-attention module and the reduced-dimensional time series features respectively; Fusing each local fusion feature with the global fusion feature to generate multiple local-global fusion features; Generate multiple offset parameters and action parameters according to the multiple local-global fusion features, the regression model, and the classification model; Calculating a target loss value of the action detection network according to the plurality of offset parameters and action parameters, and determining whether to generate a pre-trained action detection network based on the target loss value; wherein, The method of constructing a plurality of local fusion features and global fusion features based on the local self-attention module, the global self-attention module and the reduced-dimensional time series features respectively includes: Sliding a plurality of sliding windows of different window sizes preset on the reduced-dimensional time series features along the time series direction to obtain a plurality of sliding window features; Input each sliding window feature into the local self-attention module in sequence, and output each sliding window local feature set; Splicing each local feature in each sliding window local feature set along the time series direction to obtain multiple local fusion features; The reduced-dimensional time series features are input into the global self-attention module, and a global fusion feature is output; wherein the multiple offset parameters and action parameters are generated according to the multiple local-global fusion features, the regression model, and the classification model, including: Inputting the multiple local-global fusion features in parallel into a pre-trained BMN network for nomination, and outputting a starting boundary point and an ending boundary point; Obtaining a starting local area feature, an ending local area feature, and a main local area feature according to the starting boundary point and the ending boundary point; Inputting the starting local area features and the ending local area features into the regression model respectively, and outputting the starting point offset and the ending point offset; After splicing the starting local area features, the ending local area features and the main local area features, a local area fusion feature is generated; Inputting the local area fusion features into the regression model, and outputting the segment center point offset, segment length offset, and the probability of containing an action; Inputting the local area fusion features into the classification model and outputting the action category; The starting point offset, the ending point offset, the segment center point offset and the segment length offset are determined as a plurality of offset parameters, and the probability of including the action and the action category are determined as action parameters; wherein, The obtaining of the starting local area feature, the ending local area feature and the main local area feature according to the starting boundary point and the ending boundary point includes: Expanding a starting local interest region and an ending local interest region with the starting boundary point and the ending boundary point as center points; Determine the area between the starting boundary point and the ending boundary point as the main local interest area; The starting local region of interest, the ending local region of interest, and the main local region of interest are respectively aligned with the corresponding time series features to obtain the starting local region feature, the ending local region feature, and the main local region feature.

2. The method according to claim 1, It is characterized in that The generating of reduced-dimensional temporal features according to the I3D network, the convolution module and the uncut source video includes: Sample RGB images and optical flow images of preset time sequence length in the uncut source video; Input the RGB image and the optical flow image into the I3D network respectively, and output the RGB image sequence features and the optical flow image sequence features; Splicing the optical flow map sequence features with the RGB map sequence features to obtain spliced ​​time series features; The spliced ​​time series features are convolved multiple times through the convolution module, so that the feature channels of the spliced ​​time series features are reduced to preset feature channels to obtain reduced-dimensional time series features.

3. The method according to claim 1, It is characterized in that The step of calculating a target loss value of the action detection network according to the plurality of offset parameters and the action parameters, and determining whether to generate a pre-trained action detection network based on the target loss value, comprises: Inputting the multiple offset parameters into a preset offset regression loss function, and outputting an offset regression loss value; Input the action parameters into a preset classification loss function and output a classification loss value; Add the offset regression loss value and the classification loss value to obtain a target loss value of the action detection network; When the target loss value reaches a minimum, generating a pre-trained action detection network; or, When the target loss value does not reach the minimum, the network parameters of the action detection network are adjusted, and the step of obtaining the uncut source video is continued.

4. A timing action detection device, It is characterized in that The device comprises: A video acquisition module, used to acquire action videos; A video input module, used for inputting the action video into a pre-trained action detection network; wherein the pre-trained action detection network is generated based on a plurality of local-global fusion features, and the plurality of local-global fusion features are constructed according to a local self-attention module and a global self-attention module; A time sequence action information output module is used to output the time sequence action information in the action video; wherein, Follow these steps to generate a pre-trained action detection network, including: Constructing an action detection network; wherein the action detection network includes at least an I3D network, a convolution module, a local self-attention module, a global self-attention module, a regression model, and a classification model; Obtaining an uncut source video, and generating a reduced-dimensional temporal feature according to the I3D network, the convolution module, and the uncut source video; Constructing a plurality of local fusion features and a global fusion feature according to the local self-attention module, the global self-attention module and the reduced-dimensional time series features respectively; Fusing each local fusion feature with the global fusion feature to generate multiple local-global fusion features; Generate multiple offset parameters and action parameters according to the multiple local-global fusion features, the regression model, and the classification model; Calculating a target loss value of the action detection network according to the plurality of offset parameters and action parameters, and determining whether to generate a pre-trained action detection network based on the target loss value; wherein, The method of constructing a plurality of local fusion features and global fusion features based on the local self-attention module, the global self-attention module and the reduced-dimensional time series features respectively includes: Sliding a plurality of sliding windows of different window sizes preset on the reduced-dimensional time series features along the time series direction to obtain a plurality of sliding window features; Input each sliding window feature into the local self-attention module in sequence, and output each sliding window local feature set; Splicing each local feature in each sliding window local feature set along the time series direction to obtain multiple local fusion features; The reduced-dimensional time series features are input into the global self-attention module, and a global fusion feature is output; wherein the multiple offset parameters and action parameters are generated according to the multiple local-global fusion features, the regression model, and the classification model, including: Inputting the multiple local-global fusion features in parallel into a pre-trained BMN network for nomination, and outputting a starting boundary point and an ending boundary point; Obtaining a starting local area feature, an ending local area feature, and a main local area feature according to the starting boundary point and the ending boundary point; Inputting the starting local area features and the ending local area features into the regression model respectively, and outputting the starting point offset and the ending point offset; After splicing the starting local area features, the ending local area features and the main local area features, a local area fusion feature is generated; Inputting the local area fusion features into the regression model, and outputting the segment center point offset, segment length offset, and the probability of containing an action; Inputting the local area fusion features into the classification model and outputting the action category; The starting point offset, the ending point offset, the segment center point offset and the segment length offset are determined as a plurality of offset parameters, and the probability of including the action and the action category are determined as action parameters; wherein, The obtaining of the starting local area feature, the ending local area feature and the main local area feature according to the starting boundary point and the ending boundary point includes: Expanding a starting local interest region and an ending local interest region with the starting boundary point and the ending boundary point as center points; Determine the area between the starting boundary point and the ending boundary point as the main local interest area; The starting local region of interest, the ending local region of interest, and the main local region of interest are respectively aligned with the corresponding time series features to obtain the starting local region feature, the ending local region feature, and the main local region feature.

5. A computer storage medium, It is characterized in that The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method according to any one of claims 1 to 3.

6. A terminal, It is characterized in that include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method according to any one of claims 1 to 3.