Temporal action localization method based on global perception and nomination relationship mining

Abstract

An embodiment of the present invention provides a temporal action localization method based on global perception and nomination relationship mining, which relates to the field of video recognition technology. Among them, this temporal action localization method includes steps S1 to S4. S1. Obtain the video to be identified. S2. Extract visual features based on the video to be identified. S3. Input the visual features into a pre-trained temporal nomination generation network model to obtain temporal nominations. S4. Input the temporal nominations into a pre-trained temporal nomination classification network model to obtain positioning results. The positioning results include action categories and action start and end times. The temporal nomination generation network model dynamically captures the correlation between features through graph modeling, realizes information interaction between action boundaries and action features, and achieves the purpose of global perception of the relationship between action, background, and boundary. The temporal nomination classification network model mines the potential relationship between nominations to achieve more accurate positioning and action classification.

Description

Technical Field

[0001] The present invention relates to the field of video recognition technology, and in particular to a temporal action positioning method based on global perception and nomination relationship mining. Background Art

[0002] Temporal action localization is a method for identifying the category, start time, and end time of an action of interest within a long, unedited video. In entertainment, film, and sports videos, temporal action localization can be used to edit highlights or generate video covers. In surveillance scenarios like stations and squares, temporal action localization can be used to detect unusual behaviors such as falls, fights, and crowds.

[0003] In temporal action localization, the detected actions are subject to various complexities, such as large time span differences, complex background composition, small differences between different actions, and blurred action boundaries. Existing technologies are unable to accurately locate and classify these actions.

[0004] In view of this, the applicant filed this application after studying the existing technology. Summary of the Invention

[0005] The present invention provides a temporal action localization method based on global perception and nomination relationship mining, so as to improve the problem that the prior art cannot accurately localize temporal actions in videos.

[0006] An embodiment of the present invention provides a temporal action localization method based on global perception and nomination relationship mining, which includes steps S1 to S4.

[0007] S1. Obtain the video to be identified.

[0008] S2. Extract visual features based on the video to be identified.

[0009] S3. Input the visual features into the pre-trained temporal nomination generation network model to obtain the temporal nomination.

[0010] S4: Input the temporal nomination into the pre-trained temporal nomination classification network model to obtain the positioning result. The positioning result includes the action category and the start and end time of the action.

[0011] The temporal nomination generation network model is used to execute steps A1 to A3 to obtain temporal nominations based on visual features.

[0012] A1. Obtain abstract semantic features based on visual features.

[0013] A2. Based on the abstract semantic features, graph modeling is performed with video unit features as nodes and the relationships between video unit features as edges to obtain the output feature stream.

[0014] A3. Based on the output feature stream, a feature relationship graph is constructed with boundary features as nodes and action features as edges. The feature relationship graph is updated to obtain new edge features and node features. Then, based on the new edge features and new node features, a temporal nomination is obtained.

[0015] The temporal nomination classification network model is used to execute steps B1 to B3 to obtain positioning results based on the temporal nomination.

[0016] B1. Nominate according to time sequence and obtain nomination features.

[0017] B2. Based on the nomination features, a nomination relationship graph is constructed with the nomination features as nodes and the relationships between nominations as edges.

[0018] B3. Obtain positioning results based on the nomination relationship diagram.

[0019] By adopting the above technical solution, the present invention can achieve the following technical effects:

[0020] In the temporal action localization method of this embodiment, the temporal nomination generation network model dynamically captures the correlation between features through graph modeling, enabling information exchange between action boundaries and action features, and achieving global perception of the relationship between action, background, and boundaries. The temporal nomination classification network model explores the potential relationships between nominations, achieving more accurate localization and action classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 It is a flowchart of the sequential action positioning method.

[0023] Figure 2 It is a logical diagram of the temporal action positioning method.

[0024] Figure 3 It is the network structure diagram of the temporal nomination generation network model.

[0025] Figure 4 This is the network structure diagram of the graph convolution module.

[0026] Figure 5 It is the network structure diagram of the temporal graph convolution branch.

[0027] Figure 6It is a network structure diagram for graph reasoning.

[0028] Figure 7 It is the network structure diagram of the temporal nomination classification network model. DETAILED DESCRIPTION

[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0030] See also Figures 1 to 7 The first embodiment of the present invention provides a method for temporal action localization based on global perception and nomination relationship mining, which can be performed by a temporal action localization device (hereinafter referred to as the action localization device). In particular, one or more processors in the action localization device are used to implement steps S1 to S4.

[0031] S1. Obtain the video to be identified.

[0032] Specifically, the video to be identified can be a video shot in real time by a camera, or a video shot in advance containing motion. The present invention does not specifically limit the source and type of the video to be identified.

[0033] It is understandable that the motion positioning device can be an electronic device with computing capabilities, such as a portable notebook computer, a desktop computer, a server, a smart phone, or a tablet computer.

[0034] S2. Extract visual features based on the video to be identified.

[0035] Specifically, feature extraction can convert video data into a language that can be recognized by computers, which has great practical significance. Preferably, step S2 specifically includes: obtaining visual features based on the video to be recognized through a dual-stream network model.

[0036] In this embodiment, a dual-stream network is used as the feature extraction network. In other embodiments, other existing feature extraction networks may be used, and the present invention does not specifically limit this. The dual-stream network has excellent performance in the field of action recognition, as well as high feature abstraction and expression capabilities.

[0037] The two-stream network consists of two branch networks: the spatial network extracts appearance features based on RGB video frames, and the temporal network captures motion information based on stacked optical flow frames. j , middle frame Input spatial network to extract appearance features a j , fragment s j The corresponding stacked optical flow frames are used as the input of the optical flow network to extract the motion features m of the video clip. j The appearance features and motion features are expressed as and Finally, the obtained appearance features and motion features are added together to achieve feature fusion, which is expressed as X = M + A. The final output of the feature extraction network is X, Where C represents the output dimension size (usually 1024), L s It is the time sequence length corresponding to the video V.

[0038] It should be noted that, since the lengths of videos vary, in order to enable parallel computation of the network, the videos are sampled before feature extraction. Preferably, in this embodiment, a uniform sampling strategy is adopted. In other embodiments, other existing sampling strategies may be adopted, or no sampling may be performed.

[0039] S3. Input the visual features into the pre-trained temporal nomination generation network model to obtain temporal nominations. Specifically,

[0040] S4: Input the temporal nomination into the pre-trained temporal nomination classification network model to obtain the positioning result. The positioning result includes the action category and the start and end time of the action.

[0041] In the temporal action localization method of this embodiment, the temporal nomination generation network model dynamically captures the correlation between features through graph modeling, enabling information exchange between action boundaries and action features, and achieving global perception of the relationship between action, background, and boundaries. The temporal nomination classification network model explores the potential relationships between nominations, achieving more accurate localization and action classification.

[0042] The temporal nomination generation network model uses graph convolution to model video clip features, dynamically captures the correlation between features, and transforms the temporal action nomination generation task into a graph generation task with action boundaries as nodes and action features as edges. It realizes information interaction between action boundaries and action features, achieves the purpose of global perception of the relationship between action, background, and boundaries, and generates temporal action nominations with more accurate action boundaries.

[0043] The temporal nomination classification network model uses nomination features as nodes and nomination relationships as edges to construct a dynamic nomination relationship graph, explore the potential relationships between nominations, and achieve the purpose of more accurate positioning and action classification.

[0044] This embodiment of the present invention uses a graph convolutional integration approach to model features, dynamically capturing the correlations between features and generating temporal action nominations with more accurate action boundaries. It also constructs a dynamic nomination relationship graph to explore potential relationships between nominations, enabling more precise positioning and action classification. This not only generates accurate action nominations but also enables the identification of action categories.

[0045] The embodiment of the present invention realizes phased optimization through the "nomination + classification" paradigm, first obtaining the approximate position of the action with a high recall rate, and then performing action recognition and position fine-tuning, which can locate the action more accurately.

[0046] Prior art techniques for spatiotemporal feature extraction generally use temporal convolution to extract local features, without considering the overall situation and fully utilizing global information to assist in action localization. In this embodiment of the present invention, the temporal nomination generation network model uses graph convolution to model video clip features, capturing the correlation between features. This transforms the temporal action nomination generation task into a graph generation task with action boundaries as nodes and action features as edges, enabling information exchange between action boundaries and action features, and generating more accurate temporal action nominations.

[0047] Prior art techniques directly fine-tune and classify actions based on generated sequential action nominations, without considering the potential relationships between nominations. In this embodiment, the sequential nomination classification network model uses nominations as nodes and nomination relationships as edges to construct a dynamic nomination relationship graph, exploring the potential relationships between nominations for more accurate positioning and action classification.

[0048] like Figure 3 As shown in the figure, the network structure of the temporal nomination generation network model includes: a basic module connected in sequence, a three-layer graph convolution module and a nomination output module.

[0049] The basic module includes two one-dimensional convolutions in a stacked arrangement.

[0050] The graph convolution module includes a temporal graph convolution branch, a self-attention graph convolution branch, and a fusion operation;

[0051] The fusion operation is used to fuse the output of the temporal graph convolution branch and the output of the self-attention graph convolution branch.

[0052] The timing graph convolution branch includes 32 branch paths and an addition and fusion operation; the branch path includes a "256,4" graph convolution, a "4,4" convolution, and a "4,256" convolution connected in sequence; and the fusion operation is used to fuse the outputs of the 32 branch paths.

[0053] The self-attention graph convolution branch includes two self-attention convolution branches, a dot product operation, an activation operation, a one-dimensional convolution branch and a graph convolution operation; the self-attention convolution branch includes two one-dimensional convolutions connected in sequence; the dot product operation is used to perform dot product on the outputs of the two self-attention convolution branches, and the activation operation is used to activate the dot product result through the sigmoid activation function; the one-dimensional convolution branch includes two one-dimensional convolution operations connected in sequence; the graph convolution operation uses the result of the activation operation output as a street matrix to perform graph convolution on the result of the one-dimensional convolution branch output.

[0054] The nomination output module includes three one-dimensional convolution modules, a mapping module and a map update module;

[0055] Specifically, the temporal nomination generation network model is used to perform steps A1 to A3 to obtain temporal nominations based on visual features.

[0056] A1. Obtaining abstract semantic features based on visual features. Preferably, step A1 specifically includes: obtaining abstract semantic features based on visual features through convolution.

[0057] Specifically, this module is the first module in the temporal nomination generation network model. As the backbone of the entire network, it directly processes the network's input features, further abstracting semantic features based on the original features and expanding the receptive field in the temporal dimension. This module consists of two stacked one-dimensional convolutions, each with a kernel size of 3 and a stride of 1. Given the input feature f, the module's output is expressed as:

[0058] F = conv1d2(conv1d1(f))

[0059] A2: Based on the abstract semantic features, we construct a graph model using video unit features as nodes and the relationships between them as edges to obtain the output feature stream. Specifically, step A2 is the core of the temporal nomination generation network model. Through graph modeling, we transform the temporal action localization task into a subgraph localization problem.

[0060] like Figure 4 As shown, based on the above embodiment, in an optional embodiment of the present invention, step A2 includes steps A21 to A23.

[0061] A21. Based on the abstract semantic features, a static edge feature stream is constructed through temporal graph convolution.

[0062] like Figure 5As shown, preferably, step A21 includes steps A211 to A212. A211: Based on the abstract semantic features, extract static feature streams through 32 branch paths. Each branch path includes one graph convolution operation and two convolution operations. A212: Fusion the 32 static feature streams in an additive manner to construct a static edge feature stream.

[0063] Specifically, the temporal graph convolution branch adopts the "decomposition-transformation-fusion" strategy. This process is similar to the group convolution, which decomposes the complex calculation into multiple units with smaller computational complexity and finally fuses them together. Here, 32 branch paths are used to increase the diversity of the transformation. Figure 5 This process is demonstrated.

[0064] Figure 5 In the example, "256,4" represents a graph convolution operation, while "4,4" and "4,256" represent 1×1 convolution operations. The first number represents the feature input dimension, and the following numbers represent the feature output dimension. The output feature streams of the 32 branch paths are finally fused by summing.

[0065] A22. Based on the abstract semantic features, a dynamic edge feature flow is constructed through the self-attention module and graph convolution. Figure 4 As shown, based on the above embodiment, in an optional embodiment of the present invention, step A22 includes steps A221 to A222.

[0066] A221. Based on the abstract semantic features, the attention matrix is ​​obtained through the self-attention module.

[0067] like Figure 4 As shown, preferably, the self-attention module includes two one-dimensional convolutions. Step A221 includes steps A2211 to A2213. A2211: Obtain query Q through one-dimensional convolution based on abstract semantic features. A2212: Obtain key value K through one-dimensional convolution based on abstract semantic features. A2213: Multiply query Q and key value K by point, then activate with a sigmoid activation function to obtain an attention matrix. A222: Based on abstract semantic features, perform two layers of one-dimensional convolution, then perform graph convolution using the attention matrix as the adjacency matrix to construct a dynamic edge feature stream.

[0068] Specifically, the self-attention graph convolution branch consists of two parts: the self-attention module and the graph convolution operation.

[0069] Self-attention module: Models the potential correlation between nodes in the graph and adaptively constructs edges between nodes. The self-attention module consists of two one-dimensional convolutions. For the input feature N, one convolution operation obtains the query Q = conv1d3 (N), and the other convolution operation obtains the key value K = conv1d4 (N), Q, K∈R L , multiply Q and K, and then pass the sigmoid activation function to get the attention matrix A.

[0070] A=QK T , A∈R L×L

[0071] The attention matrix serves as the adjacency matrix in the graph convolution operation, representing the edges and connection strengths between nodes.

[0072] Graph convolution operation: The input features are first processed through two layers of one-dimensional convolution, and then input into the graph convolution operation to obtain the output feature stream of this part.

[0073] A23. Based on the abstract semantic features, static edge feature flow and dynamic edge feature flow, a feature map is constructed by fusion to obtain the output feature flow.

[0074] Specifically, the output of step A2 is the fusion of the temporal graph convolution branch, the self-attention graph convolution branch, and the input features. Therefore, step A23 performs the fusion, which is expressed as:

[0075]

[0076] Where A t Represents the adjacency matrix of the time series edge, A d Represents the adjacency matrix of dynamic edges, W t and W d Represents the corresponding learnable parameters, and Relu is the activation function.

[0077] In step A2, the graph convolution module constructs a graph with video unit features as nodes and the relationships between video unit features as edges. in E=E s ∪E d , N and E represent the nodes and edges of the graph respectively. Among them, there are two types of edges, one is static edge, denoted by E s , one is the dynamic edge, denoted as E d The following is a detailed introduction to the construction of these two types of edges and the graph processing process.

[0078] Static edge: Static edge encodes the temporal relationship between video units. Each node n i ∈N has a connection n i+1 The forward edge and a connection ni-1 Therefore, the static edge The forward and reverse edges are defined as:

[0079]

[0080]

[0081] Here, L represents the number of video units.

[0082] Dynamic edge: Dynamic edge is used to model the long-distance dependency between video units and extract global information. The construction of dynamic edge is realized by the self-attention module, which consists of two one-dimensional convolutions. The module generates a weight matrix of size L×L, denoted as A. The element A in the weight matrix i，j It represents the connection edge between the i-th node and the j-th node. The weight value represents the connection strength of the edge. Generally, the higher the correlation, the greater the connection strength, and vice versa.

[0083] In this embodiment, the graph convolution operation is used to represent the features of all nodes in the graph as N=[n1, n2, ..., n L ]∈R C×L , the graph convolution operation is implemented using a single layer of edge convolution, denoted as The formula is:

[0084]

[0085] In the formula, “||” represents the concatenation operation on the feature dimension, A∈R L×L is the adjacency matrix, W∈R 2C×c′ are learnable parameters, C and C′ are the input and output dimensions of node features, respectively, and I is an identity matrix.

[0086] The element A in row i and column j of the adjacency matrix i，j Represents the edge between node i and node j. When the value is 1, it means there is an edge connection, and when it is 0, there is no edge connection, that is:

[0087]

[0088] In dynamic edges, the adjacency matrix directly uses the weight matrix generated by the self-attention module, A i，j ∈(0,1).

[0089] The graph convolution operation is divided into two parts, N T Maintaining global information, (AI)N T Maintain local information associated through edges.

[0090] A3. Based on the output feature stream, a feature relationship graph is constructed with boundary features as nodes and action features as edges. The feature relationship graph is updated to obtain new edge features and node features. Then, based on the new edge features and new node features, a temporal nomination is obtained.

[0091] like Figure 6 As shown, based on the above embodiment, in an optional embodiment of the present invention, step A3 includes steps A31 to A34.

[0092] A31. According to the output feature flow, three one-dimensional convolution operations are performed to obtain the action start feature F. s , end feature F e and action feature F a .

[0093] Specifically, after the three graph convolution modules process and update the graph, an information-rich feature stream is obtained and input into the nomination output module. The structure of the nomination output module is as follows: Figure 6 shown.

[0094] The nomination output module first performs three one-dimensional convolutions on the input features to obtain the action start features F s , end feature F e and action feature F a , the sizes of these three features are R D×L , L is the time series length, and D is the feature dimension.

[0095] A32, start feature F according to action s , end feature F e and action feature F a , starting with action feature F s and end feature F e As a node, the action feature F a As edges, construct a feature relationship graph.

[0096] like Figure 6 As shown, in this embodiment, dots are used to represent the action start node and the action end node, and the edges between them are represented by the action features between the start node and the end node. The boundary features and action features are organically connected through the construction of a graph. Specifically:

[0097] Calculate F s and F e Cartesian product of the start node feature and the end node feature to construct a feature pair set S se :

[0098] S se =F s ×F e ={(fs，i , f e，j )|(∈F s ∧f e，j ∈F e ∧i<j)}

[0099] Since the start time of an action must be less than the end time, we filter out these invalid feature pairs by ensuring that i < j. s，i , f e，j ) related action content features, feature F is used a The feature representation from the i-th time sequence to the j-th time sequence position is transformed into a uniform size by linear interpolation, and the feature dimension is transformed into D×1 using a single-layer full connection, which is expressed as f c，ij ∈R D×1 , all action content features together constitute the content feature set S c .

[0100] It is understandable that in step A2, the graph convolution module constructs a graph of the feature segments, which is a feature dimension. In step A3, the nomination output module constructs a graph in the action nomination dimension, and the node and edge structures of the two are different.

[0101] A33. Update according to the feature relationship graph to obtain new edge features and node features.

[0102] Specifically, in order to realize information interaction between nodes and edges, node update and edge update operations are adopted. Preferably, step A33 includes steps A331 and A332.

[0103] A331. Update the edge according to the feature relationship graph to obtain new edge features. The edge update model is: e′ ij =σ(θ e ×(e ij *f s，i *f e，j )+e ij ), where e′ ij is the updated edge, σ is the Relu activation function, θ e is a learnable parameter, e ij is the edge before update, f s，i is the feature representation of the i-th time series, f e，j is the feature representation between the jth temporal positions.

[0104] In this embodiment, the edge between the starting node i and the ending node j is represented by e ij If e i，j =f c，ij , the edge update calculation formula is as follows:

[0105] e′ ij =σ(θ e ×(e ij *f s，i *f e，j )+e ij )

[0106] In the formula, “×” and “*” represent matrix multiplication and element-wise multiplication respectively, θ e ∈R D×D is a learnable parameter, and σ is the ReLU activation function. Through this operation, the node information is integrated into the edge representation.

[0107] A332. According to the feature relationship graph, normalize the edge features of each node, and then perform graph convolution operations to obtain new node features. The node update model is: Where, is the updated node, σ is the Relu activation function, θ n is a learnable parameter, I is the unit matrix, e ij Indicates the starting node n i and end node n j The edge between.

[0108] In this embodiment, during the node update operation, since the number of edges associated with each node is different, in order to prevent the output features from changing in magnitude, the edge features are first normalized. The formula is as follows:

[0109]

[0110] Where m represents the element at the mth position in the edge feature vector, and K represents the total number of edges associated with node i, regardless of the direction of the edge.

[0111] After feature normalization, the graph convolution operation is performed, and the formula is as follows:

[0112]

[0113] Where θ n ∈R D×D is a learnable parameter, which is divided into θ according to the node type start and θ end ,σ is the activation function, n i is the node in the graph, and I represents the total number of head nodes.

[0114] The purpose of this graph convolution operation is to fuse the edge information into the node representation. Through the above process, new edge features and node features are obtained.

[0115] A34. Obtain time series nominations based on the new edge features and node features.

[0116] Specifically, the time series nomination is calculated and expressed as follows:

[0117]

[0118] Public, θ sv ,θ ev and θ cv is a learnable parameter vector, and “×” represents vector dot product.

[0119] The temporal nomination classification network model models the relationship between nominations and calculates the final positioning and classification results. The structure is as follows Figure 7 As shown in Figure 2, the network consists of three modules: nomination feature generation module, nomination relationship module and output module.

[0120] The temporal nomination classification network model is used to execute steps B1 to B3 to obtain positioning results based on the temporal nomination.

[0121] B1. Nominate according to time sequence and obtain nomination features.

[0122] Preferably, step B1 specifically includes steps B11 and B12. B11. Obtain the start region, action region, and end region based on the time series nomination. B12. Uniformly sample the three regions using linear interpolation, and perform a two-dimensional convolution operation and dimensional transformation on the sampled features to obtain the nominated features. The number of samples for the start and end regions is 4, and the number of samples for the action region is 8.

[0123] Specifically, this module is used to generate corresponding feature expressions for the nominations obtained in the previous stage. This part of the network uses the temporal nomination network of the first stage as the backbone network and the output of the aforementioned graph convolution module as the input of this module, which is expressed as

[0124] For nomination ψ=(t s , t e ), that is, the starting time is t s , the end time is t e The nomination is divided into three parts: the starting area, the action area and the ending area. The starting area is defined as r s =[t s -d s / k,t s +d s / k], the end area is defined as r e =[t s -d s / k,t s +d s / k], the action area is r a=[t s , t e ], where d s =t e -t s .

[0125] The three areas are uniformly sampled using the linear interpolation method, with 4 samples in the start area and the end area and 8 samples in the action area.

[0126] The sampling features undergo a layer of two-dimensional convolution and dimension transformation to obtain the nominated features. Among them D p Represents the feature dimension, N p Indicates the total number of nominations.

[0127] B2. Based on the nomination features, a nomination relationship graph is constructed with the nomination features as nodes and the relationships between nominations as edges. Figure 7 As shown, preferably, step B2 specifically includes: according to the nomination features, taking the nomination features as nodes and the relationships between nominations as edges, building edges between nodes and adjacent nodes of the nodes to obtain a nomination relationship graph.

[0128] Specifically, this module is the core part of the network, and its purpose is to model the relationship between nominations. The nomination features are nodes, and the relationships between nominations are edges.

[0129] In order to avoid excessive computation and the introduction of too much irrelevant information, this method only constructs edges between nodes and their adjacent nodes.

[0130] In this embodiment, when nominating ψ i With ψ j There is an overlap between them and r(ψ i , ψ j )>θ1, an edge is constructed. Obviously, the non-overlapping part can provide rich context information to help more accurate positioning. Here θ1 is set to 0.7. i , ψ j ) is the regional intersection-union ratio, and the formula is as follows:

[0131] r(ψ i , ψ j )=(ψ i ∩ψ j ) / (ψ i ∪ψ j )

[0132] In addition, even if ψ i Nomination and ψ j There is no overlap between them, and the distance satisfies d(ψ i , ψ j)<θ2, construct an edge, the purpose is to take into account the possible causal relationship between different actions, and use this relationship to help classify the actions. Here θ2 is set to 1. i , ψ j ) is calculated as:

[0133] d(ψ i , ψ j )=|c i -c j | / (ψ i ∪ψ j )

[0134] Where c i and c j Respectively represent the nomination ψ i and ψ j The center point of .

[0135] Preferably, step B2 consists of two layers of graph convolution, and the graph convolution operation is implemented by the following formula:

[0136] X (k) =AX (k-1) W (k)

[0137] Where k = {1, 2}, represents the kth layer. A is the adjacency matrix, is a learnable parameter, is the hidden layer feature of the kth layer, X (0) is the input feature, that is, the output feature F of the nomination generation module p .

[0138] More preferably, a ReLU activation function is used after each layer of graph convolution. The adjacency matrix is ​​used to represent the edges and their connection strengths, and the cosine similarity is calculated using the formula:

[0139]

[0140] B3. Obtain positioning results based on the nomination relationship diagram. Figure 7 As shown, preferably, step B3 specifically includes: obtaining a positioning result through a three-layer fully connected network according to the nomination relationship graph.

[0141] Specifically, through a simple design of three-layer full connection, the output features of the nomination relationship module are used as input to output the action category scores p cls , start and end time (t′ s , t′ e ) and confidence score p conf Three prediction results.

[0142] The temporal action localization method of the embodiment of the present invention is a multi-stage multi-task network, and its loss function is composed of multiple parts.

[0143] The mathematical expression of the loss function of the temporal nomination generation network model is as follows:

[0144] L tag =L bl (p s )+L bl (p e )+L bl (p c )

[0145] Where, L bl is the logistic regression loss function, p is the prediction score, is the label score. The three losses are the start, end and action completeness losses.

[0146] The loss function of the temporal nomination classification network model consists of three parts: classification loss, regression loss, and action integrity loss. The mathematical expression is as follows:

[0147] L rel =L ce (p cls )+L bl (p conf )+L l1 (t′ s , t′ e )

[0148] Where, L ce is the cross entropy loss function, L l1 is the L1 loss function.

[0149] In the several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of the devices, methods, and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified functions or actions, or can be implemented using a combination of dedicated hardware and computer instructions.

[0150] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0151] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, electronic device, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk. It should be noted that, in this article, the terms "include", "comprising" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without further constraints, an element defined by the phrase "comprises a..." does not preclude the existence of additional identical elements in the process, method, article or apparatus that includes the element.

[0152] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "an", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0153] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0154] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0155] The "first" and "second" mentioned in the embodiments are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that the specific order or precedence of "first" and "second" can be interchanged where appropriate. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0156] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A temporal action localization method based on global perception and nomination relationship mining, characterized by: Include: Get the video to be identified; Extracting visual features based on the video to be identified; Inputting the visual features into a pre-trained temporal nomination generation network model to obtain temporal nominations; Input the time series nomination into a pre-trained time series nomination classification network model to obtain a positioning result; the positioning result includes the action category and the start and end time of the action; The time series nomination generation network model is used to: Obtaining abstract semantic features based on the visual features; Based on the abstract semantic features, graph modeling is performed with video unit features as nodes and relationships between video unit features as edges to obtain an output feature stream; According to the output feature stream, a feature relationship graph is constructed with boundary features as nodes and action features as edges, and the feature relationship graph is updated to obtain new edge features and node features, and then the time series nomination is obtained based on the new edge features and new node features; The temporal nomination classification network model is used to: Nominations are made according to the time sequence, and nomination features are obtained; According to the nomination features, a nomination relationship graph is constructed with the nomination features as nodes and the relationships between nominations as edges; Obtaining the positioning result according to the nomination relationship graph; According to the feature relationship graph, update to obtain new edge features and node features, specifically including: According to the feature relationship graph, edge updates are performed to obtain new edge features. The edge update model is: , where is the updated edge, is the Relu activation function, are learnable parameters, is the edge before update, For the The feature representation of a time series, For the Feature representation between temporal positions; According to the feature relationship graph, normalize the edge features of each node respectively, and then perform graph convolution operation to obtain new node features; the node update model is: , where For the updated node, is the Relu activation function, are learnable parameters, is the identity matrix, Indicates the start node and end node The edge between.

2. The temporal action localization method based on global perception and nomination relationship mining according to claim 1 is characterized in that: Obtaining abstract semantic features based on the visual features, specifically including: Obtaining abstract semantic features through convolution based on the visual features; Based on the abstract semantic features, graph modeling is performed with video unit features as nodes and relationships between video unit features as edges to obtain an output feature stream, specifically including: According to the abstract semantic features, a static edge feature stream is constructed through temporal graph convolution; Based on the abstract semantic features, a dynamic edge feature stream is constructed through a self-attention module and graph convolution; fusing the abstract semantic features, the static edge feature stream, and the dynamic edge feature stream to construct a feature map and obtain the output feature stream; According to the output feature stream, a feature relationship graph is constructed with boundary features as nodes and action features as edges, and the feature relationship graph is updated to obtain new edge features and node features. Then, according to the new edge features and new node features, the time series nomination is obtained, specifically including: According to the output feature flow, three one-dimensional convolution operations are performed to obtain the action start feature. , end feature and action characteristics ; Start the feature according to the action , end feature and action characteristics , starting with action features and end features As nodes, action features For edges, construct a feature relationship graph; Update according to the feature relationship graph to obtain new edge features and node features; The time series nomination is obtained according to the new edge feature and node feature.

3. The temporal action localization method based on global perception and nomination relationship mining according to claim 2 is characterized in that: Based on the abstract semantic features, a static edge feature stream is constructed through temporal graph convolution, specifically including: According to the abstract semantic features, static feature streams are extracted through 32 branch paths respectively; wherein each branch path includes one graph convolution operation and two convolution operations; The 32 static feature streams are fused in an additive manner to construct the static edge feature stream; Based on the abstract semantic features, a dynamic edge feature flow is constructed through the self-attention module and graph convolution, specifically including: According to the abstract semantic features, an attention matrix is ​​obtained through a self-attention module; According to the abstract semantic features, two layers of one-dimensional convolution are performed, and then a graph convolution operation is performed using the attention matrix as the adjacency matrix to construct the dynamic edge feature flow.

4. The temporal action localization method based on global perception and nomination relationship mining according to claim 3 is characterized in that: The self-attention module includes two one-dimensional convolutions; According to the abstract semantic features, the attention matrix is ​​obtained through the self-attention module, which specifically includes: According to the abstract semantic features, query Q is obtained through one-dimensional convolution; According to the abstract semantic features, a key value K is obtained through one-dimensional convolution; The query Q is multiplied by the key value K, and then activated with the sigmoid activation function to obtain the attention matrix.

5. The temporal action localization method based on global perception and nomination relationship mining according to claim 1 is characterized in that: According to the time series nomination, nomination features are obtained, including: According to the time sequence nomination, a start area, an action area and an end area are obtained; The three regions are uniformly sampled using a linear interpolation method, and a two-dimensional convolution operation and dimensional transformation are performed on the sampled features to obtain the nominated features; wherein the number of samples in the start region and the end region is 4, and the number of samples in the action region is 8; Based on the nomination features, a nomination relationship graph is constructed with the nomination features as nodes and the relationships between nominations as edges, specifically including: According to the nomination features, taking the nomination features as nodes and the relationships between nominations as edges, edges are constructed between nodes and their adjacent nodes to obtain the nomination relationship graph; Obtaining the positioning result according to the nomination relationship graph specifically includes: According to the nomination relationship graph, the positioning result is obtained through a three-layer fully connected network.

6. The temporal action localization method based on global perception and nomination relationship mining according to any one of claims 1 to 5, characterized in that: Extracting visual features from the video to be identified includes: According to the video to be identified, visual features are obtained through a dual-stream network model.

Images