Online action detection algorithm, system, and equipment based on directed graph information interaction

Abstract

The present invention discloses an online action detection algorithm based on directed graph information interaction, which is used to model temporal information and complete the task of temporal action positioning. For an action category, first, each feature of all action instances of this type is collected and clustered, and representative features are selected as typical action segments. Secondly, the typical action segments are divided into three stages and embedded in a directed graph in chronological order, and a typical action network is constructed globally from the category level. Thirdly, four information interaction steps are designed in the typical action network: internal interaction within the stage, transmission between adjacent stages, typical information aggregation and video frame broadcasting, and the temporal information is processed in sequence. The features of each typical action segment in the typical action network are adjusted to provide appropriate temporal information guidance for each frame of video processed. Finally, the online action detection algorithm based on directed graph information interaction is combined with the existing temporal action positioning method to achieve performance gains under supervision.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision algorithms, and in particular to an online action detection algorithm, system, and device based on directed graph information interaction. Background Art

[0002] In the information age of the Internet of Things, video, as an effective medium for recording the objective world and transmitting information, has been widely used in production and life, which has made the demand for video interpretation increasingly urgent.

[0003] To effectively perceive the structure of actions, researchers in the field of video action interpretation have conducted in-depth exploration and achieved a series of results. In their 2020 work, G-tad: Sub-graph localization for temporal action detection, Mengmeng Xu et al. modeled temporal and semantic neighborhood relationships between video clips. In their 2021 work, Graph convolutional module for temporal action localization in videos, Runhao Zeng et al. used graph topology to represent the relationships between neighboring action proposals. Recently, the self-attention mechanism has demonstrated excellent ability to model the structure of actions. In their 2017 work, Attention is all you need, Ashish Vaswani et al. used dense connections to enable video frames to interact with any neighboring frames through a single sequential operation. This rich interaction between neighboring frames provides rich temporal context for the video frames. However, while graph convolutional and self-attention mechanisms aim to learn a unique representation for each action type, they often overlook intra-class diversity and the fact that an action typically consists of multiple stages, with the video frames of each stage exhibiting different appearance and motion characteristics. In addition, existing methods are usually limited to a single video, or even a sliding window in a video, for action structure modeling, and have not yet fully explored the relationship between cross-video category levels.

[0004] To characterize the diversity of similar actions and model action category-level relationships across videos, researchers have explored some meaningful methods. For example, Wang Luo et al. proposed a learning memory pool (Memory Bank) in their 2021 work "Action unit memory network for weakly supervised temporal action localization," which serves as a prototype for action learning categories. Le Yang et al. selected representative sub-actions from each action category and considered them as typical action segments in their 2022 work "Colar: Effective and efficient online action detection by consulting exemplars." These models have also achieved corresponding performance improvements on specific tasks.

[0005] However, the above existing methods only process representative typical action clips at the category level, but have never explored the temporal relationship between multiple typical action clips. This will result in the omission of three meaningful information: (1) Since typical action clips only represent part of the complete action, multiple typical action clips from different stages can usually contain different temporal information. (2) If typical action clips are organically organized in time sequence, they can represent the complete concept of such action. We call the typical action clips organically organized in time sequence the Exemplar Complex, abbreviated as ExCo. (3) Through the effective interaction between video frames and the Exemplar Complex, the video interpretation task can perceive reliable action structure information, thereby achieving stable performance gains. Summary of the Invention

[0006] In response to the above-mentioned problems, the present invention aims to provide an online action detection algorithm based on directed graph information interaction, which is targeted at supervised training, used to model temporal information, complete the task of temporal action positioning, and thus solve the task of supervised online action detection.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] An online action detection algorithm based on directed graph information interaction, characterized by comprising the following steps:

[0009] Step 1: Input the video data to be detected, and use the I3D model to extract the video features of the video data to obtain the high-level semantic features of the video;

[0010] Step 2: Generate typical actions based on the K-Means algorithm for a type of action to be detected;

[0011] Step 3: For the generated typical actions, a directed graph is constructed based on the information transfer mechanism to explicitly represent the structure between the typical actions.

[0012] Step 4: Combine the directed graph representing typical actions with the Colar method to perform action detection in an online manner;

[0013] Step 5: Collect video frame features from typical actions of all categories, compare the dot product similarity between the features of the video frame to be detected and the features of the typical actions, and obtain the video frame classification score;

[0014] Step 6: Based on the obtained predicted classification scores, perform video action detection and output the action positioning results.

[0015] Furthermore, step 2 includes the following specific steps:

[0016] Step 21: For a type of action, obtain high-level semantic features of all action instances;

[0017] Step 22: Cluster the obtained features based on the K-Means algorithm to obtain M clusters. The feature closest to the cluster center in each cluster is used as the typical feature, and M typical features are obtained;

[0018] Step 23: Divide each action instance evenly into three stages. According to the position of the typical feature in its corresponding action instance, determine the stage to which the typical feature belongs, and obtain multiple typical actions in each stage.

[0019] Furthermore, the action instance is divided into three stages: action start, action evolution and action end.

[0020] Furthermore, step 3 includes the following specific steps:

[0021] Step 31: Connect multiple typical actions belonging to the same stage, and perform two-way information exchange between each two typical actions through the internal interactive information transmission mechanism of the stage;

[0022] Step 32: Using the adjacent stage information transfer mechanism, each typical feature of the action start stage is connected with all the typical features of the action evolution stage, and each typical feature of the action evolution stage is connected with all the typical features of the action end stage, thereby constructing a directed graph between the typical features of different stages to explicitly represent the structural information of the action;

[0023] Step 33: Compare the video frame features with all typical features one by one, and use the attention mechanism to adaptively aggregate features from various stages of action instance division, thereby aggregating the typical information to the video frame through the typical information aggregation message passing mechanism;

[0024] Step 34: Using the video frame broadcast operation via the video frame broadcast message delivery mechanism, the characteristics of the current video frame are delivered to the constructed directed graph, so as to enable the typical actions to make adaptive adjustments according to the characteristics of the video frame.

[0025] Furthermore, the specific steps of step 4 are: replacing the static typical branches used in the traditional Colar method with directed graph information to interactively represent the action structure information, so as to perform the action detection task in an online form while keeping other settings consistent with Colar.

[0026] Furthermore, step 5 includes the following specific steps:

[0027] Step 51: Collect video frame features from all categories of typical actions represented by directed graphs to obtain feature representations

[0028] Step 52: Perform L2 regularization on all features;

[0029] Step 53: Calculate the dot product similarity between the video features and the typical features, average the similarity scores of the cth category, and obtain the classification score s c , and then the classification scores s of various action categories c The sum is finally added to obtain the classification score s of the video feature.

[0030] Furthermore, a loss function is defined, and training and testing are performed according to the loss function. The loss function is:

[0031] L=L van +L exp +λ·L cons

[0032] Among them, L exp represents the loss term corresponding to the typical branch; L cons represents the consistency loss term; λ is used to adjust the impact of consistency loss; L van represents the loss term calculated by the Colar algorithm, and:

[0033]

[0034] Where, Represents Focal loss, which is used for action category classification; represents DIoU loss, which is used for the regression of action boundaries; T + Indicates the number of positive samples in a video; is an indicator function, used to indicate whether the current moment belongs to a certain action; reg Indicates the equalization coefficient, used for balancing and

[0035] An online action detection system based on directed graph information interaction, characterized by comprising a video feature extraction module, a typical action generation module, an action modeling module, a weakly supervised video action detection module, a prediction classification score module, and a detection result output module;

[0036] The video feature extraction module is used to extract video features of the video data to be detected using the I3D model to obtain a high-level semantic representation of the video;

[0037] The typical action generation module is used to generate typical actions based on the K-Means algorithm for a type of action to be detected;

[0038] The action modeling module is used to construct a directed graph based on the information transmission mechanism, and to explicitly represent the structure between typical actions through the directed graph;

[0039] The online action detection module is used to combine the directed graph representing typical actions with the Colar method to perform action detection in an online form;

[0040] The prediction classification score module is used to collect video frame features from typical actions of all categories, compare the dot product similarity between the video frame features and the typical features, and obtain the video frame classification score;

[0041] The detection result output module is used to perform video action detection and output action positioning results based on the obtained prediction classification scores.

[0042] A computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the program implements the steps of the online action detection algorithm based on directed graph information interaction as described in any one of claims 1 to 7.

[0043] A computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the processor implements the steps of the online action detection algorithm based on directed graph information interaction as described in any one of claims 1 to 7.

[0044] The beneficial effects of the present invention are:

[0045] First, the online action detection algorithm with directed graph information interaction proposed in the present invention is an algorithm with both representational and discriminative power, which can effectively serve the video interpretation tasks involving action structure. As a representational module, first of all, each selected typical action clip corresponds to a series of video frames with clear appearance features and motion patterns. The clear features corresponding to such typical action clips can represent many similar features. Secondly, the algorithm divides a type of action into three stages: the beginning of the action, the evolution of the action, and the end of the action, which can explicitly characterize the structure of the same type of action. Furthermore, each stage of the algorithm contains multiple typical action clips, which can appropriately reflect the appearance and motion diversity of the actions in the same stage;

[0046] Secondly, as a discriminative module, compared to the features of most training videos, the directed graph information interaction network participates in the training of all video frames. It is fully trained through interaction with all samples, making it a discriminative module. In addition, given video features, traditional methods usually learn a classifier with additional parameters to predict the classification score. However, this algorithm predicts the classification score by comparing the similarity between video features and each typical action. This approach, without introducing additional parameters, further enhances the algorithm's feature discriminative power.

[0047] Second, the online action detection algorithm based on directed graph information interaction proposed in this paper can be efficiently combined with the Collar method to effectively model action structure and cross-video dependencies, serving the task of temporal action localization. The traditional Collar method treats all typical actions equally, thus lacking structural modeling. However, the proposed method organizes typical actions in the form of a directed graph, which can accurately represent the action structure.

[0048] Therefore, the method proposed in the present invention is aimed at supervised training and handles supervised online action detection tasks. By efficiently combining the online action detection algorithm based on directed graph information interaction with the Colar method, it can achieve a performance gain of 2.1% on the TVSeries dataset while saving computational overhead. With limited computational complexity, it can classify and locate video actions with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 The flowchart of the method proposed in the present invention is shown in FIG.

[0050] Figure 2 A visualization of some of the training data.

[0051] Figure 3 This is a diagram of the experimental results of the method proposed in the present invention. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0053] The core idea of the present invention is: considering an action category, first, this method collects each feature of all action instances of this type and clusters them, and selects representative features as typical action clips. Then, this method divides the typical action clips into three stages and embeds them into a directed graph in chronological order, and constructs a typical action network globally from the category level. Next, this method designs four information interaction steps in the typical action network: internal interaction within the stage, adjacent stage transmission, typical information aggregation and video frame broadcasting. These four steps process the timing information in sequence, adjust the features of each typical action clip in the typical action network, and provide appropriate timing information guidance for each frame of video processed. Finally, this online action detection algorithm based on directed graph information interaction is combined with the existing temporal action localization method to achieve performance gains.

[0054] Based on the above technical ideas, the present invention proposes an online action detection algorithm based on directed graph information interaction, which includes the following steps:

[0055] Step 1: Extract video features:

[0056] For a given video, we use the I3D model proposed by Joao Carreira in his 2017 work Action recognition a new model and the kinetics dataset to extract a series of video features and obtain the high-level semantic representation of the video F = [f1, f2, ..., f T ],in represents the t-th video feature, f t Reflects the information of a continuous video frame.

[0057] Step 2: Generate typical actions based on clustering algorithm

[0058] Step 21: Consider a class of actions. First, collect the feature representations of all action instances. Then use the K-Means algorithm to cluster the features to obtain M clusters. For each cluster of feature representations, take the feature closest to the cluster center as the typical feature, and obtain M typical features.

[0059] Step 22: Divide each action instance into three phases: the beginning of the action, the evolution of the action, and the end of the action. Determine the phase to which the typical feature belongs based on its position in the corresponding action instance. Different action categories typically have different numbers of typical actions in the three phases.

[0060] Step 3: Action modeling based on structural information transfer

[0061] Given the typical actions corresponding to the three stages, we construct a directed graph G using four information transmission mechanisms: internal interaction within the stage, adjacent stage transmission, typical information aggregation, and video frame broadcasting. c (V c , E c ) to explicitly characterize the structure between typical actions.

[0062] Step 31: Connect multiple representative actions belonging to the same stage and perform bidirectional information exchange between each two representative actions. Considering a specific action stage, multiple representative actions can reflect the diversity of actions in this stage in terms of appearance characteristics and motion patterns. Therefore, intra-stage interaction facilitates the representative actions to capture diversity within the same time sequence.

[0063] Step 32: This involves two types of information transfer: from "action start" to "action evolution" and from "action evolution" to "action end." The present invention connects each typical feature at the start stage with all typical features at the evolution stage, and connects each typical feature at the evolution stage with all typical features at the end stage. Ultimately, a directed graph is established between the typical features at different stages, constructing a graph transformation network. Adjacent stage transfers enable each typical feature to explicitly perceive its predecessor and successor typical features, thereby explicitly characterizing the structural information of the action.

[0064] Step 33: After sufficient information exchange within and between stages, the typical action network (the constructed graph transformation network) can accurately convey structural information for the processed video frame. At this time, the typical information is aggregated into the video frame to provide temporal information support for the action interpretation task. The video frame features are compared with all the typical features one by one, and the attention mechanism is used to adaptively aggregate features from each action stage. After the typical information aggregation, the video frame features can fully perceive the structural information of the current action;

[0065] Step 34: Use the video frame broadcast operation to pass the characteristics of the current video frame to the graph transformation network module. Each representative action comprehensively considers its own characteristics and the characteristics of the video frame being processed, and updates its own characteristics according to the attention mechanism. The video frame broadcast operation enables the representative action to make adaptive adjustments based on the characteristics of the video frame, thereby generating more representative characteristics during the next layer of information propagation.

[0066] Step 4: Typical action network module combined with Collar method

[0067] Combine the online detection algorithm for directed graph information interaction (steps 1-3 and step 3) with the temporal action localization method Colar proposed by Yang Le et al. in their 2022 work Colar: Effective and efficient online action detection by consulting exemplars. Specifically:

[0068] While traditional Colar uses a dynamic canonical branch to process adjacent historical frames and a static canonical branch to compare the similarity between the current frame and the canonical action and perform cross-video category-level modeling, this paper proposes using directed graph information interaction to represent action structure information, replacing the static canonical branch with a directed graph interaction module. While maintaining consistency with Colar in other settings, the paper performs action detection online.

[0069] Step 5: Classification estimation based on similarity comparison

[0070] In this algorithm, the node features of the last layer of graph transformation network can be expressed as in, Represents the characteristics of M typical nodes in the cth category, Represents the video frame features represented by category c. Collecting video frame features from typical action network modules of all categories can obtain feature representation

[0071] In order to obtain the predicted classification score, all features are L2 regularized, and then the dot product similarity between the video frame features and the typical features is calculated, and the similarity scores of the cth class are averaged to obtain the classification score s c , and finally obtain the classification score s of the video frame.

[0072] Step 6: Training and Testing

[0073] During the training phase, set the loss function:

[0074] L=L van +L exp +λ·L cons (1)

[0075] Among them, L van Represents the loss term calculated by the Colar algorithm, which is calculated as follows:

[0076]

[0077] Where, It represents the Focal loss proposed by Tsung-Yi Lin et al. in their 2017 work Focal loss for dense object detection, which is used for action category classification; It represents the DIoU loss proposed by Zhaohui Zheng et al. in their 2020 work Distance-IoU loss: Faster and better learning for bounding box regression, which is used for the regression of action boundaries. + Indicates the number of positive samples in a video. Is an indicator function, used to indicate whether the current moment belongs to a certain action. reg Indicates the equalization coefficient, used for balancing and

[0078] Given the prediction results of a typical action branch, refer to the Colar algorithm to calculate the corresponding loss term L of the typical branch exp Constrain the prediction results of the Colar algorithm to be consistent with the prediction results of the typical action branch, and calculate the consistency loss term L cons ,λ is used to adjust the impact of consistency loss.

[0079] In the test phase, the prediction results of the Colar algorithm are fused using the hyperparameter β. van And the prediction results of typical action branches s exp , and get the final result of the relevant video interpretation task:

[0080] s=β·s van +(1-β)·s exp .

[0081] Example

[0082] In order to further verify the effect of the method proposed in the present invention, the method was implemented using PyTorch1.5 software.

[0083] 1. Experimental environment

[0084] The hardware environment for implementing the present invention is: Intel Xeon E5-2600v3 @ 2.6GHz 8-core CPU processor, 128GB memory, equipped with GeForce GTX 1600 Ti AN2080 Ti GPU. The software environment is: Linux 16.04 64-bit operating system.

[0085] 2. Implementation process

[0086] (1) Constructing a dataset

[0087] In this embodiment, two datasets, THUMOS14 and ActivityNet-v1.3, are used for experiments. The THUMOS14 dataset comes from: http: / / crcv.ucf.edu / THUMOS14 / , and the ActivityNet-v1.3 dataset comes from: http: / / activity-net.org / . The THUMOS14 training dataset contains 20 action categories, and the ActivityNet-v1.3 training dataset contains 200 action categories. Each training video contains multiple action instances. Figure 2 All videos are processed individually using the method of the present invention.

[0088] (2) Extracting video features

[0089] Using the I3D model proposed by Joao Carreira et al. in their 2017 work Quo vadis, action recognition a new model and the kinetics dataset[C] / / proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:6299-6308., we extract a series of video features and obtain a high-level semantic representation of the video F = [f1, f2, ..., f T ],in represents the t-th video feature, f t Reflects the information of a continuous video frame.

[0090] The I3D model is pre-trained on the Kinetics-400 dataset, which is available from: https: / / deepmind.com / research / open-source / kinetics.

[0091] (3) Constructing a typical action network module

[0092] This example uses Deep GraphLibray 0.8.2, proposed by Minjie Wang et al. in their 2019 work Wang M, Zheng D, Ye Z, et al. Deep graph library: A graph-centric, high-performant package for graph neural networks [J]. arXiv preprint arXiv:1909.01315, 2019, to implement the typical action network module. The typical action is run in the form of a graph transformer network. Specifically, the following steps are performed:

[0093] First, initialize the node features and edge features of the graph transformation network. Use a linear transformation layer to transform the typical action features Projected into the latent space, Furthermore, consider a line consisting of node v i Points to node v j The edge of the edge is represented by e ij .

[0094] In the initialization phase, splicing x i and x j The features of the two nodes and apply a linear transformation layer for feature projection, In addition, the random walk strategy proposed by Pan Li et al. in their 2020 work Li P, Wang Y, Wang H, et al. Distance encoding: Design provably more powerful neural networks for graph representation learning [J]. Advances in Neural Information Processing Systems, 2020, 33: 4465-4478. is used to generate position encoding for each node and a linear transformation layer is used to project the position encoding into the latent space.

[0095] The online action detection algorithm based on directed graph information interaction uses a multi-layer graph transformation network. For the graph transformation network at layer l, the feature of its i-th node is The corresponding position code is Connect node v i and v j The feature corresponding to the edge of Using the projection matrix To process node features and corresponding position encoding, use the projection matrix To process edge features.

[0096] To calculate the node v i and its adjacent node v j , j∈N i The correlation of , estimated correlation vector

[0097]

[0098] in represents element-wise multiplication, and [·] represents concatenation in the channel dimension.

[0099] Then, the similarity is calculated as follows:

[0100]

[0101] Update the features of node vi according to formula (4):

[0102]

[0103] Finally, standard operations in the graph transformation network are used to process node features, using residual connections, layer normalization, and feed forward networks in sequence.

[0104] To simplify the representation process, the above operations only consider the single-head self-attention mechanism, but these operations can be easily extended to the multi-head self-attention mechanism, thereby giving the graph transformation network module stronger representation capabilities.

[0105] At the same time, the present invention comprehensively considers the position code of the node itself, the position code of the neighboring nodes, and the characteristics of the edges connecting the neighboring nodes to update the position code pi of each node. The update process can be expressed as:

[0106]

[0107] Here, Γ(·) indicates that in the information propagation process similar to the node features, only the node position code is considered in the calculation, without considering the node feature information.

[0108] (4) Online action detection algorithm combining directed graph information interaction and Collar

[0109] First, Colar uses a dynamic canonical branch to process adjacent historical frames, and a static canonical branch to compare the similarity between the current frame and the canonical action and perform cross-video category-level modeling. Compared to the static canonical branch used by Colar, the proposed method uses directed graph information interaction to represent action structure information. Therefore, this method replaces the static canonical branch with a directed graph interaction module, and while maintaining the same settings as Colar, it performs action detection tasks in an online manner.

[0110] Secondly, in this algorithm, the node features of the last layer of graph transformation network can be expressed as in, Represents the characteristics of M typical nodes in the cth category, represents the video frame features represented by the typical nodes of type c, Represents the video frame features characterized by the background class.

[0111] Collecting video frame features from all categories of directed graph information interaction modules can obtain feature representation

[0112] Again, in order to obtain the predicted classification score, this method regularizes all features, then calculates the dot product similarity between the video frame features and the typical features, averages the similarity scores of the class, obtains the classification score, and finally obtains the classification score of the video frame.

[0113] (5) Training and testing

[0114] During the training phase, the loss function is calculated as follows:

[0115] L=L van +L exp +λ·L cons

[0116] Among them, L van Represents the loss term calculated by the colar algorithm, which is calculated as follows:

[0117]

[0118] Where, It represents the Focal loss proposed by Tsung-Yi Lin et al. in their 2017 work Lin TY, Goyal P, Girshick R, et al. Focal loss for dense object detection, which is used for action category classification; Indicates the DIoU loss proposed by Zhaohui Zheng et al. in their 2020 work Zheng Z, Wang P, Liu W, et al. Distance-IoU loss: Faster and better learning for bounding box regression, which is used for the regression of action boundaries. + Indicates the number of positive samples in a video. Is an indicator function, used to indicate whether the current moment belongs to a certain action. reg Indicates the equalization coefficient, used for balancing and

[0119] Given the prediction results of a typical action branch, we can refer to the Colar algorithm to calculate the corresponding loss term L of the typical branch. exp The prediction results of the constrained Colar algorithm are consistent with the prediction results of the typical action branch, and the consistency loss term L is calculated using the mean square error function cons ,λ is used to adjust the impact of consistency loss.

[0120] In the test phase, the prediction results of the Colar algorithm are fused using the hyperparameter β. van And the prediction results of typical action branches s exp , and get the result of the temporal action localization task:

[0121] s=β·s van +(1-β)·s exp ·

[0122] This paper uses the Soft-NMS method proposed by Navaneeth Bodla et al. in their 2017 work Bodla N, Singh B, Chellappa R, et al. Soft-NMS--improving object detection with one line of code[C] / / Proceedings of the IEEE international conference on computer vision.2017:5561-5569. to post-process the temporal action localization results, remove highly overlapping action instances, and generate the final result of temporal action localization, such as Figure 3 shown.

[0123] (6) Parameter settings

[0124] This method of the present invention refers to the work of Devin Kreuzer et al. in 2021, Kreuzer D, Beaini D, Hamilton W, et al. Rethinking graph transformers with spectral attention [J]. Advances in Neural Information Processing Systems, 2021, 34: 21618-21629. and the work of Vijay Prakash Dwivedi et al. in 2022, Dwivedi VP, Luu AT, Laurent T, et al. Graph neural networks with learnable structural and positional representations [J]. arXiv preprint ar Xiv: 2110.07875, 2021. The relevant parameters of the graph transformation network layer are set, and the dimensions of the node features and position encoding features are set to 64.

[0125] In addition, this method adjusts hyperparameters based on the experimental performance of the validation set. Ultimately, the typical number M is set to 10, the number of typical action layers is set to 3, the loss coefficient λ is set to 1.0, and the fusion coefficient β is set to 0.5.

[0126] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. An online action detection method based on directed graph information interaction, characterized in that: The following steps are involved: Step 1: Input the video data to be detected, and use the I3D model to extract the video features of the video data to obtain the high-level semantic features of the video; Step 2: Generate typical actions based on the K-Means algorithm for a type of action to be detected; Step 3: For the generated typical actions, a directed graph is constructed based on the information transfer mechanism to explicitly represent the structure between the typical actions. Step 3 includes the following specific steps: Step 31: Connect multiple typical actions belonging to the same stage, and perform two-way information exchange between each two typical actions through the internal interactive information transmission mechanism of the stage; Step 32: Using the adjacent stage information transfer mechanism, each typical feature of the action start stage is connected with all the typical features of the action evolution stage, and each typical feature of the action evolution stage is connected with all the typical features of the action end stage, thereby constructing a directed graph between the typical features of different stages to explicitly represent the structural information of the action; Step 33: Compare the video frame features with all typical features one by one, and use the attention mechanism to adaptively aggregate features from various stages of action instance division, thereby aggregating the typical information to the video frame through the typical information aggregation message passing mechanism; Step 34: Using a video frame broadcast operation through a video frame broadcast message delivery mechanism, the characteristics of the current video frame are delivered to the constructed directed graph, so that the typical actions can be adaptively adjusted according to the characteristics of the video frame; Step 4: Combine the directed graph representing typical actions with the Colar method to perform action detection in an online manner; Step 5: Collect video frame features from typical actions of all categories, compare the dot product similarity between the features of the video frame to be detected and the features of the typical actions, and obtain the video frame classification score; Step 6: Based on the obtained predicted classification scores, perform video action detection and output the action positioning results.

2. The online action detection method based on directed graph information interaction according to claim 1, characterized in that: Step 2 includes the following specific steps: Step 21: For a type of action, obtain high-level semantic features of all action instances; Step 22: Cluster the obtained features based on the K-Means algorithm to obtain M clusters. The feature closest to the cluster center in each cluster is used as the typical feature, and M typical features are obtained; Step 23: Divide each action instance evenly into three stages. According to the position of the typical feature in its corresponding action instance, determine the stage to which the typical feature belongs, and obtain multiple typical actions in each stage.

3. The online action detection method based on directed graph information interaction according to claim 2, characterized in that: The action instance is divided into three stages: action start, action evolution and action end.

4. The online action detection method based on directed graph information interaction according to claim 3 is characterized in that: The specific steps of step 4 are: replacing the static typical branches used in the traditional Colar method with directed graph information to interactively represent the action structure information, so as to perform the action detection task in an online form while keeping other settings consistent with Colar.

5. The online action detection method based on directed graph information interaction according to claim 4 is characterized in that: Step 5 includes the following specific steps: Step 51: Collect video frame features from all categories of typical actions represented by directed graphs to obtain feature representations ; Step 52: Perform Regularization; Step 53: Calculate the dot product similarity between the video features and the typical features, average the similarity scores of the cth category, and obtain the classification score , and then classify the scores of various action categories Add up and finally get the classification score of the video feature .

6. The online action detection method based on directed graph information interaction according to claim 5, characterized in that: Define the loss function, and perform training and testing based on the loss function. The loss function is: in, represents the loss term corresponding to the typical branch; represents the consistency loss term; Used to adjust the impact of consistency loss; represents the loss term calculated by the Colar algorithm, and: Where, Represents Focal loss, which is used for action category classification; represents DIoU loss, which is used for the regression of action boundaries; Indicates the number of positive samples in a video; It is an indicator function, used to indicate whether the current moment belongs to a certain action; Indicates the equalization coefficient, used for balancing and .

7. An online action detection system based on directed graph information interaction, characterized in that: The online action detection method based on directed graph information interaction according to any one of claims 1 to 6 is implemented, wherein the detection system includes a video feature extraction module, a typical action generation module, an action modeling module, a weakly supervised video action detection module, a prediction classification score module, and a detection result output module; The video feature extraction module is used to extract video features of the video data to be detected using the I3D model to obtain a high-level semantic representation of the video; The typical action generation module is used to generate typical actions based on the K-Means algorithm for a type of action to be detected; The action modeling module is used to construct a directed graph based on the information transmission mechanism, and to explicitly represent the structure between typical actions through the directed graph; The online action detection module is used to combine the directed graph representing typical actions with the Colar method to perform action detection in an online form; The prediction classification score module is used to collect video frame features from typical actions of all categories, compare the dot product similarity between the video frame features and the typical features, and obtain the video frame classification score; The detection result output module is used to perform video action detection and output action positioning results based on the obtained prediction classification scores.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the online action detection method based on directed graph information interaction as described in any one of claims 1 to 6 are implemented.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the online action detection method based on directed graph information interaction according to any one of claims 1 to 6 are implemented.

Images