Weakly Supervised Video Action Detection Method, System, and Device Based on Graph Transformation Network

Abstract

The present invention discloses a weakly supervised video action detection method based on a graph transformation network, which studies weakly supervised video action detection from the perspective of learning discriminative features of action background, and constructs a novel graph transformation network module. The module selects representative typical features from each type of action, embeds them into a directed graph according to the spatiotemporal structure, and designs four information propagation mechanisms: internal interaction within the stage, transmission between adjacent stages, typical information aggregation, and video frame broadcasting. Through the method of the present invention, effective information interaction can be carried out between the video frame and the graph transformation network, providing action structure information guidance for the weakly supervised video action detection task. In addition, the graph transformation network module of the present invention can effectively model the action structure and can be deployed as a plug-and-play module into the existing weakly supervised video action detection framework, significantly improving its performance on related tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision algorithms, and in particular to a method, system, and device for weakly supervised video action detection based on a graph transformation network. Background Art

[0002] In recent years, the rapid development of social media and video sharing websites has led to an increasingly strong demand for video processing, and the research on video interpretation methods has significant application value.

[0003] To effectively perceive the structure of actions, researchers in the field of video action interpretation have conducted in-depth exploration and achieved a series of results. In their 2020 work, G-tad: Sub-graph localization for temporal action detection, Mengmeng Xu et al. modeled temporal and semantic neighborhood relationships between video clips. In their 2021 work, Graph convolutional module for temporal action localization in videos, Runhao Zeng et al. used graph topology to represent the relationships between neighboring action proposals. Recently, the self-attention mechanism has demonstrated excellent ability to model the structure of actions. In their 2017 work, Attention is all you need, Ashish Vaswani et al. used dense connections to enable video frames to interact with any neighboring frames through a single sequential operation. This rich interaction between neighboring frames provides rich temporal context for the video frames. However, while graph convolutional and self-attention mechanisms aim to learn a unique representation for each action type, they often overlook intra-class diversity and the fact that an action typically consists of multiple stages, with the video frames of each stage exhibiting different appearance and motion characteristics. In addition, existing methods are usually limited to a single video, or even a sliding window in a video, for action structure modeling, and have not yet fully explored the relationship between cross-video category levels.

[0004] To characterize the diversity of similar actions and model action category-level relationships across videos, researchers have explored some meaningful methods. For example, Wang Luo et al. proposed a learning memory pool (Memory Bank) in their 2021 work "Action unit memory network for weakly supervised temporal action localization," which serves as a prototype for action learning categories. Le Yang et al. selected representative sub-actions from each action category and considered them as typical action segments in their 2022 work "Colar: Effective and efficient online action detection by consulting exemplars." These models have also achieved corresponding performance improvements on specific tasks.

[0005] However, existing methods only process representative typical action clips at the category level, but have not explored the temporal relationship between multiple typical action clips. Ignoring the structural relationship between typical actions may miss the following three meaningful information: (1) Since typical action clips only represent part of the complete action, multiple typical action clips from different stages can usually contain different temporal information. (2) If typical action clips are organically organized in time sequence, they can represent the complete concept of such action. We call the typical action clips organically organized in time sequence the Exemplar Complex, abbreviated as ExCo. (3) Through the effective interaction between video frames and the Exemplar Complex, the video interpretation task can perceive reliable action structure information, thereby achieving stable performance gains. Summary of the Invention

[0006] In response to the above-mentioned problems, the present invention aims to provide a weakly supervised video action detection method based on graph transformation network, which is used to model temporal information and complete the weakly supervised video action detection task.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] A weakly supervised video action detection method based on a graph transformation network is characterized by comprising the following steps:

[0009] Step 1: Input the video data to be detected, and use the I3D model to extract the appearance features and motion features of the video data to obtain the video appearance semantic features and motion semantic features respectively;

[0010] Step 2: Generate typical actions based on the K-Means algorithm for a type of action to be detected;

[0011] Step 3: For the generated typical actions, a directed graph is constructed based on the information transmission mechanism, a graph transformation network module is established, and the graph transformation network module is used to model the actions;

[0012] Step 4: Introduce the graph transformation network module into the weakly supervised video action detection method CO2-Net, using two branches to process the video appearance semantic features and action semantic features in turn;

[0013] Step 5: By comparing the feature similarity between the video to be detected and the typical action, the predicted classification scores of the video appearance semantic features and the action semantic features are obtained respectively;

[0014] Step 6: Based on the obtained predicted classification scores, perform video action detection and output the detection results.

[0015] Furthermore, step 2 includes the following specific steps:

[0016] Step 21: For a type of action, obtain high-level semantic features of all action instances;

[0017] Step 22: Cluster the obtained features based on the K-Means algorithm to obtain M clusters. The feature closest to the cluster center in each cluster is used as the typical feature, and M typical features are obtained;

[0018] Step 23: Divide each action instance evenly into three stages, and determine the stage to which the typical feature belongs based on the position of the typical feature in its corresponding action instance.

[0019] Furthermore, the action instance is divided into three stages: action start, action evolution and action end.

[0020] Furthermore, the information transmission mechanism in step 3 includes four types: intra-stage interaction, adjacent stage transmission, typical information aggregation and video frame broadcasting.

[0021] Furthermore, step 3 includes the following specific steps:

[0022] Step 31: Connect multiple typical actions belonging to the same stage, and perform two-way information exchange between each two typical actions through the internal interactive information transmission mechanism of the stage;

[0023] Step 32: Using the adjacent stage information transfer mechanism, each typical feature of the action start stage is connected with all the typical features of the action evolution stage, and each typical feature of the action evolution stage is connected with all the typical features of the action end stage, thereby constructing a directed graph between the typical features of different stages and obtaining a graph transformation network module;

[0024] Step 33: Compare the video frame features with all typical features one by one, and use the attention mechanism to adaptively aggregate features from various stages of action instance division, thereby aggregating typical information to the video frame;

[0025] Step 34: Use the video frame broadcast operation to pass the characteristics of the current video frame to the graph transformation network module, so that the typical actions can be adaptively adjusted according to the characteristics of the video frame.

[0026] Furthermore, step 5 includes the following specific steps:

[0027] Step 51: Collect video frame features from the graph transformation network modules of all action categories to obtain the video frame feature representation represented by the c-th action

[0028] Step 52: Perform L2 regularization on all features obtained in step 51;

[0029] Step 53: Calculate the dot product similarity between the video features and the typical features, average the similarity scores of the cth category, and obtain the classification score s c , and then the classification scores s of various action categories c The sum is finally added to obtain the classification score s of the video feature.

[0030] Furthermore, during the training phase of the graph transformation network module, training and testing are performed according to the loss function, and the loss function is defined as:

[0031] L=L mil +L cas +L ml +λ1L oppo +λ2L norm

[0032] Among them, L mil and L cas They represent top-k multiple-instance learning loss and co-activity similarity loss, L ml represents the mutual learning loss, L oppo Indicates the similarity between the attention weight distribution and the background class probability distribution, L norm represents the normalized loss, and λ1 and λ2 are hyperparameters for balancing.

[0033] A weakly supervised video action detection system based on a graph transformation network is characterized by comprising a video feature extraction module, a typical action generation module, an action modeling module, a weakly supervised video action detection module, a prediction classification score module, and a detection result output module;

[0034] The video feature extraction module is used to extract the appearance features and motion features of the video data to be detected using the I3D model, and obtain high-level semantic features of the video appearance and motion respectively;

[0035] The typical action generation module is used to generate typical actions based on the K-Means algorithm for a type of action to be detected;

[0036] The action modeling module is used to construct a directed graph based on the information transmission mechanism for the generated typical actions, establish a graph transformation network module, and use the graph transformation network module to perform action modeling;

[0037] The weakly supervised video action detection module is used to introduce the graph transformation network module into the weakly supervised video action detection method CO2-Net, using two branches to process appearance features and motion features in sequence;

[0038] The prediction classification score module is used to obtain the prediction classification scores of appearance features and motion features by comparing the feature similarities of the video and the typical nodes;

[0039] The detection result output module is used to perform video action detection based on the obtained predicted classification score and output the detection result.

[0040] A computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the weakly supervised video action detection method based on a graph transformation network as described in any one of claims 1 to 7 are implemented.

[0041] A computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the weakly supervised video action detection method based on a graph transformation network are implemented as described in any one of claims 1 to 7.

[0042] The beneficial effects of the present invention are:

[0043] While reducing computational overhead, the present invention can achieve a 2.1% performance gain on the Thumos dataset. This allows for highly accurate classification and localization of video actions with limited computational effort. This is due to the following:

[0044] First, the graph transformation network proposed in the present invention has both representational and discriminative power, and can effectively serve the video interpretation tasks involving action structure. As a representational module, first of all, each selected typical action clip corresponds to a series of video frames with clear appearance features and motion patterns. The clear features corresponding to such typical action clips can represent many similar features. Secondly, the graph transformation network divides a type of action into three stages: the beginning of the action, the evolution of the action, and the end of the action, which can explicitly characterize the structure of the same type of action. Moreover, each stage of the graph transformation network contains multiple typical action clips, which can appropriately reflect the appearance and motion diversity of the actions in the same stage;

[0045] Furthermore, as a discriminative module, the graph transformer network participates in the training of all video frames, rather than the features of the majority of training videos. This allows it to be fully trained through interaction with all samples, making it a discriminative module. Furthermore, given video features, traditional methods typically learn a classifier with additional parameters to predict the classification score. However, the graph transformer network predicts the classification score by comparing the similarity between video features and various typical actions. This approach, without introducing additional parameters, further enhances the feature discriminative power of the graph transformer network.

[0046] Second, as a module with both representational and discriminative capabilities, the Graph Transformation Network can be effectively combined with the CO2-Net method to effectively model action structure and cross-video dependencies, serving the task of weakly supervised video action detection. Compared to existing weakly supervised video action detection methods, this method can achieve higher localization accuracy with limited computational overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Flowchart of the method of the present invention.

[0048] Figure 2 A visualization of some of the training data.

[0049] Figure 3 Graph showing the experimental results of the method of the present invention. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0051] The core idea of this invention is as follows: for an action category, first, every feature of all action instances of this category is collected and clustered, and representative features are selected as typical action segments. Second, the typical action segments are divided into three stages and embedded in a directed graph in chronological order, constructing a graph transformation network globally from the category level. Third, four information interaction steps are designed in the graph transformation network: intra-stage interaction, adjacent stage transmission, typical information aggregation, and video frame broadcasting. These four steps sequentially process temporal information, adjust the features of each typical action segment in the graph transformation network, and provide appropriate temporal information guidance for each processed video frame. Finally, the graph transformation network module is combined with existing weakly supervised video action detection methods to achieve performance gains.

[0052] Based on the above technical ideas, the present invention proposes a weakly supervised video action detection method based on graph transformation network, which includes the following steps:

[0053] Step 1: Extract video features

[0054] For a given video, we use the I3D model proposed by Joao Carreira in his 2017 work “action recognition? a new model and the kinetics dataset” to extract a series of appearance features and motion features from the video, and obtain high-level semantic representations of the video appearance and action F = [f1, f2, ..., f T ],in represents the t-th video feature, f t Reflects the information of a continuous video frame.

[0055] Step 2: Generate typical actions based on clustering algorithm

[0056] Step 21: Consider a class of actions. First, collect high-level semantic representations of all action instances. Then, use the K-Means algorithm proposed by MacQueen J in his 1967 work "Classification and analysis of multivariate observations" to cluster the features, resulting in M clusters. For each cluster, consider the feature closest to the cluster center as the representative feature, resulting in M representative features.

[0057] Step 22: Divide each action instance evenly into three stages: action start, action evolution, and action end; determine the stage to which the typical feature belongs based on the position of the typical feature in its corresponding action instance; for different action categories, the three stages usually contain different numbers of typical actions.

[0058] Step 3: Action modeling based on structural information transfer

[0059] Given the typical actions corresponding to the three stages, we construct a directed graph G using four information transmission mechanisms: internal interaction within the stage, adjacent stage transmission, typical information aggregation, and video frame broadcasting. c (V c , E c ) to explicitly characterize the structure between typical actions.

[0060] Step 31: Connect multiple typical actions belonging to the same stage and conduct two-way information exchange between every two typical actions;

[0061] For a specific action phase (i.e., one of the action onset, action evolution, and action ending phases), multiple representative actions can reflect the diversity of the actions in this phase in terms of appearance characteristics and motion patterns. Therefore, intra-phase interaction facilitates representative actions to capture diversity within the same time sequence.

[0062] Step 32: This includes two types of information transmission: from the "action start" to the "action evolution" and from the "action evolution" to the "action end". Each typical feature of the "action start" stage is connected with all the typical features of the "action evolution" stage, and each typical feature of the "action evolution" stage is connected with all the typical features of the "action end" stage. Finally, a directed graph is established between the typical features of different stages.

[0063] It can be seen that the adjacent stage transmission can help each typical feature explicitly perceive its predecessor typical and successor typical, thereby explicitly representing the structural information of the action.

[0064] Step 33: After sufficient information exchange within and between stages, the graph transformation network can accurately convey structural information for the processed video frames; at this time, typical information is aggregated into the video frames to provide temporal information support for the action interpretation task; the video frame features are compared with all typical features one by one, and the attention mechanism is used to adaptively aggregate features from each action stage; after typical information aggregation, the video frame features can fully perceive the structural information of the current action.

[0065] Step 34: Use the video frame broadcast operation to pass the characteristics of the current video frame to the graph transformation network module. Each typical action comprehensively considers its own characteristics and the characteristics of the video frame being processed, and updates its own characteristics according to the attention mechanism;

[0066] The video frame broadcasting operation enables typical actions to be adaptively adjusted according to the characteristics of the video frame, thereby generating more representative typical features in the next layer of information propagation.

[0067] Step 4: Graph Transformation Network Module Combined with CO2-Net Method

[0068] In a weakly supervised setting, since this algorithm can only learn from video-level classification labels, the mainstream weakly supervised video action detection algorithm first predicts a classification score for each frame, and then regards multiple consecutive frames with scores above a threshold as an action instance. Therefore, accurate frame-by-frame classification scores are the core factor affecting the effectiveness of weakly supervised video action detection algorithms. In their 2021 work "Cross-modal consensus network for weakly supervised temporal action localization", Fa-Ting Hong et al. proposed a weakly supervised video action detection method CO2-Net, which can achieve accurate video action detection by exploring the consistency of inter-modal representations.

[0069] Therefore, this paper introduces the graph transformation network module into CO2-Net to further improve the algorithm and provide structured guidance for the prediction classification score step, such as Figure 1 As shown, the present invention uses two branches to process appearance features and motion features in sequence, and predicts the classification score by comparing the feature similarity between the video frame and the typical node as described in step 5. Referring to CO2-Net, this method also uses a mutual learning mechanism to allow the prediction results (appearance prediction classification score and motion prediction classification score) of the two modalities (appearance features and motion features) to guide each other.

[0070] Step 5: Classification estimation based on similarity comparison

[0071] In the graph transformation network module, the node features of the last layer of graph transformation network are represented as in, Represents the characteristics of M typical nodes in the cth category, represents the video frame features represented by class c, Represents the video frame features characterized by the background class;

[0072] Collect video frame features from the graph transformation network module of all action categories to obtain the video frame features represented by the c-th action

[0073] In order to obtain the predicted classification score, all features are L2 regularized, and then the dot product similarity between the video frame features and the typical features is calculated, and the similarity scores of the cth class are averaged to obtain the classification score s c The classification scores of different action categories are summed up to finally get the classification scores s, s of the video frame. C+1 Indicates the classification score of the feature with respect to the background class. If the background score is the highest, it means that the current video does not contain action.

[0074] Step 6: Training and Testing

[0075] During the training phase, set the loss function:

[0076] L=L mil +L cas +L ml +λ1L oppo +λ2L norm (1)

[0077] Among them, L mil and L cas They represent the top-k multiple-instance learning loss and co-activity similarity loss proposed by Sujoy Paul et al. in "W-talc: Weakly-supervised temporal activity localization and classification" in 2018, respectively. ml represents the mutual learning loss, L oppo Indicates the similarity between the attention weight distribution and the background class probability distribution, L norm represents the normalized loss, and λ1 and λ2 are hyperparameters for balancing.

[0078] Example

[0079] In order to further verify the effectiveness of the method proposed in the present invention, the method was implemented using PyTorch 1.5 software.

[0080] 1. Experimental environment

[0081] The hardware environment for implementing this invention is: Intel Xeon E5-2600 v3 @ 2.6GHz 8-core CPU processor, 128GB memory, equipped with GeForce GTX TITAN 2080Ti GPU. The software environment is: Linux 16.04 64-bit operating system.

[0082] 2. Implementation process

[0083] (1) Constructing a dataset

[0084] In this example, three datasets, THUMOS14, ActivityNet-v1.2, and ActivityNet-v1.3, are used for experiments. The THUMOS14 dataset is from: http: / / crcv.ucf.edu / THUMOS14 / , and the ActivityNet-v1.2 and ActivityNet-v1.3 datasets are from: http: / / activity-net.org / . The THUMOS14 training dataset contains 20 action categories, while the ActivityNet-v1.2 and ActivityNet-v1.3 training datasets contain 100 and 200 action categories, respectively. Each training video contains multiple action instances, such as Figure 2 All videos are processed individually using the method of the present invention.

[0085] (2) Extracting video features

[0086] Using the I3D model proposed by Joao Carreira et al. in 2017, “Quo vadis, action recognition? a new model and the kinetics dataset[C] / / proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:6299-6308,” we extract a series of video appearance features and motion features, and obtain high-level semantic representations of video appearance and action, respectively. T ],in, represents the t-th video feature, f t Reflects the information of a continuous video frame.

[0087] The I3D model is pre-trained on the Kinetics-400 dataset, which is available from: https: / / deepmind.com / research / open-source / kinetics.

[0088] (3) Constructing a graph transformation network module

[0089] This example uses Deep Graph Library 0.8.2, proposed by Minjie Wang et al. in "Wang M, Zheng D, Ye Z, et al. Deepgraph library: A graph-centric, highly-performant package for graph neural networks[J].arXiv preprint arXiv:1909.01315,2019," to implement the graph transformation network module, which specifically includes:

[0090] First, initialize the node features and edge features of the graph transformation network. Use a linear transformation layer to transform the typical action features Projected into the latent space, In addition, a line from node v i Points to node v j The edge feature is represented by e ij .

[0091] In the initialization phase, splicing x i and x j The features of the two nodes and apply a linear transformation layer for feature projection, In addition, the random walk strategy proposed by Pan Li et al. in "Li P, Wang Y, Wang H, et al. Distance encoding: Design provably more powerful neural networks for graph representation learning [J]. Advances in Neural Information Processing Systems, 2020, 33: 4465-4478." in 2020 is used to generate position encoding for each node, and a linear transformation layer is used to project the position encoding into the latent space.

[0092] The graph transformation network module uses a graph transformation network at each layer. For the graph transformation network at layer l, the feature of the i-th node is The corresponding position code is Connect node v i and v j The feature corresponding to the edge of Using the projection matrix To process node features and corresponding position encoding, use the projection matrix To process edge features.

[0093] To calculate the node vi and its adjacent node v j , j∈N i The correlation of , estimated correlation vector

[0094]

[0095] in represents element-wise multiplication, and [·] represents concatenation in the channel dimension.

[0096] Then, the similarity is calculated as follows:

[0097]

[0098] Update node v according to formula (4) i Features:

[0099]

[0100] Finally, standard operations in the graph transformation network are used to process node features, using residual connections, layer normalization, and feed forward networks in sequence.

[0101] To simplify the representation process, the above operations only consider the single-head self-attention mechanism, but these operations can be easily extended to the multi-head self-attention mechanism, thereby giving the graph transformation network module stronger representation capabilities.

[0102] At the same time, the present invention comprehensively considers the position code of the node itself, the position code of the adjacent nodes and the characteristics of the edges connecting the adjacent nodes, and updates the position code p of each node. i The update process can be expressed as:

[0103]

[0104] Here, Γ(·) indicates that in the information propagation process similar to the node features, only the node position code is considered in the calculation, without considering the node feature information.

[0105] (4) Combining Graph Transformation Network and CO2-Net

[0106] First, the graph transformation network module is introduced into the weakly supervised video action detection method CO2-Net proposed by Fa-Ting Hong et al. in "Hong FT, Feng JC, Xu D, et al. Cross-modal consensus network for weakly supervised temporal action localization[C] / / Proceedings of the 29th ACM International Conference on Multimedia.2021:1591-1599." in 2021, providing structured guidance for the step of predicting classification scores. The present invention uses two branches to process appearance features and motion features in sequence, and predicts classification scores by comparing the feature similarity between video frames and typical nodes. The prediction results of the two modalities guide each other according to the mutual learning mechanism.

[0107] Secondly, the frame-by-frame classification score of the video is obtained by fusing the classification scores of the appearance classification branch and the motion branch.

[0108] Again, using temporal aggregation, weakly supervised video action detection results are obtained from the frame-by-frame classification scores.

[0109] The proposed graph transformation network module does not need to modify the network structure of the existing CO2-Net method. Instead, it adds an additional graph transformation network module to the appearance and motion branches respectively, which can serve the existing method in a plug-and-play manner and significantly improve its performance in the weakly supervised video action detection task. The obtained weakly supervised video action detection results are as follows: Figure 3 shown.

[0110] (5) Training and testing

[0111] During the training phase, the graph transformation network is trained using the following loss function:

[0112] L=L mil +L cas +L ml +λ1L oppo +λ2L norm

[0113] L mil and L cas They represent top-k multiple-instance learning loss and co-activitysimilarity loss, L ml represents the mutual learning loss, L oppo Indicates the similarity between the attention weight distribution and the background class probability distribution, Lnorm represents the normalized loss, λ1 and λ2 are hyperparameters for balancing, and L ml , L oppo and L norm The specific calculation formula is:

[0114]

[0115]

[0116]

[0117] in, represents the appearance attention weight, represents the motion attention weight, represents the fusion attention weight, which is and The mean of , φ(·) is used to limit the input gradient, δ(·) represents the mean square error function, α is a hyperparameter; |·| represents the absolute value function, s C+1 represents the classification score of each segment with respect to the background class; ||·||1 represents the L1 norm.

[0118] (6) Parameter settings

[0119] This paper sets the relevant parameters of the graph transformation network layer with reference to Devin Kreuzer et al. "Kreuzer D, Beaini D, Hamilton W, et al. Rethinking graph transformers with spectral attention [J]. Advances in Neural Information Processing Systems, 2021, 34: 21618-21629." and Vijay Prakash Dwivedi et al. "Dwivedi VP, Luu AT, Laurent T, et al. Graph neural networks with learnable structural and positional representations [J]. arXiv preprint arXiv: 2110.07875, 2021." in 2022, and adjusts the hyperparameters according to the experimental performance of the validation set. Finally, the number of graph transformation network layers is set to 2, the typical number M is set to 10, the number of typical action layers is set to 2, the loss coefficient λ is set to 1.0, and the fusion coefficient β is set to 0.5.

[0120] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A weakly supervised video action detection method based on graph transformation network, characterized in that The following steps are involved: Step 1: Input the video data to be detected, and use the I3D model to extract the appearance features and motion features of the video data to obtain the video appearance semantic features and motion semantic features respectively; Step 2: Generate typical actions based on the K-Means algorithm for a type of action to be detected; Step 3: For the generated typical actions, a directed graph is constructed based on the information transmission mechanism, a graph transformation network module is established, and the graph transformation network module is used to model the actions; Step 4: Introduce the graph transformation network module into the weakly supervised video action detection method CO2-Net, using two branches to process the video appearance semantic features and action semantic features in turn; Step 5: By comparing the feature similarity between the video to be detected and the typical action, the predicted classification scores of the video appearance semantic features and the action semantic features are obtained respectively; Step 6: Based on the obtained predicted classification scores, perform video action detection and output the detection results; Wherein, step 3 includes the following specific steps: Step 31: Connect multiple typical actions belonging to the same stage, and perform two-way information exchange between each two typical actions through the internal interactive information transmission mechanism of the stage; Step 32: Using the adjacent stage information transfer mechanism, each typical feature of the action start stage is connected with all the typical features of the action evolution stage, and each typical feature of the action evolution stage is connected with all the typical features of the action end stage, thereby constructing a directed graph between the typical features of different stages and obtaining a graph transformation network module; Step 33: Compare the video frame features with all typical features one by one, and use the attention mechanism to adaptively aggregate features from various stages of action instance division, thereby aggregating typical information to the video frame; Step 34: Use the video frame broadcast operation to pass the characteristics of the current video frame to the graph transformation network module, so that the typical actions can be adaptively adjusted according to the characteristics of the video frame.

2. The weakly supervised video action detection method based on graph transformation network according to claim 1 is characterized in that: Step 2 includes the following specific steps: Step 21: For a type of action, obtain high-level semantic features of all action instances; Step 22: Cluster the obtained features based on the K-Means algorithm to obtain M clusters. The feature closest to the cluster center in each cluster is used as the typical feature, and M typical features are obtained; Step 23: Divide each action instance evenly into three stages, and determine the stage to which the typical feature belongs based on the position of the typical feature in its corresponding action instance.

3. A weakly supervised video action detection method based on graph transformation network according to claim 2, characterized in that The action instance is divided into three stages: action start, action evolution and action end.

4. A weakly supervised video action detection method based on graph transformation network according to claim 3, characterized in that The information transmission mechanisms in step 3 include four types: intra-stage interaction, adjacent-stage transmission, typical information aggregation, and video frame broadcasting.

5. The weakly supervised video action detection method based on graph transformation network according to claim 4 is characterized in that Step 5 includes the following specific steps: Step 51: Collect video frame features from the graph transformation network module of all action categories to obtain the first Video frame feature representation of action-like characters ; Step 52: All features obtained in step 51 are Regularization; Step 53: Calculate the dot product similarity between the video features and the typical features. The class similarity scores are averaged to obtain the classification score , and then classify the scores of various action categories Add up and finally get the classification score of the video feature .

6. A weakly supervised video action detection method based on graph transformation network according to claim 5, characterized in that During the training phase of the graph transformation network module, training and testing are performed according to the loss function. The loss function is defined as: in, and They represent top-k multiple-instance learning loss and co-activitysimilarity loss respectively. represents the mutual learning loss, Indicates the similarity between the attention weight distribution and the background class probability distribution, represents the normalized loss, and is a hyperparameter used for balancing.

7. A weakly supervised video action detection system based on graph transformation network, characterized in that The method according to any one of claims 1 to 6 is implemented, wherein the system comprises a video feature extraction module, a typical action generation module, an action modeling module, a weakly supervised video action detection module, a prediction classification score module, and a detection result output module; The video feature extraction module is used to extract the appearance features and motion features of the video data to be detected using the I3D model, and obtain high-level semantic features of the video appearance and motion respectively; The typical action generation module is used to generate typical actions based on the K-Means algorithm for a type of action to be detected; The action modeling module is used to construct a directed graph based on the information transmission mechanism for the generated typical actions, establish a graph transformation network module, and use the graph transformation network module to perform action modeling; The weakly supervised video action detection module is used to introduce the graph transformation network module into the weakly supervised video action detection method CO2-Net, using two branches to process appearance features and motion features in sequence; The prediction classification score module is used to obtain the prediction classification scores of appearance features and motion features by comparing the feature similarities of the video and the typical nodes; The detection result output module is used to perform video action detection based on the obtained predicted classification score and output the detection result.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the weakly supervised video action detection method based on a graph transformation network as described in any one of claims 1 to 6 are implemented.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps in the weakly supervised video action detection method based on a graph transformation network according to any one of claims 1 to 6 are implemented.

Images