Video motion detection method, system, device and storage medium

By using short-term and long-term interaction modules combined with attention mechanism in video action detection, the spatial and temporal interaction between action performers is modeled, and the category relationship module is used to deal with multi-label problems, the action classification in the existing technology is solved The problem of insufficient identification accuracy has achieved higher identification effect and accuracy.

CN114495282BActive Publication Date: 2024-11-05UNIV OF SCI & TECH OF CHINA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210135464.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-14
Publication Date
2024-11-05
Estimated Expiration
2042-02-14

AI Technical Summary

Technical Problem

Existing video action detection technology is difficult to effectively handle the spatial and temporal interaction between the action performer and the environment, resulting in insufficient accuracy of action classification recognition, especially in multi-label classification scenarios.

Method used

A video action detection method is adopted to model the interaction between spatial dimensions and time dimensions through short-term interaction modules and long-term interaction modules, combine attention mechanisms and feature splicing to obtain enhanced features, and mine action classes through category relationship modules. , the dependency relationship of classification detection results are integrated.

Benefits of technology

It significantly improves the recognition effect of interactive actions and the accuracy of multi-label classification, enhances the expression ability of target features and the robustness of category representation, and reduces the calculation cost.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114495282B_ABST
    Figure CN114495282B_ABST
Patent Text Reader

Abstract

The present invention discloses a video action detection method, system, device and storage medium. On the one hand, the present invention models the interactive relationship between action performers in a targeted manner, fully utilizes the spatiotemporal characteristics of video signals, can greatly enhance the expression ability of target features, and thus greatly improves the recognition effect of interactive actions; on the other hand, the present invention utilizes a self-attention mechanism to mine the dependency relationship between different action classes, so that the robustness and discrimination of the original category representation are further improved, and the multi-label problem is solved at a lower computational cost.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of video analysis technology, and in particular to a video action detection method, system, device and storage medium. Background Art

[0002] With the popularization of electronic shooting equipment, massive amounts of video data are generated every day, most of which contain human-centered actions. These videos have been widely used in many fields such as intelligent robots, security monitoring, and autonomous driving. How to understand and analyze the content in massive videos has become a hot topic. Video action detection is one of the key technologies. It means that given a video, the spatial positions of all people in the video need to be detected and their corresponding action categories need to be determined. It has great application value and research significance in real-world scenarios.

[0003] At present, most of the research work on video action detection tasks is carried out in two steps: first, perform frame-level target detection on the key frames of the video clip to obtain the position coordinates and action score of the action performer; then, use a three-dimensional convolutional neural network to extract the global spatiotemporal representation of the video clip, so as to obtain the regional features corresponding to the detection box for subsequent action classification and recognition. In the above process, target detection for images is already very mature and can achieve relatively ideal performance. Therefore, the main difficulty of the video action detection task lies in the action classification and recognition part. In practical applications, since many action categories are interactive in nature, that is, the subjects involved in the action are not only the action performers, but also the surrounding environment context, it is inaccurate to infer the action only based on the regional features of the action performers themselves. In order to solve this technical problem, the existing technology proposes the following solutions:

[0004] 1) ACRN (Actor-Centric Relation Network) automatically mines spatiotemporal elements related to the performers of actions in the scene through the network and generates relational features for action classification, but does not explicitly model the interactions between entities.

[0005] 2) LFB (Long-Term Feature Banks for Detailed Video Understanding) provides long-term temporal support for the model and models global temporal dependencies by calculating long-range interactions between entities, but it does not emphasize the use of local temporal correlations of actions.

[0006] 3) LSTC (paper "LSTC: Boosting Atomic Action Detection with Long-Short-Term Context") uses the temporal dependency between video signals for action recognition, decomposing it into short-term dependency and long-term dependency independent inference.

[0007] The basic research ideas of the above three types of solutions are to model the potential interaction relationships around the action performers, but they do not deal with spatial and temporal interactions in a targeted manner, and ignore the heterogeneity of spatial and temporal dimensions.

[0008] In addition, in real-world scenarios, an action performer may have multiple action labels at the same time. For multi-label classification problems, existing models usually use binary cross entropy combination as the loss function, which is simple to operate and has good results, and has therefore been widely used. However, this implementation means that the multi-label classification problem is directly regarded as a binary classification problem on multiple classes. In the final prediction, each class is independently predicted, without considering the intrinsic correlation between action classes. Summary of the invention

[0009] The purpose of the present invention is to provide a video action detection method, system, device and storage medium, which not only takes into account the heterogeneity of spatial and temporal dimensions, but also takes into account local and global information in time sequence, thereby improving the recognition effect of interactive actions; at the same time, it mines the dependency relationship between different action classes and improves the accuracy of multi-label classification.

[0010] The objective of the present invention is achieved through the following technical solutions:

[0011] A video motion detection method, comprising:

[0012] Obtaining a video clip and determining key frames in the video clip;

[0013] The video clip is input into the feature extraction network part, and the regional features corresponding to all the detection frames of the key frame are obtained through target detection and feature extraction;

[0014] The regional features corresponding to all detection boxes of the key frame are respectively input into the short-term interaction module and the long-term interaction module, and the interactions in the spatial dimension and the temporal dimension are respectively modeled by the short-term interaction module to obtain enhanced features; a feature library containing historical and future moments centered on the key frame is constructed by the long-term interaction module, and the enhanced features are interacted with by the attention mechanism, and then the final features are obtained by splicing with the enhanced features;

[0015] The final feature is subjected to classification detection to obtain a first classification detection result; and the final feature is subjected to classification detection again by calculating the dependency relationship between categories to obtain a second classification detection result; the first classification detection result is merged with the second classification detection result to obtain a final classification detection result.

[0016] A video motion detection system, the system comprising:

[0017] A video data acquisition module, used to acquire video clips and determine key frames in the video clips;

[0018] The feature extraction network part, whose input is the video clip, is used to obtain the regional features corresponding to all detection boxes of the key frame through object detection and feature extraction;

[0019] The short-term interaction module takes as input the regional features corresponding to all detection boxes of the keyframes, and is used to model the interactions in the spatial and temporal dimensions to obtain enhanced features.

[0020] A long-term interaction module, whose input is the regional features corresponding to all detection boxes of the key frame and the enhanced features, is used to construct a feature library containing historical and future moments centered on the key frame, and interacts with the enhanced features through an attention mechanism;

[0021] The feature fusion module is used to concatenate the outputs of the short-term interaction module and the long-term interaction module to obtain the final features;

[0022] The classification detection module is used to perform classification detection on the final feature to obtain a first classification detection result; and to perform classification detection on the final feature again by calculating the dependency relationship between categories to obtain a second classification detection result; and to merge the first classification detection result with the second classification detection result to obtain a final classification detection result.

[0023] A processing device, comprising: one or more processors; a memory for storing one or more programs;

[0024] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0025] A readable storage medium stores a computer program, which implements the above method when the computer program is executed by a processor.

[0026] It can be seen from the technical solution provided by the present invention that the present invention specifically models the interactive relationship between action executors, fully utilizes the spatiotemporal characteristics of video signals, can greatly enhance the expression ability of target features, and thus greatly improves the recognition effect of interactive actions; secondly, the present invention uses the self-attention mechanism to mine the dependency relationship between different action classes, so that the robustness and discrimination of the original category representation are further improved, and the multi-label problem is solved at a lower computational cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0028] Figure 1 A schematic diagram of a framework of a video action detection method provided by an embodiment of the present invention;

[0029] Figure 2 A schematic diagram of a time-space decoupling mechanism provided by an embodiment of the present invention;

[0030] Figure 3 A schematic diagram of the GAT network header structure provided by an embodiment of the present invention;

[0031] Figure 4 A schematic diagram of the Non-local module structure provided in an embodiment of the present invention;

[0032] Figure 5 A schematic diagram of the CRM module structure provided by an embodiment of the present invention;

[0033] Figure 6 A schematic diagram of a video motion detection system provided by an embodiment of the present invention;

[0034] Figure 7 A physical diagram of a processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0036] First, the terms that may be used in this article are explained as follows:

[0037] The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0038] The video motion detection scheme provided by the present invention is described in detail below. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professionals in the field. If no specific conditions are specified in the embodiments of the present invention, the conventional conditions in the field or the conditions recommended by the manufacturer shall be followed.

[0039] Embodiment 1

[0040] In order to address the deficiencies of the prior art, an embodiment of the present invention provides a video action detection method, which is a video action detection method that integrates interaction relationships and category associations. This method specifically models the spatial interaction, short-term time interaction, and long-term time interaction between action performers to enhance the expressiveness of target features and improve the recognition effect of interactive actions. This method not only considers the heterogeneity of spatial and temporal dimensions, but also takes into account local and global information in time series. For multi-label problems, a category relationship module is designed to mine the dependency relationship between different action classes, and this relationship is used to fuse the original category representation, so that the learned representation has stronger robustness and discrimination, further improving the accuracy of multi-label classification.

[0041] like Figure 1 FIG. 1 is a schematic diagram of a framework of a video action detection method provided by an embodiment of the present invention. The method mainly includes the following steps:

[0042] Step 1: Get a video clip and determine the key frames in the video clip.

[0043] In the embodiment of the present invention, the key frames in the video clip can be determined by conventional methods, which will not be elaborated in the present invention.

[0044] Step 2: Input the video clip into the feature extraction network part, and obtain the regional features corresponding to all detection boxes of the key frame through target detection and feature extraction.

[0045] This step can be accomplished by conventional methods, such as Figure 1As shown in the upper part, the key frame obtains several detection boxes through the object detector, each detection box corresponds to an action performer, the video clip extracts global features through 3D CNN, and the regional features corresponding to all detection boxes (i.e., detection box features) are obtained through ROIAlign.

[0046] In the embodiment of the present invention, the feature matrix of the video clip at the current moment is recorded as It is composed of the regional features of all detection boxes on the corresponding key frame. Among them, t represents the current moment, that is, the position of the video clip in the time dimension, and N t represents the number of detection frames on the key frame in the current video clip, d represents the dimension of the regional feature corresponding to a single detection frame, Represents the set of real numbers.

[0047] For each video clip, a series of frames can be obtained according to the frame rate of the video itself. Assuming the frame rate is 30fps (frame per second), a video clip with a duration of 1 second can obtain 30 consecutive video frames. Generally, the middle frame is taken as the key frame. The video clip at the current time t is recorded as clip t, and the corresponding middle frame is the key frame t. Figure 1 In the video, segment t-1 refers to the video segment at the previous moment (i.e., moment t-1), and segment t+1 refers to the video segment at the next moment (i.e., moment t+1). The feature library corresponding to the video clip representing the current moment, including information of historical and future moments, is composed of feature matrices of other video clips, generally containing information from the previous w moments to the next w moments, that is, the feature matrix set of video clips tw~t+w.

[0048] Step 3: Input the regional features corresponding to all detection boxes of the key frame into the short-term interaction module and the long-term interaction module respectively, and model the interactions in the spatial dimension and the temporal dimension respectively through the short-term interaction module to obtain enhanced features; construct a feature library containing historical and future moments centered on the key frame through the long-term interaction module, and interact with the enhanced features through the attention mechanism, and then splice with the enhanced features to obtain the final features.

[0049] This step includes three parts: short-term interaction module, long-term interaction module, and feature splicing part.

[0050] 1) Short-Term Interaction Module (STIM)

[0051] The short-term interaction module models the short-term spatiotemporal interactions between action executors. Based on the heterogeneity of spatial and temporal dimensions, the short-term interaction module uses a decoupling mechanism to specifically handle spatial interactions and short-term temporal interactions.

[0052] Specifically, the short-term interaction module aims to model the interaction between two action performers in a short period of time. Considering that the actions performed by different action performers at the same time are interactive, and the actions performed by the same action performer in a short period of time are continuous, and spatial interaction and temporal interaction are essentially different, the interaction in the spatial dimension and the temporal dimension are modeled separately. Figure 2 As shown in the figure, the short-term interaction module adopts a decoupling mechanism to specifically process spatial and temporal interactions through a two-layer graph attention network (GAT), while also reducing the learning difficulty of the network. Figure 2 middle, It refers to the kth action performer node at the current time t. Each action performer node is the detection box of the corresponding action performer. The superscript is the time and the subscript is the sequence number of the action performer.

[0053] The head structure of the graph attention network is as follows Figure 3 As shown in the figure, M represents the total number of detection frames on the key frames of all video clips in a batch. If the batch size is equal to 8, then it contains the key frame detection frames of 8 consecutive video clips. The adjacency matrix Adj is a binary matrix. Adj ij Represents the adjacency relationship between node i and node j. If they are adjacent, it is 1, and if they are not adjacent, it is 0. Usually, the adjacency matrix is ​​predefined, and only adjacent nodes need to be considered in subsequent feature updates. The correlation matrix E is obtained by calculating the mutual correlation coefficients between nodes. E ij represents the importance of node j to node i; the weight matrix W is obtained by multiplying the adjacency matrix Adj and the correlation matrix E, and then normalized by the Softmax function. ij Represents the weight coefficient between node i and node j.

[0054] Based on the above principle, the regional features L corresponding to all detection frames of the key frame are t Model the interactions in spatial and temporal dimensions sequentially to obtain enhanced features When considering the interaction in the spatial dimension, it is necessary to connect the nodes of different action performers in the same frame in pairs. In an embodiment of the present invention, the action performers are represented by nodes of a graph network, and the interaction relationship between the action performers is represented by the edges of the graph network. Since this part involves the description of the graph network, the relevant definition of the graph network is used. Specifically, the action performer node is the detection box of the corresponding action performer, and the feature vector of the node is the regional feature of the corresponding detection box. Similarly, when considering the interaction in the time dimension, the nodes of the same action performer at different time steps are connected in pairs, and the calculation process involved includes:

[0055] The feature vector of node i is denoted as h i , the set of adjacent nodes (determined by the edges of the nodes) is denoted as N i , then after the interaction in the spatial dimension of graph attention network modeling, the updated feature vector of node i is obtained, which is expressed as:

[0056]

[0057] W ij =Softmax(σ(w b T [h i ||h j ]))

[0058] Among them, h j represents the feature vector of the neighboring node j of node i, [h i ||h j ] represents the feature vector h i With h j Concatenate (concat) ij represents the weight coefficient between node i and its adjacent node j, w b represents a learnable vector, T represents the transposed symbol; σ represents the linear rectification function ReLU;

[0059] Then, in the same way, the graph attention network is used to model the interaction in the time dimension to obtain the enhanced feature vector of each node. The calculation principle is the same as the above two formulas, and finally the enhanced features are formed.

[0060] 2) Long-Term Interaction Module (LTIM).

[0061] In the embodiment of the present invention, the long-term interaction module is intended to provide long-term support information in time sequence, thereby helping the model (i.e. Figure 1 The framework model shown) can more accurately infer the behavioral state of the action performer at the current moment.

[0062] In the embodiment of the present invention, a sliding window is used to flexibly construct a long-term feature library containing historical and future moments, which alleviates the problem of short video input due to GPU memory limitations. Assuming the current moment is t, the feature matrix of video clip t is expressed as It is composed of the regional features corresponding to all detection boxes on key frame t; taking the current time t as the center, the feature matrix of the video clips taken before and after w moments constitutes a long-term feature library, which is expressed as Among them, L t-w , L t+w Respectively represent the feature matrices of the video clips corresponding to the twth time and the t+wth time; in Represents the number of all action executors in the feature library of length 2w+1.

[0063] The features of the target in the key frame enhanced by the short-term interaction module are recorded as S t , the target feature S t And the corresponding long-term feature library Act and extract the relevant global temporal context information.

[0064] The operator used in the present invention is a modified Non-local module, such as Figure 4 As shown, for the enhanced feature S t and feature library Perform linear transformation to reduce to d dimensions respectively, use the enhanced features after linear transformation as the query matrix in the attention mechanism, and use the feature library after different linear transformations as the key matrix and value matrix in the attention mechanism; use the inner product in the embedding space to calculate the semantic relevance of the query matrix and the key matrix, and then scale them in turn (divided by ) and normalization; the Softmax function is used for normalization, and the purpose of scaling is to reduce the order of magnitude of the dot product value to avoid the gradient vanishing problem of the subsequent Softmax function. After that, the normalized result is used as the weight coefficient to weight the value matrix, and then processed by the LayerNorm function (to make the distribution in each sample consistent), and then linear transformation is used to extract the global information on the time series.

[0065] In the embodiment of the present invention, a two-layer Non-local superposition structure may be used, namely:

[0066]

[0067]

[0068] in, and Represents two different Non-local modules, and their parameters are denoted as θ1 and θ2 respectively.

[0069] 3) Feature splicing part.

[0070] This part mainly combines the results obtained from the previous two parts to obtain the final features.

[0071] Step 4: classify and detect the final features to obtain a first classification detection result; and classify and detect the final features again by calculating the dependency between categories to obtain a second classification detection result; and merge the first classification detection result with the second classification detection result to obtain a final classification detection result.

[0072] This step is mainly for classification detection, which mainly includes two branches:

[0073] 1) The first branch adopts the conventional method and inputs the final feature L′ t , output the probability score for each action category As the first classification detection result, C here is the number of all action categories.

[0074] 2) In the second branch, a Class Relationship Module (CRM) is designed to calculate the dependencies between categories to enhance the representation.

[0075] In the embodiment of the present invention, the category relationship module is intended to allow the model to automatically learn the semantic correlation between different action classes of the action performer at the same time, and adaptively enhance the representation of each action class, thereby improving the performance of multi-label classification. Especially for action classes with a relatively small number of positive samples, it may be difficult for the network to learn effective and robust category representations. At this time, other action categories with strong correlation or high co-occurrence can be used to enhance their own representations through feature fusion to improve recognition accuracy.

[0076] The structural design of the category relationship module is as follows Figure 5 As shown, for the features corresponding to each detection box in the final features, C linear transformations are performed independently, C represents the number of action categories, and each linear transformation obtains the representation of the corresponding action category (i.e. Figure 5); for the representation of all action categories, the semantic correlation between different action categories is calculated through the self-attention mechanism, and the weight coefficients on all categories are obtained; the representation of all action categories is weighted and summed using the corresponding weight coefficients, and the obtained results are classified and detected, and the probability score of each action category is predicted as the classification detection result of each detection box, and the classification detection results of all detection boxes are combined as the second classification detection result.

[0077] like Figure 5 As shown, after passing through the Non-local module, M×C×d (M is the number of samples, C is the number of action categories, and d is the dimension of the category representation) will be obtained. The C linear layers after the Non-local module are C action classifiers. The representation of each action category passes through a separate linear layer to obtain a classification score (i.e., predicting the probability score of each action category), which represents the possibility of executing each action category. Each input of the C linear layer corresponds to the representation of the action class M×1×d, and the output is M×1×1, which is the score of the M samples in each action category, and finally M×C×1 can be obtained. Unlike other methods, the classifier parameters of the CRM module in the present invention are not shared for all categories, because the representation of each action category has been decoupled before, so each action category can be learned based on its own category representation.

[0078] For the detection results of the above two branches, the embodiment of the present invention proposes a two-stage score fusion strategy, which can fuse the classification detection results of the two branches according to the complementarity of different branches. Since the first branch has a weak ability to distinguish minority class samples, the first classification detection result is conservative, and the CRM module improves this problem in a targeted manner in the second branch.

[0079] In the embodiment of the present invention, the two-stage score fusion strategy fuses the predicted scores of each action class, and the first classification detection result and the second classification detection result can be fused by taking the minimum value, the maximum value or the average value to obtain the final classification detection result, which is expressed as:

[0080]

[0081] in, Indicates the first classification test result, It represents the second classification detection result, min, max, and avg represent the minimum function, maximum function, and average function respectively.

[0082] The above is the main principle of the embodiment of the present invention. In the training stage, the short-term interaction module, the long-term interaction module, and the classification detection part are jointly trained. The total loss function Loss includes: the loss function Loss1 of the short-term interaction module and the long-term interaction module, and the loss function Loss2 of the classification detection part, which is expressed as:

[0083] Loss = Loss1 + Loss2

[0084] in:

[0085]

[0086]

[0087] In the above formula, M represents the number of samples, and a detection box is regarded as a sample. is the label of the cth action category of the mth sample, represents the prediction score corresponding to the cth action category in the first classification result of the mth sample, Represents the prediction score corresponding to the cth action category in the second classification result of the mth sample.

[0088] Compared with the prior art, the above scheme of the embodiment of the present invention mainly achieves the following beneficial effects: 1) The interactive relationship between the action performers is modeled in a targeted manner, which makes full use of the spatiotemporal characteristics of the video signal, can greatly enhance the expression ability of the target features, and thus greatly improve the recognition effect of interactive actions. 2) The self-attention mechanism is used to mine the dependency relationship between different action classes, so that the robustness and discrimination of the original category representation are further improved, and the multi-label problem is solved at a lower computational cost.

[0089] The above solution provided by the embodiment of the present invention can be easily embedded into the existing video action detection framework to improve the recognition accuracy of the algorithm. For ease of understanding, the implementation process of the above solution is introduced below in conjunction with an example.

[0090] 1. Prepare video action detection dataset.

[0091] Each video is processed into continuous video segments of equal length, and frames are extracted according to the corresponding frame rate. Then, a number of frame images are selected as key frames in each video segment. The spatial positions of all people appearing in the key frames are marked and a certain number of action labels are assigned. Finally, the dataset is divided into a training set and a test set.

[0092] 2. Establish a network model.

[0093] In the embodiment of the present invention, a deep learning framework is used to establish a network model, and the network structure parameters are determined according to the selected data set, such as Figure 1 As shown. The network framework is mainly composed of a backbone network, a target detection network, STIM, LTIM and CRM. The backbone network and target detection network here are the feature extraction network parts mentioned above. The backbone network can be a mainstream three-dimensional convolutional neural network in the current video processing field, such as I3D, S3D, SlowFast, etc. In this example, SlowFast is used as the backbone network for feature extraction; the target detection network can choose Faster R-CNN or SSD, etc. In this example, Faster R-CNN is used as the target detector to detect people appearing in the key frame.

[0094] 3. Target detection.

[0095] The object detection network is first pre-trained on MS COCO and then fine-tuned on the selected dataset. The action performer in each key frame is detected to obtain a series of detection frame position coordinates and action scores. In addition, based on the object detection results on the key frame, the detection frame of the same person can be tracked. The greedy strategy adopted is to select the detection frame with the largest Intersection over Union (IoU) with the current frame in the next frame during matching, and the IoU must be greater than the preset threshold, which is 0.5 in this example.

[0096] 4. Train the network model.

[0097] The backbone network is first pre-trained on Kinetics-600 and then fine-tuned on the selected dataset. The backbone network and the object detection network remain fixed while training the rest of the model. The regional features corresponding to each detection box on the video key frame t are extracted and recorded as Among them, N t represents the number of detection boxes on key frame t, and d is the feature dimension. t As the input of the subsequent modules, the enhanced features after the short-term interaction module are Then S t and long-term feature library To obtain relevant long-term temporal context information and to t Perform feature concatenation to obtain L′ t , and sent to the classifier. The first branch of the classifier is a normal fully connected layer, with input features Output the probability score for each action category C is the total number of action categories. The second branch of the classifier is CRM, which can also get the probability score Then s 1 and 2 Fusion is performed as the final prediction score s *At the same time, the loss function of the model is also the sum of the long-term and short-term interaction part and the classifier part.

[0098] 5. Test the network model.

[0099] For each video clip in the test set, target detection is performed on the key frames in the same way to obtain the position coordinates and action scores of all detection frames, and then the backbone network is used to extract the regional features corresponding to each detection frame. The target features are then enhanced through the long-term and short-term interaction part, and finally the classifier part outputs the probability score of each action class and calculates the mean average precision (mAP). The mean average precision is used as an evaluation indicator during testing to evaluate the performance of the network model.

[0100] Embodiment 2

[0101] The present invention also provides a video action detection system, which is mainly implemented based on the method provided in the above embodiment 1. Figure 6 As shown, the system mainly includes:

[0102] A video data acquisition module, used to acquire video clips and determine key frames in the video clips;

[0103] The feature extraction network part, whose input is the video clip, is used to obtain the regional features corresponding to all detection boxes of the key frame through object detection and feature extraction;

[0104] The short-term interaction module takes as input the regional features corresponding to all detection boxes of the keyframes, and is used to model the interactions in the spatial and temporal dimensions to obtain enhanced features.

[0105] A long-term interaction module, whose input is the regional features corresponding to all detection boxes of the key frame and the enhanced features, is used to construct a feature library containing historical and future moments centered on the key frame, and interacts with the enhanced features through an attention mechanism;

[0106] The feature fusion module is used to concatenate the outputs of the short-term interaction module and the long-term interaction module to obtain the final features;

[0107] The classification detection module is used to perform classification detection on the final feature to obtain a first classification detection result; and to perform classification detection on the final feature again by calculating the dependency relationship between categories to obtain a second classification detection result; and to merge the first classification detection result with the second classification detection result to obtain a final classification detection result.

[0108] Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0109] In addition, the main technical details involved in each module of the above system have been introduced in detail in the previous embodiment 1, so they will not be repeated here.

[0110] Embodiment 3

[0111] The present invention also provides a processing device, such as Figure 7 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the methods provided in the aforementioned embodiments.

[0112] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0113] In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example:

[0114] The input device may be a touch screen, an image acquisition device, a physical button or a mouse, etc.;

[0115] The output device may be a display terminal;

[0116] The memory may be a random access memory (RAM) or a non-volatile memory, such as a disk memory.

[0117] Embodiment 4

[0118] The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0119] In the embodiment of the present invention, the readable storage medium as a computer-readable storage medium can be set in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk, etc., which can store program codes.

[0120] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A video action detection method, characterized in that: include: Obtaining a video clip and determining key frames in the video clip; The video clip is input into the feature extraction network part, and the regional features corresponding to all detection frames of the key frame are obtained through target detection and feature extraction; The regional features corresponding to all detection boxes of the key frame are respectively input into the short-term interaction module and the long-term interaction module, and the interactions in the spatial dimension and the temporal dimension are respectively modeled by the short-term interaction module to obtain enhanced features; A feature library including historical and future moments centered on the key frame is constructed through the long-term interaction module, and the feature library interacts with the enhanced feature through an attention mechanism, and then is concatenated with the enhanced feature to obtain the final feature; Performing classification detection on the final features to obtain a first classification detection result; and performing classification detection again on the final features by calculating the dependency relationship between categories to obtain a second classification detection result; The first classification detection result is combined with the second classification detection result to obtain a final classification detection result; The modeling of the interactions in the spatial dimension and the temporal dimension by the short-term interaction module to obtain enhanced features includes: The feature matrix of the current video clip is recorded as It is composed of the regional features of all detection boxes on the key frames in the current video clip; where t represents the current moment, that is, the position of the video clip in the time dimension; N t represents the number of detection frames on the key frame in the current video clip, d represents the dimension of the regional feature corresponding to a single detection frame, represents the set of real numbers; The decoupling mechanism is adopted and the graph attention network is used to analyze the regional features L corresponding to all detection boxes of the key frame. t Model the interactions in spatial and temporal dimensions sequentially to obtain enhanced features The step of constructing a feature library including historical and future moments centered on the key frame through the long-term interaction module, and interacting with the enhanced features through an attention mechanism includes: The feature matrix of the current video clip is recorded as It is composed of the regional features of all detection boxes on the key frames in the current video clip; where t represents the current moment, that is, the position of the video clip in the time dimension; N t represents the number of detection frames on the key frame in the current video clip, d represents the dimension of the regional feature corresponding to a single detection frame, represents the set of real numbers; Taking the current time t as the center, the feature matrix of the video clips taken at w times before and after constitutes the feature library, which is expressed as Among them, L t-w , L t+w Respectively represent the feature matrices of the video clips at time tw and time t+w; For enhanced feature S t and feature library Perform linear transformations respectively, use the enhanced features after linear transformation as the query matrix in the attention mechanism, and use the feature libraries after different linear transformations as the key matrix and value matrix in the attention mechanism; Using the inner product in the embedding space to calculate the semantic relevance between the query matrix and the key matrix, and then scaling and normalizing them in turn; The normalized result is used as a weight coefficient to weight the value matrix, and then processed by the LayerNorm function, and then the global information on the time series is extracted using linear transformation.

2. A video action detection method according to claim 1, characterized in that: The graph attention network is used to analyze the regional features L corresponding to all detection boxes of the key frame. t Modeling the interactions in the spatial and temporal dimensions in turn includes: The detection box is regarded as a node in the graph network. Different detection boxes correspond to different action performers. The interaction relationship between action performers is regarded as the edge of the graph network. The feature vector of the node is the regional feature of the corresponding detection box. The feature vector of node i is denoted as h i , the set of adjacent nodes is recorded as N i , then after the interaction in the spatial dimension of graph attention network modeling, the updated feature vector of node i is obtained, which is expressed as: W ij =Softmax(σ(w b T [h i ||h j ])) Among them, h j represents the feature vector of the neighboring node j of node i, [h i ||h j ] represents the feature vector h i With h j For splicing, W ij represents the weight coefficient between node i and its adjacent node j, w b represents a learnable vector, T represents the transposed symbol; σ represents the linear rectification function ReLU; Then, in the same way, the graph attention network is used to model the interaction in the time dimension, and the enhanced feature vector of each node is obtained to form the enhanced feature.

3. A video action detection method according to claim 1, characterized in that: The final feature is subjected to classification detection by calculating the dependency relationship between categories, and obtaining the second classification detection result includes: For the features corresponding to each detection box in the final features, C linear transformations are performed independently, where C represents the number of action categories, and each linear transformation obtains a representation of the corresponding action category; For the representation of all action categories, the semantic correlation between different action categories is calculated through the self-attention mechanism to obtain the weight coefficients of all categories; The representations of all action categories are weighted and summed using the corresponding weight coefficients, and the obtained results are classified and tested to predict the probability score of each action category.

4. A video action detection method according to claim 1, characterized in that: The fusing the first classification detection result and the second classification detection result to obtain a final classification detection result includes: The first classification detection result and the second classification detection result are fused by taking the minimum value, the maximum value or the average value to obtain the final classification detection result, which is expressed as: in, Indicates the first classification test result, It represents the second classification detection result, min, max, and avg represent the minimum function, maximum function, and average function respectively.

5. A video action detection method according to claim 1, characterized in that: In the training phase, the total loss function Loss includes: the loss function Loss1 of the short-term interaction module and the long-term interaction module, and the loss function Loss2 of the classification detection part, which is expressed as: Loss = Loss1 + Loss2 in: In the above formula, M represents the number of samples, a detection box is regarded as a sample, C represents the number of action categories, is the label of the cth action category of the mth sample, represents the prediction score corresponding to the cth action category in the first classification result of the mth sample, Represents the prediction score corresponding to the cth action category in the second classification result of the mth sample.

6. A video motion detection system, characterized in that: The method according to any one of claims 1 to 5 is implemented, and the system comprises: A video data acquisition module, used to acquire video clips and determine key frames in the video clips; The feature extraction network part, whose input is the video clip, is used to obtain the regional features corresponding to all detection boxes of the key frame through object detection and feature extraction; The short-term interaction module takes as input the regional features corresponding to all detection boxes of the keyframes, and is used to model the interactions in the spatial and temporal dimensions to obtain enhanced features. A long-term interaction module, whose input is the regional features corresponding to all detection boxes of the key frame and the enhanced features, is used to construct a feature library containing historical and future moments centered on the key frame, and interacts with the enhanced features through an attention mechanism; The feature fusion module is used to concatenate the outputs of the short-term interaction module and the long-term interaction module to obtain the final features; The classification detection module is used to perform classification detection on the final feature to obtain a first classification detection result; and to perform classification detection on the final feature again by calculating the dependency relationship between categories to obtain a second classification detection result; and to merge the first classification detection result with the second classification detection result to obtain a final classification detection result.

7. A processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.