Video behavior identification method and system based on graph convolutional network

Abstract

The invention relates to the technical field of video behavior recognition, in particular to a video behavior recognition method and system based on a graph convolutional network, and the method comprises the steps: obtaining a plurality of videos with behavior tags; decomposing the video into a plurality of video segments, wherein each video segment comprises a plurality of shots; obtaining time information, space information and entity information of each shot; constructing space-time diagram connection based on the time information, the space information and the entity information to obtain a diagram structure; inputting the graph structure into a graph convolutional network, and extracting node features, spatial features and global features by using different convolution and pooling operations; fusing the node features, the spatial features and the global features to obtain fused features; according to the method, the fusion features are input into a full-connection layer, a behavior recognition result is generated, the long-term dependency relationship in the video is effectively captured by constructing a space-time diagram structure, the continuity in time is considered, and semantic association in space is fused, so that the model can comprehensively understand the behavior pattern in the whole video sequence.

Description

Technical Field

[0001] The present invention relates to the technical field of video behavior recognition, and more specifically, to a video behavior recognition method and system based on a graph convolutional network. Background Art

[0002] With the rapid development of artificial intelligence technology, video behavior recognition, as an important research direction in the field of computer vision, has received widespread attention in recent years. Video behavior recognition has broad application prospects in security monitoring, smart home, human-computer interaction and other fields. However, due to the complexity and diversity of video data, accurately identifying human behavior in videos still faces many challenges.

[0003] Traditional video action recognition methods mainly rely on manually designed feature extraction algorithms, such as optical flow method, 3D gradient histogram, etc. Although these methods have achieved certain results in some simple scenarios, they often perform poorly in complex practical applications. In recent years, with the rise of deep learning technology, methods based on convolutional neural networks (CNN) and recurrent neural networks (RNN) have made significant progress in video action recognition tasks. These methods can automatically learn the spatiotemporal features in videos, which improves the accuracy of recognition to a certain extent.

[0004] However, existing deep learning-based methods still have some obvious limitations. First, most methods find it difficult to effectively capture long-term spatiotemporal dependencies in videos. Behaviors in videos often span long time spans, while existing methods can usually only process local spatiotemporal information and cannot fully understand the behavioral patterns in the entire video sequence. Second, existing methods often view videos as simple sequences of frames, ignoring the complex structural information in the video. In fact, there are rich semantic associations between different parts of the video, which are crucial for accurate recognition of behaviors. In addition, existing methods also have difficulties in dealing with behavioral patterns of different scales. Some behaviors may be completed in a short time, while others may last for a long time. How to handle such multi-scale behavioral patterns in a unified framework remains a challenge.

[0005] In view of the above problems, there is an urgent need for new video behavior recognition methods that can effectively capture long-term spatiotemporal dependencies in videos, make full use of video structural information, and handle multi-scale behavior patterns. Summary of the invention

[0006] The video behavior recognition method and system based on graph convolutional network proposed in this invention are innovative solutions to the problems existing in the prior art. This method effectively captures the long-term dependencies and structural information in the video by representing the video as a spatiotemporal graph structure. At the same time, the multi-scale graph convolutional network is used for feature extraction, which can adapt to behavior patterns of different scales.

[0007] The present invention provides a video behavior recognition method based on a graph convolutional network, comprising:

[0008] The acquisition steps include:

[0009] Get multiple videos with behavior labels;

[0010] Decomposing the video into a plurality of video segments, each of the video segments comprising a plurality of shots;

[0011] Acquire time information, space information and entity information of each of the shots;

[0012] Processing steps include:

[0013] Based on the time information, the space information and the entity information, a spatiotemporal graph connection is constructed to obtain a graph structure;

[0014] Input the graph structure into a graph convolutional network, and use different convolution and pooling operations to extract node features, spatial features, and global features;

[0015] Fusing the node feature, the spatial feature and the global feature to obtain a fused feature;

[0016] Output steps include:

[0017] The fused features are input into the fully connected layer to generate behavior recognition results.

[0018] Preferably, the obtaining step specifically includes:

[0019] Obtaining an original video and a corresponding label, wherein the label includes a behavior name and a corresponding category;

[0020] Splitting the original video into a plurality of video segments according to a preset time interval;

[0021] Each of the video segments is further divided into a plurality of the shots based on scene changes.

[0022] Preferably, the step of constructing the space-time graph connection specifically includes:

[0023] Establishing connections between entities in the shot based on spatial relationships, wherein the spatial relationships include whether they overlap, whether they interact, whether they move in the same direction, whether they are in relative motion, or multiple types thereof;

[0024] Establishing connections between entities in the shot based on a temporal relationship, wherein the temporal relationship includes whether the interaction is continuous, whether the interaction occurs simultaneously, or a combination thereof;

[0025] Establishing a connection between entities of the shots based on the spatial relationship and the temporal relationship.

[0026] Preferably, the step of constructing the graph structure specifically includes:

[0027] The shots are represented as nodes, and each node stores the spatial information and entity information of the corresponding shot;

[0028] Adjacent nodes between different shots are connected by edges, and weights of the edges are set according to the spatial information and the temporal relationship.

[0029] Preferably, the step of extracting features using the graph convolutional network specifically includes:

[0030] Set the node convolution filter to aggregate the information of adjacent nodes and obtain the node features;

[0031] Set a spatial pooling filter to selectively aggregate nodes based on their spatial structure and extract spatial features;

[0032] Perform global attention operations on all nodes in the graph to extract global features.

[0033] Preferably, after the acquisition step, a preprocessing step is further included:

[0034] Calculate the similarity matrix between different shots to measure the similarity between two shots;

[0035] Merging the similarity matrix in the time dimension and the space dimension to obtain a comprehensive similarity matrix;

[0036] Selecting adjacent shots to form a cluster based on the comprehensive similarity matrix;

[0037] Temporal features and spatial features are extracted from each cluster, each shot is classified, and the predicted event of each shot is obtained.

[0038] Preferably, the step of calculating the similarity matrix specifically includes:

[0039] For shot m and shot n, compare the corresponding visual features therein, count the differences between the visual features, and obtain a difference value;

[0040] When the difference value is less than a preset difference threshold, the corresponding position in the similarity matrix Smn is set to 1;

[0041] When the difference value is greater than or equal to a preset difference threshold, the corresponding position in the similarity matrix Smn is set to 0.

[0042] Preferably, the step of calculating the comprehensive similarity matrix specifically includes:

[0043] Get the time characteristics of each shot and get the time vector Tm ;

[0044] Get the spatial features of each shot and get the spatial vector S m ;

[0045] Transpose the time vector and the space vector to obtain a transposed vector and S A ;

[0046] Calculate the time similarity matrix S AAT and the spatial similarity matrix S AS ;

[0047] Calculate the mean of the time similarity matrix and the space similarity matrix to obtain the time-space similarity matrix mean matrix S T ;

[0048] The time-space similarity matrix mean matrix is ​​normalized to obtain the comprehensive similarity matrix S.

[0049] Preferably, the convolution process of the graph convolutional network includes:

[0050] represents the embedding vector of the kth action node in the tth frame, and its update formula is:

[0051]

[0052] in, is the l+1th layer feature of the kth action node, is the weight calculated by node attention, W l is the weight matrix of the lth layer, represents the embedding vector of the i-th node in the l-th layer, N(k) represents the set of nodes connected to the k-th action node, and σ(·) represents the activation function.

[0053] The video behavior recognition system based on graph convolutional network includes:

[0054] Video segment splitting module, used for:

[0055] Get multiple videos with behavior labels;

[0056] Decomposing the video into a plurality of video segments, each of the video segments comprising a plurality of shots;

[0057] Acquire time information, space information and entity information of each of the shots;

[0058] Graph building blocks for:

[0059] Based on the time information, the space information and the entity information, a spatiotemporal graph connection is constructed to obtain a graph structure;

[0060] Feature extraction module for:

[0061] Input the graph structure into a graph convolutional network, and use different convolution and pooling operations to extract node features, empty features, and global features;

[0062] Feature fusion module, used for:

[0063] Fusing the node feature, the spatial feature and the global feature to obtain a fused feature;

[0064] Identify output modules for:

[0065] The fused features are input into the fully connected layer to generate behavior recognition results.

[0066] The beneficial effects of the present invention are mainly reflected in the following aspects:

[0067] First, the method of the present invention effectively captures the long-term dependencies in the video by constructing a spatiotemporal graph structure. This graph structure not only considers the temporal continuity, but also integrates the spatial semantic association, so that the model can fully understand the behavior pattern in the entire video sequence. This global perspective feature extraction greatly improves the ability to recognize complex behaviors.

[0068] Secondly, the multi-scale graph convolutional network architecture introduced in this paper can process local details and global semantic information at the same time. By extracting and fusing features at different scales, this method can adapt to various behavioral patterns of different time spans, thereby improving the flexibility and accuracy of recognition.

[0069] In addition, the method of the present invention provides valuable prior information for subsequent fine recognition by designing innovative preprocessing steps, such as shot clustering and prejudgment event generation. This hierarchical processing strategy not only improves the accuracy of recognition, but also reduces the computational complexity to a certain extent.

[0070] Finally, the end-to-end system design proposed in the present invention has strong flexibility and scalability. Each module can be independently optimized according to specific application requirements while maintaining the consistency of the overall system. This design facilitates the deployment and optimization of the method in practical applications.

[0071] In general, the video behavior recognition method and system based on graph convolutional network proposed in this invention effectively solve the problems existing in the prior art through innovative graph structure representation, multi-scale feature extraction and end-to-end system design. This method can not only accurately identify behaviors in complex video scenes, but also has strong adaptability and scalability, opening up a new direction for the development of video behavior recognition technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 The present invention is a flow chart of the method.

[0073] Figure 2 It is a logic block diagram of the preprocessing step of the present invention.

[0074] Figure 3 This is a logical block diagram of the graph convolution process of the present invention. DETAILED DESCRIPTION

[0075] Please refer to Figure 1-3 The present invention provides a method and system for video behavior recognition based on graph convolutional networks. This method innovatively integrates temporal information, spatial information, and entity information in the video into a unified graph structure, and uses a multi-scale graph convolutional network for feature extraction and behavior recognition, effectively solving the problem of behavior recognition in complex video scenes.

[0076] First, the method of the present invention comprises the following key steps:

[0077] The method of the present invention first obtains multiple videos with behavior tags. These videos can come from various sources, such as surveillance cameras, videos shot by mobile phones, etc. Then, these videos are decomposed into multiple video segments, each of which contains multiple shots. Preferably, the division of video segments can be based on time intervals or scene changes. For example, a video segment can be divided every 30 seconds, or a new video segment can be divided when the scene changes significantly.

[0078] For each shot, the method obtains its time information, spatial information and entity information. Time information may include the start time and duration of the shot; spatial information may include the position, size and movement trajectory of each entity in the shot; entity information may include feature descriptions of people and objects appearing in the shot.

[0079] After obtaining the necessary information, the method of the present invention enters the core processing stage. First, based on the previously obtained time information, spatial information and entity information, a spatiotemporal graph connection is constructed to obtain a graph structure. This graph structure is one of the key innovations of the present method, which effectively represents the complex spatiotemporal relationship in the video as a unified mathematical model.

[0080] In one embodiment, the process of constructing the graph structure can be performed as follows: each shot is represented as a node in the graph, and the edges between the nodes represent the spatiotemporal relationship between the shots. For example, if the same person appears in two shots, or if the two shots are continuous in time, an edge can be established between the corresponding nodes. The weight of the edge can be set according to the strength of the relationship, for example, the higher the probability of the same person appearing, the greater the weight of the edge.

[0081] Next, the constructed graph structure is input into the graph convolutional network. The present invention designs a multi-scale graph convolutional network architecture, and extracts node features, spatial features, and global features through different convolution and pooling operations. After extracting these multi-scale features, the method fuses the node features, spatial features, and global features to obtain fused features. The fusion process can use simple feature splicing or more complex fusion strategies, such as attention mechanisms or gating units.

[0082] Finally, the method of the present invention inputs the fused features into the fully connected layer to generate the final behavior recognition result. The fully connected layer can use a softmax activation function to output the probability distribution of each possible behavior.

[0083] The method of the present invention realizes behavior recognition in complex video scenes through the above steps. Compared with traditional methods, this method has the following advantages:

[0084] 1. It can effectively capture the long-term spatiotemporal dependencies in videos.

[0085] 2. Through multi-scale feature extraction, both local details and global semantic information are taken into account.

[0086] 3. The graph structure representation enables the model to process video inputs of different lengths and structures.

[0087] 4. The end-to-end training method enables the entire model to adaptively learn the optimal feature representation.

[0088] In practical applications, this method can be used in a variety of scenarios, such as security monitoring, behavior analysis, human-computer interaction, etc. For example, in intelligent security systems, this method can automatically identify abnormal behaviors in surveillance videos and issue alarms in a timely manner; in human-computer interaction systems, it can identify user gestures and actions to achieve a more natural interaction method.

[0089] In summary, the video behavior recognition method based on graph convolutional network provided by the present invention effectively improves the accuracy of behavior recognition in complex video scenes through innovative graph structure representation and multi-scale feature extraction strategy, and has broad application prospects.

[0090] In a preferred embodiment of the present invention, the acquisition step is further refined into the following process: First, the original video and the corresponding labels are obtained. These labels contain the names of behaviors and the corresponding categories, which provide important prior information for subsequent behavior recognition. For example, the labels may include the names of behaviors such as "running", "playing basketball", and "driving", as well as their corresponding category numbers.

[0091] Next, this method splits the original video into multiple video segments according to the preset time interval. The choice of this time interval has an important impact on the subsequent behavior recognition effect. Usually, we can choose 2-5 minutes as the length of the video segment. The reason for choosing this range is that a video segment that is too short may not contain complete behavior information, while a video segment that is too long may contain multiple different behaviors, increasing the difficulty of recognition. In actual applications, this time interval can be adjusted according to specific scenarios and needs.

[0092] After obtaining the video segments, the method of the present invention further splits each video segment into multiple shots based on scene changes. This step is introduced to capture the behavioral information in the video more finely. The judgment of scene changes can be based on a variety of factors, such as the color distribution of the picture, the movement of the object, the change of the background, etc. For example, the image difference method can be used to detect the changes between consecutive frames. When the change exceeds a certain threshold, it is considered that a scene change has occurred, and a new shot is divided.

[0093] This multi-level video decomposition strategy enables the proposed method to better handle behavioral information of different scales. The segment-level division helps capture behavioral patterns over a longer time span, while the shot-level division is conducive to identifying more fine-grained actions and events.

[0094] When constructing the spatiotemporal graph connection, the method of the present invention takes into account a variety of spatial and temporal relationships.

[0095] In terms of spatial relationships, the following aspects are mainly considered:

[0096] 1. Overlap: Determine whether two entities overlap in space. This can be achieved by calculating the intersection of the bounding boxes of the entities.

[0097] 2. Whether there is interaction: Determine whether there is interaction between two entities. This can be determined by analyzing the distance change, contact status, etc. between the entities.

[0098] 3. Whether they are moving in the same direction: Determine whether multiple entities have similar motion trajectories. This can be determined by comparing the movement directions and speeds of the entities.

[0099] 4. Is there any relative motion? Analyze the relative position changes between entities. This is very helpful for identifying some complex interactive behaviors.

[0100] In terms of time relationship, this method mainly considers the following two aspects:

[0101] 1. Whether the interaction is continuous: Determine whether two entities maintain an interactive state in multiple consecutive shots. This helps to identify some complex behaviors that last for a long time.

[0102] 2. Whether they appear at the same time: Determine whether multiple entities appear in the same time period. This is very helpful for identifying group behavior or multi-person collaborative behavior.

[0103] Based on these spatial and temporal relationships, our method not only establishes connections between entities within a shot, but also between entities in different shots. This cross-shot connection enables the model to capture behavioral patterns over a longer time span, thereby improving recognition accuracy.

[0104] In specific implementation, quantitative indicators such as distance threshold, overlap ratio, speed vector, etc. can be used to judge various relationships. For example, a distance threshold d (such as 10% of the screen width) can be set. When the distance between two entities is less than d, it is considered that they may interact. For another example, the angle between the motion direction vectors of two entities can be calculated. When the angle is less than a certain threshold (such as 30 degrees), they are considered to be traveling in the same direction.

[0105] This graph structure construction method based on multiple relationships enables the present invention to more comprehensively capture the complex spatiotemporal dependencies in the video, providing rich structured information for subsequent behavior recognition.

[0106] When constructing a specific graph structure, the method of the present invention adopts the representation of nodes and edges. Each shot is represented as a node in the graph, and this node stores the spatial information and entity information of the shot. The spatial information may include the main scene type of the shot (such as indoors, outdoors), the layout structure of the scene, etc. The entity information may include the feature description of the person or object appearing in the shot, such as the person's posture, expression, the type and size of the object, etc.

[0107] Nodes are connected by edges, which represent the neighboring relationships between different shots. Setting the weight of the edge is a key point, which determines the strength of information transmission between different shots. This method sets the weight of the edge based on the spatial information and temporal relationship mentioned above. For example, if the same person appears in two shots, the weight of the edge connecting the two nodes can be set larger to reflect the strong correlation between them.

[0108] Specifically, the edge weight can be calculated by the following formula:

[0109] w ij =α·S ij +β·T ij ,

[0110] Among them, w i j is the weight of the edge connecting node i and node j, S i j is the spatial relationship score, T ijis the temporal relationship score, and α and β are coefficients that balance the two relationships.

[0111] S ij It can be further expressed as:

[0112] S ij =γ 1 ·O ij +γ 2 I ij +γ 3 ·M ij +γ 4 ·R ij ,

[0113] Among them, O ij ,I ij 、M ij , R ij denote the scores of overlap, interaction, peer and relative motion, respectively, and γ 1 , γ 2 , γ 3 , γ 4 is the corresponding weight coefficient.

[0114] Likewise, T i j can be expressed as:

[0115] T ij =δ 1 ·C ij +δ 2 ·A ij ,

[0116] Among them, C ij and A ij denote the scores of continuous interaction and co-occurrence, δ 1 and δ 2 is the corresponding weight coefficient.

[0117] The graph structure constructed in this way not only contains rich spatiotemporal information, but also reflects the strength of association between different shots through the weight of the edges. This provides an ideal input for the subsequent graph convolutional network, enabling the network to more effectively extract and transmit key behavioral features. In a preferred embodiment of the present invention, the process of extracting features using a graph convolutional network is further refined into the following key steps:

[0118] First, this method sets a node convolution filter to aggregate the information of adjacent nodes, thereby obtaining rich node features. This process can be regarded as the extraction and integration of local information in the graph structure. Specifically, for each node in the graph, we consider the node and its directly connected neighboring nodes, and perform a weighted sum of the features of these nodes through a convolution operation. The weights here can be pre-defined or learned through the network.

[0119] For example, for node i, its feature update can be expressed as:

[0120]

[0121] in, represents the feature of node i in the l+1th layer, is the neighbor set of node i, d i and d j are the degrees of nodes i and j respectively, and W (l) is the learnable weight matrix of the lth layer, and σ is the nonlinear activation function. Next, this method sets the spatial pooling filter. The purpose of this step is to extract spatial features by selectively summarizing based on the spatial structure of the nodes. The spatial pooling operation can help the model capture spatial dependencies in a larger range, while also reducing the dimension of the features and improving computational efficiency. In the specific implementation, different pooling strategies can be used, such as maximum pooling, average pooling, or attention pooling. For example, using the maximum pooling operation can be expressed as:

[0122]

[0123] Among them, p i is the pooled feature of node i, is the spatial neighborhood of node i, which can be defined according to actual needs. For example, all nodes directly connected to i can be selected, or the k nodes closest to i in spatial distance.

[0124] Finally, this method performs a global attention operation on all nodes in the graph to extract global features. The global attention mechanism allows the model to focus on the most relevant parts of the entire graph, which is very effective for capturing long-range dependencies and global contextual information.

[0125] The global attention operation can be implemented by the following steps: first, the attention scores between each pair of nodes are calculated; then, these scores are used to perform a weighted sum of the features of all nodes. Specifically:

[0126] e ij =a(Wh i ,Wh j ),

[0127]

[0128] g i =σ(∑ j∈v α ij W j ),

[0129] Where a is the attention function, which can be a simple dot product or a more complex neural network, W is the learnable weight matrix, and α ij is the attention coefficient of node j to node i, g i is the global feature of node i, is the set of all nodes in the graph.

[0130] Through these three steps, this method can comprehensively extract node features, spatial features and global features, providing a rich information basis for subsequent behavior recognition tasks.

[0131] After the acquisition step, the method of the present invention also includes an innovative preprocessing step, which aims to improve the accuracy and efficiency of subsequent behavior recognition. This preprocessing step mainly includes the following key operations:

[0132] First, this method calculates the similarity matrix between different shots. This similarity matrix is ​​used to measure the similarity between two shots and is the basis for subsequent shot clustering. When calculating similarity, multiple features can be considered, such as visual features, motion features, audio features, etc. For example, a pre-trained deep convolutional neural network can be used to extract the visual features of each shot, and then the cosine similarity between the feature vectors can be calculated.

[0133] Specifically, for shot i and shot j, their similarity can be expressed as:

[0134]

[0135] Among them, f i and f j are the feature vectors of lens i and j respectively, · represents the dot product operation, and ||·|| represents the L2 norm of the vector.

[0136] Next, this method merges the similarity matrix in the time dimension and the space dimension to obtain a comprehensive similarity matrix. The purpose of this step is to consider the temporal continuity and spatial similarity in a unified way, so as to obtain a more comprehensive similarity measure. The merging method can be a simple weighted average or a more complex fusion strategy.

[0137] For example, you can use the following formula to merge:

[0138] S′ ij =λSij +(1-λ)T ij ,

[0139] Among them, S′ ij is the comprehensive similarity, S ij is the spatial similarity, T ij is the temporal similarity (which can be calculated based on the distance between shots on the time axis), and λ is a balancing parameter used to adjust the relative importance of spatial similarity and temporal similarity.

[0140] After obtaining the comprehensive similarity matrix, this method selects adjacent shots to form clusters based on this matrix. This process can be regarded as soft clustering, the purpose of which is to organize similar shots together to form larger semantic units. Clustering can use classic clustering algorithms such as K-means or spectral clustering. It is also possible to use graph-based clustering methods such as community detection algorithms, which can better utilize the already constructed graph structure.

[0141] Finally, in each cluster, this method extracts temporal and spatial features and classifies each shot to obtain the predicted event of each shot. This step can be regarded as a preliminary behavior recognition process, and its results will serve as prior knowledge for subsequent fine recognition.

[0142] When extracting features, various existing feature extraction methods can be used. For example, for temporal features, optical flow methods can be used to extract motion information; for spatial features, deep convolutional neural networks can be used to extract high-level semantic features. After obtaining these features, simple classifiers (such as SVM or random forest) can be used to perform preliminary classification on each shot.

[0143] The introduction of this preprocessing step has several important advantages: first, through shot clustering, related shots can be organized together, which helps to capture behavioral patterns over a longer time span; second, the generation of predicted events provides valuable prior information for subsequent fine recognition, which can improve the accuracy of recognition; finally, this step can be regarded as coarse-grained recognition, which can reduce the complexity of subsequent processing to a certain extent. In a preferred embodiment of the present invention, the process of calculating the similarity matrix is ​​further refined into the following steps:

[0144] For any two shots m and n, the method first compares their corresponding visual features. These visual features can be high-dimensional feature vectors extracted by deep learning models (such as ResNet or VGG networks). The comparison process is essentially to calculate the difference between the two feature vectors. Preferably, the Euclidean distance or cosine distance can be used to quantify this difference.

[0145] For example, suppose the eigenvectors of lenses m and n are f m and fn , then the difference between them can be expressed as:

[0146] d mn =||f m -f n || 2

[0147] where ||·|| 2 represents the L2 norm.

[0148] Next, this method introduces a preset difference threshold τ. The choice of this threshold has an important impact on the final similarity matrix. Usually, the value of τ can be optimized on the validation set through cross-validation. In practice, τ can be set to the median or a certain percentile of the feature vector difference distribution.

[0149] When the calculated difference value is less than the preset threshold, this method converts the similarity matrix S mn The corresponding position in is set to 1, indicating that the two shots are considered similar. Conversely, when the difference value is greater than or equal to the preset threshold, the corresponding position is set to 0, indicating that the two shots are not similar. Formally, this process can be expressed as:

[0150]

[0151] This binary similarity matrix helps to highlight important similarity relationships and also simplifies subsequent processing. However, in some cases, using continuous similarity values ​​may provide more detailed information. Therefore, in another embodiment of the present invention, it is possible to consider using a soft threshold function, such as a sigmoid function, to map the difference value to the [0,1] interval:

[0152]

[0153] where σ is a parameter that controls the steepness of the function.

[0154] The similarity matrix constructed in this way provides an important basis for subsequent shot clustering and graph structure construction. It not only reflects the visual similarity between shots, but also makes it possible to capture long-term dependencies in the video.

[0155] After obtaining the preliminary similarity matrix, the method of the present invention further calculates the comprehensive similarity matrix, and this process includes the following key steps:

[0156] First, this method obtains the time characteristics of each shot and obtains the time vector T mThe temporal features may include the start time, duration, relative position of the shot in the video, etc. For example, the start time of the shot can be normalized to the interval [0,1] to represent the temporal position of the shot in the entire video. Next, the spatial features of each shot are obtained to obtain the spatial vector S m Spatial features can describe the position, size, motion trajectory, and other information of the main objects or characters in the shot. For example, an object detection algorithm can be used to locate the main object, and then its position and size information can be encoded into a feature vector.

[0157] Then, this method transposes the time vector and the space vector to obtain the transposed vector and S A The purpose of this step is for subsequent matrix multiplication operations.

[0158] Next, calculate the temporal similarity matrix S AAT and the spatial similarity matrix S AS Specifically:

[0159]

[0160]

[0161] Among them, T A and S A are matrices composed of the time vectors and space vectors of all shots respectively. Afterwards, this method calculates the mean of the time similarity matrix and the space similarity matrix to obtain the time-space similarity matrix mean matrix S T :

[0162]

[0163] Finally, the time-space similarity matrix mean matrix is ​​normalized to obtain the final comprehensive similarity matrix S. The purpose of normalization is to control the similarity value within the interval [0,1] for subsequent processing. The commonly used normalization method is min-max normalization:

[0164]

[0165] This method of calculating the comprehensive similarity matrix takes into account information in both time and space dimensions, and can more comprehensively describe the similarity relationship between shots. It provides an important basis for subsequent shot clustering and behavior recognition.

[0166] In the core embodiment of the present invention, the convolution process of the graph convolutional network is the key link in behavior recognition. This process can be formally expressed as follows:

[0167] For the kth action node in the tth frame, the update formula of its embedding vector is:

[0168]

[0169] In this formula, represents the feature representation of the k-th action node at the l+1 layer. σ(·) is a nonlinear activation function, and commonly used choices include ReLU, LeakyReLU, or tanh. N(k) represents the set of nodes connected to the k-th action node. The definition of this set directly affects the range of information propagation. It is the weight calculated by the node attention mechanism, which reflects the importance of node i to node k in layer l. The introduction of this attention weight enables the model to adaptively adjust the importance of different neighbor nodes, thereby aggregating information more effectively. The calculation of attention weight can be adopted in many ways, such as:

[0170]

[0171] in, It can be a learnable attention function, such as:

[0172]

[0173] Here a is a learnable vector, and || represents a vector concatenation operation. l is the learnable weight matrix of layer l, which linearly transforms the input features. By using different weight matrices at different layers, the model is able to extract more advanced feature representations layer by layer. Represents the embedding vector of the i-th node in the l-th layer. These vectors contain the feature information of the nodes, and through the graph convolution operation, this information will be propagated and aggregated in the graph structure.

[0174] A key advantage of this convolution process is that it can effectively capture local and global dependencies in the graph structure. By stacking multiple layers, the model can gradually expand the receptive field to capture longer-distance dependencies. At the same time, the introduction of the attention mechanism enables the model to dynamically adjust the flow of information between different nodes, which is particularly effective for processing complex spatiotemporal relationships.

[0175] In practical applications, this convolution process can be adjusted according to the needs of specific tasks. For example, residual connections can be introduced to alleviate the gradient vanishing problem of deep networks; different types of graph convolution operations can be used, such as GraphSAGE or GAT; additional regularization techniques such as dropout or batch normalization can be added to the convolution process to improve the generalization ability of the model.

[0176] Through this carefully designed graph convolution process, our method can effectively extract and integrate complex spatiotemporal information in videos, providing powerful feature representation for the final action recognition task.

[0177] Finally, the present invention also proposes a video behavior recognition system based on graph convolutional networks. This system is a systematic implementation of the above method, which includes multiple key modules, each of which has a specific function and works together to complete the task of video behavior recognition.

[0178] Specifically, the system includes the following modules:

[0179] Video segment splitting module 1 is responsible for obtaining multiple videos with behavior labels and decomposing them into multiple video segments. Each video segment contains multiple shots. In addition, this module is also responsible for obtaining the time information, spatial information and entity information of each shot. This information provides basic data for subsequent processing.

[0180] Graph construction module 2: This module builds spatiotemporal graph connections based on the temporal information, spatial information, and entity information obtained from the video segment splitting module, and finally obtains a complete graph structure. This graph structure is the key input for subsequent graph convolutional network processing.

[0181] Feature extraction module 3: This module inputs the constructed graph structure into the graph convolutional network. Through the designed different convolution and pooling operations, this module can effectively extract node features, spatial features and global features. These multi-scale features provide a rich information basis for subsequent behavior recognition.

[0182] Feature fusion module 4 is responsible for fusing the node features, spatial features and global features obtained by the feature extraction module to obtain a comprehensive fusion feature. This fusion process can adopt different strategies, such as simple feature concatenation or more complex attention mechanisms.

[0183] The recognition output module 5 is the last link of the system. This module inputs the fused features into the fully connected layer to generate the final behavior recognition result. This result can be the probability distribution of different behavior categories or the prediction of specific behavior categories.

[0184] This modular system design is highly flexible and scalable. Each module can be independently optimized and improved according to specific application requirements, while the interfaces between modules remain consistent, ensuring the stability and maintainability of the entire system. For example, in the feature extraction module, different graph convolutional network structures can be tried; in the feature fusion module, more efficient fusion strategies can be explored; in the recognition output module, different classifiers or regressors can be selected according to specific tasks.

[0185] In addition, this system design also makes end-to-end training possible. Through the back-propagation algorithm, the parameters of all modules can be optimized simultaneously, thereby achieving joint optimization of the entire system. This end-to-end training method helps the model capture more complex patterns and improve overall recognition performance.

[0186] In general, the video behavior recognition method and system based on graph convolutional network proposed in this paper provide an effective solution for behavior recognition tasks in complex video scenes through innovative graph structure representation, multi-scale feature extraction and end-to-end system design. This method and system can not only effectively capture the long-term spatiotemporal dependencies in the video, but also have strong flexibility and scalability, providing a solid foundation for future deployment and optimization in various practical applications.

[0187] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A video behavior recognition method based on graph convolutional network, characterized in that: include: The acquisition steps include: Get multiple videos with behavior labels; Decomposing the video into a plurality of video segments, each of the video segments comprising a plurality of shots; Acquire time information, space information and entity information of each of the shots; Processing steps include: Based on the time information, the space information and the entity information, a spatiotemporal graph connection is constructed to obtain a graph structure; Input the graph structure into a graph convolutional network, and use different convolution and pooling operations to extract node features, spatial features, and global features; Fusing the node feature, the spatial feature and the global feature to obtain a fused feature; Output steps include: The fused features are input into the fully connected layer to generate behavior recognition results.

2. The method according to claim 1, characterized in that The acquisition step specifically includes: Obtaining an original video and a corresponding label, wherein the label includes a behavior name and a corresponding category; Splitting the original video into a plurality of video segments according to a preset time interval; Each of the video segments is further divided into a plurality of the shots based on scene changes.

3. The method according to claim 1, characterized in that: The steps of constructing the space-time graph connection specifically include: Establishing connections between entities in the shot based on spatial relationships, wherein the spatial relationships include whether they overlap, whether they interact, whether they move in the same direction, whether they are in relative motion, or multiple types thereof; Establishing connections between entities in the shot based on a temporal relationship, wherein the temporal relationship includes whether the interaction is continuous, whether the interaction occurs simultaneously, or a combination thereof; Establishing a connection between entities of the shots based on the spatial relationship and the temporal relationship.

4. The method according to claim 3, characterized in that: The steps of constructing the graph structure specifically include: The shots are represented as nodes, and each node stores the spatial information and entity information of the corresponding shot; Adjacent nodes between different shots are connected by edges, and weights of the edges are set according to the spatial information and the temporal relationship.

5. The method according to claim 1, characterized in that The steps of extracting features using the graph convolutional network specifically include: Set the node convolution filter to aggregate the information of adjacent nodes and obtain the node features; Set a spatial pooling filter to selectively aggregate nodes based on their spatial structure and extract spatial features; Perform global attention operations on all nodes in the graph to extract global features.

6. The method according to claim 1, characterized in that After the acquisition step, a pre-processing step is also included: Calculate the similarity matrix between different shots to measure the similarity between two shots; Merging the similarity matrix in the time dimension and the space dimension to obtain a comprehensive similarity matrix; Selecting adjacent shots to form a cluster based on the comprehensive similarity matrix; Temporal features and spatial features are extracted from each cluster, each shot is classified, and the predicted event of each shot is obtained.

7. The method according to claim 6, characterized in that The steps of calculating the similarity matrix specifically include: For shot m and shot n, compare the corresponding visual features therein, count the differences between the visual features, and obtain a difference value; When the difference value is less than a preset difference threshold, the corresponding position in the similarity matrix Smn is set to 1; When the difference value is greater than or equal to a preset difference threshold, the corresponding position in the similarity matrix Smn is set to 0.

8. The method according to claim 6, characterized in that The step of calculating the comprehensive similarity matrix specifically includes: Get the time characteristics of each shot and get the time vector T m ; Get the spatial features of each shot and get the spatial vector S m ; Transpose the time vector and the space vector to obtain a transposed vector and S A ; Calculate the time similarity matrix S AAT and the spatial similarity matrix S AS ; Calculate the mean of the time similarity matrix and the space similarity matrix to obtain the time-space similarity matrix mean matrix S T ; The time-space similarity matrix mean matrix is ​​normalized to obtain the comprehensive similarity matrix S.

9. The method according to claim 1, characterized in that: The convolution process of the graph convolutional network includes: represents the embedding vector of the kth action node in the tth frame, and its update formula is: in, is the l+1th layer feature of the kth action node, is the weight calculated by node attention, W l is the weight matrix of the lth layer, represents the embedding vector of the i-th node in the l-th layer, N(k) represents the set of nodes connected to the k-th action node, and σ(·) represents the activation function.

10. The video behavior recognition system based on graph convolutional network is characterized by: include: Video segment splitting module, used for: Get multiple videos with behavior labels; Decomposing the video into a plurality of video segments, each of the video segments comprising a plurality of shots; Acquire time information, space information and entity information of each of the shots; Graph building blocks for: Based on the time information, the space information and the entity information, a spatiotemporal graph connection is constructed to obtain a graph structure; Feature extraction module for: Input the graph structure into a graph convolutional network, and use different convolution and pooling operations to extract node features, empty features, and global features; Feature fusion module, used for: Fusing the node feature, the spatial feature and the global feature to obtain a fused feature; Identify output modules for: The fused features are input into the fully connected layer to generate behavior recognition results.

Images