Group behavior identification method based on Markov decision process and extension graph convolution

Abstract

The invention discloses a group behavior recognition method based on a Markov decision process and extension graph convolution, and relates to the technical field of computers. The method comprises the following steps: extracting a key frame of a video to be identified based on a Markov decision process, and sequentially extracting a convolution feature map of the key frame, bounding boxes of a plurality of target individuals in the convolution feature map and behavior features of each target individual; in each layer of graph convolutional network, calculating a cosine distance between behavior characteristics of a node and an adjacent node in the human body target relation graph so as to determine a connection weight and update the behavior characteristics of the node; calculating a relation feature vector between any two nodes based on the behavior features obtained by the last layer of graph convolutional network, and dividing a plurality of sub-target interaction graphs; classifying each sub-target interaction diagram to obtain a predicted behavior label; and determining the predicted behavior tag of the subgroup with the highest occurrence probability in all the key frames as a group behavior tag. According to the method, the group behavior identification precision is improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method for group behavior recognition based on Markov decision processes and extended graph convolution. Background Technology

[0002] Group behavior recognition is an important research area in computer vision, primarily applied in fields such as intelligent surveillance, video retrieval, and human-computer interaction. Group behavior recognition refers to the automatic identification of activities performed jointly by multiple people from video images using algorithms.

[0003] Existing group behavior recognition methods sparsely sample video sequences containing group behavior events to obtain video sample frames. They then use object detection networks and dimensionality reduction to obtain individual target features from each sample frame. A graph model is constructed based on the appearance and positional relationships between individuals, and a graph convolutional neural network is used to extract the group behavior representation features from each frame. Finally, the group behavior representation features of the entire video are obtained by fusing the group behavior features from multiple frames. While this method utilizes a graph structure to model individual relationships, its recognition accuracy is significantly limited in practical applications, especially in complex scenarios where group behavior recognition accuracy is low. Summary of the Invention

[0004] Therefore, it is necessary to provide a group behavior recognition method based on Markov decision processes and extended graph convolution to address the aforementioned technical problems. This method improves the accuracy of group behavior recognition.

[0005] The present invention adopts the following technical solution: This invention provides a group behavior recognition method based on Markov decision processes and extended graph convolution, comprising: Based on the Markov decision process, a preset number of key frames are extracted from the video to be identified. For each key frame, the convolutional feature map of the key frame is extracted by a convolutional neural network, the bounding boxes of multiple target individuals in the convolutional feature map are extracted by a region proposal network, and the behavioral features of each target individual are extracted by region pooling. Using individual targets as nodes and the connections between nodes as edges, a human target relationship graph is constructed, and the human target relationship graph is input into a multi-layer graph convolutional network. In each layer of the graph convolutional network, the cosine distance between the behavioral features of a node and the behavioral features of its neighboring nodes in the human target relationship graph is calculated to determine the connection weights between the node and its neighboring nodes. The behavioral features of the node are updated based on the connection weights between the node and its neighboring nodes, and the updated behavioral features are passed to the next layer of the graph convolutional network. Based on the behavioral features obtained from the last layer of the graph convolutional network, the relationship feature vector between any two nodes in the human target relationship graph is calculated, and multiple sub-target interaction graphs are divided based on the relationship feature vector; each sub-target interaction graph corresponds to a subgroup. The interaction graph of each sub-target is classified by a classifier to obtain the predicted behavior label of each subgroup; the predicted behavior label of the subgroup with the highest probability of occurrence in all keyframes is determined as the group behavior label of the video to be identified.

[0006] Preferably, based on a Markov decision process, a predetermined number of keyframes are extracted from the video to be identified, specifically including: The video to be identified is uniformly sampled to obtain the frame sequence of the video to be identified; Visual features of each frame in the frame sequence are extracted using a deep convolutional neural network, and observation probabilities are generated based on the visual features of all frames. The keyframe extraction problem is defined as the decoding problem of a hidden Markov model; the state transition probabilities of the Markov chain represent the temporal continuity between frames; Using the Viterbi algorithm, the globally optimal keyframe sequence is calculated under the joint constraints of observation probability and state transition probability; The globally optimal keyframe sequence is determined as the selected frames; Based on the global information of the video to be identified and the selected frames, a preset number of keyframes are extracted from the video to be identified through a pre-trained keyframe extraction network.

[0007] Preferably, the keyframe extraction network includes a first branch and a second branch connected in parallel, a third fully connected layer, and a softmax activation function; the first branch includes multiple sets of convolutional layers and pooling layers connected in series, as well as a first fully connected layer; the second branch includes a second fully connected layer; based on the global information of the video to be identified and the selected frames, a preset number of keyframes of the video to be identified are extracted through the pre-trained keyframe extraction network, specifically including: The global information of the video to be identified and the selected frames are input into the first branch for convolution and pooling processing to obtain the result of the first branch; The binary mask of the selected frame is input into the second branch for processing, and the result of the second branch is obtained. The results of the first branch and the second branch are concatenated and then input into the third fully connected layer; The output of the third fully connected layer is regularized using the softmax activation function to obtain a preset number of keyframes.

[0008] Preferably, the training sample images of the region proposal network include positive anchor points and negative anchor points; the loss function of the region proposal network is: ; in, The loss value for the region proposal network. anchor point i It is the probability of the target individual. anchor point i The tag, For the parameters of the prediction box, For the parameters of the calibration box, For classification loss function, For regression loss function, This indicates that the regression is performed only on samples with positive anchor points. This represents the number of cells in the feature map after convolution in a batch of data. For the number of anchor points, These are the balancing parameters.

[0009] Preferably, the formula for calculating the relation feature vector is: ; in, For nodes i With nodes j In the l Layer relational feature vectors For node-based i behavioral characteristics Calculate the representative node i Scalar value of query intent Based on nodes j behavioral characteristics Calculate representative node j The scalar value of the queried value.

[0010] Preferably, the interaction graph is divided into multiple sub-targets based on relational feature vectors, specifically including: When the relation feature vector is greater than 0, it is determined that the two nodes belong to the same behavior category; When the relation feature vector is equal to 0, it is determined that the two nodes do not belong to the same behavior category; Multiple sub-goal interaction graphs are constructed by using nodes that belong to the same behavior category.

[0011] Preferably, the update formula for the node's behavioral characteristics is: ; in, For nodes i Updated features The weight matrix is ​​a learnable matrix. For the aggregated nodes i global context features For nodes i The initial behavioral characteristics.

[0012] Preferably, the loss function of the multi-layer graph convolutional neural network is: ; in, For the loss of a graph convolutional neural network, , The total loss for group behavior, For the true tags of group activities, Predictive labels for group activities For the loss of the first classification group, For the first m The losses of each classification group, The parameters to be learned , For losses of individual activities, As a predictive label for individual activities, As a true label for individual activities, For the first i Individual activity loss in each classification group k For the number of people in each classification group, For the first j Authentic labels of individual activities of each group member For the first j Predictive labels for individual activities of group members m For the number of classification groups, k This represents the number of members in the classification group.

[0013] This invention provides a group behavior recognition device based on Markov decision process and extended graph convolution, comprising: The extraction module is used to extract a preset number of key frames from the video to be identified based on the Markov decision process. For each key frame, the convolutional feature map of the key frame is extracted sequentially through a convolutional neural network, the bounding boxes of multiple target individuals in the convolutional feature map are extracted through a region proposal network, and the behavioral features of each target individual are extracted through region pooling of interest. The module is used to construct a human target relationship graph with the target individual as a node and the connection relationship between the nodes as an edge, and then input the human target relationship graph into a multi-layer graph convolutional network. The graph convolution module is used in each layer of the graph convolutional network to calculate the cosine distance between the behavioral features of a node and the behavioral features of its neighboring nodes in the human target relationship graph, so as to determine the connection weight between the node and its neighboring nodes, update the behavioral features of the node based on the connection weight between the node and its neighboring nodes, and pass the updated behavioral features to the next layer of the graph convolutional network. The partitioning module is used to calculate the relationship feature vector between any two nodes in the human target relationship graph based on the behavioral features obtained from the last layer of the graph convolutional network, and to partition multiple sub-target interaction graphs based on the relationship feature vector; each sub-target interaction graph corresponds to a subgroup. The determination module is used to classify each sub-target interaction graph through a classifier to obtain the predicted behavior label of each subgroup; the predicted behavior label of the subgroup with the highest probability of occurrence in all keyframes is determined as the group behavior label of the video to be identified.

[0014] The present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described group behavior recognition method based on Markov decision process and extended graph convolution.

[0015] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described group behavior recognition method based on hash encoding and extended graph convolution.

[0016] The above-mentioned at least one technical solution adopted in this invention can achieve the following beneficial effects: Based on Markov decision processes, a predetermined number of keyframes are extracted from the video to be identified. For each keyframe, the convolutional feature map of the keyframe, the bounding boxes of multiple target individuals in the convolutional feature map, and the behavioral features of each target individual are extracted sequentially. Markov decision processes can extract the most representative keyframes from the video to be identified while reducing redundant information and retaining key behavioral features. In each layer of the graph convolutional network, the cosine distance between the behavioral features of a node and the behavioral features of its neighboring nodes in the human target relationship graph is calculated to determine the connection weights between the node and its neighboring nodes. The behavioral features of the node are updated based on the connection weights between the node and its neighboring nodes, and the updated behavioral features are passed to the next layer of the graph convolutional network. Based on the last layer of the graph convolutional network, the following is obtained: The method calculates the relationship feature vector between any two nodes in the human target relationship graph based on behavioral features, and divides the graph into multiple sub-target interaction graphs based on these feature vectors. Connection weights are calculated using cosine distance, dynamically adjusting the importance of nodes and their neighbors to avoid bias caused by fixed weights. This ensures the graph convolutional network can adaptively learn the interaction relationships between target individuals. Furthermore, by updating behavioral features layer by layer, higher-level behavioral patterns are gradually extracted, improving classification accuracy. Each sub-target interaction graph corresponds to a subgroup. A classifier classifies each sub-target interaction graph to obtain the predicted behavioral label for each subgroup. The predicted behavioral labels of the subgroups with the highest probability of occurrence in all keyframes are determined as the group behavior labels for the video to be identified. This method improves the accuracy of group behavior recognition. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0018] Figure 1 A schematic diagram of the process for a group behavior recognition method based on Markov decision process and extended graph convolution provided by the present invention; Figure 2 The keyframe extraction network structure diagram provided by this invention; Figure 3 This is a schematic diagram of the RPN processing procedure provided by the present invention; Figure 4 The RPN principle structure diagram provided by this invention; Figure 5 This is a schematic diagram of the RoI Pooling principle provided by the present invention; Figure 6 The RPN network structure diagram provided by this invention; Figure 7 The flowchart of the group behavior recognition method provided by the present invention; Figure 8 A schematic diagram of a group behavior recognition device based on Markov decision process and extended graph convolution provided by the present invention; Figure 9 This is a schematic diagram of a computer device for implementing a group behavior recognition method based on Markov decision process and extended graph convolution, as provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0020] Devices such as desktop computers, servers, and laptops are capable of executing the present invention. For ease of explanation, the following description will focus on servers as the executing entity.

[0021] The technical solutions provided by the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0022] Figure 1 This is a schematic diagram of a group behavior recognition method based on Markov decision process and extended graph convolution in this invention, which specifically includes the following steps: S101: Based on the Markov decision process, a preset number of key frames are extracted from the video to be identified. For each key frame, the convolutional feature map of the key frame is extracted sequentially through a convolutional neural network, the bounding boxes of multiple target individuals in the convolutional feature map are extracted through a region proposal network, and the behavioral features of each target individual are extracted through region pooling of interest.

[0023] In an exemplary embodiment, based on a Markov decision process, a predetermined number of keyframes are extracted from the video to be identified. Specifically, this includes: uniformly sampling the video to be identified to obtain a frame sequence; extracting the visual features of each frame in the frame sequence using a deep convolutional neural network, and generating observation probabilities based on the visual features of all frames; defining the keyframe extraction problem as a decoding problem of a hidden Markov model; the state transition probabilities of the Markov chain characterize the temporal continuity between frames; using the Viterbi algorithm, under the joint constraints of observation probabilities and state transition probabilities, calculating the globally optimal keyframe sequence; determining the globally optimal keyframe sequence as the selected frames; and extracting the predetermined number of keyframes from the video to be identified using a pre-trained keyframe extraction network based on the global information of the video to be identified and the selected frames.

[0024] In an exemplary embodiment, the keyframe extraction network includes a first branch and a second branch connected in parallel, a third fully connected layer, and a softmax activation function. The first branch includes multiple sets of convolutional layers and pooling layers connected in series, as well as a first fully connected layer. The second branch includes a second fully connected layer. Based on the global information of the video to be identified and the selected frames, a preset number of keyframes of the video to be identified are extracted through the pre-trained keyframe extraction network. Specifically, this includes: inputting the global information of the video to be identified and the selected frames into the first branch for convolution and pooling processing to obtain the first branch result; inputting the binary mask of the selected frames into the second branch for processing to obtain the second branch result; concatenating the first branch result and the second branch result and inputting them into the third fully connected layer; and using the softmax activation function to regularize the output of the third fully connected layer to obtain the preset number of keyframes.

[0025] Specifically, Figure 2 The keyframe extraction network structure diagram provided by this invention is as follows: Figure 2 As shown, the keyframe extraction network receives the state respectively. and As input, This includes global information about the video to be identified and the selected frames. This includes the binary mask of the selected frame; then, it is processed using three sets of sequentially concatenated convolutional and pooling layers, plus a first fully connected layer (fc1). The processing involves three sets of convolutional and pooling layers connected in series: the first convolutional layer (conv1), the first pooling layer (pool1), the second convolutional layer (conv2), the second pooling layer (pool2), the third convolutional layer (conv3), and the third pooling layer (pool3). The features extracted from fc1 and fc2 are processed by a second fully connected layer (fc2), and then concatenated with each other to input into a third fully connected layer (fc3). Finally, the output of fc3 is regularized using the softmax function, and the highest probability is selected to determine the next action. Figure 2 In M For keyframe selection module, F For all video frames, conv is the convolution operation, pool is the pooling operation, fc is the fully connected operation, concat is the concatenation operation, softmax is the normalized exponential function, and binary mask of selected indices is the masked filter frame.

[0026] In an exemplary embodiment, the keyframe extraction network is used as the agent of the Markov decision process; the adjustment direction of the selected frames based on the state change is taken as the action of the Markov decision process; the direction includes left, right and remain unchanged; the global information of the video to be identified, the selected frames and the binary mask of the selected frames are taken as the state of the Markov decision process; the result of the agent taking action is taken as the reward of the Markov decision process.

[0027] Rewards are recorded as r ( S,A This reflects the outcome of an agent taking an action in state S. The reward is obtained using a pre-trained graph convolutional neural network. In the first iteration, if the prediction is correct, then... r =1, otherwise r =-1.

[0028] The reward for a Markov decision process is given by formula (1): (1); in, Let be the reward value of the Markov decision process, where the reward is the value when the prediction changes from incorrect to correct. The reward is - when the prediction result changes from correct to incorrect. The reward is when the prediction result has no jump. The prediction result determines whether to retain the current frame. , sgn is the activation function of the neuron. For the first n In the next iteration, the result was correctly predicted as c The probability, For the first n The result was correctly predicted in the -1st iteration. c The probability, c For the first n -1 is the probability generated in the first iteration.

[0029] The goal of training the keyframe extraction network is to maximize the reward, i.e., to maximize the reward. , It is a number greater than 0 and less than 1. With each iteration, the reward decreases because the main adjustments are concentrated in the early iterations, and the final iteration tends to converge. To maximize... R The present invention proposes that the loss function of the keyframe extraction network is formula (2):

[0030] (2); in, Extract the network's loss value for keyframes. m The duration of a video frame. For a parameter The defined model, given input The error value calculated at that time. For a specific point in time t The spatial relationships and states among group members in the scene. For the previous moment t -1 to the current time t Changes and actions that occur in the scene, This represents all the learnable parameters of the model. This means that the magnitude of the loss value depends on the current parameter settings of the model.

[0031] The updated formula is (3): (3); in, For the first i+ In the first iteration The value, For the first i During the next iteration The value, Indicates the gradient sign. A learning rate or adaptive scaling factor determines the step size for parameter updates.

[0032] In one embodiment of the present invention, the network structure for keyframe extraction uses a frame distillation network (FDNet). The main function of FDNet is to generate actions. Actions are generated based on the network... Parameters in Produced.

[0033] Specifically, the selection of keyframes is described as a Markov Decision Process (MDP), which is a cyclical process in which an agent takes action to change its state, obtains a reward, and interacts with the environment. By maximizing the reward, the agent learns to adjust the selected frames, ultimately obtaining a given number of video frames. In this process, the selected frames are gradually adjusted by considering two main factors: (1) the quality of the selected frames; and (2) the relationship between the selected frames and the entire video.

[0034] An agent is a keyframe extraction network structure (FDNet). An action, based on changes in state and the already selected frames, determines the direction of frame selection, including three types of actions: left, remain unchanged, and right. The state consists of two parts. and , It contains information about the entire video sequence and the selected video frames. It is a binary mask of the selected frame index; the index of the selected frame is 1, otherwise it is 0. The reward is determined based on the impact of the extracted keyframes on action recognition, i.e., how good the result will be if the agent takes an action. The network architecture of action recognition can be considered as the environment that interacts with the agent.

[0035] The state S of MDP consists of two parts { , } =[F, M], yes F and M There are two tensors. F Representing global information about the video, is f × N A tensor of shape ×3. Here, f , N 3 and 3 represent the number of frames, the number of keypoints, and the coordinate dimension, respectively. For those that are not entirely... f For video with a frame length of [length], export using bicubic interpolation. fA video of frames, where the first and last frames are identical to the original frames. F similar, M yes m × N ×3 tensors represent m Information from a selected frame. It is a binary mask of the selected frame index to explicitly let FDNet know what has been selected. It is f dimensional vector, m One element is 1, and the rest are 0. In one embodiment of the present invention, the following is used: f Set to 100. m Set it to 30.

[0036] Action, or FDNet output, represents the adjustment direction for each selected frame. Three types of actions are defined: "move left" (Action 0), "stay still" (Action 1), and "move right" (Action 2), with a movement step size set to 1 frame. For example... Figure 2 As shown, FDNet emits a vector during each iteration. ,in, Represented as the first i Select action for selected frames j The probability of.

[0037] When taking an Action, it is important to pay attention to the order of the frame sequence. For example, if the first frame moves significantly to the right while the second frame keeps moving to the left, causing the first frame to end up to the right of the second frame, the order of different frames may become disordered with each movement. Therefore, it is necessary to set boundaries for the Action to ensure that all frames are arranged in order. The upper limit is the frame between the current frame and the next frame, and the boundary of the Action is shown in formula (4):

[0038] (4); in, For the first i The upper limit of frame movement, that is, the limit for rightward movement, is the [number]th frame. i Frame and the i +1 frame, round up to the nearest integer. i equal m ,but i Called the selected frame m The last frame in the file, then the upper limit of adjustment is... f That is, the total number of frames in the video sequence. For the first i Frame video frame, For the first i +1 video frame.

[0039] The lower limit for adjustment is the midpoint between this frame and the previous frame; if it is not divisible, it is rounded down. The lower limit is given by formula (5):

[0040] (5); in, For the first i The lower limit of frame shift, For the first i- 1 video frame.

[0041] No. i The frame adjustment range is ,use If the change is represented by the formula (6), then the adjusted frame is represented by the formula (6): (6); in, For the first i Frame video frame, The variable is the amount of change.

[0042] in, For formula (7): (7); In an exemplary embodiment, the convolutional feature map of the keyframe, the bounding boxes of multiple target individuals in the convolutional feature map, and the behavioral features of each target individual are extracted sequentially, including: extracting the convolutional feature map of the keyframe through a convolutional neural network; extracting the bounding boxes of the target individuals in the convolutional feature map through a region proposal network; and extracting the behavioral features of each bounding box of a preset size through region pooling of interest.

[0043] Specifically, Figure 3 This is a schematic diagram of the RPN processing procedure provided by the present invention, as shown below. Figure 3 As shown, a Convolutional Neural Network (CNN) model with a backbone network (such as VGG16) is used as the feature extractor. Each input video frame is... H × W × D In this way, each video frame is processed by a pre-trained CNN model to obtain a convolutional feature map.

[0044] The Region Propose Network (RPN) processes the extracted convolutional feature maps to find bounding boxes that may contain the target individuals.

[0045] Specifically, Figure 4 The RPN principle structure diagram provided by this invention is as follows: Figure 4 As shown, the goal is to first detect each individual in the video, and then analyze the interactions between these individuals to determine the behavior of the entire group. A convolutional feature map is an abstract feature map extracted from video frames using a deep convolutional network; a sliding window systematically scans every possible position on the abstract feature map, row by row and column by column, using a small window. k Each anchor box / prior box is predefined at the center of each sliding window. k 1. Default bounding boxes of different shapes and sizes; 2. An intermediate layer is a fully connected or convolutional layer used to further process the features extracted from the sliding window; 3. The 256-dimensional (256-d) feature vector output by the intermediate layer represents the richness of the features. k 2k scores & classification layer (cls layer). cls is short for classification, which refers to the classification of scores at each position. k With one anchor box, the classification layer will output 2. k A fraction, usually 2 k One, because of the background / foreground binary classification, 2 k Each score represents the probability that each anchor box is a "player" or a "background"; 4 k 4k coordinates® layer (reg is short for regression), which refers to the coordinates and regression layer at each location. k With one anchor box, the regression layer will output 4. k Each box requires four coordinate values ​​for fine-tuning: center point. x , y offset, width w (and the scaling factor of the height), its function is to fine-tune the position and shape of the anchor frame so that it more accurately frames the target.

[0046] Each point in the convolutional feature map will be predicted. k Nine anchor boxes are generated on the original image, representing pre-selected Regions of Interest (ROIs). These anchor boxes are centered on a point in the convolutional feature map (which can be mapped onto the input image), and a sliding window operation (3×3) is performed on the convolutional feature map. The sliding window only selects the location and has no other function. For each sliding window, nine anchor boxes are generated in the original image. These anchor boxes share the same center but have three different aspect ratios and three different scales, calculated relative to the original image size. The specific operation process is as follows: Figure 4 As shown in the diagram, a small network slides across this convolutional feature map to score (determine if each anchor box is foreground) and regress (position corrections for each anchor box) the nine anchor boxes corresponding to each point on the normalized image. This corresponds to Figure 2. k and 4 k Output ( k =9), 2 represents the foreground / background, and 4 represents the four values ​​for the proposed corrections.

[0047] For the classification layer, each anchor box outputs two predicted values: the probability of background and the probability of foreground. For the regression layer, also known as the bounding box adjustment layer, each anchor box outputs four predicted values. The final result is achieved by adjusting the anchor frame. .

[0048] The training process of a Region Proposal Network (RPN) includes: randomly selecting multiple anchor points from a sample image; ensuring a 1:1 ratio of positive to negative anchor points among the multiple anchor points; using these multiple anchor points as a training set to train the RPN; and optimizing the parameters of the RPN using the Adam algorithm until the loss function value is minimized, thus obtaining a trained RPN.

[0049] Specifically, Figure 5 The principle structure diagram of RoI Pooling provided by this invention is as follows: Figure 5 As shown, in the context of deep learning, "projection" usually refers to a projection or transformation operation, typically a fully connected layer or a 1x1 convolutional layer, used to change the dimension of features or map them to another space. A Region of Interest (RoI) is a rectangular region defined on the original image that surrounds a specific target (such as a person). This is often used as an analogy in object detection: like drawing the bounding box of each player on a video screen, each bounding box being a RoI. RoI pooling is an operation that converts feature maps extracted from different RoIs into a single fixed-size (e.g., 7x7x512) output. 7x7 represents the spatial dimensions (height and width), ensuring that the output of each RoI is spatially consistent; 512 represents the channel dimension, indicating that each spatial location (each in the 7x7 grid) is described by a 512-dimensional vector representing its rich features.

[0050] RoIPooling is used to extract fixed-size feature maps for each proposal. Specifically, the convolutional feature maps are cropped for each proposal, and then an interpolation algorithm (e.g., bilinear interpolation) is used to crop each feature map to a fixed size of 14×14×channels. After cropping, max pooling with a 2×2 kernel is performed to obtain a final 7×7×channel feature map for each proposal. Convolutional feature maps are obtained through a CNN backbone network, such as the conv, pooling, and ReLU layers of the Visual Geometry Group (VGG). The feature maps are used to generate region proposals via RPN. Softmax is used to determine whether the anchor boxes (9 boxes) are foreground or background, and anchor box regression is used for further refinement to define the proposals. RoIPooling is used to extract proposal feature maps, and the category is determined using these proposal feature maps and the initial proposals.

[0051] Figure 6 The RPN network structure diagram provided by this invention is as follows: Figure 6 As shown, Conv.Net is the backbone convolutional network; Feature Map is the high-level feature map extracted from the input image by the backbone convolutional network; Reshape is an operation that adjusts the data dimensions, rearranging the feature maps output by the convolutional layers in terms of dimensions; Softmax is a mathematical function that transforms the raw scores (logits) output by the classification branch into a probability distribution with a sum of 1; Proposal is the final output of RPN, combining which anchor boxes are likely targets (cls_prob) and how to adjust these anchor boxes (bbox_pred), and after a post-processing step (e.g., Non-Maximum Suppression, NMS) to generate a set of high-quality candidate regions to be detected; Region of Interest Pooling (RoI Pooling) is an operation that transforms all feature maps extracted from different RoIs into a fixed-size output (e.g., 7x7x512); Image Information (im_info) is a vector containing basic information about the input image, typically [height, width, ... [scale], scale is the scaling ratio; 1x1 is a convolutional layer with a kernel size of 1x1, 3x3 is a convolutional layer with a kernel size of 3x3, used to change the dimension of features or map them to another space, 18 and 36 are the channel dimensions.

[0052] In an exemplary embodiment, the loss function of the region proposal network is Equation (8): (8); in, The loss value for the region proposal network. anchor point i It is the probability of the target individual. anchor point i The tag, For the parameters of the prediction box, For the parameters of the calibration box, For classification loss function, For regression loss function, This indicates that the regression is performed only on samples with positive anchor points. This represents the number of cells in the feature map after convolution in a batch of data. For the number of anchor points, These are the balancing parameters.

[0053] Specifically, the region proposal network is trained end-to-end using backpropagation and stochastic gradient descent. A loss function for a patch is computed by randomly sampling 256 anchor points from an image, with a 1:1 ratio of positive to negative anchor points. If an image has fewer than 128 positive samples, it is padded with negative samples. The network is initialized using a pre-trained ImageNet classification model, with weights initialized using a Gaussian distribution with a standard deviation of 0.01 and a mean of 0.

[0054] Based on the classification and regression probability values ​​obtained from the RPN network, the loss function is constructed as shown in Equation (8). To minimize the loss value, the Adam algorithm is used as the optimization function of the model. The learning rate for the 60k mini-batch is 0.001, and the learning rate for the next 20k mini-batch is 0.0001. A step size of 0.9 and a decay of 0.0005 are used.

[0055] S102: Construct a human target relationship graph using the target individual as a node and the connection relationship between nodes as an edge, and input the human target relationship graph into a multi-layer graph convolutional network.

[0056] In the human target relationship diagram G middle, V It is a set of nodes. V = { |t=1, . . . , T, i=1, . . . ,N}, E It is the set of edges, and , Representing nodes respectively i Feature representations before and after the l-th convolutional layer. Node iGraph-based convolutional propagation can be divided into two steps.

[0057] S103: In each layer of the graph convolutional network, calculate the cosine distance between the behavioral features of a node and the behavioral features of its neighboring nodes in the human target relationship graph to determine the connection weights between the node and its neighboring nodes. Update the behavioral features of the node based on the connection weights between the node and its neighboring nodes, and pass the updated behavioral features to the next layer of the graph convolutional network.

[0058] First, using a learnable parameter matrix For nodes i The feature representation is transformed; secondly, the transformed nodes are fused. i and its neighboring nodes j ∈N(i), plus a non-linear function (e.g., the Sigmoid function). If all nodes are represented as matrices. The convolution operation is given by formula (9):

[0059] (9); in, , A Given the adjacency matrix of the relational graph, if the nodes i and nodes j If they are connected, then the nodes j For nodes i The adjacent nodes, A The corresponding position is 1 if it is in the middle, otherwise it is 0. I It is a self-connected identity matrix. It is a nonlinear function. , For the first l +1 feature dimension of the convolutional layer for l Feature dimensions of convolutional layers For the first l Features of +1 convolutional layers For the first l Features of convolutional layers This is the adjacency matrix between pedestrians in the graph model. R It indicates the real number field, meaning that the parameter belongs to the range of real numbers.

[0060] The loss function of the graph convolutional neural network is given by formula (10): (10); in, For the loss of a graph convolutional neural network, , The total loss for group behavior, For the true tags of group activities, Predictive labels for group activities For the loss of the first classification group, For the first m The losses of each classification group, The parameters to be learned , For losses of individual activities, As a predictive label for individual activities, As a true label for individual activities, For the first i Individual activity loss in each classification group k For the number of people in each classification group, For the first j Authentic labels of individual activities of each group member For the first j Predictive labels for individual activities of group members m For the number of classification groups, k This represents the number of members in the classification group.

[0061] i Add with other nodes j The new feature after the correlation degree is formula (11): (11); in, For nodes j In the l+1 Features of convolutional layers For nodes j In the l Features of convolutional layers These are learnable weight parameters. The total number of nodes. for i Other nodes j The relationship between them This is a learnable message function, typically implemented using a linear transformation (fully connected layer) or a small neural network. Its input is a node. j eigenvectors, For nodes j In the l Characteristics of convolutional layers.

[0062] The behavioral feature vectors of nodes need to be normalized, that is, their magnitude is adjusted to 1 to eliminate the influence of vector length on similarity calculation, so that the attention mechanism can focus more on the directional differences of features. After normalization, the cosine similarity between the feature vectors of two nodes can be directly obtained by calculating the dot product of the nodes. For a normalized vector, its dot product is numerically equal to the cosine similarity between nodes. The Softmax function is used to normalize the calculated cosine similarity of all neighboring nodes, converting it into a probability distribution, thereby obtaining the attention score as the connection weight.

[0063] S104: Calculate the relationship feature vector between any two nodes in the human target relationship graph based on the behavioral features obtained from the last layer of the graph convolutional network, and divide the graph into multiple sub-target interaction graphs based on the relationship feature vector; each sub-target interaction graph corresponds to a subgroup.

[0064] In an exemplary embodiment, multiple sub-target interaction graphs are divided based on relation feature vectors, specifically including: when the relation feature vector is greater than 0, determining that two nodes belong to the same behavior category; when the relation feature vector is equal to 0, determining that two nodes do not belong to the same behavior category; and constructing multiple sub-target interaction graphs through nodes that belong to the same behavior category.

[0065] node i With nodes j The relational feature vector is calculated using formula (12): (12); in, For nodes i With nodes j In the l Layer relational feature vectors For node-based i behavioral characteristics Calculate the representative node i Scalar value of query intent Based on nodes j behavioral characteristics Calculate representative node j The scalar value of the queried value.

[0066] In function f middle, This invention only considers functions of linear embedding form. g , g For formula (13): (13); in, It is the weight matrix to be learned, either a spatial 1×1 convolution or a spatiotemporal 1×1×1 convolution.

[0067] If let The update formula for the behavior characteristics of nodes is formula (14): (14); in, For nodes i The new feature map after global context information enhancement The weight matrix is ​​a learnable matrix. For the aggregated nodes i Global context information, For nodes i initial characteristics, This is the transpose representation of the learnable weight matrix. This is a learnable weight matrix.

[0068] This invention employs Graph Convolutional Networks (GCNs) for graph reasoning, using the human target interaction graph G as input to the GCN, and determining the feature vector of the relationship between two nodes. Whether the value is zero determines whether a connection edge exists between two target nodes. If the relationship value between the two nodes is zero... =0, then determine i and j They are independent of each other and do not belong to the same category of behavior; conversely, if >0, then it is considered i and j For interactions belonging to the same behavior category, an interaction graph is divided into multiple interaction subgraphs, each containing a behavior. After passing through multiple layers of GCN, the resulting sub-target interaction graph is output. , m This indicates the number of subgroups contained in a scene.

[0069] For a target node in the graph i Based on the weights of adjacent edges, the features of all adjacent nodes are aggregated. Formally, a single-layer GCN structure is represented by formula (15):

[0070] (15); in, It is a matrix representation of a graph. It is the first l Feature representation of layer nodes, For the first l The feature representation of the +1 layer node, and , It is the first l The layer can learn a weight matrix, where σ(∙) is an activation function, and in this invention, the ReLU function is used. The layered propagation structure can stack multiple layers of GCN.

[0071] S105: Classify each sub-target interaction graph using a classifier to obtain the predicted behavior label for each subgroup; determine the predicted behavior label of the subgroup with the highest probability of occurrence in all keyframes as the group behavior label of the video to be identified.

[0072] The sub-interaction graph output after passing through GCN Each subgraph is represented by a one-dimensional vector. To indicate, will As input to the softmax classifier, the behavior score for each subgroup and the predicted behavior label for that subgroup are obtained. .

[0073] The loss function of the graph convolutional neural network is given by formula (16): (16); in, For the loss of a graph convolutional neural network, , The total loss for group behavior, For the true tags of group activities, Predictive labels for group activities For the loss of the first classification group, For the first m The losses of each classification group, The parameters to be learned , For losses of individual activities, As a predictive label for individual activities, As a true label for individual activities, For the first i Individual activity loss in each classification group k For the number of people in each classification group, For the first k Authentic labels of individual activities of each group member For the first k Predictive labels for individual activities of group members m For the number of classification groups, k This represents the number of members in the classification group.

[0074] A video clip is randomly selected from the test set as an input sample and fed into the trained network to obtain individual behavioral features. , k The total number of human subjects is given. Then, a softmax classifier is used to classify the behavioral features, obtaining the probability value for each behavior. The behavior label corresponding to the highest probability value is used as the predicted label for the human target.

[0075] Using relationship graphs for group behavior identification, based on Determine the behavior for each subgroup. Finally, the behavior that contains the most subgroups in the scene is taken as the final group behavior.

[0076] This invention addresses the problem of group behavior recognition in video sequences using graph convolutional networks. First, keyframes are extracted from the video frame sequence to reduce redundancy. Second, a convolutional neural network (CNN) is used to identify and extract each volumetric feature in each frame. Third, a relationship graph between individuals is established, and a graph convolutional network (GCN) is used to fuse and infer features. Finally, based on the spatial features of each frame, the video sequence is input into an LSTM to extract temporal features, and its softmax classifier classifies the previously fused spatiotemporal features, thus completing individual and group behavior recognition.

[0077] In one exemplary embodiment, Figure 7 The flowchart of the group behavior recognition method provided by the present invention is shown.

[0078] The core innovations of this invention are threefold: (1) Use semantic hashing to encode video summaries to improve behavior recognition rate.

[0079] In video sequences used for action recognition, each frame has different temporal importance. For most videos, the completion of an action from start to finish results in numerous redundant frames, impacting the performance of action recognition. Therefore, it is essential to select video frames that are information-rich and have application value. Keyframe selection is primarily considered from two aspects. First, the keyframe itself must be of high quality, such as clear, unblurred images and a sufficient number and even distribution of feature points. Second, the relationship between keyframes must be considered. Appropriate co-occurrence relationships and new feature points should exist, while minimizing information redundancy. Based on this, this patent mainly employs semantic hashing encoding, which transforms frame images through hash learning. Simultaneously, to avoid the "semantic gap" phenomenon, semantic features of the extracted images are fused.

[0080] (2) Add a global extension module to the graph convolutional network to increase the receptive field of the model.

[0081] In processing video images using graph convolutional networks, each individual element in the image is treated as a node. Typically, an adjacency matrix is ​​built around each node and its first-order neighborhood to fuse input features. While this establishes a graph of relationships between points, it suffers from a small receptive field, learning only local features and failing to fully utilize the information between points to learn global features. Therefore, a global extension module is added to the model to calculate the proximity between any two points. This elevates the receptive field from the first-order neighborhood to a higher-order neighborhood, not only increasing the model's receptive field but also making the model more flexible.

[0082] (2) The addition of dynamic connections solves the weight sharing problem in graph convolutional networks and improves the recognition accuracy of the model.

[0083] To enable graph convolutional networks (CNNs) to be applied to arbitrary topologies, the kernel function W shares weights across all edges in the graph. This results in underutilization of adjacent edges and the overall internal structure of the graph. Considering the ability of a single convolutional kernel in a CNN to extract multi-dimensional features, a separate weight vector 'a' is first learned for each position, and then the different vectors 'a' are combined using a shared weight matrix W. Therefore, in graph convolutional networks, solving the weight sharing problem involves assigning different weights to the connections between nodes. This invention primarily employs a method of calculating the cosine distance scalar between a point and its neighboring nodes to dynamically update the connection weights.

[0084] When applying the group behavior recognition method based on Markov decision process and extended graph convolution provided by this invention, it is not necessary to rely on... Figure 1 The steps shown are executed in sequence. The specific execution order of each step can be determined as needed, and this invention does not impose any restrictions on it.

[0085] The above describes a group behavior recognition method based on hash encoding and extended graph convolution, provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding group behavior recognition device based on Markov decision processes and extended graph convolution, such as... Figure 8 As shown.

[0086] Figure 8 A schematic diagram of a group behavior recognition device based on Markov decision process and extended graph convolution provided by the present invention includes: The extraction module 801 is used to extract a preset number of key frames from the video to be identified based on the Markov decision process. For each key frame, the convolutional feature map of the key frame is extracted sequentially through a convolutional neural network, the bounding boxes of multiple target individuals in the convolutional feature map are extracted through a region proposal network, and the behavioral features of each target individual are extracted through region pooling of interest.

[0087] Module 802 is used to construct a human target relationship graph with the target individual as a node and the connection relationship between the nodes as an edge, and input the human target relationship graph into a multi-layer graph convolutional network.

[0088] The graph convolution module 803 is used to calculate the cosine distance between the behavioral features of nodes and the behavioral features of adjacent nodes in each layer of the graph convolutional network, so as to determine the connection weight between the node and the adjacent node, update the behavioral features of the node based on the connection weight between the node and the adjacent node, and pass the updated behavioral features to the next layer of the graph convolutional network.

[0089] The partitioning module 804 is used to calculate the relationship feature vector between any two nodes in the human target relationship graph based on the behavioral features obtained from the last layer of the graph convolutional network, and to partition multiple sub-target interaction graphs based on the relationship feature vector; each sub-target interaction graph corresponds to a subgroup.

[0090] The determination module 805 is used to classify each sub-target interaction graph through a classifier to obtain the predicted behavior label of each subgroup; the predicted behavior label of the subgroup with the highest probability of occurrence in all keyframes is determined as the group behavior label of the video to be identified.

[0091] Specific limitations regarding the group behavior recognition device based on Markov decision processes and extended graph convolutions can be found in the limitations of the group behavior recognition method based on Markov decision processes and extended graph convolutions mentioned above, and will not be repeated here. Each module in the aforementioned group behavior recognition device based on Markov decision processes and extended graph convolutions can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0092] The present invention also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 The proposed method is based on Markov decision process and extended graph convolution for group behavior recognition.

[0093] The present invention also provides Figure 9 The schematic diagram of the computer device shown is as follows: Figure 9 As shown, at the hardware level, this computer device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then executes it to achieve the above. Figure 1 The proposed method is based on Markov decision process and extended graph convolution for group behavior recognition.

[0094] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0095] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this invention.

Claims

1. A method for group behavior recognition based on Markov decision processes and extended graph convolution, characterized in that, include: Based on the Markov decision process, a preset number of key frames are extracted from the video to be identified. For each key frame, the convolutional feature map of the key frame is extracted by a convolutional neural network, the bounding boxes of multiple target individuals in the convolutional feature map are extracted by a region proposal network, and the behavioral features of each target individual are extracted by region pooling. Using the target individual as nodes and the connection relationships between nodes as edges, a human target relationship graph is constructed, and the human target relationship graph is input into a multi-layer graph convolutional network; In each layer of the graph convolutional network, the cosine distance between the behavioral features of a node and the behavioral features of its neighboring nodes in the human target relationship graph is calculated to determine the connection weight between the node and its neighboring nodes. The behavioral features of the node are updated based on the connection weight between the node and its neighboring nodes, and the updated behavioral features are passed to the next layer of the graph convolutional network. Based on the behavioral features obtained from the last layer of the graph convolutional network, the relationship feature vector between any two nodes in the human target relationship graph is calculated, and multiple sub-target interaction graphs are divided based on the relationship feature vector; each sub-target interaction graph corresponds to a subgroup. The interaction graph of each sub-target is classified by a classifier to obtain the predicted behavior label of each subgroup; the predicted behavior label of the subgroup with the highest probability of occurrence in all keyframes is determined as the group behavior label of the video to be identified.

2. The method as described in claim 1, characterized in that, The step of extracting a predetermined number of keyframes from the video to be identified based on a Markov decision process specifically includes: The video to be identified is uniformly sampled to obtain the frame sequence of the video to be identified; Visual features of each frame in the frame sequence are extracted using a deep convolutional neural network, and observation probabilities are generated based on the visual features of all frames. The keyframe extraction problem is defined as the decoding problem of a hidden Markov model; the state transition probabilities of the Markov chain represent the temporal continuity between frames; Using the Viterbi algorithm, under the joint constraints of the observation probability and the state transition probability, the globally optimal keyframe sequence is calculated; The globally optimal keyframe sequence is determined as the selected frames; Based on the global information of the video to be identified and the selected frames, a preset number of keyframes of the video to be identified are extracted through a pre-trained keyframe extraction network.

3. The method as described in claim 2, characterized in that, The keyframe extraction network includes a first branch and a second branch connected in parallel, a third fully connected layer, and a softmax activation function; the first branch includes multiple sets of convolutional layers and pooling layers connected in series, as well as a first fully connected layer; the second branch includes a second fully connected layer; the extraction of a preset number of keyframes from the video to be identified based on global information of the video to be identified and the selected frames, using a pre-trained keyframe extraction network, specifically includes: The global information of the video to be identified and the selected frames are input into the first branch for convolution and pooling processing to obtain the result of the first branch; The binary mask of the selected frame is input into the second branch for processing to obtain the result of the second branch; The results of the first branch and the second branch are concatenated and then input into the third fully connected layer; The output of the third fully connected layer is regularized using the softmax activation function to obtain a preset number of keyframes.

4. The method as described in claim 1, characterized in that, The training sample images of the region proposal network include positive and negative anchor points; the loss function of the region proposal network is: ； in, The loss value for the region proposal network. anchor point i It is the probability of the target individual. anchor point i The tag, For the parameters of the prediction box, For the parameters of the calibration box, For classification loss function, For regression loss function, This indicates that the regression is performed only on samples with positive anchor points. This represents the number of cells in the feature map after convolution in a batch of data. For the number of anchor points, These are the balancing parameters.

5. The method as described in claim 1, characterized in that, The formula for calculating the relation feature vector is as follows: ； in, For nodes i With nodes j In the l Layer relational feature vectors For node-based i behavioral characteristics Calculate the representative node i Scalar value of query intent Based on nodes j behavioral characteristics Calculate representative node j The scalar value of the queried value.

6. The method as described in claim 1, characterized in that, The method of dividing the interaction graph into multiple sub-targets based on relational feature vectors specifically includes: When the relation feature vector is greater than 0, it is determined that the two nodes belong to the same behavior category; When the relation feature vector is equal to 0, it is determined that the two nodes do not belong to the same behavior category. Multiple sub-goal interaction graphs are constructed by using nodes that belong to the same behavior category.

7. The method as described in claim 1, characterized in that, The formula for updating the behavioral characteristics of a node is: ； in, For nodes i Updated features The weight matrix is ​​a learnable matrix. For the aggregated nodes i global context features For nodes i The initial behavioral characteristics.

8. The method as described in claim 1, characterized in that, The loss function of the multi-layer graph convolutional neural network is: ; in, For the loss of a graph convolutional neural network, , The total loss of group behavior, For the true tags of group activities, Predictive labels for group activities For the loss of the first classification group, For the first m The losses of each classification group, The parameters to be learned , For losses of individual activities, As a predictive label for individual activities, As a true label for individual activities, For the first i Individual activity loss in each classification group k For the number of people in each classification group, For the first j Authentic tags of individual activities for each group member For the first j Predictive labels for individual activities of group members m For the number of classification groups, k This represents the number of members in the classification group.

Images