A volleyball match group behavior recognition method fusing space-time information in a frame loss environment

Abstract

The application belongs to the technical field of computer vision, image processing, group behavior recognition and the like, and discloses a volleyball match group behavior recognition method fusing space-time information under a missing frame environment. The method uses a VGG16 network to process an input volleyball video frame sequence to obtain global features, then inputs the feature vectors and individual bounding boxes into a RoiAlign layer to obtain individual features, inputs the individual features into an inference network to obtain initial group features and individual space-time interaction features, obtains original features, inputs the original features into a space-time Transformer module to model space-time interaction information, and processes the effective inference network module to effectively improve the complexity of discontinuous feature transition. The application can complete overall modeling of space-time dependence, reduce the influence of missing frames, capture complex interaction relationships of individuals, and effectively improve the group behavior recognition capability of the volleyball match.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary fields of computer vision, image processing, and group behavior recognition. Specifically, it relates to a method for recognizing group behavior in volleyball matches by fusing spatiotemporal information in a frame-missing environment. Background Technology

[0002] Group behavior recognition typically refers to the identification of collective activities involving multiple people in the same scene. It has a wide range of applications, including sports event analysis, social scene understanding, and social relationship recognition. Unlike action recognition methods, group behavior recognition relies on spatiotemporal modeling of the interactions between individuals within a scene to obtain a representation of the group activity. Due to various environmental interferences, videos are prone to frame loss, increasing the complexity of spatial and temporal information transitions within the scene. In environments with missing frames, relying solely on the spatiotemporal relationships between individuals cannot accurately determine group behavior because individual behaviors, positions, and interactions are constantly changing and discontinuous.

[0003] Traditional methods involve manually extracting features and then processing them using probabilistic graphical models to infer group activities. Recently, some researchers have used Long Short-Term Memory (LSTM) neural networks to process group behavior from a temporal perspective for identification models, and Recurrent Neural Networks (RNNs) to model individual features, but these methods require significant computational resources. Hierarchical LSTM models only reflect temporal relationships while ignoring spatial relationships between individuals. Later methods began to apply attention-based mechanisms to model individual relationships and infer group activities, such as using attention mechanisms to capture spatial and temporal self-attention separately. Some newer works use standard Transformer encoders to extract features, selectively extracting spatial role relationships without considering temporal dynamics. However, these methods either decompose spatiotemporal information or simply fuse spatiotemporal information without considering spatiotemporal interaction relationships.

[0004] Currently, there is no effective method to solve the problem of group behavior recognition in frame-deficient environments. Therefore, it is particularly important to design a method that can fuse spatiotemporal information in frame-deficient environments. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method for recognizing group behavior in volleyball matches by fusing spatiotemporal information in a frame-missing environment. This method first extracts features from a discontinuous volleyball match video sequence, inputs the feature vectors into an inference network to infer initial individual interaction features to compensate for lost feature information, and finally inputs them into a spatiotemporal Transformer based on a self-attention mechanism to achieve overall modeling of spatiotemporal relationships, thereby improving the ability to recognize group behavior in volleyball matches.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0007] This invention is a method for recognizing group behavior in volleyball matches by fusing spatiotemporal information in a frame-missing environment, comprising the following steps:

[0008] Step 1) Input a non-continuous volleyball match video sequence;

[0009] Step 2) Use the VGG16 network to extract features from the volleyball match video sequence obtained in Step 1. The output of the last layer of the VGG16 model is used as the feature representation and can be used as global features.

[0010] Step 3) Input the individual bounding box information and the obtained global features into the RoiAlign network module. This module will obtain all the individual features in each image based on the input bounding box information.

[0011] Step 4) Input individual features into the inference network to obtain initial group features and refined individual spatiotemporal interaction features. Use the FC layer to perform feature mapping. The result is called the original features.

[0012] Step 5) Input the extracted raw features into the spatiotemporal Transformer module to model the spatiotemporal interaction information and complete the overall modeling of spatiotemporal dependencies;

[0013] Step 6) uses residual connections to fuse the final output features and the original features in steps 5) and 4), and uses a group classification layer to obtain the final recognition result.

[0014] A further improvement of the present invention is that step 3) extracting individual features from the image specifically includes the following steps:

[0015] Step 31) Input the image and region coordinates, and crop out the corresponding sub-regions from the convolutional feature map according to the size and position of RoiAlign;

[0016] Step 32) Divide the RoiAlign sub-region into a grid as in Step 31), calculate the position of each grid, and calculate the pixel value within the RoiAlign sub-region on each grid using bilinear interpolation;

[0017] Step 33) Use the pixel value in each grid as the feature value of the corresponding position in the RoiAlign sub-region. Stack the feature values ​​of all grids together to form a feature map of a fixed size.

[0018] Step 34) Return the feature map as an individual feature.

[0019] A further improvement of the present invention is that step 4) obtaining the initial group characteristics and the refined individual spatiotemporal interaction characteristics includes the following steps:

[0020] Step 41) First, the network initializes a spatiotemporal interaction graph based on spatial location information and time step. The regions adjacent to each individual in the graph are called spatiotemporal interaction domains.

[0021] Step 42) As the network's inference process continues, a relation matrix will be predicted for the interaction domains of individuals in the graph to represent the interaction relationships, and the output features of the inference module will be... It includes spatiotemporal information; during the reasoning process, the relationship between two individuals is calculated through dot product and normalization, and the relationships between all individuals in the interaction domain of a given individual are updated. Finally, it uses... This represents the output characteristic. The formula is as follows:

[0022]

[0023]

[0024]

[0025] Where R t,k Indicates individual relationships, X t W represents the time characteristic. k This represents the weight parameter matrix, K is the size of the interaction domain, σ represents the ReLU activation function, and r i,j w represents a pair of individual relationships. i with w j The weight parameter matrix, and For individual characteristics, R i,j This represents the relationship between individuals, where N represents the number of individuals;

[0026] Step 43) Obtain the initial population feature representation through global spatiotemporal pooling.

[0027] A further improvement of the present invention is that step 5) includes the following steps:

[0028] Step 51) Before inputting the original features, position encoding is performed to embed position information into the input sequence. The center point of the bounding box represents the spatial position information of each participant, and the spatial position is encoded using the PE function, as shown in the following formula;

[0029]

[0030]

[0031] Where PE is a two-dimensional matrix, pos represents the position of the participant, D represents the feature dimension of the participant, and i is the vector dimension. Then, the position encoding is directly concatenated with the original feature to represent the input.

[0032] Step 52) Input the features containing location information into the spatial Transformer network module to model the spatial and appearance features of the individual. Each layer in the spatial Transformer network has a multi-head attention layer and a feedforward layer. The calculation formula for one of the attention heads is as follows:

[0033]

[0034]

[0035]

[0036] Y s =U s +M *s

[0037] in All dimensions are R D×D By applying a shared three-layer feedforward neural network to each node, predictions based on learned node features are improved, as shown in equation M. *s =X *s +M s It is a residual connection. It is a three-layer weight matrix, and finally the features are fused to obtain Y. s S s U represents a dynamic spatial dependency. s Representing spatial characteristics, Y s Q represents the final fusion feature. s K s and V s Represents three matrices: query, key, and value;

[0038] Step 5-3: Perform max pooling on the features containing location information to obtain temporal features, then input them into the temporal Transformer network module to complete the modeling of spatiotemporal information. The temporal Transformer network consists of multiple layers, each with two sub-layers: a multi-head attention layer and a feedforward layer. The calculation formula for one of the attention heads is as follows:

[0039] X *t =G t ([X T D T ])∈R T×N×D

[0040]

[0041]

[0042]

[0043] Y T =U T +M *T

[0044] Where: X *t G represents the temporal characteristics. t It is a 1×1 convolutional layer, S T U represents dynamic temporal dependencies. T Representing time series characteristics, Y T Q represents the final fusion feature. T K T and V T G represents three matrices: query, key, and value. t The convolutional layer generates a D-dimensional vector at each time point and uses a self-attention mechanism to model the temporal dependency. In each layer of the temporal Transformers, long-distance bidirectional temporal dependencies can be effectively obtained, and it can be easily extended to long sequences while maintaining high computational efficiency.

[0045] Step 54) The last layer adds a decoder to enhance the group behavior representation. The input is the initial group feature representation output by the inference network. The fusion of the output results realizes the prediction of group behavior. The decoder follows the standard Transformer decoder.

[0046] A further improvement of the present invention is that step 6) includes:

[0047] The output of the spatiotemporal Transformer is fused with the output of the inference network to obtain the final feature representation. The individual loss and the group loss together constitute the loss function, and its formula is as follows:

[0048]

[0049] Where L1 and L2 represent the cross-entropy loss function, y g and y a Represents the prediction scores for group behavior and individual behavior. and These are labels representing different real-world behaviors.

[0050] The beneficial effects of this invention are as follows: First, a discontinuous volleyball match video sequence is acquired. The acquired video sequence is then used to extract features using a backbone network VGG16. The extracted feature vectors are input into an inference network to infer initial individual interaction features to compensate for lost feature information. Finally, the vectors are input into a spatiotemporal Transformer based on a self-attention mechanism to achieve overall modeling of spatiotemporal relationships. Using this model for group behavior recognition can jointly extract the spatiotemporal information of participants, ensuring the consistency of the participants' spatiotemporal information, effectively enhancing the individual and group representations of volleyball matches, and improving the accuracy of group behavior recognition in volleyball matches under frame-missing conditions.

[0051] Specifically:

[0052] (1) In this invention, the VGG network is used to extract features from the initial video sequence. The output of the last layer of the model is used as the feature representation, and the RoiAlign layer is used to obtain all individual features in each image based on the input bounding box information.

[0053] (2) This invention uses an inference network module to process missing frame data. The inference network combines this information with historical data to perform inference and predict future group behavior. Through network inference, lost feature information can be compensated, and individual interaction features and initial group behavior features with spatiotemporal information can be obtained.

[0054] (3) This invention uses a spatiotemporal Transformer based on a self-attention mechanism to achieve overall modeling of spatiotemporal relationships. Features with spatiotemporal information are used as inputs to the spatial Transformer and the temporal Transformer to complete the overall modeling of spatiotemporal dependencies, so as to capture the complex interaction relationships of individuals. Attached Figure Description

[0055] Figure 1 This is a flowchart of the volleyball match group behavior recognition method of the present invention.

[0056] Figure 2 This is a schematic diagram of the spatial Transformer principle of this invention. Detailed Implementation

[0057] The embodiments of the present invention will be disclosed below with reference to the drawings. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details are not intended to limit the invention. That is, in some embodiments of the invention, these practical details are not essential.

[0058] like Figure 1-2 As shown, this invention is a multi-target tracking and recognition method based on nonlocal gait features, comprising the following steps:

[0059] Step 1: Input a non-continuous volleyball match video sequence.

[0060] Step 2: Use the VGG16 network to extract features from the volleyball match video sequence obtained in Step 1. The output of the last layer of the VGG16 model is used as the feature representation and can be used as global features.

[0061] Step 3: Input the individual bounding box information and the global features obtained in Step 2 into the RoiAlign network module. This module will obtain all the individual features in each image based on the input bounding box information.

[0062] The RoiAlign network module extracts individual features from the graph through the following steps:

[0063] Step 31: Input the image and region coordinates, and crop out the corresponding sub-regions from the convolutional feature map according to the size and position of RoiAlign;

[0064] Step 32: Divide the RoiAlign sub-region into grids and calculate the position of each grid. Calculate the pixel value within the RoiAlign sub-region on each grid using bilinear interpolation.

[0065] Step 33: Use the pixel value in each grid as the feature value of the corresponding position in the RoiAlign sub-region. Stack the feature values ​​of all grids together to form a feature map of a fixed size.

[0066] Step 34: Return the feature map and use it as an individual feature.

[0067] Step 4: Input individual features into the inference network to obtain initial group features and refined individual spatiotemporal interaction features. Use the FC layer to perform feature mapping, and the result is called the original features.

[0068] The specific steps involved in inputting individual features into the inference network to obtain initial group features and refined individual spatiotemporal interaction features are as follows:

[0069] Step 41: First, the network initializes a spatiotemporal interaction graph based on spatial location information and time step. The regions adjacent to each individual in the graph are called spatiotemporal interaction domains.

[0070] Step 42: As the network's inference process continues, a relation matrix will be predicted for the interaction domains of individuals in the graph to represent the interaction relationships. Furthermore, the output features of the inference module... It includes spatiotemporal information; during the reasoning process, the relationship between two individuals is calculated through dot product and normalization, and the relationships between all individuals in the interaction domain of a given individual are updated. Finally, it uses... This represents the output characteristic. The formula is as follows:

[0071]

[0072]

[0073]

[0074] Where R t,k Indicates individual relationships, X t W represents the time characteristic. k This represents the weight parameter matrix, K is the size of the interaction domain, σ represents the ReLU activation function, and r i,j w represents a pair of individual relationships. i with w j The weight parameter matrix, and For individual characteristics, R i,j This represents the relationship between individuals, where N represents the number of individuals;

[0075] Step 43: Obtain the initial population feature representation through global spatiotemporal pooling.

[0076] Step 5: Input the extracted raw features into the spatiotemporal Transformer module to model the spatiotemporal interaction information, completing the overall modeling of spatiotemporal dependencies. This includes the following steps:

[0077] Step 51: Before inputting the original features, perform position encoding to embed the position information into the input sequence. The center point of the bounding box represents the spatial position information of each participant, and the spatial position is encoded using the PE function, as shown in the following formula;

[0078]

[0079]

[0080] Where PE is a two-dimensional matrix, pos represents the participant's position, D represents the feature dimension of the participant, and i is the vector dimension. Then, the position encoding is directly concatenated with the original features to represent the input.

[0081] Step 52: Input the features containing location information into the spatial Transformer network module to model the spatial and appearance features of the individual. Each layer in the spatial Transformer network has a multi-head attention layer and a feedforward layer. The calculation formula for one of the attention heads is as follows:

[0082]

[0083]

[0084]

[0085] Y s =U s +M *s

[0086] in All dimensions are R D×D By applying a shared three-layer feedforward neural network to each node, predictions based on learned node features are improved, as shown in equation M. *s =X *s +M s It is a residual connection. It is a three-layer weight matrix, and finally the features are fused to obtain Y. s S s U represents a dynamic spatial dependency. s Representing spatial characteristics, Y s Q represents the final fusion feature. s K s and V s Represents three matrices: query, key, and value;

[0087] Step 5-3: Perform max pooling on the features containing location information to obtain temporal features, then input them into the temporal Transformer network module to complete the modeling of spatiotemporal information. The temporal Transformer network consists of multiple layers, each with two sub-layers: a multi-head attention layer and a feedforward layer. The calculation formula for one of the attention heads is as follows:

[0088] X *t =G t ([X T D T ])∈R T×N×D

[0089]

[0090]

[0091]

[0092] Y T =U T +M *T

[0093] Where: X *t G represents the temporal characteristics. t It is a 1×1 convolutional layer, S T U represents dynamic temporal dependencies. T Representing time series characteristics, Y T Q represents the final fusion feature.T K T and V T G represents three matrices: query, key, and value. t The convolutional layer generates a D-dimensional vector at each time point and uses a self-attention mechanism to model the temporal dependency. In each layer of the temporal Transformers, long-distance bidirectional temporal dependencies can be effectively obtained, and it can be easily extended to long sequences while maintaining high computational efficiency.

[0094] Step 54: The final layer adds a decoder to enhance the group behavior representation. The input is the initial group feature representation output by the inference network. The fusion of the output results enables the prediction of group behavior. The decoder follows the standard Transformer decoder.

[0095] Step Six: Residual connections are used to fuse the final output features and the original features. A group classification layer is then used to obtain the final recognition result. Specifically, the output of the spatiotemporal Transformer is fused with the output of the inference network to obtain the final feature representation. Individual loss and group loss together constitute the loss function, and its formula is as follows:

[0096]

[0097] Where L1 and L2 represent the cross-entropy loss function, y g and y a Represents the prediction scores for group behavior and individual behavior. and These are labels representing different real-world behaviors.

[0098] This invention can perform overall modeling of spatiotemporal dependencies, reduce the impact of missing frames, capture complex interactions between individuals, and effectively improve the ability to recognize group behaviors in volleyball matches.

[0099] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A method for recognizing group behavior in volleyball matches by fusing spatiotemporal information in a frame-missing environment, characterized in that: The identification method includes the following steps: Step 1: Input a non-continuous volleyball match video sequence; Step 2: Extract features from the volleyball match video sequence obtained in Step 1 using the VGG16 model. The output of the last layer of the VGG16 model is used as the feature representation, which is used as the global feature. Step 3: Input the individual bounding box information and the obtained global features from Step 2 into the RoiAlign network module. The RoiAlign network module obtains all individual features in each image based on the input individual bounding box information. Step 4: Input the individual features obtained in Step 3 into the inference network to obtain the initial group features and the refined individual spatiotemporal interaction features. Use a fully connected layer (FC) to perform feature mapping to obtain the original features. Step 5: Input the raw features extracted in Step 4 into the spatiotemporal Transformer module to model the spatiotemporal interaction information, complete the overall modeling of spatiotemporal dependencies, and obtain the output features; Step 6: Residual connections are used to fuse the output features obtained in Step 5 and the original features obtained in Step 4, and a group classification layer is used to obtain the final recognition result; where: Step 4, obtaining the initial group features and refined individual spatiotemporal interaction features, includes the following steps: Step 4-1: First, the inference network initializes a spatiotemporal interaction graph based on spatial location information and time step. The regions adjacent to each individual in the graph are called spatiotemporal interaction domains. Step 4-2: As the inference network runs, it predicts a relation matrix representing the interaction relationships for the interaction domains of individuals in the graph. The output features of the inference module are... Incorporating spatiotemporal information, the relationship between two individuals is calculated through dot product and normalization during the inference process. This updates the relationships between all individuals within the interaction domain of a given individual, and finally... The characteristics of the output are represented by the following formula: in Indicates individual relationships. Indicates time characteristics, This represents the weight parameter matrix, where K is the size of the interaction domain. Represents the ReLU activation function. It indicates a pair of individual relationships. and The weight parameter matrix, and For individual characteristics, This represents the relationship between individuals, where N represents the number of individuals; Step 4-3: Obtain the initial population feature representation through global spatiotemporal pooling; Step 5 includes the following steps: Step 5-1: Before inputting the original features, perform positional encoding to embed positional information into the input sequence. The center point of the bounding box represents the spatial position information of each participant, and the PE function is used to encode the spatial position, as shown in the following formula: Where PE is a two-dimensional matrix, pos represents the position of the participant, D represents the feature dimension of the participant, and i is the vector dimension. Then, the position encoding is directly concatenated with the original feature to represent the input. Step 5-2: Input the features containing location information from Step 5-1 into the spatial Transformer network module to model the spatial and appearance features of the individual. Each layer in the spatial Transformer network has a multi-head attention layer and a feedforward layer. The calculation formula for one of the attention heads is as follows: in , , All dimensions are Applying a shared three-layer feedforward neural network to each node improves predictions based on learned node features, equation It is a residual connection. It is a three-layer weight matrix, and finally the features are fused to obtain the result. , Represents dynamic spatial dependencies. Represents spatial characteristics, This indicates the final fusion characteristics. and Represents three matrices: query, key, and value; Step 5-3: Perform max pooling on the features containing location information to obtain temporal features, then input them into the temporal Transformer network module to complete the modeling of spatiotemporal information. The temporal Transformer network consists of multiple layers, each with two sub-layers: a multi-head attention layer and a feedforward layer. The calculation formula for one of the attention heads is as follows: in: Representing temporal characteristics, It is convolutional layers, Indicates dynamic temporal dependencies. Represents temporal characteristics, This indicates the final fusion characteristics. , and This represents three matrices: query, key, and value. A D-dimensional vector is generated for each convolutional layer at each time point, and a self-attention mechanism is used to model the temporal dependency. Step 5-4: The last layer adds a decoder to enhance the group behavior representation. The input is the initial group feature representation output by the inference network. The fusion of the output results realizes the prediction of group behavior. The decoder follows the standard Transformer decoder. Step 6 uses residual connections to fuse the output features and the original features, and uses a population classification layer to obtain the final recognition result. The output of the spatiotemporal Transformer is fused with the output of the inference network to obtain the final feature representation. The individual loss and the group loss together constitute the loss function, and its formula is as follows: in, and Represents the cross-entropy loss function. and Represents the prediction scores for group behavior and individual behavior. and These are labels representing different real-world behaviors.

2. The method for recognizing group behavior in volleyball matches by fusing spatiotemporal information in a frame-missing environment as described in claim 1, characterized in that: Step 3, extracting all individual features from the image, specifically includes the following steps: Step 3-1: Input the image and region coordinates, and crop out the corresponding sub-regions from the convolutional feature map according to the size and position of the RoiAlign network module; Step 3-2: Divide the sub-regions cropped in Step 3-1 into grids and calculate the position of each grid. Calculate the pixel values ​​within the RoiAlign sub-regions on each grid using bilinear interpolation. Step 3-3: Use the pixel value in each grid obtained in Step 3-2 as the feature value of the corresponding position in the RoiAlign sub-region. Stack the feature values ​​of all grids together to form a feature map. Step 3-4: Return the feature map from step 3-3 and use it as individual features.

Images