Video crowd grouping method considering number of groups and relation matrix between individuals

Abstract

The invention provides a video crowd grouping method considering the number of groups and a relation matrix between individuals. In one population, a plurality of groups without individual overlapping exist. In a period of time, the individuals in the same group have similar behaviors, and the position information of the individuals has time consistency. The method mainly comprises the following two aspects: firstly, in continuous video frames, extracting action sequence features and motion track features of individuals in a crowd, and representing behavior information of the individuals by using combined features of the action sequence features and the motion track features; secondly, a graph is used as a representation mode of the crowd, nodes correspond to individuals, action sequence features and motion trail features are used as attributes and edges of the nodes to represent relations among the individuals, a similarity matrix among the individuals is constructed, and prediction of the group number in the crowd and prediction of a relation matrix among the individuals are achieved through a feed-forward network; and video crowd grouping is completed by using an unsupervised clustering method. On a Social-CAD data set, the Lam coefficient of a grouping result reaches 0.8113.

Description

Technical Field

[0001] The present invention belongs to the field of video analysis in computer vision, and in particular relates to a video crowd grouping method that considers the number of groups and a relationship matrix between individuals. Background Art

[0002] Video crowd grouping is one of the important research topics in the field of video analysis. In crowd behavior analysis, the crowd in the video is usually considered as a whole and the behavior of the crowd as a whole is analyzed. However, the behavior of some individuals is often inconsistent with the behavior of the crowd, which affects the performance of crowd behavior analysis. In contrast, when the crowd is divided into groups, individuals in the same group can often show consistent behavior.

[0003] A group is an information unit between an individual and a crowd. In a crowd, there are usually several groups without individual overlap. Current crowd grouping methods are mainly divided into two categories: one is to select feature points on the video frame, track them, and group them according to the position and movement speed of the feature points. The method based on feature points for grouping does not fully utilize the temporal and spatial information of individuals in the video, ignores the importance of individual behavior, and cannot perform subsequent analysis of the behavior. The other category is to study individual behavior in the video, extract behavioral information from the video, and perform clustering and grouping based on this information. This grouping method often requires setting the number of groups, and the selection of the number of groups has a great influence on the grouping results.

[0004] In summary, the existing video crowd grouping methods have much room for improvement and enhancement. Summary of the invention

[0005] The purpose of the present invention is to provide a video crowd grouping method that takes into account the number of groups and the relationship matrix between individuals, aiming to solve the problems of insufficient extraction of individual behavior information features in existing video crowd grouping methods, the need to set the number of groups and the inability to dynamically group people, etc.

[0006] To solve the above problems, the present invention adopts the following technical solutions:

[0007] A video crowd grouping method considering the number of groups and the relationship matrix between individuals, the method comprising:

[0008] Step 1: Divide the video into frames and determine the individual bounding boxes and unique identifiers of individuals in consecutive video frames.

[0009] Step 2: Extract the apparent feature map of the video frame from the continuous video frames.

[0010] Step 3: Extract the individual's appearance feature map based on the individual's bounding box coordinate information.

[0011] Step 4: Calculate the individual's temporal action sequence features based on the individual's unique identifier and the individual's apparent feature map obtained in step 3. After obtaining the individual's action sequence features, calculate the similarity of the action sequence features between individuals.

[0012] Step 5: Based on the bounding box coordinate information of the individual, construct the individual's motion trajectory data, extract the individual's motion trajectory features, and calculate the motion trajectory similarity between individuals.

[0013] Step 6: Connect the individual's action sequence features and the individual's motion trajectory features, and use the connected combined features as the features to characterize the individual's behavior.

[0014] Step 7: Calculate the initial adjacency weights between individuals based on the similarity of action sequence features and motion trajectories between individuals.

[0015] Step 8: Use individual behavior features as the input of the graph attention network, update the individual behavior features through message passing, and calculate the similarity between the updated individual behavior features after all individual features are updated.

[0016] Step 9: Use the initial adjacency weights between individuals and the updated individual behavior characteristics as inputs to the feedforward neural network to predict the number of groups in the population.

[0017] Step 10: Connect the initial adjacency weights between individuals and the updated similarities between individual behavioral characteristics as inputs of the feedforward neural network to predict the relationship matrix between individuals.

[0018] Step 11: Based on the number of groups and the relationship matrix between individuals, a clustering algorithm is used to obtain the results of population grouping. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a step structure diagram of a video crowd grouping method that takes into account the number of groups and the relationship matrix between individuals of the present invention. DETAILED DESCRIPTION

[0020] A video crowd grouping method considering the number of groups and the relationship matrix between individuals, the method comprising:

[0021] Step 1: Process the video by dividing it into frames, determine the individual bounding boxes and the unique identifiers of the individuals in the continuous video frames, and use the pedestrian target detection and tracking method to determine the coordinate information of the individual bounding boxes and the unique identifiers of the individuals;

[0022] Step 2: In continuous video frames, convert the video frame data into tensor data, input it into the residual network, and extract the apparent feature map of the video frame;

[0023] Step 3: Extract the individual’s apparent feature map based on the individual’s bounding box coordinate information, calculate the individual’s apparent feature map on the current video frame using the individual’s bounding box and the video frame’s apparent feature map, and use RoIAlign to normalize the individual’s feature map size to a uniform size.

[0024] Step 4: According to the unique identification of the individual and the individual appearance feature map in the continuous frames obtained in step 3, the action sequence features of the individual in time are calculated; after the action sequence features of the individual are obtained, the similarity of the action sequence features between the individuals is calculated;

[0025] Step 4.1: Connect the individual appearance feature maps of uniform size in the time dimension, and use (C, H, W) to represent the dimensions of the individual appearance feature map, where C is the number of feature map channels, H is the feature map height, and W is the feature map width.

[0026] Step 4.2: According to the unique identifier of the individual, the appearance feature graphs of the same individual are connected in the time dimension to obtain the action sequence features of the individual. The feature dimension of the action sequence features is (T*C, H, W), where T is the number of frames.

[0027] Step 4.3: Calculate the distance between the action sequence features of different individuals to obtain the similarity of the action sequence features between individuals.

[0028] Step 5: Based on the bounding box coordinate information of the individual, construct the individual's motion trajectory data, extract the individual's motion trajectory features, and calculate the motion trajectory similarity between individuals;

[0029] Step 5.1: The motion trajectory of an individual can be represented by a sequence of continuous bounding box coordinate quadruplets (x1, y1, x2, y2), where x1 represents the horizontal coordinate of the upper left corner of the bounding box, y1 represents the vertical coordinate of the upper left corner of the bounding box, x2 represents the horizontal coordinate of the lower right corner of the bounding box, and y2 represents the vertical coordinate of the lower right corner of the bounding box. N} represents the motion trajectory of an individual, box i Represents the bounding box coordinate quadruple of the individual in the i-th frame, and N represents the number of frames. LSTM is applied to the motion trajectory of the individual to obtain the motion trajectory features of the individual.

[0030] Step 5.2: Calculate the similarity between the motion trajectories of different individuals to obtain the similarity of the motion trajectories between individuals.

[0031] Step 6: Connect the action sequence features of the individual and the motion trajectory features of the individual. The combined features are used as the features that characterize the individual behavior. Connect the action sequence features of the individual and the motion trajectory features of the individual to obtain the individual behavior features. Use a graph structure to describe the crowd, where nodes correspond to individuals and edges represent the connections between individuals. Use individual behavior features as node attributes.

[0032] Step 7: Calculate the initial adjacency weights between individuals based on the similarity of action sequence features and motion trajectories between individuals;

[0033] Formula (1) is used to multiply the action sequence feature similarity between individuals and the motion trajectory similarity between individuals by their corresponding adjustable weights, and the results of the two multiplications are added together to calculate the initial adjacency weights between individuals.

[0034] D int = pD act + qD track (1)

[0035] Among them, p and q are adjustable weights, D int is the adjacency weight matrix between individuals, D act is the action sequence feature similarity matrix between individuals, D track is the motion trajectory similarity matrix between individuals.

[0036] Step 8: Use individual behavior features as the input of the graph attention network, update individual behavior features through message passing, and calculate the similarity between updated individual behavior features after all individual features are updated; use the graph attention network and message passing mechanism to consider the association between individuals and update individual behavior features. After all individual features are updated, calculate the similarity between updated individual behavior features.

[0037] Step 9: Use a feedforward neural network to take the initial adjacency weights between individuals and the updated individual behavior characteristics as inputs of the feedforward neural network to predict the number of groups in the population.

[0038] Step 10: Connect the initial adjacency weights between individuals and the updated similarities between individual behavioral characteristics as inputs of the feedforward neural network to predict the relationship matrix between individuals.

[0039] Step 11: Based on the number of groups in the population and the relationship matrix between individuals, a clustering algorithm is used to obtain the population grouping results.

[0040] In this embodiment, if Figure 1As shown in Figure 1, the video is split into frames, and the residual network is used to extract the appearance features of the video frames. The channel dimension of the video frame appearance feature map is high. The appearance feature map of the video frame is calculated by 1×1 convolution to reduce the number of channels of the video frame appearance feature map and keep the height and width of the video frame appearance feature map unchanged. The individual bounding box on the video frame is determined by detection and tracking methods. The individual bounding box is represented by a four-tuple (x1, y1, x2, y2). In the video frame coordinate system, x1 represents the horizontal coordinate of the upper left corner of the bounding box, y1 represents the vertical coordinate of the upper left corner of the bounding box, x2 represents the horizontal coordinate of the lower right corner of the bounding box, and y2 represents the vertical coordinate of the lower right corner of the bounding box. In order to obtain the individual appearance feature map from the appearance feature map of the video frame, it is necessary to convert the coordinate four-tuple of the bounding box on the video frame into the coordinate four-tuple of the bounding box on the appearance feature map of the video frame. The coordinates of the bounding box are reduced by calculating the reduction ratio of the video frame size to the size of its appearance feature map, as shown in formula (2).

[0041]

[0042] Among them, w represents the width of the video frame appearance feature map, h represents the height of the video frame appearance feature map, w rgb Indicates the width of the video frame, h rgb Indicates the height of the video frame. In the video frame appearance feature map, x1′ represents the horizontal coordinate of the upper left corner of the bounding box, y1′ represents the vertical coordinate of the upper left corner of the bounding box, x2′ represents the horizontal coordinate of the lower right corner of the bounding box, and y2′ represents the vertical coordinate of the lower right corner of the bounding box. The (x1′, y1′, x2′, y2′) quadruple represents the bounding box of the individual appearance feature map on the video frame.

[0043] After obtaining the feature map of the video frame and the coordinates of the bounding box on the feature map, determine the feature map of the individual in the video frame. The steps are as follows:

[0044] First, according to the reduction ratio from the video frame size to the size of the video frame apparent feature map, after reducing the bounding box, the floating point coordinate values ​​of the bounding box are not rounded, and the reduced bounding box is directly divided into n×n rectangular grid areas. 2×2 sampling points are equally taken in each grid area, that is, each grid area is equally divided into 2×2 small area blocks, and each sampling point is the center point of the small area block. Then, the four feature points closest to each sampling point are found respectively, and the feature value of the current sampling point is calculated using the bilinear interpolation method. The same feature value calculation is performed on all sampling points. In each grid, 2×2 feature values ​​will be obtained. Finally, the maximum pooling operation is performed in each grid. After performing the same operation on all grids, the apparent feature map of the bounding box area of ​​n×n size is obtained. A 1×1 convolution operation is performed on the bounding box feature map after the unified size to reduce the channel dimension of the bounding box feature map and obtain the apparent feature map of the individual on the video frame.

[0045] Then, the appearance feature maps of the same individual in different video frames are fused in the channel dimension, and a 1×1 convolution operation is performed on the fused feature map to reduce the channel dimension to 1 to obtain the individual's action sequence features, thereby reducing the number of channels of the individual action sequence features, summarizing the information of the individual's appearance features in the time dimension, and maintaining the contextual coherence of the individual's appearance features in the video. The distance between each pair of individual action sequence features is calculated to obtain the N*N size action sequence feature similarity matrix D act .

[0046] The action sequence features of an individual are extracted through the apparent features of the individual in the video frame. The action sequence features already contain the spatial information of the individual in the scene. Therefore, in order to reduce the reuse of spatial information and reduce the amount of calculation, the continuous bounding box coordinate sequence of the individual in a video is used to represent the individual's motion trajectory.

[0047] The steps of individual motion trajectory representation and individual motion trajectory feature extraction are as follows:

[0048] The continuous bounding box coordinates in the video are numerical data with time series. The continuous bounding box coordinates of the individual in the video are used as the individual's motion trajectory. The bounding box of the individual on the video frame is represented by a (x1, y1, x2, y2) quaternary group. The individual's motion trajectory is represented by a continuous sequence of bounding box coordinate quaternary groups, represented by the set Track = {box1, box2, ..., box N} represents the motion trajectory of an individual, box i Represents the bounding box coordinate quadruple of the individual on the i-th frame, and N represents the number of video frames. Since the motion trajectory is temporal, the extracted motion trajectory features should have the feature information of the early time step. LSTM, which can extract temporal features, is used to extract features from the individual's motion trajectory.

[0049] Use the set All_track = {track1, track2, ..., track M} represents the motion trajectories of all individuals in a video, where M is the total number of individuals in the video. After obtaining the motion trajectory set of individuals in the video, the similarity of the motion trajectories of each pair of individuals is calculated, and an M*M size similarity matrix D between individual motion trajectories is obtained. track .

[0050] It can be observed in crowd videos that although there is a certain regularity in the behavior of individuals, from the perspective of the entire crowd, the distribution of other individuals around each individual is random and irregular, and the number of individuals around each individual is also not fixed. For individuals in the same group, their influence or interaction with each other is simultaneous and mutual, and this influence has no order or direction. Therefore, this patent uses an undirected graph to represent the crowd in the video. The nodes of the graph represent the behavior information of each individual, and the edges of the graph represent the relationship between the nodes. In order to characterize the behavior of individuals in time and space, the combined features of the extracted individual action sequence features and individual motion trajectory features are used as the node features of the graph.

[0051] The association between individuals is represented by the edges of the graph, and the similarity between two individuals is used as the adjacency weight between two nodes. The action sequence feature similarity matrix D between individuals is obtained act and the similarity matrix D of the motion trajectories between individuals track , add the corresponding elements weightedly to get the initial adjacency matrix D of the graph init , as shown in formula (3).

[0052] D init = pD act + qD track (3)

[0053] Among them, p and q are two adjustable weight coefficients. The corresponding elements in the matrix are weighted and summed to obtain the initial adjacency matrix D of the graph with a size of M*M init , M is the number of nodes, that is, the number of individuals.

[0054] For each graph node, the importance weight of each neighbor node to the current node is calculated through the attention mechanism. The attention weight is normalized and multiplied with the feature vector of the corresponding neighbor node to obtain the weighted neighbor node feature, and then the weighted features of all neighbor nodes are added. In this way, the features of each node are updated, and the similarity between each two nodes is calculated for the updated node features to obtain the similarity matrix D of the updated node features. graph .

[0055] The updated node feature similarity matrix D graph and the initial adjacency weight matrix D of the graph init Connect and input the connected matrix into the convolution layer to establish D graph and D init The relationship matrix between individuals is predicted based on the relationship between them, as shown in formula (4).

[0056] RM = conv(concat(D graph , Dinit )) (4)

[0057] Among them, RM is the inter-individual relationship matrix, concat() is a matrix concatenation operation, which connects two matrices in the channel dimension to obtain a matrix dimension of (2, M, M), and conv() is a convolution operation to establish a connection between the two matrices, and the output is an inter-individual relationship matrix of dimension (1, M, M).

[0058] The steps to predict the number of groups in a population are as follows:

[0059] Through D graph and D init Establish connections and predict the number of groups in the crowd through convolutional neural networks. Initialize the adjacency matrix D init Flatten to obtain a one-dimensional matrix, which is input into the fully connected layer for parameterization. The updated node features are input into another fully connected layer for feature dimension reduction, and then the node features after dimension reduction are flattened, and the feature graphs output by the two fully connected layers are connected to establish the connection between the updated node features and the initial adjacency matrix. Finally, the connected features are input into another fully connected layer, and the predicted number of groups is output through this fully connected layer. The group number prediction process is shown in formula (5).

[0060] K = A T (concat(B T D init , flatten(C T nodefeatures))) (5)

[0061] Where K is the number of groups in the population, b T and c T They represent the parameter vectors of the fully connected layer for extracting the initial adjacency matrix features and the fully connected layer for extracting graph node features, respectively. T It indicates that the parameter vector of the fully connected layer is established to initialize the adjacency matrix and the graph node features, and the parameter matrix of the fully connected layer that predicts the number of groups is established. Flatten() indicates the flattening of the feature graph, and concat() indicates the concatenation of the feature graph.

[0062] After obtaining the relationship matrix between individuals and the number of groups in the population, the individuals in the population are grouped and calculated. The steps are as follows:

[0063] The predicted inter-individual relationship matrix RM is used as the adjacency matrix of the graph. The degree matrix of the graph is obtained through the adjacency matrix RM. Then, the Laplacian matrix of the graph is obtained by subtracting the corresponding elements of the two matrices. Finally, the eigenvalues ​​of the Laplacian matrix of the graph are calculated and arranged by size. Among the eigenvalues ​​of the Laplacian matrix of the graph, the eigenvectors corresponding to the first u smallest eigenvalues ​​are selected, and these eigenvectors are combined into new features. The dimension of the new eigenvector is (u, M), where M is the number of pedestrians in the crowd.

[0064] Then, the feature vector is spatially transformed to obtain a feature vector with a dimension of (M,u). Each row of the transformed feature vector is regarded as a sample feature, and the dimension of each sample feature is (1,u). There are M data samples in total, that is, the total number of individuals in the population is M. In this new feature space, the K-Means clustering algorithm is used to cluster these samples. The number of clusters is the number of groups obtained by formula (5), and the clustering result is obtained. This result is the grouping result of individuals in the population.

[0065] The method of the present invention studies the extraction of individual behavioral features with high representation ability from videos, and dynamically groups individuals in the crowd according to individual behavioral features. In order to verify the performance of the method, it is compared with other crowd grouping methods, and the Socail-CAD data set is used to measure the grouping performance by the Rand coefficient. The comparison results are shown in Table 1. It can be found that when using grouping accuracy (GC-Acc) as a metric, the crowd grouping method of the present invention shows better grouping performance than other methods. The method SARF with slightly lower performance uses a graph attention network to calculate the attention coefficient between individual features, and uses a spectral clustering algorithm to group the crowd on the attention coefficient matrix. Since the number of clusters K needs to be set in advance, it cannot meet the universality on the data set. After all, the K value is variable, and the crowd grouping method of the present invention dynamically predicts the K value, which is more universal. Compared with the method that uses spatial information such as individual position and direction and detects F-formation in the crowd to complete the crowd grouping task, the grouping method of the present invention shows advantages by extracting continuous appearance information and motion trajectory characteristics.

[0066] Table 1 Performance comparison of different methods

[0067]

[0068] The method of the present invention groups individuals in a crowd by extracting individual action sequence features and motion trajectory features. In order to verify the influence of different features and feature combinations on the grouping performance, an ablation experiment was carried out on the Social-CAD dataset, and the grouping performance was measured by the Rand coefficient. The test results are shown in Table 2. It can be found that: using motion trajectory features to group the crowd, the grouping performance is poor; using action sequence features to group the crowd, the grouping performance is better; using the combined features of action sequence and motion trajectory to group the crowd, the grouping performance is the best, which means that in the crowd grouping, the action sequence features and motion trajectory features promote each other and enrich the individual behavior characteristics.

[0069] Table 2 Ablation experiment results

[0070]

Claims

1. A video crowd grouping method considering the number of groups and the relationship matrix between individuals, characterized in that: The method includes: Step 1: Process the video by dividing it into frames and determining the individual bounding boxes and unique identifiers of the individuals in the continuous video frames; Step 2: extracting the apparent feature map of the video frame from the continuous video frames; Step 3: extract the individual's appearance feature map based on the individual's bounding box coordinate information; Step 4: According to the unique identification of the individual and the individual appearance feature map in the continuous frames obtained in step 3, the action sequence features of the individual in time are calculated; after the action sequence features of the individual are obtained, the similarity of the action sequence features between the individuals is calculated; Step 5: Based on the bounding box coordinate information of the individual, construct the individual's motion trajectory data, extract the individual's motion trajectory features, and calculate the motion trajectory similarity between individuals; Step 6: Connect the action sequence features of the individual and the motion trajectory features of the individual, and use the combined features as the features that characterize the individual behavior; Step 7: Calculate the initial adjacency weights between individuals based on the similarity of action sequence features and motion trajectories between individuals; Step 8: Use individual behavior features as the input of the graph attention network, update the individual behavior features through message passing, and calculate the similarity between the updated individual behavior features after all individual features are updated; Step 9: Using the initial adjacency weights between individuals and the updated individual behavior characteristics as inputs to the feedforward neural network to predict the number of groups in the population; Step 10: Connect the initial adjacency weights between individuals and the updated similarities between individual behavior characteristics as inputs of the feedforward neural network to predict the relationship matrix between individuals; Step 11: Based on the number of groups and the relationship matrix between individuals, a clustering algorithm is used to obtain the results of population grouping.

2. The video crowd grouping method considering the number of groups and the relationship matrix between individuals according to claim 1 is characterized in that: In the step 2: After splitting the video into frames, the residual network is used to extract the apparent features of the video frames. The channel dimension of the video frame appearance feature map is relatively high. A 1×1 convolution calculation is performed on the video frame appearance feature map to reduce the number of channels of the video frame appearance feature map, while keeping the height and width of the video frame appearance feature map unchanged. The individual bounding box on the video frame is determined by detection and tracking methods. The individual bounding box is represented by a four-tuple (x1, y1, x2, y2). In the video frame coordinate system, x1 represents the horizontal coordinate of the upper left corner of the bounding box, y1 represents the vertical coordinate of the upper left corner of the bounding box, x2 represents the horizontal coordinate of the lower right corner of the bounding box, and y2 represents the vertical coordinate of the lower right corner of the bounding box. The coordinate four-tuple of the bounding box on the video frame is converted into the coordinate four-tuple of the bounding box on the video frame appearance feature map. The coordinates of the bounding box are reduced by calculating the reduction ratio of the video frame size to the size of its appearance feature map. The following formula is used for calculation: Among them, w represents the width of the video frame appearance feature map, h represents the height of the video frame appearance feature map, w rgb Indicates the width of the video frame, h rgb Indicates the height of the video frame; In the video frame appearance feature map, x1′ represents the horizontal coordinate of the upper left corner of the bounding box, y1′ represents the vertical coordinate of the upper left corner of the bounding box, x2′ represents the horizontal coordinate of the lower right corner of the bounding box, and y2′ represents the vertical coordinate of the lower right corner of the bounding box. The (x1′, y1′, x2′, y2′) quadruple is used to represent the bounding box of the individual appearance feature map on the video frame.

3. The video crowd grouping method considering the number of groups and the relationship matrix between individuals according to claim 2 is characterized in that: In step 3, after obtaining the feature map of the video frame and the coordinates of the bounding box on the feature map, the feature map of the individual in the video frame is determined, and the steps are as follows: First, according to the reduction ratio from the video frame size to the size of the video frame apparent feature map, after reducing the bounding box, the floating point coordinate values ​​of the bounding box are not rounded, and the reduced bounding box is directly divided into n×n rectangular grid areas on average. 2×2 sampling points are equally taken in each grid area, that is, each grid area is equally divided into 2×2 small area blocks, and each sampling point is the center point of the small area block; Then, find the four feature points closest to each sampling point, use bilinear interpolation to calculate the feature value of the current sampling point, and perform the same feature value calculation on all sampling points; in each grid, 2×2 feature values ​​will be obtained; Finally, a maximum pooling operation is performed in each grid. After the same operation is performed on all grids, an apparent feature map of the bounding box area of ​​size n×n is obtained. A 1×1 convolution operation is performed on the bounding box feature map of uniform size to reduce the channel dimension of the bounding box feature map and obtain the apparent feature map of the individual in the video frame.

4. The video crowd grouping method considering the number of groups and the relationship matrix between individuals according to claim 1 is characterized in that: The fourth step comprises: Step 4.1, connect the individual appearance feature maps of uniform size in the time dimension, and use (C, H, W) to represent the dimension of the individual appearance feature map, where C is the number of feature map channels, H is the feature map height, and W is the feature map width; Step 4.2: According to the unique identifier of the individual, the appearance feature graphs of the same individual are connected in the time dimension to obtain the action sequence feature of the individual. The feature dimension of the action sequence feature is (T*C,H,W), where T is the number of frames. Step 4.3: Calculate the distance between the action sequence features of different individuals to obtain the similarity of the action sequence features between individuals.

5. The video crowd grouping method considering the number of groups and the relationship matrix between individuals according to claim 1 is characterized in that: The step five comprises: Step 5.1: The motion trajectory of an individual can be represented by a sequence of continuous bounding box coordinate quadruplets (x1, y1, x2, y2), where x1 represents the horizontal coordinate of the upper left corner of the bounding box, y1 represents the vertical coordinate of the upper left corner of the bounding box, x2 represents the horizontal coordinate of the lower right corner of the bounding box, and y2 represents the vertical coordinate of the lower right corner of the bounding box; the set Track = {box1, box2, ..., box N } represents the motion trajectory of an individual, box i represents the bounding box coordinate quadruple of the individual in the i-th frame, and N represents the number of frames; LSTM is applied to the motion trajectory of the individual to obtain the motion trajectory features of the individual; Step 5.2: Calculate the similarity between the motion trajectories of different individuals to obtain the similarity of the motion trajectories between individuals.

6. The video crowd grouping method considering the number of groups and the relationship matrix between individuals according to claim 1 is characterized in that: The step six comprises: The continuous bounding box coordinates in the video are numerical data with time series. The continuous bounding box coordinates of the individual in the video are used as the individual's motion trajectory. The bounding box of the individual on the video frame is represented by a (x1, y1, x2, y2) quadruple. The individual's motion trajectory is represented by a continuous sequence of bounding box coordinate quadruple sequences, represented by the set Track = {box1, box2, ..., box N } represents the motion trajectory of an individual, box i represents the bounding box coordinate quadruple of the individual on the i-th frame, and N represents the number of video frames. Since the motion trajectory has temporal characteristics, the extracted motion trajectory features should have the feature information of the early time step. The LSTM that can extract temporal features is used to extract the features of the individual's motion trajectory. Use the set All_track = {track1, track2, ..., track M } represents the motion trajectories of all individuals in a video, where M is the total number of individuals in the video. After obtaining the motion trajectory set of individuals in the video, the similarity of the motion trajectories of each pair of individuals is calculated, and an M*M size similarity matrix D of the motion trajectories between individuals is obtained. track ; An undirected graph is used to represent the people in the video. The nodes of the graph represent the behavior information of each individual, and the edges of the graph represent the relationship between nodes. In order to characterize the behavior of individuals in time and space, the combined features of the extracted individual action sequence features and individual motion trajectory features are used as the node features of the graph.

7. The video crowd grouping method considering the number of groups and the relationship matrix between individuals according to claim 1 is characterized in that: The step seven comprises: The similarity of action sequence features between individuals and the similarity of motion trajectories between individuals are multiplied by their corresponding adjustable weights, and the results of the two multiplications are added together to calculate the initial adjacency weights between individuals: D int =pD act +qD track Among them, p and q are adjustable weights, D int is the adjacency weight matrix between individuals, D act is the action sequence feature similarity matrix between individuals, D track is the motion trajectory similarity matrix between individuals.

Images