Pedestrian trajectory stream multi-level clustering algorithm considering multi-camera information

Abstract

This invention discloses a multi-level clustering algorithm for pedestrian trajectory flows that takes into account information from multiple cameras. It relates to a semi-supervised multi-level clustering algorithm for video target geographic flows. The method is based on a flow space and uses a hierarchical representation of video target trajectories to obtain geographic flow segment sets at different levels. Using camera IDs as labels, semi-supervised clustering is performed on these segments at different levels to obtain a set of cluster centers for each level. These cluster centers effectively describe the overall trend of a large number of geographic flows. This method overcomes the limitations of traditional algorithms, which can only perform single-level clustering and do not consider camera information. A series of experiments demonstrate the effectiveness of this algorithm.

Description

Technical Field

[0001] This invention relates to the field of video-geographic scene data fusion organization, trajectory OD stream space, and trajectory clustering in streaming mode, specifically to a multi-level clustering algorithm for pedestrian trajectory streams that takes into account information from multiple cameras. Background Technology

[0002] As surveillance video systems evolve from single-camera to multi-camera networks, traditional video target trajectory clustering algorithms suffer from poor clustering results because they fail to consider camera position information and field of view.

[0003] Advances in video target localization devices and location tracking technologies have generated a vast amount of trajectory data, recording the movement of people, vehicles, and animals in various applications. This data is characterized by convenient deployment, intuitive information, and rich media expression. Trajectory clustering defines the similarity of trajectories across spatiotemporal dimensions, assigning each trajectory to a suitable cluster. How to effectively cluster video target trajectories has remained a research hotspot. Existing video target trajectory clustering methods have certain shortcomings at both the object analysis level and the method analysis level. Specifically, at the object analysis level, they are limited to image trajectories, neglecting the actual geographic trajectory of video targets; at the method analysis level, they fail to consider the geographic spatial relationships of multiple cameras, performing clustering only on small datasets under a single camera, resulting in poor real-time performance and accuracy in large-scale cross-camera trajectory data streams.

[0004] The actual trajectories of video targets are complex, requiring high computational costs for clustering analysis. Furthermore, it's difficult to unify the dimensionality between different target trajectories. Flow space, on the other hand, tends to ignore actual paths or short-term single-target motion states, focusing instead on analyzing the flow or interaction of target objects between two geographically defined origins and destinations. It is not constrained by geographic spatial distribution and hierarchical structure, reflecting the interactive characteristics of different locations. Combining flow space with video target trajectory analysis can improve video target analysis methods, enhance the value of existing video target analysis, and improve the analytical capabilities of multi-camera video targets in geospatial environments. This study, based on geographic flow space and focusing on moving video targets, investigates a multi-level pedestrian trajectory flow clustering algorithm.

[0005] In summary, this invention proposes a semi-supervised trajectory flow clustering algorithm that takes into account camera spatial information. This method labels trajectory flow segments within the camera's field of view with the camera ID, and labels trajectory flow segments in the camera's blind zone with the IDs of the previous and next cameras. Hierarchical clustering is then performed on combinations of flow segments at different levels. Experiments demonstrate the algorithm's clustering performance and time efficiency, proving its effectiveness. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention aims to provide a multi-level clustering algorithm for pedestrian trajectory flows that considers information from multiple cameras. Based on the flow space, the algorithm uses a geographic flow hierarchical representation to represent video target trajectories, obtaining geographic flow segment sets at different levels. Using camera IDs as labels, semi-supervised clustering is performed on these flow segment sets at different levels, resulting in a set of cluster centers for each level. These cluster centers effectively describe the overall trend of a large number of geographic flows. This method overcomes the limitations of traditional algorithms, which can only perform single-level clustering and do not consider camera information. A series of experiments have demonstrated the effectiveness of this algorithm.

[0007] To achieve the above objectives, the present invention provides a multi-level clustering algorithm for pedestrian trajectory flow that considers information from multiple cameras, including:

[0008] 1. First, the video target dataset is transformed into a stream space to obtain geographic stream segment sets at different levels;

[0009] 2. Perform semi-supervised clustering on the flow segment sets at different levels to obtain several sets of cluster centers.

[0010] The representation of the video target in the stream space in step 1:

[0011] A video trajectory OD stream consists of a series of stream segments, and the video target Obj is represented in the stream space as:

[0012] f Obj (i)={f i，1 f i，2 f i，3 , ..., f i，n}, (1)

[0013] in,

[0014] f i，j =(x i，j y i，j , t i，j (2)

[0015] Where, x i，j and y i，j Represents map coordinates, t i，j Represents a timestamp.

[0016] In order to obtain x i，j and y i，j The contact point between the video target sub-image and the ground is used as the positioning point. The coordinates of the video target in the entry and exit areas of each camera are taken as the time points for storing the stream segments.

[0017] A mapping model is constructed using the homography matrix method. Assuming the image coordinates of the exit / entry area are q and the geospatial coordinates are Q, then the homogeneous coordinates of q and Q can be expressed as:

[0018] q = [xy1] T (3)

[0019] Q = [XYZ1] T (4)

[0020] Let the mapping matrix be M, then the relationship between q and Q is:

[0021] q = MQ. (5)

[0022] The mapping matrix M can be decomposed into: (The transformation from the image plane to the geospatial plane involves scaling, translation, and rotation.)

[0023] M = s·W·R. (6)

[0024] In the formula, s is the scaling factor; W is the camera translation transformation matrix; and R is the 3×4 dimensional rotation transformation matrix.

[0025]

[0026] R = [r1r2r3e], (8)

[0027] Among them, f u f v represents the product of the physical focal length of the lens and the size of the sensor in each unit along the horizontal and vertical axes, respectively; u and v represent the offset of the image imaging center relative to the principal optical axis along the horizontal and vertical axes, respectively; r1, r2, and r3 represent the rotational relationship of the coordinate system in the X, Y, and Z axes in physical space, respectively; e represents the translational relationship between the coordinate systems.

[0028] When using the homography matrix method, it is assumed that the camera's field of view in geographic space is a horizontal plane, i.e., Z = 0 at this plane. Therefore, the mapping relationship from image space to geographic space can be viewed as a mapping from one plane to another. To simplify the calculation, Z in Q and r3 representing rotation around the Z-axis in R are removed. The homography matrix M then simplifies to:

[0029]

[0030] The geospatial coordinates of the video target's exit / entry area can be obtained from the solution of matrix M.

[0031] Step 1 addresses the scenario where the same video target trajectory exists across multiple camera viewpoints. It involves hierarchical clustering of different levels of flow segments to represent the trajectory flow in a multi-layered manner.

[0032] Let f1 pass through Cam 1, Cam 2, and Cam 3, with a level number of 5; f2 pass through Cam 1 and Cam 2, with a level number of 3; and f3 pass through Cam 3, with a level number of 1. Both f1 and f2 contain a flow sub-segment with a level number of 3, starting from Cam 1 and ending at Cam 2. Both f1 and f3 contain a flow sub-segment under Cam 3, with a level number of 1.

[0033] Step 2 yields the cluster centers of the λ group.

[0034]

[0035] in,

[0036]

[0037] c 2h-1，k =average({f 2h-1，m f 2h-1，m+1 , ..., f 2h-1，m+2(λ-1)}), (10)

[0038] {f 2h-1，m f 2h-1，m+1 , ..., f 2h-1，m+2(λ-1)}∈c 2h-1，k (11)

[0039] Where λ is the number of cameras the trajectory flow crosses, C(2h-1) represents the set of cluster centers when the number of cameras the trajectory flow crosses is 2h-1, and k represents the kth cluster center.

[0040] The pseudocode steps of the multi-level clustering algorithm are as follows:

[0041] Given a set of cross-camera video target trajectories, perform multi-level flow segment decomposition based on the number of cameras the video target passes through. For the set of flow segments at the same level, arbitrarily select q samples as initial cluster centers, calculate the distances between other samples in the flow segment set and all cluster centers, and assign each sample to the nearest cluster center.

[0042] argmin d(s, c i (12)

[0043] d(s,c) i )=||s,ci||2, (13)

[0044] Where s represents any sample, c i Represents the i-th cluster center

[0045] Finally, check if the label of each sample is the same as the label of the current cluster center. If the labels are the same, perform a sample-to-center operation:

[0046]

[0047] Otherwise, perform a sample distance operation:

[0048]

[0049] The sample labels here are the camera numbers that the flow segment passed through.

[0050] This invention uses camera numbers (with camera blind spots using the numbers above and below the camera) as labels to perform semi-supervised clustering of flow segments at different levels, taking into account camera information, and obtains reasonable cluster centers.

[0051] The beneficial effects of this invention are as follows: A semi-supervised video target flow hierarchical clustering algorithm—SCACI. This method uses camera numbers (with camera blind spots using the numbers above and below the camera) as labels to perform semi-supervised clustering of flow segments at different levels, taking into account camera information, and obtains reasonable cluster centers. Experimental results demonstrate that this method can efficiently and accurately extract geographic flow centers. Attached Figure Description

[0052] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments;

[0053] Figure 1 This is a schematic diagram illustrating the representation of the video target in the stream space according to the present invention;

[0054] Figure 2 This is a schematic diagram representing the flow segment of the present invention;

[0055] Figure 3 This is a flowchart of the hierarchical clustering algorithm of the present invention;

[0056] Figure 4 This is a schematic diagram of the eight-channel monitoring video images of the present invention;

[0057] Figure 5 This is a schematic diagram of the field of view distribution of the surveillance camera according to the present invention;

[0058] Figure 6 This is a schematic diagram of the cross-camera video target trajectory of the DukeMTMC of the present invention;

[0059] Figure 7 A schematic diagram illustrating the elbow criterion for determining the number of cluster centers (level 1-3) according to the present invention;

[0060] Figure 8 The following are schematic diagrams illustrating the flow segment clustering results at different levels according to the present invention: (a) Flow segment clustering results at level 1; (b) Flow segment clustering results at level 3; (c) Flow segment clustering results at level 5.

[0061] Figure 9 The contour coefficient comparison line graphs of the present invention are shown below: (a) contour coefficient comparison curve when level is 1; (b) contour coefficient comparison curve when level is 3; (c) contour coefficient comparison curve when level is 5. Detailed Implementation

[0062] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0063] Reference Figure 1-9 The specific implementation adopts the following technical solution: a multi-level clustering algorithm for pedestrian trajectory flow that takes into account information from multiple cameras, specifically including the following:

[0064] 1. Representation of video targets in the stream space

[0065] A video trajectory OD stream consists of a series of stream segments, and the video target Obj is represented in the stream space as:

[0066] f Obj(i) ={f i，1 f i，2 f i，3 , ..., f i，n}, (1)

[0067] in,

[0068] f i，j =(x i，j y i，j , t i，j (2)

[0069] Where, x i，j and y i，j Represents map coordinates, t i，j Represents a timestamp.

[0070] In order to obtain x i，j and y i，k The contact point between the video target sub-image and the ground is used as the positioning point. The coordinates of the video target in the entry and exit areas of each camera are taken as the time points for storing the stream segments, such as... Figure 1 As shown.

[0071] This specific implementation uses the homography matrix method to construct the mapping model. Assuming the image coordinates of the exit / entry area are q and the geospatial coordinates are Q, then the homogeneous coordinates of q and Q can be expressed as:

[0072] q = [xy1] T (3)

[0073] Q = [XYZ1] T (4)

[0074] Let the mapping matrix be M, then the relationship between q and Q is:

[0075] q = MQ. (5)

[0076] The mapping matrix M can be decomposed into: (The transformation from the image plane to the geospatial plane involves scaling, translation, and rotation.)

[0077] M = s·W·R. (6)

[0078] In the formula, s is the scaling factor; W is the camera translation transformation matrix; and R is the 3×4 dimensional rotation transformation matrix.

[0079]

[0080] R = [r1r2r3e], (8)

[0081] Among them, f u f v represents the product of the physical focal length of the lens and the size of the sensor in each unit along the horizontal and vertical axes, respectively; u and v represent the offsets of the image imaging center relative to the principal optical axis along the horizontal and vertical axes, respectively; r1, r2, and r3 represent the rotational relationships of the coordinate system in the X, Y, and v axes of the physical space, respectively; e represents the translational relationship between the coordinate systems.

[0082] When using the homography matrix method, it is assumed that the camera's field of view in geographic space is a horizontal plane, i.e., Z = 0 at this plane. Therefore, the mapping relationship from image space to geographic space can be viewed as a mapping from one plane to another. To simplify the calculation, Z in Q and r3 representing rotation around the Z-axis in R are removed. The homography matrix M then simplifies to:

[0083]

[0084] The geospatial coordinates of the video target's exit / entry area can be obtained from the solution of matrix M.

[0085] 2. Multi-layered representation of trajectory flow

[0086] For situations where the trajectory of the same video target exists in the field of view of multiple cameras, in order to perform hierarchical clustering of flow segments at different levels, the trajectory flow needs to be represented in multiple layers:

[0087] Figure 2The diagram illustrates three trajectory flows f1, f2, and f3 within cameras Cam 1-3. f1 passes through Cam 1, Cam 2, and Cam 3, with a level of 5; f2 passes through Cam 1 and Cam 2, with a level of 3; and f3 passes through Cam 3, with a level of 1. Both f1 and f2 contain a flow sub-segment with a level of 3, starting from Cam 1 and ending at Cam 2. Both f1 and f3 contain a flow sub-segment below Cam 3, with a level of 1.

[0088] 3. SCACI Clustering Algorithm

[0089] To facilitate reasonable analysis of cross-camera trajectory flows, this specific implementation proposes a multi-level trajectory flow clustering algorithm. The algorithm aims to obtain λ cluster centers:

[0090]

[0091] in,

[0092]

[0093] c 2h-1，k =average({f 2h-1，m f 2h-1，m+1 , ..., f 2h-1，m+2(λ-1)}), (10)

[0094] {f 2h-1，m f 2h-1，m+1 , ..., f 2h-1，m+2(λ-1)}∈c 2h-1，k (11)

[0095] Where λ is the number of cameras the trajectory flow crosses, C(2h-1) represents the set of cluster centers when the number of cameras the trajectory flow crosses is 2h-1, and k represents the kth cluster center.

[0096] The specific process of the algorithm is as follows: Figure 3 As shown, the video target dataset is first transformed into a stream space to obtain geographic stream segments at different levels. Then, the stream segments at different levels are clustered to obtain several sets of cluster centers.

[0097] The pseudocode for the SCACI algorithm is shown below:

[0098]

[0099]

[0100] Given a set of cross-camera video target trajectories, perform multi-level flow segment decomposition based on the number of cameras the video target passes through. For the set of flow segments at the same level, arbitrarily select q samples as initial cluster centers, calculate the distances between other samples in the flow segment set and all cluster centers, and assign each sample to the nearest cluster center.

[0101] argmin d(s, c i (12)

[0102] d(s,c) i )=||s,ci||2, (13)

[0103] Where s represents any sample, c i Represents the i-th cluster center

[0104] Finally, check if the label of each sample is the same as the label of the current cluster center. If the labels are the same, perform a sample-to-center operation:

[0105]

[0106] Otherwise, perform a sample distance operation:

[0107]

[0108] The sample labels here are the camera numbers that the flow segment passed through.

[0109] Example 1:

[0110] 1. Experimental conditions and data

[0111] The experimental data in this embodiment is open-source data provided by DukeMTMC: including video image data synchronously captured by 8 geospatial adjacent cameras with fixed spatial positions and orientations (e.g., Figure 4 ), and all camera body and camera field of view geographic location data, camera correction parameters (such as Figure 5 On the other hand, it also includes more than 1,845 cross-camera dynamic target trajectories, marked by image bounding boxes generated by automatic detection and tracking via computer vision and manual annotation and correction (e.g., Figure 6This experiment selected a 50-minute video sequence from 4:25 to 5:15 from the original dataset as the experimental data for this embodiment. Experimental environment: Software (Windows 10, Python 3.6 + sklearn 0.0), Hardware (Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz 2.30GHz, RAM 12.0GB, NVIDIA GeForce MX250). The algorithms for preprocessing to obtain the video target trajectory are as follows: the video dynamic target detection algorithm is Mask-RCNN, the tracking algorithm is CSRT, and the cross-camera re-identification algorithm is an improved method based on GAN to generate unlabeled samples.

[0112] 2. Determining the number of cluster centers

[0113] The number of cluster centers is obtained using the elbow rule. Specifically, it is determined by calculating the sum of squared distances from each point to its cluster center after each K-means clustering operation. As the cluster centers are continuously optimized, the sum of squares gradually decreases. The first inflection point in the process of the sum of squares change is identified as the optimal K value.

[0114] mi n(SSE), (16)

[0115]

[0116] The cluster centers of the flow segments at levels 1-3 were determined using the elbow rule, and the results are shown below:

[0117] pass Figure 7 It can be determined that the number of trajectory cluster centers is between 60 and 70 when the level is 1, between 40 and 50 when the level is 2, and between 30 and 35 when the level is 3.

[0118] 3. Evaluation of SCACI Algorithm Results and Performance

[0119] This embodiment involves three tasks. First, the number of flow segments at different levels is determined:

[0120] Table 1. Elbow Criterion for Determining the Number of Cluster Centers (Levels 1-3)

[0121]

[0122] This embodiment only performs cluster analysis on flow segments with 1, 3, and 5 levels. In reality, there are flow segments with 7, 9, 11, or even more levels; however, in these cases, the number of flow segments is too small, and cluster analysis becomes meaningless because the goal of clustering is to extract the overall trend from a large number of trajectories. This embodiment only studies flow segments with 1, 3, and 5 levels. Parameter settings Itermax =5000, η=0.01. In 3(a), the clustering visualization results of the SCACI algorithm with different numbers of flow segments are shown; in 3(b), the advantages of the SCACI algorithm compared with traditional clustering algorithms are shown by evaluating the silhouette coefficient; in 3(b), its time complexity is analyzed to prove the effectiveness of the algorithm.

[0123] 3(a) Clustering results display: such as Figure 8 The results show the clustering of flow segments with 1, 3, and 5 levels. According to section 4.2, the initial number of cluster centers was selected as K1 = 60, K2 = 40, and K3 = 30, respectively.

[0124] 3(b) Evaluation of clustering results

[0125] To verify the effectiveness of the method in this embodiment, the traditional K-means method is compared with the SCACI algorithm of this embodiment. Silhouette coefficients are used to compare and analyze the detection performance. The evaluation is based on a combination of the average distance between a sample and its neighbors in the same cluster, and the average distance between a sample and its second nearest cluster.

[0126] Silhouette Coefficient(ζ)=|1-a(ζ) / b(ζ)|∈[-1,1]. (18)

[0127] pass Figure 9 Analysis of the line graph shows that the contour coefficient reaches its maximum value near the K value obtained through the elbow rule; furthermore, the SCACI algorithm outperforms existing algorithms at different levels.

[0128] 3(c) Time Complexity Analysis

[0129] The SCACI algorithm yields λ sets of prototype vectors, containing a total of obj_num video targets. For each video target obj(u), after passing through Cnum(f)... obj(u) There are ) cameras, therefore, when the number of levels is u, there should be u-λ-1 flow segments at that level, randomly select K t samples require O(K) t Time. For the learning phase, if the number of iterations reaches Iter_max or the cluster centers no longer change, generally speaking, the following timeframe is required:

[0130]

[0131] Among them, Iter max Represents the maximum number of iterations, obj num Cnum(f) represents the number of video targets. obj(u) K represents the number of cameras spanned in the geographic stream corresponding to video target u. t The algorithm calculates the number of cluster centers. Maintaining linear time complexity, the algorithm is highly efficient. Based on the experimental conditions of this embodiment, the CPU time required for different numbers of cluster centers is shown in Table 2.

[0132] Table 2. CPU time required for different levels of SCACI

[0133]

[0134] This embodiment proposes a semi-supervised video target flow hierarchical clustering algorithm—SCACI. This method uses camera IDs (with camera blind spots using the upper and lower camera IDs) as labels to perform semi-supervised clustering of flow segments at different levels, taking into account camera information, and obtains reasonable cluster centers. Experimental results demonstrate that this method can efficiently and accurately extract geographic flow centers.

[0135] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A multi-level clustering method for pedestrian trajectory flow that takes into account information from multiple cameras, characterized in that, include: (1) Transform the video target dataset into a stream space to obtain a set of geographic stream segments at different levels, including: inputting a set of cross-camera video target trajectories and performing multi-level stream segment decomposition based on the number of cameras the video target passes through; (2) Perform semi-supervised clustering on the flow segment sets at different levels to obtain The set of cluster centers includes: for a set of flow segments at the same level, arbitrarily select q samples as initial cluster centers, calculate the distance between other samples in the flow segment set and all cluster centers, and assign each sample to the nearest cluster center: (12) (13) Where s represents any sample, c i Represents the i-th cluster center Finally, check if the label of each sample is the same as the label of the current cluster center. If the labels are the same, perform a sample-to-center operation: (14) Otherwise, perform a sample distance operation: (15) The sample labels here are the camera numbers that the flow segment passed through; The aforementioned The cluster centers are as follows: . in, (9) (10) (11) in, For trajectory flow across camera number, The number of cameras across which the trajectory flow crosses is 1 The set of cluster centers at time Indicates the first Cluster centers.

2. The multi-level clustering method for pedestrian trajectory flow considering multi-camera information as described in claim 1, characterized in that, The representation of the video target in the stream space in step (1) is as follows: A video trajectory OD stream consists of a series of stream segments, and the video target... In the stream space, it is represented as: (1) in, (2) in, and Represents map coordinates, Represents a timestamp; In order to obtain and The contact point between the video target sub-image and the ground is used as the positioning point; the entry and exit coordinates of the video target at each camera are taken as the storage time of the stream segment. A mapping model is constructed using the homography matrix method; assuming the image coordinates of the exit / entry regions are... The geographic spatial coordinates are ,So and Homogeneous coordinates can be represented as: (3) (4) Let the mapping matrix be ,but and The relation is: (5) The mapping matrix can be transformed from the image plane to the geospatial plane through scaling, translation, and rotation. Decomposed into: (6) In the formula, This is the scaling factor; The translation transformation matrix is ​​the camera translation matrix. The rotation transformation matrix is ​​3×4 dimensional: (7) (8) in, , These represent the product of the lens's physical focal length and the sensor's dimensions in the horizontal and vertical directions of each unit, respectively. , The offset of the image center relative to the principal optical axis on the horizontal and vertical axes, respectively; , , These represent the coordinate systems in physical space. axis, axis, Rotational relationship along the axis; Indicates the translation relationship between coordinate systems; When using the homography matrix method, it is assumed that the camera's field of view in geographic space is a horizontal plane, that is, it is assumed that this plane is located at... Therefore, the mapping relationship from image space to geographic space can be regarded as a mapping from one plane to another. To simplify the calculation, we will... In and The middle indicates the circle Rotation of axis Remove; then the homography matrix Simplified to: (9) According to the matrix The solution can be used to obtain the geospatial coordinates of the video target's exit / entry area.

3. The multi-level clustering method for pedestrian trajectory flow considering multi-camera information according to claim 1, characterized in that, Step (1) addresses the situation where the same video target trajectory exists in multiple camera fields of view. To perform hierarchical clustering of different levels of flow segments, the trajectory flow is represented in a multi-layered manner. set up go through , , The number of levels is 5; Pass , The number of levels is 3. Pass The number of levels is 1; Simultaneously includes from Start to The ending sub-segment with a level of 3; Simultaneously includes The sub-segment below has a level of 1.

Images