A method for identifying clusters in low signal-to-noise ratio OD streaming data

Abstract

This invention relates to a method for identifying clusters in low signal-to-noise ratio (SNR) OD (Organizational Stream) data, belonging to the field of data stream clustering technology. It solves the problems of existing clustering techniques, such as high dependence on parameter settings and poor sensitivity to weak clustering signals, leading to low identification accuracy and insufficient robustness in complex data environments. The method includes: acquiring the OD stream dataset to be clustered and obtaining its clustering scale range to generate several initial clustering scale values; generating several random OD stream datasets with the same spatial range as the OD stream dataset to be clustered; obtaining a potential clustering scale set based on the initial clustering scale values, the OD stream dataset to be clustered, and the several random OD stream datasets; performing local spatial clustering analysis on the OD stream dataset to be clustered at each potential clustering scale to obtain a set of candidate center streams; obtaining the corresponding optimal clustering scale for each candidate center stream, and obtaining the corresponding cluster based on the optimal clustering scale of each candidate center stream.

Description

Technical Field

[0001] This invention relates to the field of data stream clustering technology, and in particular to a method for identifying clusters in low signal-to-noise ratio (OD) stream data. Background Technology

[0002] With the rapid development of information technology, data flow plays a crucial role in many fields. Origin-Destination (OD) flow data, as an important spatiotemporal data type, effectively characterizes the spatial interaction and connection between regions by abstracting the movement behavior of mobile objects (such as people and vehicles) from their origin to their destination. Cluster analysis of OD flow data can identify flow clusters with similar travel patterns and close spatial connections, thus providing key decision support for applications such as urban transportation planning, commuting behavior analysis, infrastructure optimization, and business site selection. In real-world data, the spatial distribution of OD flows is often highly heterogeneous. While some regions have strong connections, many others exhibit weak flow patterns, with minimal density differences between their flow clusters and background noise, exhibiting a so-called "low signal-to-noise ratio" characteristic (the density ratio of clusters to noise is close to 1). Although these low signal-to-noise ratio OD clusters are not particularly dense, they often contain weak but stable travel connections within the city, such as commuting patterns between peripheral areas, stable flows during off-peak hours, and gradually increasing interactions between emerging areas. Therefore, the ability to identify these weak but meaningful flow clusters is of great research and application value for understanding the evolution of urban spatial structure and the diversity of traffic behavior.

[0003] Most mainstream flow clustering methods are inspired by and extended from spatiotemporal point clustering techniques. They generally rely on preset density thresholds and neighborhood radii. These methods are less sensitive to weak clustering structures and are difficult to adapt to the highly heterogeneous distribution characteristics of OD data. When the density of the target cluster is similar to that of noise, traditional methods often cannot effectively separate the real clusters from the noisy data, making it difficult to accurately identify low signal-to-noise ratio flow clusters. Summary of the Invention

[0004] Based on the above analysis, the embodiments of the present invention aim to provide a method for identifying clusters in low signal-to-noise ratio (OD) streaming data, in order to solve the problems of existing clustering techniques being highly dependent on parameter settings and having poor sensitivity to weak clustering signals, resulting in low identification accuracy and insufficient robustness in complex data environments.

[0005] The objective of this invention is mainly achieved through the following technical solutions:

[0006] This invention provides a method for identifying clusters in low signal-to-noise ratio (SNR) OD streaming data, comprising the following steps:

[0007] Obtain the OD stream dataset to be clustered;

[0008] Based on the coordinates of each OD flow data in the OD flow data to be clustered, the clustering scale range of the OD flow data to be clustered is obtained and several initial clustering scale values ​​are generated to form an initial clustering scale sequence.

[0009] Generate several random OD stream datasets that have the same spatial range as the OD stream dataset to be clustered;

[0010] Based on the initial clustering scale sequence, the OD flow dataset to be clustered, and several random OD flow datasets, a global spatial clustering analysis is performed to obtain a potential clustering scale set.

[0011] Based on the OD flow dataset to be clustered and several random OD flow datasets, local spatial clustering analysis is performed at each potential clustering scale of the potential clustering scale set to obtain a set of candidate center flows.

[0012] For each candidate center flow in the candidate center flow set, the corresponding optimal clustering scale is obtained according to the change of its local spatial clustering intensity with each potential clustering scale, and the corresponding cluster is obtained according to the optimal clustering scale of each candidate center flow.

[0013] Furthermore, based on the initial clustering scale sequence, the OD flow dataset to be clustered, and several random OD flow datasets, a global spatial clustering analysis is performed to obtain a potential clustering scale set, including:

[0014] Using several random OD stream datasets as a reference, calculate the FL function value of the OD stream dataset to be clustered at each initial clustering scale of the initial clustering scale sequence;

[0015] The FL function values ​​at each initial aggregation scale are fitted to obtain the global FL function curve;

[0016] Calculate the second derivative of the global FL function curve to obtain at least one local minimum point in the second derivative curve of the global FL function curve;

[0017] The scale values ​​corresponding to each minimum point are used as potential clustering scales to obtain a potential clustering scale set.

[0018] Furthermore, the FL function values ​​for each initial aggregation scale are obtained using the following formula:

[0019] ;

[0020] ;

[0021] Where r represents the r-th initial aggregation scale; This represents the distance between the i-th OD stream data to be clustered and the j-th OD stream data to be clustered; n represents the total number of OD stream data to be clustered. Indicates an indicator function; This represents the density estimate of the OD flow data to be clustered; Let FK represent the expected value of a random OD stream dataset with the r-th initial clustering scale value; This represents the distance between the p-th random OD stream data and the q-th random OD stream data in the m-th random OD stream dataset.

[0022] Furthermore, the step of performing local spatial clustering analysis at each potential clustering scale of the potential clustering scale set based on the OD flow dataset to be clustered and the random OD flow dataset to obtain a candidate center flow set includes: performing the following operations on each potential clustering scale of the potential clustering scale set:

[0023] Using several random OD stream datasets as a benchmark, calculate the local FL function value of each OD data stream to be clustered at the current potential clustering scale;

[0024] Based on the local FL function values ​​of each OD data stream to be clustered, an initial candidate center stream is obtained, forming an initial candidate center stream set;

[0025] Redundant initial candidate center flows are removed from the initial candidate center flow set to obtain the candidate center flow set corresponding to the current potential clustering scale.

[0026] Furthermore, the local FL function value of each OD data stream to be clustered at the current potential clustering scale is calculated using the following formula:

[0027] ;

[0028] ;

[0029] Where s represents the s-th potential clustering scale; This represents the density estimate of the OD flow data to be clustered; This represents the distance between the i-th OD stream data to be clustered and the j-th OD stream data to be clustered; Let FK represent the expected value of the function for several random OD flow datasets with the s-th potential clustering scale value; This represents the distance between the p-th random OD stream data and the q-th random OD stream data in the m-th random OD stream dataset.

[0030] Furthermore, the initial candidate center stream is obtained based on the local FL function values ​​of each OD data stream to be clustered, forming an initial candidate center stream set, including:

[0031] Sort all OD stream data in the dataset to be clustered from high to low based on their corresponding local FL function values;

[0032] Select the top K% of the OD flow data to be clustered as the initial candidate center flow set; where K is a preset integer.

[0033] Furthermore, the step of removing redundant candidate center flows from the initial candidate center flow set to obtain a candidate center flow set includes:

[0034] Select the OD flow data with the highest local FL function value in the initial candidate center flow set, add it to the candidate center flow set, and remove it from the initial candidate center flow set;

[0035] The initial candidate center flows whose distance from the aforementioned initial candidate center flows is less than the current potential clustering scale value are removed from the initial candidate center flow set;

[0036] Repeat the above process until the initial candidate center flow set is empty, thus obtaining the candidate center flow set.

[0037] Furthermore, for each candidate center flow in the candidate center flow set, obtaining the corresponding optimal clustering scale based on the change of its local spatial clustering intensity with each potential clustering scale, and obtaining the corresponding cluster based on the optimal clustering scale of each candidate center flow, includes: for each candidate center flow in the candidate center flow set, performing the following operations:

[0038] Fit the local FL function values ​​of the current candidate center flow at each potential clustering scale to obtain the local FL function curve corresponding to the current candidate center flow;

[0039] Calculate the second derivative of the local FL function curve corresponding to the current candidate center flow, and obtain all the minimum points in the second derivative curve of the local FL function curve;

[0040] The maximum potential clustering scale corresponding to each of the aforementioned minimum points is taken as the optimal clustering scale for the current candidate center flow.

[0041] Using the current candidate center flow as the center, the OD flow data to be clustered that are within the optimal clustering scale of the current candidate center flow are taken as the clusters of the current candidate center flow. When an OD flow data to be clustered belongs to multiple clusters, it is assigned to the cluster closest to the candidate center flow.

[0042] Furthermore, generating several random OD stream datasets that are consistent with the spatial range of the OD stream dataset to be clustered includes: randomly and independently generating start coordinates and end coordinates within the spatial range of the OD stream dataset to be clustered, and obtaining random OD stream data with the same number as the OD stream data as a set of random OD stream datasets.

[0043] Furthermore, the step of obtaining the clustering scale range of the OD flow data to be clustered and generating several initial clustering scale values ​​based on the coordinates of each OD flow data in the OD flow data to be clustered dataset includes:

[0044] Based on the starting and ending coordinates of the OD flow data of each cluster, the minimum bounding rectangle region including the OD flow data of each cluster is obtained and its diagonal length is calculated.

[0045] Based on the diagonal length, the clustering scale range of the OD flow dataset to be clustered is obtained;

[0046] Within the said aggregation scale range, several initial aggregation scale values ​​are generated at equal intervals.

[0047] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0048] 1. This invention generates a random OD stream dataset with the same spatial range as the OD stream dataset to be clustered as a statistical benchmark. By combining global spatial clustering analysis and local spatial clustering analysis, it can effectively distinguish between noisy data and valid signals. In low signal-to-noise ratio environments, it can accurately identify true clusters, thereby providing a reliable basis for analysis and decision-making based on clustering results and significantly improving the accuracy of clustering.

[0049] 2. This invention objectively determines the potential clustering scale and the optimal clustering range of each cluster in the data by calculating the second derivative of the spatial clustering function (FL function) and finding its minimum point. This replaces the traditional method of subjectively setting density thresholds and neighborhood radii, which not only avoids performance fluctuations caused by improper parameter settings, but also has excellent robustness and generalization ability across datasets.

[0050] 3. This invention adaptively assigns a unique optimal clustering scale to each candidate center flow, enabling simultaneous and accurate identification of clustering structures across different spatial scales, effectively solving the multi-scale clustering problem. Furthermore, the algorithm ensures mutual exclusivity between clusters through post-processing steps, resulting in each final cluster being an independent pattern with a clear spatial structure and explicit semantics.

[0051] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0052] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0053] Figure 1 This is a flowchart illustrating a method for identifying clusters in low signal-to-noise ratio (OD) streaming data according to an embodiment of the present invention.

[0054] Figure 2 This is a schematic diagram of the structure of the random OD stream data generation model in an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of OD stream data distance in an embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram of OD stream data distribution in an embodiment of the present invention;

[0057] Figure 5 This is a schematic diagram of the FL function curve and derivative curve in an embodiment of the present invention. Detailed Implementation

[0058] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0059] One specific embodiment of the present invention discloses a method for identifying clusters in low signal-to-noise ratio (SNR) OD streaming data, such as... Figure 1 As shown, it includes the following steps S1-S6:

[0060] Step S1: Obtain the OD stream dataset to be clustered.

[0061] Specifically, an OD flow is a movement path consisting of an origin and a destination, typically represented by a pair of spatial coordinates. In human mobility behavior research, OD flows can be used to reflect the travel connections of individuals or groups in urban space and are a commonly used simplified modeling unit when studying trajectory data. The spatial extent of an OD flow cluster is usually measured by the maximum distance between all OD flows in the cluster and its central flow (i.e., the cluster radius), reflecting the spatial size of the cluster.

[0062] More specifically, the OD flow data to be clustered can be obtained from mobile phone signaling data, vehicle trajectory data, and public transportation card swipe data provided by telecommunications operators. Among them, the data in mobile phone signaling data is usually a spatiotemporal record left by mobile phone numbers continuously attached to different base stations. Obtaining OD flow data from mobile phone signaling data provided by telecommunications operators includes: analyzing the continuous signaling data of a single user, identifying the base station locations where the stay time exceeds a certain threshold (such as 30 minutes), determining them as the starting point (O) or ending point (D) of a trip, and connecting two consecutive stay points in chronological order to form an OD flow data from the coordinates of point O to the coordinates of point D.

[0063] Obtaining OD (Original Discharge) flow data from vehicle trajectory data includes: Based on the vehicle's GPS trajectory data, for vehicles such as ride-hailing vehicles with an operating label ("0" when idling and "1" when there are passengers), OD flow data is directly extracted based on the status change points. For example, if a taxi's status changes from 0 to 1, it can be considered that a passenger has boarded (the start of a trip), and from 1 to 0, it indicates that a passenger has alighted (the end of a trip). For vehicles without status labels, passenger pick-up and drop-off points are inferred by calculating features such as speed and dwell time. Points with prolonged low-speed dwell or those that restart after reaching a speed of 0 can be identified as points O or D.

[0064] Obtaining OD flow data from public transportation card swipe data includes: based on the card swipe transaction records (AFC data) of the public transportation system, including card ID, entry station and time, and exit station and time, directly generating an OD flow data from the entry station (O point) to the exit station (D point) for a complete trip.

[0065] Step S2: Based on the coordinates of each OD stream data in the OD stream data to be clustered, obtain the clustering scale range of the OD stream data to be clustered and generate several initial clustering scale values ​​to form an initial clustering scale sequence.

[0066] Specifically, in order to accurately analyze the clustering characteristics of trajectory data at different spatial scales, this embodiment needs to determine a reasonable set of spatial scales. The clustering scale range of OD flow data refers to the possible degree of clustering between OD flow data points in space. By analyzing the coordinates of each data point in the OD flow dataset to be clustered, a reasonable clustering scale range can be determined. This range is usually determined based on the maximum and minimum distances between data points to ensure that subsequent clustering analysis can cover various situations from local small-scale clustering to large-scale clustering.

[0067] Furthermore, the step of obtaining the clustering scale range of the OD flow data to be clustered and generating several initial clustering scale values ​​based on the coordinates of each OD flow data in the OD flow data to be clustered dataset includes:

[0068] Based on the starting and ending coordinates of the OD flow data of each cluster, the minimum bounding rectangle region including the OD flow data of each cluster is obtained and its diagonal length is calculated.

[0069] Based on the diagonal length, the clustering scale range of the OD flow dataset to be clustered is obtained.

[0070] Within the said aggregation scale range, several initial aggregation scale values ​​are generated at equal intervals.

[0071] Specifically, by extracting the maximum and minimum x and y coordinates of all OD flow data to be clustered, the minimum bounding rectangle including each OD flow data to be clustered is obtained as the study region. The diagonal length of the minimum bounding rectangle is taken as the region diameter. The clustering scale range should be at least less than half of the region diameter. In this embodiment, the clustering scale range is set to one-third of the region diameter to ensure that all subsequent calculations are performed within a reliable range with minimal impact from boundary effects, thus ensuring the accuracy of the results.

[0072] More specifically, urban clusters naturally exist at different scales. For example, the cluster scale around a subway station might be 500 meters, while the commuter cluster scale of a large industrial park might be 3000 meters. Therefore, within the range of these cluster scales, generating several initial cluster scale values ​​at equal intervals can capture clusters of different sizes. For example, if the maximum scale is set to 300 meters, 30 equally spaced scale values ​​can be generated between 0 and 300 meters, with each interval being 10 meters. The scale set constructed in this way can not only reflect trajectory clustering behavior from local to medium ranges, but also provide continuous and stable numerical support for the derivative calculation of subsequent spatial statistical functions.

[0073] It should be noted that in some applications that are more sensitive to small-scale clustering, non-uniform spacing (such as exponential or logarithmic spacing) can be used to construct the initial clustering scale set. However, it is important to ensure that there is sufficient sampling density across the entire scale range to avoid problems such as numerical instability or curve oscillation.

[0074] Step S3: Generate several random OD stream datasets that are consistent with the spatial range of the OD stream dataset to be clustered, including: randomly and independently generating start coordinates and end coordinates within the spatial range of the OD stream dataset to be clustered, and obtaining random OD stream data with the same number as the OD stream data as a set of random OD stream datasets.

[0075] Specifically, the generated random OD flow dataset has the same spatial range as the OD flow dataset to be clustered. However, since its data points are randomly generated, it can provide a benchmark without clustering features for comparison with the actual OD flow data. It can effectively distinguish between the clustering effect caused by the clustering characteristics of the data itself and the pseudo-clustering effect caused by random noise.

[0076] More specifically, a random OD flow dataset obtained by randomly and independently generating start and end coordinates can detect whether there is a significant spatial clustering structure in the entire data. This method of generating random OD flow data does not consider the distribution characteristics of any existing data, but generates simulated data in the following way: First, a start point and an end point are uniformly and randomly sampled within the study area determined in step S2, and the two points constitute an OD flow; repeat this process until the number of generated OD flows is consistent with the number of OD flow data to be clustered. As a random OD flow dataset, this method reflects a "weakest clustering hypothesis", that is, assuming that the OD distribution is completely unstructured. By comparing the differences between the real data and this random data in spatial statistical indicators (such as clustering functions), it can be determined whether there is a non-random clustering pattern in the data.

[0077] In another embodiment, a deep learning-based random OD stream generation method can be used. This method is suitable for identifying local or anomalous clustering patterns while controlling existing distribution characteristics. Compared to completely random data, this simulated data can retain the macroscopic background distribution of real data, thus more sensitively detecting "abnormal" clustering patterns. Specifically, this method uses a random OD stream data generation model based on a generative adversarial network (GAN) architecture, such as... Figure 2 As shown, the random OD stream data generation model includes a generator and a discriminator.

[0078] The generator is a four-layer fully connected neural network connected sequentially. Each fully connected layer is followed by a batch normalization function and a LeakyReLU activation function. A dropout layer is added to the first and second fully connected neural networks to reduce the risk of overfitting. 64-dimensional random noise following a standard normal distribution is input into the generator, and the output is a 4-dimensional vector corresponding to the four coordinate values ​​of the OD flow data. The generator's output layer activation function is tanh, and the output result can be converted into actual OD coordinate pairs after inverse normalization.

[0079] The discriminator also includes four fully connected neural networks connected in sequence. Each fully connected neural network is followed by a batch normalization function and a LeakyReLU activation function. The input is 4-dimensional OD data, and the output is a scalar to evaluate the authenticity of the input data.

[0080] When training the random OD stream data generation model, the discriminator is updated once every 10 training iterations to maintain training balance. The optimizer uses AdamW with a learning rate of 0.0001 and a momentum parameter. and The values ​​are set to 0.5 and 0.999 respectively; the learning rate is halved every 100 epochs during training. The loss function for training the random OD stream data generation model is WGAN-GP (Wasserstein GAN with gradient penalty), and its form is as follows:

[0081] ;

[0082] ;

[0083] in, This represents the discriminator loss function; This indicates the distribution of the OD stream data output by the generator; This represents the distribution of the actual OD stream data; This indicates the distribution of the interpolated samples; This represents the distribution of the OD stream data output by the generator. Data sampled from the middle; Represents the distribution of real OD stream data Data sampled from the middle; This represents the interpolated sample obtained by randomly sampling along the line connecting the real data and the generated data. This represents the scoring function of the discriminator; Represent the expected function; This represents the interpolated sample obtained by the discriminator. The gradient; Represents the generator loss function; This represents the weight of the gradient penalty term, which is set to 10 in this embodiment; This represents the weight of the L1 loss term, which is set to 0.1 in this embodiment to constrain the generated results to be close to the real data.

[0084] To ensure that the generated random OD stream data closely approximates the real data in terms of spatiotemporal distribution, the optimal generation epoch needs to be selected during the training process. This involves: every 100 epochs, saving a snapshot of the generator, sampling 199 times from the generator output, sampling P OD pairs each time (to reduce computational cost without affecting evaluation performance, P is set to 300 in this embodiment), and randomly sampling P OD pairs from the input data of the discriminator as real data for comparison. The comparison method is as follows: calculating the FK function values ​​of the generated random OD stream data and the real data at each initial aggregation scale value, and obtaining the 95% confidence interval strips of the function based on the 199 sampling results. Then, the overlap area is used to evaluate the degree of coincidence between the confidence interval strips of the generated random OD stream data and the real data for each epoch. The higher the overlap, the more consistent the spatial distribution of the generated random OD stream data is with the real data, and the closer the epoch is to the optimal epoch. The generator with the highest overlap is used as the generator for random OD stream data to generate random OD stream data.

[0085] Preferably, when the background is significantly non-uniform, a random OD stream data generation model based on a generative adversarial network (GAN) architecture is preferred to generate the random OD stream dataset.

[0086] It should be noted that this embodiment needs to generate several random OD stream datasets. For example, it can be 199 random OD stream datasets.

[0087] Step S4: Based on the initial clustering scale sequence, the OD flow dataset to be clustered, and several random OD flow datasets, perform global spatial clustering analysis to obtain a potential clustering scale set.

[0088] Specifically, the initial clustering scale sequence provides a series of possible clustering scales, covering various situations from local small-scale clustering to large-scale clustering. The global spatial clustering analysis, by comparing the clustering characteristics of real data and random data, can distinguish which clustering features are caused by the characteristics of the data itself and which are caused by random noise, so as to identify the overall clustering characteristics of OD stream data at different clustering scales.

[0089] Furthermore, the step of performing global spatial clustering analysis based on the initial clustering scale sequence, the OD flow dataset to be clustered, and the random OD flow dataset to obtain a potential clustering scale set includes steps S41-S44:

[0090] Step S41: Using several random OD stream datasets as a reference, calculate the FL function value of the OD stream dataset to be clustered at each initial clustering scale of the initial clustering scale sequence.

[0091] Specifically, the FL function value of the OD stream dataset to be clustered is calculated at each initial clustering scale. By introducing a random OD stream dataset as a benchmark for comparison, weak and real clustering signals can be extracted from a background of strong noise.

[0092] Furthermore, the FL function values ​​for each initial aggregation scale are obtained using the following formula:

[0093] ;

[0094] ;

[0095] Where r represents the r-th initial aggregation scale; This represents the distance between the i-th OD stream data to be clustered and the j-th OD stream data to be clustered, such as... Figure 3 As shown, this represents the maximum Euclidean distance between point O of the i-th OD stream data to be clustered and point O of the j-th OD stream data to be clustered, and the maximum Euclidean distance between point D of the i-th OD stream data to be clustered and point D of the j-th OD stream data to be clustered; n represents the total number of OD stream data to be clustered. This indicates an indicator function that returns 1 if the condition is true and 0 otherwise. The density estimate of the OD stream data to be clustered is calculated as: the total number of OD stream data to be clustered / the area of ​​the smallest bounding rectangle containing all OD stream data to be clustered. Let FK represent the expected value of the FK function of several random OD stream datasets with the r-th initial clustering scale value. In this embodiment, FK represents the mean of the FK function of the M random OD stream datasets. This represents the distance between the p-th random OD stream data and the q-th random OD stream data in the m-th random OD stream dataset. This distance is also the Euclidean distance between each starting point and each ending point.

[0096] Step S42: Fit the FL function values ​​at each initial aggregation scale to obtain the global FL function curve.

[0097] Specifically, the FL function values ​​at each initial aggregation scale are fitted using a smoothing algorithm to obtain a continuous and smooth original curve, which facilitates the observation of the aggregation intensity's trend with scale. In this embodiment, spline interpolation can be used for fitting.

[0098] Step S43: Calculate the second derivative of the global FL function curve to obtain at least one minimum point in the second derivative curve of the global FL function curve.

[0099] Specifically, the second derivative of the global FL function curve obtained in step S42 is calculated to identify all local minima (i.e., troughs) on the second derivative curve. For example... Figure 4 As shown, the OD stream data contains a low signal-to-noise ratio cluster within the r=2 range, with a cluster noise ratio of approximately 4.4; further as... Figure 5 As shown, although the first derivative of the global FL function curve can reflect the changes of some clusters with scale, its second derivative is the smallest when r=2, which also indicates that the cluster stops expanding after r=2.

[0100] More specifically, the global FL function value changes with the increase of the initial clustering scale *r*, and its second derivative characterizes the rate of change of the first derivative of the global FL function curve, i.e., the acceleration of the clustering strength growth. When the second derivative curve has a minimum value, it indicates that the growth pattern of the clustering strength has fundamentally changed near this scale. This turning point indicates that at the current scale, the effective expansion of the target cluster has reached its limit, and further increasing the scale will introduce a large amount of irrelevant noise rather than cluster members. Therefore, the scale corresponding to this minimum point is the optimal spatial boundary of the cluster. The minimum point in the second derivative curve of the global FL function curve can objectively identify the characteristic scale of the cluster.

[0101] It should be noted that spatiotemporal data in the real world often exhibit nested or multi-scale clustering patterns. The second derivative curve of the global FL function curve may contain multiple local minima, each corresponding to a cluster of different spatial scales. This embodiment, by identifying all these minima, can simultaneously resolve the multi-scale clustering structures inherent in the data, overcoming the limitations of single-scale analysis models.

[0102] Step S44: Use the scale values ​​corresponding to each minimum point as potential clustering scales to obtain a potential clustering scale set.

[0103] Specifically, since each minimum point corresponds to an initial clustering scale value, which marks a significant change in the spatial clustering pattern of the OD flow data to be clustered, that is, a feature scale with prominent statistical significance, these initial clustering scale values ​​serve as potential clustering scales, indicating that there may be a cluster in the data with a spatial range that matches it.

[0104] Transforming the continuous initial clustering scale set into a discrete potential clustering scale set allows subsequent local spatial clustering analysis to focus only on these potential clustering scales, thereby improving computational efficiency and the purposefulness of clustering.

[0105] Step S5: Based on the OD flow dataset to be clustered and the random OD flow dataset, perform local spatial clustering analysis at each potential clustering scale of the potential clustering scale set to obtain a candidate center flow set.

[0106] Specifically, based on the potential clustering scale determined by global analysis, the local function value of each OD flow data to be clustered is calculated to obtain the core location of each potential cluster, providing accurate candidate points for the final cluster division.

[0107] Furthermore, the step of performing local spatial clustering analysis at each potential clustering scale of the potential clustering scale set based on the OD flow dataset to be clustered and the random OD flow dataset to obtain a candidate center flow set includes: performing the following steps S51-S53 on each potential clustering scale of the potential clustering scale set:

[0108] Step S51: Using several random OD stream datasets as a benchmark, calculate the local FL function value of each OD data stream to be clustered at the current potential clustering scale.

[0109] Specifically, by introducing a random OD stream dataset, the local original neighbor density of each OD data stream to be clustered is transformed into a density that is significantly higher than that of the random background. This effectively eliminates background noise interference, allowing the OD data streams to be clustered that truly have clustering core features to highlight their high local FL function values ​​in low signal-to-noise ratio environments.

[0110] Furthermore, the local FL function value of each OD data stream to be clustered at the current potential clustering scale is calculated using the following formula:

[0111] ;

[0112] ;

[0113] Where s represents the s-th potential clustering scale; The density estimate of the OD stream data to be clustered is calculated as: the total number of OD stream data to be clustered / the area of ​​the smallest bounding rectangle containing all OD stream data to be clustered. It represents the distance between the i-th OD stream data to be clustered and the j-th OD stream data, which is the maximum value of the Euclidean distance between the O point of the i-th OD stream data and the O point of the j-th OD stream data and the Euclidean distance between the D point of the i-th OD stream data and the D point of the j-th OD stream data. Let FK represent the expected value of the FK function of several random OD stream datasets for the s-th potential clustering scale value. In this embodiment, FK represents the mean of the FK function of the M random OD stream datasets. This represents the distance between the p-th random OD stream data and the q-th random OD stream data in the m-th random OD stream dataset. This distance is also the maximum value of the Euclidean distance between each starting point and each ending point.

[0114] Step S52: Based on the local FL function values ​​of each OD data stream to be clustered, an initial candidate center stream is obtained, forming an initial candidate center stream set, including:

[0115] Sort all OD stream data in the dataset to be clustered from high to low based on their corresponding local FL function values.

[0116] Select the top K% of the OD flow data to be clustered as the initial candidate center flow set; where K is a preset integer.

[0117] Specifically, by selecting OD streams with higher local FL function values ​​as initial candidate center streams, preliminary screening is performed to identify potential cluster centers at the current potential clustering scale, providing candidate objects for subsequent clustering analysis. More specifically, in this embodiment, K can be set to 20 to ensure coverage of all potential clustering cores.

[0118] Step S53: Remove redundant initial candidate center flows from the initial candidate center flow set to obtain the candidate center flow set corresponding to the current potential clustering scale, including:

[0119] Select the OD flow data with the highest local FL function value in the initial candidate center flow set, add it to the candidate center flow set, and remove it from the initial candidate center flow set.

[0120] The initial candidate center flows whose distance from the aforementioned initial candidate center flows is less than the current potential clustering scale value are removed from the initial candidate center flow set.

[0121] Repeat the above process until the initial candidate center flow set is empty, thus obtaining the candidate center flow set.

[0122] Specifically, by removing initial candidate center flows that are too close together, the problem of multiple candidate centers being overly clustered in space and jointly representing the same cluster is avoided. This effectively prevents redundancy and overlap in the final clustering results, ensuring that each candidate center flow has a certain degree of independence and representativeness at the current potential clustering scale. It is the flow with the most significant clustering intensity in the region, which can improve the quality of the clustering results.

[0123] More specifically, in step S5, a set of candidate center flows is obtained at each potential scale s.

[0124] Step S6: For each candidate center flow in the candidate center flow set, obtain the corresponding optimal clustering scale based on the change of its local spatial clustering intensity with each potential clustering scale, and obtain the corresponding cluster based on the optimal clustering scale of each candidate center flow.

[0125] Specifically, by independently analyzing the cluster function curve characteristics of each candidate center flow and assigning it a unique optimal clustering scale, it is possible to simultaneously identify clusters of different sizes, ensuring that the clustering results accurately reflect the true distribution of the data.

[0126] Furthermore, for each candidate center flow in the candidate center flow set, obtaining the corresponding optimal clustering scale based on the change of its local spatial clustering intensity with each potential clustering scale, and obtaining the corresponding cluster based on the optimal clustering scale of each candidate center flow, includes: performing the following steps S61-S64 for each candidate center flow in the candidate center flow set:

[0127] Step S61: Fit the local FL function values ​​of the current candidate center flow at each potential aggregation scale to obtain the local FL function curve corresponding to the current candidate center flow.

[0128] Specifically, based on the local FL function values ​​of the current candidate center flow at each potential clustering scale calculated in step S51, the local clustering intensity changes of the candidate center flow at different potential clustering scales can be quantified. Simultaneously, by fitting the local FL function values ​​of the current candidate center flow at each potential clustering scale using a smoothing algorithm, a continuous and smooth original curve is obtained. In this embodiment, spline interpolation can be used for fitting.

[0129] Step S62: Calculate the second derivative of the local FL function curve corresponding to the current candidate center flow, and obtain all the minimum points in the second derivative curve of the local FL function curve.

[0130] Specifically, the second derivative of the local FL function curve of the current candidate center flow obtained in step S62 is calculated, and all the minimum points in the second derivative curve are obtained. This reflects that the clustering effect centered on the current candidate center flow has undergone a fundamental change at this scale. Therefore, the potential clustering scale corresponding to this point is the key feature scale.

[0131] Step S63: Take the maximum potential clustering scale corresponding to each minimum point as the optimal clustering scale of the current candidate center flow.

[0132] Specifically, when there are more than one minimum point in step S62, the value with the largest potential clustering scale is selected as the optimal clustering scale for the current candidate center flow to ensure the integrity of the entire cluster and avoid splitting it into multiple sub-clusters that are too small.

[0133] Step S64: Using the current candidate center flow as the center, the OD flow data to be clustered that are within the optimal clustering scale from the current candidate center flow are taken as the clusters of the current candidate center flow. When an OD flow data to be clustered belongs to multiple clusters, it is assigned to the cluster closest to the candidate center flow.

[0134] Specifically, taking the spatial coordinates of each candidate center flow as the center and the corresponding optimal clustering scale as the threshold, all OD flow data to be clustered that are less than or equal to the corresponding optimal clustering scale are initially classified as members of the cluster. The distance to the candidate center flow is the maximum value of the Euclidean distance between the starting point (point O) and the ending point (point D) of two OD flow data.

[0135] More specifically, after all the OD stream data to be clustered has been assigned, it is checked whether there is an OD stream data that has been assigned to more than one cluster. If so, it is retained in the cluster with the smallest distance to each candidate center stream, and removed from other clusters. This ensures that each OD stream data ultimately belongs to only one cluster, guaranteeing the clarity and mutual exclusivity of the clustering results.

[0136] In summary, the method for identifying clusters in low signal-to-noise ratio (SNR) OD streaming data according to embodiments of the present invention has the following beneficial effects:

[0137] 1. The embodiments of the present invention generate a random OD stream dataset with the same spatial range as the OD stream dataset to be clustered as a statistical benchmark. Combined with global spatial clustering analysis and local spatial clustering analysis, it can effectively distinguish between noisy data and valid signals. In low signal-to-noise ratio environments, it can accurately identify true clusters, thereby providing a reliable basis for analysis and decision-making based on clustering results and significantly improving the accuracy of clustering.

[0138] 2. The embodiments of the present invention objectively determine the potential clustering scale and the optimal clustering range of each cluster in the data by calculating the second derivative of the spatial clustering function (FL function) and finding its minimum point. This replaces the operation of subjectively setting the density threshold and neighborhood radius in the traditional method. It not only avoids performance fluctuations caused by improper parameter settings, but also has excellent robustness and generalization ability across datasets.

[0139] 3. By adaptively assigning a unique optimal clustering scale to each candidate center flow, this embodiment of the invention can simultaneously and accurately identify clustering structures of different spatial ranges, effectively solving the multi-scale clustering problem. Furthermore, the algorithm ensures mutual exclusivity between clusters through post-processing steps, resulting in each final cluster being an independent pattern with a clear spatial structure and explicit semantics.

[0140] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for identifying clusters in low signal-to-noise ratio (SNR) OD streaming data, characterized in that, Includes the following steps: Obtain the OD stream dataset to be clustered; Based on the coordinates of each OD flow data in the OD flow data to be clustered, the clustering scale range of the OD flow data to be clustered is obtained and several initial clustering scale values ​​are generated to form an initial clustering scale sequence. A random OD flow data generation model using a generative adversarial network (GAN) architecture is used to generate several random OD flow datasets with a spatial range consistent with the OD flow dataset to be clustered. The random OD flow data generation model includes a generator and a discriminator. The generator is a four-layer fully connected neural network connected sequentially, with each layer followed by a batch normalization function and a LeakyReLU activation function. A Dropout layer is added to both the first and second fully connected neural networks. 64-dimensional random noise following a standard normal distribution is input to the generator, and the output is a 4-dimensional vector corresponding to the four coordinate values ​​of the OD flow data. The generator's output layer activation function is tanh, and the output result can be converted into actual OD coordinate pairs after inverse normalization. The discriminator... Similarly, it includes four fully connected neural networks connected in sequence. Each fully connected neural network is followed by a batch normalization function and a LeakyReLU activation function. The input is 4-dimensional OD data, and the output is a scalar to evaluate the authenticity of the input data. The random OD stream data generation model needs to select the optimal generation round from the training process, including: calculating the FK function values ​​of the generated random OD stream data and the real data at each of the initial aggregation scale values, and obtaining the 95% confidence interval strips of the function based on the 199 sampling results; using the area of ​​intersection to evaluate the overlap between the confidence interval strips of the generated random OD stream data and the real data calculated at each FK function value in each round; and using the generator with the highest overlap as the generator of random OD stream data to generate random OD stream data. Based on the initial clustering scale sequence, the OD flow dataset to be clustered, and several random OD flow datasets, a global spatial clustering analysis is performed to obtain a potential clustering scale set. Based on the OD flow dataset to be clustered and several random OD flow datasets, local spatial clustering analysis is performed at each potential clustering scale of the potential clustering scale set to obtain a set of candidate center flows. For each candidate center flow in the candidate center flow set, the corresponding optimal clustering scale is obtained according to the change of its local spatial clustering intensity with each potential clustering scale, and the corresponding cluster is obtained according to the optimal clustering scale of each candidate center flow.

2. The method according to claim 1, characterized in that, The process involves performing global spatial clustering analysis based on the initial clustering scale sequence, the OD flow dataset to be clustered, and several random OD flow datasets to obtain a potential clustering scale set, including: Using several random OD stream datasets as a reference, calculate the FL function value of the OD stream dataset to be clustered at each initial clustering scale of the initial clustering scale sequence; The FL function values ​​at each initial aggregation scale are fitted to obtain the global FL function curve; Calculate the second derivative of the global FL function curve to obtain at least one local minimum point in the second derivative curve of the global FL function curve; The scale values ​​corresponding to each minimum point are used as potential clustering scales to obtain a potential clustering scale set.

3. The method according to claim 2, characterized in that, The FL function values ​​for each initial aggregation scale are obtained using the following formula: ； ； Where r represents the r-th initial aggregation scale; This represents the distance between the i-th OD stream data to be clustered and the j-th OD stream data to be clustered; n represents the total number of OD stream data to be clustered. Indicates an indicator function; This represents the density estimate of the OD flow data to be clustered; Let FK represent the expected value of a random OD stream dataset with the r-th initial aggregation scale value; This represents the distance between the p-th random OD stream data and the q-th random OD stream data in the m-th random OD stream dataset.

4. The method according to claim 2, characterized in that, The step of performing local spatial clustering analysis at each potential clustering scale of the potential clustering scale set based on the OD flow dataset to be clustered and the random OD flow dataset to obtain a candidate center flow set includes: performing the following operations on each potential clustering scale of the potential clustering scale set: Using several random OD stream datasets as a benchmark, calculate the local FL function value of each OD data stream to be clustered at the current potential clustering scale; Based on the local FL function values ​​of each OD data stream to be clustered, an initial candidate center stream is obtained, forming an initial candidate center stream set; Redundant initial candidate center flows are removed from the initial candidate center flow set to obtain the candidate center flow set corresponding to the current potential clustering scale.

5. The method according to claim 4, characterized in that, The local FL function value of each OD data stream to be clustered at the current potential clustering scale is calculated using the following formula: ； ； Where s represents the s-th potential clustering scale; This represents the density estimate of the OD flow data to be clustered; This represents the distance between the i-th OD stream data to be clustered and the j-th OD stream data to be clustered; Let FK represent the expected value of the function for several random OD flow datasets with the s-th potential clustering scale value; This represents the distance between the p-th random OD stream data and the q-th random OD stream data in the m-th random OD stream dataset.

6. The method according to claim 5, characterized in that, The initial candidate center streams are obtained based on the local FL function values ​​of each OD data stream to be clustered, forming an initial candidate center stream set, including: Sort all OD stream data in the dataset to be clustered from high to low based on their corresponding local FL function values; Select the top K% of the OD flow data to be clustered as the initial candidate center flow set; where K is a preset integer.

7. The method according to claim 6, characterized in that, The step of removing redundant candidate center flows from the initial candidate center flow set to obtain a candidate center flow set includes: Select the OD flow data with the highest local FL function value in the initial candidate center flow set, add it to the candidate center flow set, and remove it from the initial candidate center flow set; The initial candidate center flows whose distance from the aforementioned initial candidate center flows is less than the current potential clustering scale value are removed from the initial candidate center flow set; Repeat the above process until the initial candidate center flow set is empty, thus obtaining the candidate center flow set.

8. The method according to claim 7, characterized in that, The step of obtaining the corresponding optimal clustering scale for each candidate center flow in the candidate center flow set based on the change of its local spatial clustering intensity with each potential clustering scale, and obtaining the corresponding cluster based on the optimal clustering scale of each candidate center flow, includes: performing the following operations for each candidate center flow in the candidate center flow set: Fit the local FL function values ​​of the current candidate center flow at each potential clustering scale to obtain the local FL function curve corresponding to the current candidate center flow; Calculate the second derivative of the local FL function curve corresponding to the current candidate center flow, and obtain all the minimum points in the second derivative curve of the local FL function curve; The maximum potential clustering scale corresponding to each of the aforementioned minimum points is taken as the optimal clustering scale for the current candidate center flow. Using the current candidate center flow as the center, the OD flow data to be clustered that are within the optimal clustering scale of the current candidate center flow are taken as the clusters of the current candidate center flow. When an OD flow data to be clustered belongs to multiple clusters, it is assigned to the cluster closest to the candidate center flow.

9. The method according to any one of claims 1-8, characterized in that, The step of generating several random OD stream datasets that are consistent with the spatial range of the OD stream dataset to be clustered includes: randomly and independently generating start coordinates and end coordinates within the spatial range of the OD stream dataset to be clustered, and obtaining random OD stream data with the same number as the OD stream data as a set of random OD stream datasets.

10. The method according to claim 9, characterized in that, The method involves obtaining the clustering scale range of the OD flow data based on the coordinates of each OD flow data in the dataset to be clustered, and generating several initial clustering scale values, including: Based on the starting and ending coordinates of the OD flow data of each cluster, the minimum bounding rectangle region including the OD flow data of each cluster is obtained and its diagonal length is calculated. Based on the diagonal length, the clustering scale range of the OD flow dataset to be clustered is obtained; Within the said aggregation scale range, several initial aggregation scale values ​​are generated at equal intervals.

Images