Network digital twinning-oriented dual-view flow data sampling method and system

Abstract

The invention discloses a network digital twinning-oriented dual-view flow data sampling method and system, and mainly solves the problems of difficult key path capture and low flow reconstruction precision caused by lack of frequency domain perception of a flow sampling method in the prior art. According to the implementation scheme, historical flow interaction data in a network node sliding window are collected to form a third-order flow tensor and normalized, and node frequency domain features are extracted and then smoothly updated to obtain a smooth feature vector; mapping the vector to determine a node category, and dividing source-destination OD pair traffic categories of the whole network; calculating statistics lever scores of all OD pairs, dynamically allocating sampling budget of each category according to the statistics lever scores, and generating a sampling set and a binary sampling mask by adopting a determinacy and random combination strategy; and collecting the flow data of the corresponding position at the current moment based on the mask, constructing a tensor completion optimization model for solving completion, and outputting the recovered whole network flow data. According to the method, through double-view cooperation of the frequency domain features and the statistical lever fraction, precise coverage of a high-value traffic path is realized, and the traffic reconstruction precision of the digital twin network is effectively improved while the sampling overhead is reduced. The method can be applied to measurement and reconstruction of large-scale network traffic.

Description

Technical Field

[0001] This invention belongs to the field of network measurement and digital twin technology, and specifically relates to a method and system for traffic data sampling and recovery, which can be applied to the measurement and reconstruction of large-scale network traffic. Background Technology

[0002] Digital twin network (DTN) is a network architecture that accurately maps the topology, state attributes, and operational behavior of a physical network to a virtual space through real-time data interaction. The network traffic matrix records the traffic interaction data between all source-destination node (OD) pairs in the network, serving as the data foundation for building high-fidelity digital twin models, conducting refined traffic engineering, and detecting network anomalies. However, with the growth of network scale, performing real-time and continuous direct measurement of all OD pairs across the entire network faces severe challenges such as high bandwidth overhead and deployment costs. Therefore, employing sparse sampling techniques to collect traffic data from a small number of key OD pairs and using matrix or tensor completion algorithms to infer the overall network traffic has become a key path to achieving low-overhead, high-precision network awareness.

[0003] Patent document CN110138614A discloses "An Online Network Traffic Anomaly Detection Method and System Based on Tensor Model". It acquires current-time data in real-time through a fixed window, uses historical tensor data for iterative training and factor updates of the current data, and decomposes the observed data into a low-rank normal component and a sparse anomaly component, thereby achieving online detection of traffic anomalies. However, because this method primarily focuses on using low-rank sparse decomposition to separate anomalous traffic, it lacks an active sampling decision mechanism for scenarios with limited sampling resources. Therefore, when processing traffic data, it cannot fully utilize the structural differences of different OD flows in the frequency domain to guide the optimal allocation of sampling resources.

[0004] Patent document CN115225528A discloses a "Distributed Measurement and Scheduling Method, System, and Medium for Network Traffic Data." It introduces Jensen-Shannon divergence and variance to quantify the distribution changes and fluctuations of network traffic within adjacent time windows, calculates the importance of row vectors in each factor matrix, and constructs a system of linear equations to prioritize the collection of measurement points with high importance. Finally, it uses historical data and newly sampled data to reconstruct the entire network traffic. While this method relies on statistical differences in the time domain for sampling point selection, it fails to deeply explore the fluctuation characteristics of traffic data in the frequency domain. Therefore, it struggles to achieve refined sampling and budget allocation for traffic with the same statistical distribution but different frequency domain patterns. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the prior art by providing a dual-view traffic data sampling method and system for network digital twins, so as to achieve refined sampling of traffic in different frequency domain modes, thereby improving the traffic reconstruction accuracy in digital twin networks.

[0006] To achieve the above objectives, the technical solution adopted by the present invention includes:

[0007] 1. A dual-view traffic data sampling method for network digital twins, characterized in that it includes:

[0008] (1) Collect historical traffic interaction data of network nodes within the sliding window to form a third-order traffic tensor, and normalize it to obtain the normalized traffic tensor.

[0009] (2) Extract the node frequency domain features from the normalized flow tensor and update them smoothly to obtain a smooth feature vector;

[0010] (3) Map the smooth feature vector and combine it with the frequency domain features to determine the node category, and then divide the source-destination node OD pairs of the entire network into different traffic categories;

[0011] (4) Calculate the statistical leverage score of all OD pairs, dynamically allocate the sampling budget for each flow category based on the score, and select a portion of OD pairs using a combination of deterministic and random strategies to generate a sampling set. and the corresponding binary sampling mask ;

[0012] (5) Based on the sampling mask, the traffic data at the corresponding location at the current time is collected to construct a tensor completion optimization model, solve and complete the observed data, and output the recovered network traffic data.

[0013] Furthermore, the mapping of the smooth feature vector and the classification of traffic categories in (3) are implemented as follows:

[0014] (3a) Generate a random projection matrix that follows a standard normal distribution. Where L is the number of projections and D is the feature dimension; the smooth feature vector of each node is projected to a low-dimensional space, and mapped to multiple discrete hash buckets according to the projection value, so as to achieve preliminary clustering of nodes with similar frequency domain fluctuation patterns;

[0015] (3b) Perform semantic analysis on the nodes within the hash bucket and calculate the high-frequency energy ratio (Ratio) of the node feature vectors:

[0016] (3c) Based on the numerical range of Ratio, the source node and the destination node are respectively marked as Z traffic categories with different frequency domain characteristics;

[0017] (3d) Combine the source node category and the destination node category to obtain the set of traffic categories to which each OD pair belongs in the entire network.

[0018] Furthermore, in step (4), the statistical leverage score of all OD pairs is calculated, the sampling budget for each flow category is dynamically allocated based on the statistical leverage score, and a combination of deterministic and random strategies is used to select a subset of OD pairs, including:

[0019] (4a) Calculate the source node lever fraction and target node leverage score :

[0020] (4b) Calculate each OD pair Comprehensive statistical leverage score :

[0021] (4c) Calculate the set of the m-th traffic category The sum of the statistical leverage scores of all OD pairs included is defined as the total weight of the category. The total number of OD pairs contained in a category is counted and defined as the capacity of that category. ;

[0022] (4d) Set the total sampling budget A, based on the total weight of each category. The proportion of the total network weight is used to calculate the initial budget. :

[0023] (4e) Initial target budgets for each category Its capacity Comparison:

[0024] If the target budget for this category exceeds its capacity, execute (4f).

[0025] Otherwise, use the initial target budget;

[0026] (4f) Truncate the final sample number if the target budget exceeds its capacity category. The excess budget is then returned to the public surplus budget pool, and the quota in the surplus budget pool is iteratively allocated to categories that have not yet reached their capacity limits according to their weighted proportions, until all budgets are allocated, thus obtaining the final sample size for each category. ;

[0027] (4g) A hybrid sampling strategy is adopted within each traffic category, based on a preset deterministic sampling ratio. The final number of samples for this category Decomposed into deterministic sampling part and random sampling ,satisfy .

[0028] 2. A dual-view traffic data sampling system for network digital twins, characterized in that it comprises:

[0029] Traffic data acquisition module: used to monitor network ports or read log files in real time, collect historical traffic interaction data between source nodes and destination nodes in the network within a sliding window, and obtain the initial third-order traffic tensor;

[0030] The preprocessing module receives the initial third-order flow tensor and performs numerical scaling on it using the max-min normalization method to obtain the normalized flow tensor.

[0031] Frequency domain feature extraction module: used to perform fast Fourier transform on the normalized flow tensor along the time dimension, and to perform truncated singular value decomposition on the frequency slice to extract the instantaneous frequency domain feature vector that reflects the energy distribution of node flow fluctuations.

[0032] Frequency domain feature update module: used to receive instantaneous frequency domain feature vectors and use an exponentially weighted moving average mechanism to smoothly update the feature vectors to obtain smooth feature vectors;

[0033] The traffic category segmentation module is used to map the smooth feature vector based on local sensitive hashing and combine it with the frequency domain energy ratio feature to determine the node category, thereby dividing the source-destination OD node pairs of the entire network into different traffic category sets.

[0034] The dual-view sampling decision module calculates the comprehensive statistical leverage score for all OD node pairs. Based on this score, it dynamically allocates the sampling budget for each flow category using a weighted water injection algorithm. Within each flow category, it employs a hybrid sampling strategy combining deterministic and random sampling to select a subset of OD pairs, generating a sampling set. and the corresponding binary sampling mask tensor ;

[0035] The flow tensor completion and reconstruction module is used to complete and reconstruct the flow tensor based on the binary sampling mask tensor. Acquire sparse observation data at the current moment, construct a tensor completion optimization model using the low-rank characteristics of network traffic, and iteratively solve the model using the alternating direction multiplier method to output the restored complete network traffic data.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] Firstly, this invention introduces statistical leverage scores and weighted water injection algorithms to dynamically allocate sampling budgets, enabling sampling resources to be tilted towards critical paths with high information content. Therefore, it can effectively solve the problem of low efficiency of traditional random sampling and achieve high-precision coverage of the entire network traffic structure at low sampling rates.

[0038] Secondly, this invention employs frequency domain feature extraction and local sensitive hashing classification mechanisms to distinguish traffic patterns from the perspective of fluctuation frequency. Therefore, it breaks through the limitation of existing technologies that rely solely on numerical size for classification and significantly improves the system's ability to perceive sudden network traffic and abnormal fluctuations. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating the implementation of the dual-perspective traffic data sampling method for network digital twins according to the present invention.

[0040] Figure 2 This is a block diagram of the dual-view traffic data sampling system for network digital twins according to the present invention;

[0041] Figure 3 The figure shows the simulation results comparing the relative standard error of traffic completion of the present invention and several existing sampling methods on a real network traffic dataset. Detailed Implementation

[0042] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention and not all embodiments. Based on the embodiments of the present invention, other embodiments obtained by those skilled in the art without creative effort should all fall within the protection scope of the present invention.

[0043] It should be noted that the step numbers in the specification and claims of this invention are only for the purpose of clearly describing the embodiments of this invention and facilitating understanding, and their order is not limited.

[0044] Example 1: A dual-perspective traffic data sampling method for network digital twins.

[0045] The network digital twin refers to the construction of a digital mirror of a physical network in virtual space through real-time data interaction to achieve the mapping and simulation of network status. To obtain complete network traffic data required for constructing a digital twin under limited bandwidth resources, this invention employs a strategy combining sparse sampling and tensor completion.

[0046] Reference Figure 1 This example includes the following steps:

[0047] Step 1: Obtain historical traffic data and obtain the normalized traffic tensor:

[0048] 1.1) Collect data from the network Individual source nodes and Each destination node contains Traffic interaction data within a sliding window at each time step. Source-destination traffic records with timestamps are obtained by reading network traffic monitoring log files. As time progresses, a first-in, first-out (FIFO) strategy is used to remove data from the earliest time step within the window and incorporate traffic data arriving at the latest time, updating in real time to obtain the current third-order traffic tensor. Each element of the tensor Representing the From the source node to the first The destination node is at the _ The flow value at each time step;

[0049] 1.2) Since the traffic volumes of the backbone network and the edge network, as well as those during peak and off-peak hours, can differ by several orders of magnitude, directly using the raw data for feature extraction can lead to overfitting of the model to high-traffic nodes. Therefore, it is necessary to normalize the traffic data in the tensor. Commonly used data normalization methods include Z-Score standardization, fractional scaling standardization, and max-min normalization. Given the non-negativity of network traffic data, this example uses, but is not limited to, the max-min normalization method, which can linearly map all traffic data to... The closed interval is used to eliminate the impact of large differences in traffic volume across different links on subsequent feature extraction, effectively improving the model's numerical stability and training efficiency. Specifically, the normalization calculation formula used in this embodiment is as follows:

[0050] ,

[0051] in, Represents the normalized i-th The source node, the first The destination node is at the _ The flow value at each time step; This represents the value of the corresponding element in the original flow tensor; and ϵ represents the minimum and maximum values ​​of the flow data within the current sliding window, respectively, and is used to prevent the introduction of a minimum value when the denominator is zero.

[0052] The above processing eliminates the impact of large differences in traffic volume between different links on the convergence speed of subsequent feature extraction and tensor decomposition algorithms, effectively improving the numerical stability of the model.

[0053] Step 2: Extract and smoothly update the frequency domain feature vector of the node.

[0054] 2.1) To capture the periodic patterns of the traffic, the normalized traffic tensor... The first k time steps, i.e., historical data slices, are subjected to Fast Fourier Transform (FFT) along the time dimension to obtain the frequency domain tensor. and to Each frequency slice Perform truncated singular value decomposition:

[0055] ,

[0056] in, , , These are the left singular vector matrix, singular value matrix, and right singular vector matrix of the k-th frequency slice, respectively, with superscripts... Indicates conjugate transpose;

[0057] 2.2) Based on the above decomposition results, construct the instantaneous source node feature matrix for the current time step t. and instantaneous target node feature matrix Each row of the matrix represents the frequency domain eigenvector of a node. This vector is formed by concatenating the singular vectors of that node across all frequency slices after weighted and logarithmic transformations. The calculation formula is as follows:

[0058] ,

[0059] in, This indicates a splicing operation. Indicates the modulo value. This indicates a logarithmic transformation operation. The features extracted in this step can reflect the energy distribution of a node at different frequency components, thus reflecting the severity of its flow fluctuations;

[0060] 2.3) Frequency domain feature smoothing:

[0061] Because network traffic is dynamically changing, the instantaneous features extracted by a single sliding window may be affected by sudden noise. Common feature smoothing methods include Simple Moving Average (SMA) and Weighted Moving Average (WMA), but they have high computational costs or are not sensitive enough to immediate changes when processing streaming data. This example uses, but is not limited to, the Exponentially Weighted Moving Average (EWMA) mechanism. By assigning higher weights to recent data, it ensures the continuity and smoothness of the feature vector over time, thereby avoiding drastic fluctuations in subsequent clustering results due to sudden fluctuations at individual time steps. Specifically, the frequency feature smoothing in this embodiment uses the Exponentially Weighted Moving Average (EWMA) mechanism, and its calculation formula is as follows:

[0062] ,

[0063] in, This is a smoothing coefficient used to adjust the sensitivity to data at the current time. This represents the smoothed feature matrix updated at time t, which in actual calculation corresponds to the smoothed matrix of the source node. and the smoothing matrix of the target node , This represents the calculated instantaneous feature matrix. This represents the smooth feature matrix cached at the previous time step.

[0064] Step 3: Node clustering based on locality-sensitive hashing.

[0065] To address the problem of fast node retrieval and clustering in feature space, traditional K-Means or DBSCAN algorithms suffer from high computational complexity. This example employs, but is not limited to, the Locality Sensitive Hashing (LSH) algorithm. By constructing a random projection matrix, it ensures that similar nodes in the original space collide with the same hash bucket with a very high probability. This transforms the complex problem of comparing nodes across the entire network into an efficient table lookup operation, enabling rapid classification of OD pairs. The implementation includes the following:

[0066] 3.1) Generate a standard normal distribution random projection matrix , where L is the number of projections and D is the feature dimension. The smooth feature vector of each node is multiplied by the projection matrix P to obtain the projection values ​​of the node in the L projection directions.

[0067] 3.2) Based on the numerical range of the projected values, they are mapped to multiple discrete hash buckets. Nodes falling into the same hash bucket are considered to have similar frequency domain fluctuation patterns, thereby achieving preliminary clustering of nodes;

[0068] 3.3) To determine the specific traffic pattern of nodes within the hash bucket, such as whether it is stable or bursty traffic, semantic analysis is performed on the node features, and the high-frequency energy ratio (Ratio) of the node feature vector is calculated:

[0069] ,

[0070] in, Let be the amplitude of the eigenvector at the k-th frequency component. and These represent the preset high-frequency band set and low-frequency band set, respectively. This indicator can effectively quantify the burstiness of traffic. The larger the Ratio value, the higher the proportion of high-frequency component energy and the more violent the node traffic fluctuation.

[0071] 3.4) Based on the numerical distribution range of the Ratio, the source node and destination node are each labeled as one of Z traffic categories with different frequency domain characteristics. The source node categories and destination node categories are then combined to represent the entire network. Each OD pair is divided into A set of flow categories with different frequency domain fluctuation characteristics For example, if both the source and destination nodes are high-frequency, then the OD pair is classified as high-frequency-high-frequency. This hierarchical method provides a basis for subsequently allocating differentiated sampling budgets for different categories.

[0072] Step 4: Perform dual-view hierarchical sampling decision.

[0073] To accurately quantify the importance of each OD pair in maintaining the low-rank structure of the overall network traffic tensor, this example uses the statistical leverage score as an evaluation metric. Compared to simple traffic numerical values, the leverage score can reveal the information density of the data from an algebraic structure perspective. Its implementation is as follows:

[0074] 4.1) Using the singular tensors obtained from the T-SVD decomposition in step 2 and Calculate the source node leverage fraction and target node leverage score :

[0075] ,

[0076] in, This represents the feature vector of the i-th source node at the k-th time step. This represents the feature vector of the j-th target node at the k-th time step;

[0077] 4.2) Based on the source node leverage fraction and target node leverage score Calculate each OD pair Comprehensive statistical leverage score :

[0078] ,

[0079] This formula can comprehensively reflect the importance of the source node and the destination node. The higher the score, the more important the OD is in maintaining the low-rank structure of the tensor.

[0080] 4.3) Calculate the set of the m-th traffic category. The sum of the statistical leverage scores of all OD pairs included is defined as the weight of that category. The calculation formula is as follows:

[0081] ,

[0082] in, Indicates OD pair The comprehensive statistical leverage score, This represents the set of OD pairs belonging to the m-th traffic category, and the total number of OD pairs contained in this category is defined as the capacity of this category. ;

[0083] 4.4) Implementation of dynamic sampling budget allocation among categories:

[0084] To maximize recovery accuracy with limited sampling resources, it is necessary to consider the total weight of each category. The budget is dynamically allocated based on its size. Common budget allocation methods include uniform allocation or simple weighted allocation. Uniform allocation ignores the differences in information density between different categories, while simple weighted allocation does not consider the physical capacity limitations of each category. This example uses, but is not limited to, the weighted water-filling algorithm, which automatically redistributes the overflowing budget to other unsaturated categories through an iterative mechanism, maximizing the global sampling efficiency. Its implementation is as follows:

[0085] 4.4.1) Let the total sampling budget be A, and then calculate the sampling budget based on the weights of each category. The proportion of the total network weight is used to calculate the initial budget. :

[0086] ;

[0087] 4.4.2) Initial target budgets for each category Its capacity Comparison:

[0088] If the target budget for a category does not exceed its capacity, then its initial target budget shall be used;

[0089] If the target budget for a category exceeds its capacity, it indicates that the category is saturated, and its final sample size is truncated. The excess budget is then returned to the public surplus budget pool. The remaining budget is then iteratively allocated to categories that have not yet reached their capacity limits, according to their weighted proportions, until all budgets are allocated, resulting in the final sample size for each category. ;

[0090] 4.5) Implementation of mixed sampling strategy within categories:

[0091] Within each traffic category, while purely deterministic sampling can capture the main structure, it easily overlooks non-essential but crucial details. Purely random sampling is inefficient. Therefore, this example employs a hybrid sampling strategy to balance sampling accuracy and exploratory nature. Its implementation includes:

[0092] 4.5.1) Set the deterministic sampling ratio to According to the preset deterministic sampling ratio The final number of samples for the m-th category Decomposed into deterministic sampling part and random sampling part The specific calculation formula is as follows:

[0093] ,

[0094] in, This indicates the rounding operation, and satisfies the following conditions: ;

[0095] 4.5.2) For all OD pairs included in the m-th traffic category, calculate their comprehensive statistical leverage score. Sort in descending order and select the highest-ranking result. OD pairs constitute the deterministic sampling set for this category. ;

[0096] 4.5.3) For all OD pairs contained in the m-th traffic category, remove the deterministic sampling set. The included OD pairs constitute the remaining candidate set, and random selection is made from the remaining candidate set. OD pairs constitute a random sampling set .

[0097] 4.5.4) For the above deterministic sampling set and random sampling set Perform a union operation to obtain the complete sample set of the m-th traffic category. ,Right now This set contains both high-weight backbone traffic features and retains the global exploratory nature of random sampling;

[0098] 4.6) Traverse all M traffic categories and collect the complete sample sets of each category obtained in the above steps. Perform a union operation to obtain the total sample set of the entire network. :

[0099] ,

[0100] This set It covers all high-priority backbone nodes and randomly selected exploration nodes in the entire network, and serves as the direct basis for generating the sampling mask in the future.

[0101] 4.7) Set the total sampling set of the entire network All OD pairs included in the binary sampling mask The corresponding position is marked as 1, and the rest are marked as 0, thus generating the final sampling mask. :

[0102] ,

[0103] in, This represents the element value of the sampling mask tensor at the k-th time step, the i-th source node, and the j-th destination node.

[0104] Step 5, Flow Tensor Completion and Reconstruction:

[0105] 5.1) Use a sampling mask Perform a Hadamard product with the actual total network traffic tensor, and retain only the data at positions where the mask value is 1, to form a sparse observation tensor. :

[0106] ,

[0107] in, Represents the actual total network traffic tensor that objectively exists in the physical network at the current moment; symbol This represents the Hadamard product operation, which is the direct multiplication of elements in the same position of two tensors;

[0108] 5.2) Utilizing the low-rank characteristic of network traffic data, from sparse observation traffic tensors To restore the entire network traffic, an optimization objective function based on minimizing the tensor kernel norm is constructed:

[0109] ,

[0110] in, For the complete flow tensor to be recovered, The nuclear norm of a tensor is the sum of the singular values ​​of all the matrices expanded from the tensor. The projection operator is defined as follows: This operator is used to constrain the recovered tensor in the sampling mask. The value corresponding to the position with a value of 1 must match the observed data. To maintain consistency, while ensuring data fidelity, low-rank prior information is used to fill in the missing unsampled data;

[0111] 5.3) To solve the above objective function, i.e., the constrained convex optimization problem, an auxiliary tensor is introduced. and Lagrange multiplier tensor This problem can be transformed into an unconstrained augmented Lagrangian function form:

[0112] ,

[0113] in, For penalty parameters, This represents the inner product operation. Denotes the Frobenius norm, in which the Lagrange multiplier tensor is in this function. Used to handle low-rank constraints of nuclear norm; auxiliary tensor , used to handle low-rank constraints of the nuclear norm; This is the traffic tensor to be recovered, used to handle data consistency constraints;

[0114] 5.4) Solving the low-rank tensor completion problem:

[0115] Algorithms for solving the low-rank tensor completion problem mainly include Alternating Least Squares (ALS), Stochastic Gradient Descent (SGD), and Alternating Directional Multiplier Method (ADMM). Since ALS is prone to getting trapped in local optima, and SGD has a slow convergence speed and its accuracy is greatly affected by the step size, this example uses, but is not limited to, the Alternating Directional Multiplier Method (ADMM). By introducing the augmented Lagrangian function, the global optimization problem, which is difficult to solve directly, is transformed into a simpler subproblem that is easier to solve. This ensures that the algorithm converges to the global optimum while strictly satisfying the data consistency constraint. Its implementation includes:

[0116] 5.4.1) To strictly satisfy the data consistency constraint at the sampling location, and to perform numerical estimation for the unsampled region, the low-rank auxiliary tensor of the k-th round is fixed. and Lagrange multiplier tensor Calculate the data consistency tensor for the (k+1)th round using the projection operator. :

[0117] ,

[0118] in, This indicates a projection operation on the unsampled region. This indicates a projection operation applied to the sampling region;

[0119] This step involves dividing the tensor into two complementary regions, a sampling region and an unsampling region, and assigning values ​​to them respectively: the sampling region is the binary sampling mask. For locations with a value of 1 within the sampling area, their values ​​are forcibly reset to reflect the actual sparse observation data. The unsampled area is the binary sampling mask. For locations where the value is 0, in the unsampled region, the value of the tensor to be recovered is updated to the predicted value calculated by the current model. To fill in the missing information;

[0120] 5.4.2) To remove noise from traffic data and enhance the low-rank characteristics of the entire network traffic, the data consistency tensor of the (k+1)th round is fixed. And the Lagrange multiplier tensor of the kth round The singular value thresholding operator is applied to denoise the intermediate variables and a low-rank constraint is imposed to obtain the low-rank auxiliary tensor of the (k+1)th round. :

[0121] ,

[0122] in, Indicates the threshold is The singular value threshold operator, i.e., for the input tensor Singular values ​​perform shrinkage operations This filters out high-frequency noise.

[0123] 5.4.3) Calculate the data consistency tensor Sum and low-rank auxiliary tensors The constraint residuals between them are used as the gradient direction of the dual function to penalize the parameters. Step size tensor with respect to Lagrange multipliers Update the function to maximize the dual objective function:

[0124] ;

[0125] 5.5) Repeat steps (5.4.1) to (5.4.3) to calculate the relative error between two adjacent iterations until the error is less than a preset threshold. Then stop iterating and output the finally converged tensor. This is the restored network traffic data.

[0126] Example 2: Dual-view traffic data sampling system for network digital twins.

[0127] Reference Figure 2 This example includes a traffic data acquisition module 1, a preprocessing module 2, a frequency domain feature extraction module 3, a frequency domain feature update module 4, a traffic category classification module 5, a dual-view sampling decision module 6, and a traffic tensor completion and reconstruction module 7. The traffic tensor completion and reconstruction module 7 includes a sparse traffic acquisition submodule 71, a model building module 72, a model transformation submodule 73, and an iterative solution submodule 74. Detailed functional descriptions of each module are as follows:

[0128] The traffic data acquisition module 1 is used to monitor network ports or read log files in real time, collect historical traffic interaction data between source nodes and destination nodes in the network within a sliding window, obtain an initial third-order traffic tensor, and transmit it to the preprocessing module 2.

[0129] The preprocessing module 2 is used to numerically scale the initial third-order flow tensor using the maximum-minimum normalization method to obtain a normalized flow tensor, so as to eliminate the influence of the difference in the basis of different links on subsequent calculations, and transmit it to the frequency domain feature extraction module 3.

[0130] The frequency domain feature extraction module 3 is used to perform a fast Fourier transform on the normalized flow tensor along the time dimension, and to perform truncated singular value decomposition on the frequency slice to extract the instantaneous frequency domain feature vector that reflects the energy distribution of node flow fluctuations, and transmit it to the frequency domain feature update module 4.

[0131] The frequency domain feature update module 4 is used to smooth the received instantaneous frequency domain feature vector using an exponentially weighted moving average mechanism to obtain a smooth feature vector, so as to ensure the continuity of features in the time dimension and suppress burst noise, and transmit the smooth feature vector to the traffic category classification module 5.

[0132] The traffic category segmentation module 5 is used to map the smooth feature vector based on local sensitive hashing and determine the node category by combining the frequency domain energy ratio feature, thereby dividing the source-destination OD node pairs of the entire network into different traffic category sets and transmitting them to the dual-view sampling decision module 6.

[0133] The dual-view sampling decision module 6 is used to calculate the comprehensive statistical leverage score of all OD node pairs, dynamically allocate the sampling budget for each flow category based on the score and using a weighted water injection algorithm, and select a portion of OD pairs within each flow category using a hybrid sampling strategy combining determinism and randomness to generate a sampling set. and the corresponding binary sampling mask tensor And transmit it to the flow tensor completion and reconstruction module 7;

[0134] The traffic tensor completion and reconstruction module 7 is used to obtain sparse observation data at the current moment based on the binary sampling mask tensor, and to recover the entire network traffic using a tensor completion algorithm. The sparse traffic acquisition submodule 71 is used to receive the binary sampling mask tensor. Based on the location indicator where the value is 1, the actual traffic data of the corresponding network OD node pair at the current moment is collected in a directional manner, and the unsampled locations are set to zero to obtain a sparse observation tensor. Model building module 72 is used to receive the sparse observation tensor. The algorithm constructs a minimum nuclear norm tensor completion optimization model constrained by maintaining the consistency of observed data. A model transformation submodule 73 receives the tensor completion optimization model and transforms it into an unconstrained augmented Lagrange function form by introducing auxiliary tensors and Lagrange multipliers. An iterative solution submodule 74 solves the augmented Lagrange function using the alternating direction multiplier method. It iteratively executes data consistency tensor updates, low-rank auxiliary tensor updates, and Lagrange multiplier updates until the algorithm converges. This submodule outputs the restored complete network traffic data.

[0135] It should be noted that the above functional modules can be implemented, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, as program instruction products. A program instruction product includes one or a set of program instructions. When the program instructions are loaded and executed on a computer, the described process or function is generated, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The program instructions can be stored in a computer-readable and writable storage medium or transferred from one computer's readable and writable storage medium to another.

[0136] The direct coupling or communication connections between the modules shown or discussed in this embodiment can be achieved through indirect coupling or communication connections via interfaces, devices, or modules. The various functional modules and sub-modules in this embodiment can dynamically reside within a single processing unit, or each module can exist physically independently, or two or more modules can dynamically reside within a single processing unit. When these dynamic components are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable and writable storage medium. This storage medium can be a memory, disk, or optical disc, etc.

[0137] The technical effects of the present invention will be further explained below with reference to simulation experiments:

[0138] 1. Simulation experimental conditions:

[0139] The experimental computer runs Windows 11 (64-bit), has 24GB of RAM, and is equipped with an Intel(R) Core(TM) i3-10105 CPU @ 3.70GHz (3.70 GHz). The system is implemented using Python, version 3.8.

[0140] The experiment was conducted using JetBrains PyCharm Community Edition 2020.2, with major dependencies including NumPy, SciPy, scikit-learn, and Matplotlib.

[0141] The datasets used in the simulation include three real network traffic datasets: Abilene, Germany17, and GEANT.

[0142] The simulation comparison methods used include: Random sampling algorithm, Full ETBS (Full History Tube Sampling Algorithm), and Windowed ETBS (Tunnel Sampling Algorithm with Sliding Window Mechanism).

[0143] The formula for calculating the relative standard error (RSE) is:

[0144] ,

[0145] in, A tensor representing the actual network traffic. This represents the flow tensor recovered by the algorithm. This represents the Frobenius norm; the smaller the RSE value, the closer the recovered traffic data is to the real data, and the higher the recovery accuracy.

[0146] 2. Simulation content and result analysis:

[0147] Simulation 1: Using the present invention and the three existing methods described above, we performed completion accuracy simulations on the Abilene dataset. The results are as follows: Figure 3 (a). The horizontal axis represents the sampling rate, and the vertical axis represents the average instantaneous error (RSE).

[0148] Simulation 2: Using the present invention and the three existing methods described above, we performed completion accuracy simulations on the Germany17 dataset. The results are as follows: Figure 3 (b). The horizontal axis represents the sampling rate, and the vertical axis represents the average instantaneous error (RSE).

[0149] Simulation 3: Using the present invention and the three existing methods described above, we performed completion accuracy simulations on the GEANT dataset. The results are as follows: Figure 3 (c). The horizontal axis represents the sampling rate, and the vertical axis represents the average instantaneous error (RSE).

[0150] from Figure 3 As can be seen from the data, the RSE value of the present invention is closer to 0 than that of existing sampling methods, indicating that the present invention has high flow recovery accuracy.

Claims

1. A dual-view traffic data sampling method for network digital twins, characterized in that, include: (1) Collect historical traffic interaction data of network nodes within the sliding window to form a third-order traffic tensor, and normalize it to obtain the normalized traffic tensor. (2) Extract the node frequency domain features from the normalized flow tensor and update them smoothly to obtain a smooth feature vector; (3) Map the smooth feature vector and combine it with the frequency domain features to determine the node category, and then divide the source-destination node OD pairs of the entire network into different traffic categories; (4) Calculate the statistical leverage score of all OD pairs, dynamically allocate the sampling budget for each flow category based on the score, and select a portion of OD pairs using a combination of deterministic and random strategies to generate a sampling set. and the corresponding binary sampling mask ; (5) Based on the sampling mask, the traffic data at the corresponding location at the current time is collected to construct a tensor completion optimization model, solve and complete the observed data, and output the recovered network traffic data.

2. The method according to claim 1, characterized in that, The normalization process for the third-order flow tensor in (1) is described by the following formula: ， in, It is the normalized flow value. and These are the minimum and maximum values ​​of the flow data within the current sliding window, respectively. ϵ is a local minimum introduced to prevent the denominator from being zero.

3. The method according to claim 1, characterized in that, The extraction of node frequency domain features from the normalized flow tensor and their smooth update in step (2) include: (2a) For the normalized flow tensor The first k time steps, i.e., historical data slices, are subjected to Fast Fourier Transform (FFT) along the time dimension to obtain the frequency domain tensor. ,right Each frequency slice Perform truncated singular value decomposition: ， in, , , These are the left singular vector matrix, singular value matrix, and right singular vector matrix of the k-th frequency slice, respectively. (2b) Based on the decomposition results, construct the instantaneous source node feature matrix at the current time step t. and instantaneous target node feature matrix Each row of the matrix represents the frequency domain eigenvector of a node, which is formed by concatenating the singular vectors of that node across all frequency slices after weighted and logarithmic transformations; its calculation formula is: ， in, , , Let be the left singular vector matrix, singular value matrix, and right singular vector matrix of the k-th frequency slice, respectively. This indicates a splicing operation. Indicates the modulo value. Indicates logarithmic transformation operation; (2c) Update the feature vector using an exponentially weighted moving average mechanism: ， in For smoothing coefficients, This represents the smoothed feature matrix at the current time step, and the corresponding smoothed feature matrices of the source nodes. and the smoothing matrix of the target node , This represents the instantaneous feature matrix calculated in step (2b). This represents the smooth feature matrix cached at the previous time step.

4. The method according to claim 1, characterized in that, The mapping of smooth feature vectors and the classification of traffic categories in (3) are implemented as follows: (3a) Generate a random projection matrix that follows a standard normal distribution. Where L is the number of projections and D is the feature dimension; the smooth feature vector of each node is projected to a low-dimensional space, and mapped to multiple discrete hash buckets according to the projection value, so as to achieve preliminary clustering of nodes with similar frequency domain fluctuation patterns; (3b) Perform semantic analysis on the nodes within the hash bucket and calculate the high-frequency energy ratio (Ratio) of the node feature vectors: ， in, Let be the amplitude of the eigenvector at the k-th frequency component. and These represent the preset high-frequency band set and low-frequency band set, respectively. (3c) Based on the numerical range of Ratio, the source node and the destination node are respectively marked as Z traffic categories with different frequency domain characteristics; (3d) Combine the source node category and the destination node category to obtain the set of traffic categories to which each OD pair belongs in the entire network.

5. The method according to claim 1, characterized in that, The calculation of the statistical leverage score for all OD pairs in (4) includes: (4a) Calculate the source node lever fraction and target node leverage score : ， in, This represents the feature vector of the i-th source node at the k-th time step. This represents the feature vector of the j-th target node at the k-th time step; (4b) Calculate each OD pair Comprehensive statistical leverage score : 。 6. The method according to claim 1, characterized in that, In step (4), the sampling budget for each flow category is dynamically allocated based on the statistical leverage score, and a combination of deterministic and random strategies is used to select a subset of OD pairs. This process includes: (4c) Calculate the set of the m-th traffic category The sum of the statistical leverage scores of all OD pairs included is defined as the weight of that category. The total number of OD pairs contained in a category is counted and defined as the capacity of that category. ; (4d) Set the total sampling budget A, based on the weights of each category. The proportion of the total network weight is used to calculate the initial budget. : ； (4e) Initial target budgets for each category Its capacity Comparison: If the target budget for this category exceeds its capacity, execute (4f). Otherwise, use the initial target budget; (4f) Truncate the final sample number if the target budget exceeds its capacity category. The excess budget is then returned to the public surplus budget pool, and the quota in the surplus budget pool is iteratively allocated to categories that have not yet reached their capacity limits according to their weighted proportions, until all budgets are allocated, thus obtaining the final sample size for each category. ; (4g) A hybrid sampling strategy is adopted within each traffic category, based on a preset deterministic sampling ratio. The final number of samples for this category Decomposed into deterministic sampling part and random sampling ,satisfy .

7. The method according to claim 6, characterized in that, The (4g) figure represents the final sample size for this category. Decomposed into deterministic sampling part and random sampling Its implementation is as follows: First, for all OD pairs within this category, their comprehensive statistical leverage scores are calculated. Sort in descending order and select the top... Each OD pair constitutes a deterministic sampling set. ; Then, remove the OD pairs contained in the deterministic sampling set from all OD pairs in that category to construct the remaining candidate set, and randomly select from the remaining candidate set. OD pairs constitute a random sampling set ; Finally, the total sample set of this category All OD pairs included in the binary sampling mask The corresponding position is marked as 1, and the rest are marked as 0, thus generating the final sampling mask. The formula is as follows: ， in, This represents the element value of the sampling mask tensor at the k-th time step, the i-th source node, and the j-th destination node. For the sample set.

8. The method according to claim 1, characterized in that, The sampling mask in (5) A tensor completion optimization model is constructed by collecting traffic data at the current location, and the observed data is solved and completed. The implementation includes: (5a) Based on the low-rank characteristic of network traffic, a convex optimization objective function is constructed with the goal of minimizing the nuclear norm: ， in, For the flow tensor to be recovered, It is the actual observed tensor. The nuclear norm of a tensor is the sum of the singular values ​​of all the matrices expanded from the tensor. The projection operator is defined as follows: The tensor after constraint recovery by this operator is in the sampling mask. The value at the corresponding position that is 1 must be consistent with the observed data; (5b) Introducing an auxiliary tensor and Lagrange multiplier tensor The constrained convex optimization objective function described above is transformed into an unconstrained augmented Lagrangian function form: ； in, For penalty parameters, This represents the inner product operation. Represents the Frobenius norm; in this function, For Lagrange multiplier tensors, As an auxiliary tensor, it is used to handle low-rank constraints of the nuclear norm. This is the traffic tensor to be recovered, used to handle data consistency constraints; (5c) Solve for the above augmented Lagrangian function using the alternating direction multiplier method: 5c1) Fixed the first Low-rank auxiliary tensor of the wheel and Lagrange multiplier tensor By solving the extremum problem Get the k+1th round : ， in, This indicates that values ​​from unsampled regions will be retained. 5c2) Fix the data consistency tensor in the (k+1)th round And the Lagrange multiplier tensor of the kth round By solving the extremum problem Get the k+1th round : ， in, Indicates the threshold is Singular value threshold operator; 5c3) Calculate the data consistency tensor Sum and low-rank auxiliary tensors The constraint residuals between them are used as the gradient direction of the dual function to penalize the parameters. The Lagrange multiplier tensor is updated with a step size to maximize the dual objective function. : ； 5c4) Repeat steps (5c1) to (5c3) to calculate the relative error between two adjacent iterations until the error is less than the preset threshold. Then stop the iteration and output the finally converged tensor as the recovered network traffic data.

9. A dual-view traffic data sampling system for network digital twins, characterized in that, include: Traffic data acquisition module: used to monitor network ports or read log files in real time, collect historical traffic interaction data between source nodes and destination nodes in the network within a sliding window, and obtain the initial third-order traffic tensor; The preprocessing module receives the initial third-order flow tensor and performs numerical scaling on it using the max-min normalization method to obtain the normalized flow tensor. Frequency domain feature extraction module: used to perform fast Fourier transform on the normalized flow tensor along the time dimension, and to perform truncated singular value decomposition on the frequency slice to extract the instantaneous frequency domain feature vector that reflects the energy distribution of node flow fluctuations. Frequency domain feature update module: used to receive instantaneous frequency domain feature vectors and use an exponentially weighted moving average mechanism to smoothly update the feature vectors to obtain smooth feature vectors; The traffic category segmentation module is used to map the smooth feature vector based on local sensitive hashing and combine it with the frequency domain energy ratio feature to determine the node category, thereby dividing the source-destination OD node pairs of the entire network into different traffic category sets. The dual-view sampling decision module calculates the comprehensive statistical leverage score for all OD node pairs. Based on this score, it dynamically allocates the sampling budget for each flow category using a weighted water injection algorithm. Within each flow category, it employs a hybrid sampling strategy combining deterministic and random sampling to select a subset of OD pairs, generating a sampling set. and the corresponding binary sampling mask tensor ; The flow tensor completion and reconstruction module is used to complete and reconstruct the flow tensor based on the binary sampling mask tensor. Acquire sparse observation data at the current moment, construct a tensor completion optimization model using the low-rank characteristics of network traffic, and iteratively solve the model using the alternating direction multiplier method to output the restored complete network traffic data.

10. The system according to claim 9, characterized in that, The flow tensor completion and reconstruction module includes: The sparse flow acquisition submodule is used to receive the binary sampling mask tensor. Based on the location indicator where the value is 1, the actual traffic data of the corresponding network OD node pair at the current moment is collected in a directional manner, and the unsampled locations are set to zero, resulting in a sparse observation tensor. ; The model building module is used to receive the sparse observation tensor. We construct a minimum kernel norm tensor completion optimization model constrained by maintaining the consistency of observed data; The model transformation submodule is used to receive the tensor completion optimization model and transform the model into an unconstrained augmented Lagrange function form by introducing auxiliary tensors and Lagrange multipliers. The iterative solution submodule is used to solve the augmented Lagrange function using the alternating direction multiplier method. It iteratively performs data consistency tensor update, low-rank auxiliary tensor update and Lagrange multiplier update until the algorithm converges and outputs the restored complete network traffic data.

Images