A traffic matrix estimation method combining observed traffic and link load constraints

Abstract

The application belongs to the field of network engineering, artificial intelligence and network measurement, and discloses a traffic matrix estimation method combined with observation traffic and link load constraints, comprising the following steps: step 1, obtaining a traffic matrix dataset and a routing matrix dataset; step 2, obtaining a complete traffic matrix sample; step 3, performing main training on a diffusion model module, and updating a Transformer denoising network parameter by minimizing a loss function until convergence; step 4, generating a traffic matrix subjected to known traffic constraints; step 5, repeating steps 4-5 T times, and finally obtaining an output traffic matrix; and step 6, performing post-processing optimization on the traffic matrix output in step 5, and finally outputting a complete estimated traffic matrix. The application utilizes the probability generation capability of the diffusion model module, incorporates the constraint conditions such as observable traffic and link load into the reverse sampling process thereof, and cooperates with a training data preprocessing module to realize high-precision recovery of the traffic matrix.

Description

Technical Field

[0001] This application belongs to the fields of network engineering, artificial intelligence and network measurement, and specifically relates to a traffic matrix estimation method that combines observed traffic and link load constraints. Background Technology

[0002] In modern network systems, the complexity of network scale and traffic patterns is constantly increasing. With the widespread adoption of 5G, IoT, and AI, the number of services, connection scale, and data traffic within the network are all growing exponentially, leading to more severe challenges for network measurement, network scheduling, and traffic engineering. To effectively control and optimize the network, operators need to accurately grasp the various source-destination (SDR) connections. , The distribution of data flows between pairs constitutes the network's traffic matrix. , ). It is an important input for traffic engineering, routing control, congestion management, and capacity planning, and its accuracy directly determines the quality of network performance.

[0003] Traditional traffic acquisition methods rely on flow-level monitoring tools such as NetFlow, JFlow, and OpenTM. These tools typically measure traffic directly through counters in routers or switches, but due to limitations in sampling costs, hardware resources, privacy, and management policies, they can only obtain a small number of traffic entries in practice. Therefore, the observable traffic is often incomplete, containing only the raw, true data. Part of it.

[0004] In order to rebuild the complete Network tomography (NMT) , This has become a core technology direction. Based on the linear relationship between link load and traffic matrix, through routing matrix With link load To solve the flow matrix However, in real-world networks, The sheer number of traffic links far exceeds the number of links, leading to severe underdeterminacy in the linear equations and thus multiple possible solutions. Existing statistical methods rely on strong prior assumptions, such as the Poisson and Gaussian assumptions, but real network traffic exhibits temporal correlation, spatial structure, and nonlinear distribution. Traditional methods cannot accurately capture these characteristics, resulting in limited estimation accuracy.

[0005] In recent years, deep learning methods have been introduced into the task of flow matrix estimation, including recurrent neural networks (RNNs), convolutional neural networks (CNNs), and graph neural networks (GNNs). These methods have improved the performance of flow matrix estimation to some extent, but still have the following problems: (1) It is difficult to explicitly model the flow matrix. (1) The probability distribution and uncertainty of the model; (2) The existing model cannot effectively utilize some observable traffic; (3) Link load constraints are not added during the generation process. (4) A large number of traffic entries are missing in the training data, making it difficult for the model to converge. Summary of the Invention

[0006] To address the technical problems existing in the prior art, this application provides a traffic matrix estimation method that combines observed traffic and link load constraints. This method utilizes the probabilistic generation capability of the diffusion model to incorporate constraints such as observable traffic and link load into its backsampling process, and, in conjunction with a training data preprocessing module, achieves high-precision recovery of the traffic matrix.

[0007] To achieve the above objectives, this application employs the following technical solution:

[0008] This application discloses a flow matrix estimation method that combines observed flow with link load constraints. The flow matrix estimation method is implemented through a flow matrix estimation model, which includes a preprocessing module, a diffusion model module, and a post-processing module. The flow matrix estimation method specifically includes the following steps:

[0009] Step 1: Obtain the traffic matrix dataset and the routing matrix dataset. Divide the obtained traffic matrix dataset into training and test sets proportionally. Perform data preprocessing on the training and test sets of the traffic matrix dataset to obtain the 0-1 observation mask for training. Training flow matrix 01 observation mask used for testing With the observed flow matrix The routing matrix dataset is assumed to be known and divided into training and test sets in equal proportions. The test set of the routing matrix dataset... Perform matrix multiplication with the test set of the traffic matrix dataset to obtain link load. Link load It is assumed to be known.

[0010] Step 2: Training the flow matrix 01 observation mask used for training The input is fed into the preprocessing module, ultimately yielding a complete flow matrix sample for training the diffusion model module. ;

[0011] Step 3: Complete flow matrix sample output by the preprocessing module Based on this, the diffusion model module is trained. The diffusion model module first undergoes a forward process from time step... Calculate the training flow matrix samples at any time step. Then the diffusion model module is based on the first Training traffic matrix samples at each time step , estimate the first The noise at the nth time step is calculated simultaneously. The first noise-free estimate of the output at each time step The parameters of the Transformer denoising network in the diffusion model module are updated by minimizing the loss function. until convergence;

[0012] Step 4: After the diffusion model module has been trained, the observed flow matrix is ​​input into the diffusion model module. 01 observation mask used for testing Sampling from standard Gaussian noise Begin, according to time steps The reverse denoising process is executed sequentially, at the first... At the first time step, the Transformer denoising network of the diffusion model module first passes through the first... Sample flow matrix at each time step Generate initial flow matrix estimation And estimate based on the obtained initial flow matrix Calculate the flow matrix of the previous time step Then, based on the observed flow matrix 01 observation mask used for testing Calculate additional likelihood terms and use these likelihood terms to guide the flow matrix of the previous time step. To ensure that the generated flow matrix satisfies the known flow constraints, the final flow matrix with known flow constraints is obtained. ;

[0013] Step 5: Based solely on the known flow constraints, use the range-null space decomposition method to ensure that the generated flow matrix complies with the known flow constraints. Satisfying the link load equation Repeat steps 4-5 T times to finally obtain the output flow matrix. ;

[0014] Step 6: Introduce the expectation-maximization algorithm to process the flow matrix output in Step 5. Post-processing optimization is performed to finally output a complete estimated flow matrix. .

[0015] A further improvement of this application is that, during the training phase of the said flow matrix estimation model,

[0016] The preprocessing module includes an autoencoder, which consists of an encoder and a decoder. The encoder uses a multi-layer fully connected network, and a non-causal temporal convolutional network is introduced between the encoder and the decoder. The non-causal temporal convolutional network includes three convolutional layers with dilated structures.

[0017] The diffusion model module uses a Transformer denoising network as the basic neural network. This Transformer denoising network mainly includes a feature embedding module, a Transformer encoder module, a Transformer decoder module, and a reconstruction module. The feature embedding module consists of a 1D convolutional layer, which is responsible for embedding the feature into a 1D convolutional layer. Training traffic matrix samples at each time step Mapping to the feature space, extracting the first... Training traffic matrix samples at each time step The Transformer encoder module, composed of a multi-layer self-attention Transformer structure, captures the dependencies and short-term change patterns between adjacent time points in the data. Training traffic matrix samples at each time step The global context representation provided by the Transformer encoder module is used to complete the denoising process. The output of the Transformer decoder module is then mapped back to the original data form by the reconstruction module, finally outputting the first noise-free estimate. ;

[0018] During the sampling phase of the flow matrix estimation model,

[0019] The diffusion model module also uses a Transformer denoising network as the basic neural network. This Transformer denoising network mainly includes a feature embedding module, a Transformer encoder module, a Transformer decoder module, and a reconstruction module. The feature embedding module consists of a 1D convolutional layer, which is responsible for embedding the feature into a 1D convolutional layer. Sampled flux matrix estimated by the diffusion model module at each time step Mapping to the feature space, extracting the first... Sampled flux matrix estimated by the diffusion model module at each time step The Transformer encoder module, composed of a multi-layer self-attention Transformer structure, captures the dependencies and short-term change patterns between adjacent time points. Sampled flux matrix estimated by the diffusion model module at each time step The Transformer decoder module utilizes the global context representation provided by the Transformer encoder module to achieve the denoising process. The output of the Transformer decoder module is mapped back to the original data form by the reconstruction module, outputting an initial flow matrix estimate. Then, guided by known traffic and link load. Guided, repeated T times, the final output flow matrix is ;

[0020] The post-processing module uses the expectation-maximization algorithm, which is a parameter-free mathematical optimization algorithm. It is used to optimize the results after the diffusion model module generates the results, so that the optimization results satisfy the link constraints.

[0021] A further improvement of this application is that step 2 specifically includes the following steps:

[0022] Step 2.1: The training flow matrix with missing values... The input is fed into the encoder, which processes the flow matrix used for training. Compression and refinement are performed on the training flow matrix. Transforming raw data into core feature representations;

[0023] Step 2.2: Use a non-causal temporal convolutional network to analyze the time-varying pattern of the core feature representations from Step 2.1 and output the feature representations;

[0024] Step 2.3: After receiving the feature representation output from Step 2.2, the decoder expands the core feature representation and maps it back to the original data form, outputting the fully reconstructed traffic data. This is achieved by minimizing the reconstructed traffic matrix output by the preprocessing module compared to the training traffic matrix. 0-1 observation mask used in training The differences between known positions on the network are used to update the neural network parameters through the loss function of the preprocessing module until the loss function of the preprocessing module converges. The loss function of the preprocessing module is as follows:

[0025]

[0026] in, This indicates the overall processing flow of the preprocessing module;

[0027] Step 2.4, Loss function of the preprocessing module After updating the neural network parameters, the flow matrices of all locations in the flow matrix training set are estimated to obtain preliminary estimation results. For these preliminary estimates, the entries corresponding to the actual observation locations are replaced with real data to ensure consistency with the measured data. Finally, a complete flow matrix sample is obtained for training the diffusion model module. .

[0028] A further improvement in this application is that step 3 specifically includes the following steps:

[0029] Step 3.1, Complete Traffic Matrix Sample Using the training samples of the diffusion model module, the forward noise addition formula of the diffusion model module is reparameterized to calculate the first... Training traffic matrix samples corresponding to each time step :

[0030]

[0031] in, For the first The training traffic matrix samples obtained after adding noise at each time step and The hyperparameters of the Transformer denoising network for the diffusion model module are: It is random noise. It follows a Gaussian distribution with a mean of 0 and a variance of 1. Represents the identity matrix;

[0032] Step 3.2: Sample the training traffic matrix The Transformer denoising network of the input diffusion model module begins to gradually remove the noise added in the forward process. In the inverse denoising stage... At each time step, the Transformer denoising network estimates the noise. And according to noise Calculate the first The first noise-free estimate at each time step :

[0033]

[0034] in, The first denoising network estimated by the Transformer Noise added at each time step;

[0035] Step 3.3: Use the 0-1 observation mask for training. Only for the first noise-free estimate at the observed locations Supervised updates are performed, using the loss function to update the parameters of the Transformer denoising network in the diffusion model module. The loss function of the diffusion model module converges until the loss function of the diffusion model module is defined as:

[0036]

[0037] By minimizing the loss, while ensuring consistency with the actual observed data, the parameters of the Transformer denoising network in the diffusion model module are adjusted. Optimization was performed so that the diffusion model module generates a flow matrix that conforms to the statistical laws of the real network during sampling in step 4.2.

[0038] A further improvement of this application is that step 4 specifically includes the following steps:

[0039] Step 4.1: First, utilize the known observed flow matrix. Calculate the first step based on the forward process. Sample state at each time step Then the first Sample state at each time step Replace the current number Sampled flux matrix estimated by the diffusion model module at each time step Obtaining noisy samples :

[0040]

[0041] Step 4.2: Use the Transformer denoising network to denoise the noisy samples obtained in Step 4.1. Perform noise prediction and recover the initial flow matrix estimation Then the noisy samples and initial flow matrix estimation Substituting into the following formula, we can calculate the mean of the previous time step. variance from the previous time step The unconditional flow matrix of the previous time step is obtained through the reparameterization formula. :

[0042]

[0043]

[0044]

[0045] in, Let be the hyperparameter of the diffusion model module, representing the th Variance scheduling with added noise at each time step. Represents the identity matrix;

[0046] Step 4.3: Introduce an additional likelihood term. Guiding unconditional flow matrix :

[0047]

[0048] in, For the 01 observation mask used in the test The masking operator, For known noisy samples At that time, the observed flow matrix The likelihood distribution, For covariance:

[0049]

[0050] in, This represents the Jacobian matrix predicted by the Transformer denoising network;

[0051] Step 4.4, based on covariance The conjugate gradient method is used to solve the linear equations. For linear systems, the solution method is as follows:

[0052]

[0053] Finally, we obtain the guided flow matrix with known flow constraints. :

[0054]

[0055] in, Indicates the guiding strength. This is a transpose.

[0056] A further improvement in this application is that step 5 includes the following steps:

[0057] Step 5.1: Obtain the flow matrix subject to known flow constraints. The diffusion model module is input again, and it predicts the first... The second noiseless estimate at each time step At this point, the second noiseless estimate The information already includes known flow rates, and the range-null space decomposition method formula is used to obtain the desired result. corrected solution :

[0058]

[0059] in, Test set for the routing matrix dataset The false reversal;

[0060] In the diffusion model module During the reverse process at each time step, the corrected solution is used at each time step. Replace the second noiseless estimate This constrains the sampling trajectory to satisfy Within the solution space, a flow matrix is ​​finally generated that satisfies both the observed flow and the link load constraints. The final sampling results at each step... for:

[0061] ;

[0062] Output sampling results The sampled flow matrix from step 4.1 is then input into the diffusion model module again. Steps 4 and 5 are executed T times, and the final output is a flow matrix. .

[0063] A further improvement of this application is that, in step 6, the expectation maximization algorithm takes the flow matrix output in step 5 as input. Iteratively update the traffic vector on the test set of a given routing matrix dataset. and link load Approaching satisfaction gradually under the circumstances The maximum likelihood solution is calculated iteratively as follows:

[0064]

[0065] in, and These are the flow matrices. With link load The The, the One portion, and These are the flow matrices in the flow matrix dataset. With link load Dimensions This represents the test set located in the routing matrix dataset. No. row and number The value of the column, Test set for the routing matrix dataset No. row and number The value of the column, The flow matrix in the flow matrix dataset The Each component, as the iteration progresses, results in a flow matrix in the flow matrix dataset. The estimation will gradually converge to a feasible solution under link constraints. Based on the diffusion generation results, the estimation will be further refined and corrected, and finally a complete estimated flow matrix will be output. .

[0066] The beneficial effects of this application are as follows: This application proposes a flow matrix estimation model based on a diffusion model module. When training the flow matrix estimation model, the problem of insufficient training data is taken into account, and the problem caused by missing values ​​in the training data is effectively alleviated through a preprocessing module, providing higher quality training samples for the diffusion model module, thereby improving the training stability and effect of the model.

[0067] This application directly incorporates known traffic information into the backsampling process, which can significantly reduce the traffic matrix estimation error in scenarios where the observed traffic matrix is ​​limited, and achieve high-precision reconstruction of the internal traffic structure of the network.

[0068] This application uses the range-null projection decomposition method to ensure that the generated results satisfy the link load constraints. Compared with existing methods that only apply "soft constraints" to the link load during training or post-processing, the traffic matrix generated by this application has a high degree of matching with the link measurements and has stronger physical interpretability. Attached Figure Description

[0069] Figure 1 This is a flowchart of the training method for the flow matrix estimation model in this application.

[0070] Figure 2 This is a diagram showing the training structure of the flow matrix estimation model in this application.

[0071] Figure 3 This is a flowchart of the sampling method for the flow matrix estimation model in this application.

[0072] Figure 4 This is a sampling structure diagram of the flow matrix estimation model in this application.

[0073] Figure 5 This diagram illustrates the comparison between the method of this application and existing methods using the Abilene dataset in terms of normalized mean absolute error (NMAE).

[0074] Figure 6 This diagram illustrates the comparison between the method of this application and existing methods using the Geant dataset in terms of normalized mean absolute error (NMAE). Detailed Implementation

[0075] The embodiments of the present invention will be disclosed below with reference to the drawings. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details are not intended to limit the invention. That is, in some embodiments of the invention, these practical details are not essential.

[0076] This application is a traffic matrix estimation method that combines observed traffic and link load constraints. The traffic matrix estimation method is implemented through a traffic matrix estimation model, which includes a preprocessing module, a diffusion model module, and a postprocessing module.

[0077] like Figure 2 As shown, in the training phase of the flow matrix estimation model, the preprocessing module includes an autoencoder, which consists of an encoder and a decoder. The encoder employs a multi-layer fully connected network, and a non-causal temporal convolutional network is introduced between the encoder and decoder. This non-causal temporal convolutional network includes three convolutional layers with dilated structures. The diffusion model module uses a Transformer denoising network as the basic neural network. This Transformer denoising network mainly includes a feature embedding module, a Transformer encoder module, a Transformer decoder module, and a reconstruction module. The feature embedding module consists of a 1D convolutional layer, responsible for embedding the first feature into the second feature. Training traffic matrix samples at each time step Mapping to the feature space, extracting the first... Training traffic matrix samples at each time step The Transformer encoder module, composed of a multi-layer self-attention Transformer structure, captures the dependencies and short-term change patterns between adjacent time points in the data. Training traffic matrix samples at each time step The global context representation.

[0078] The Transformer decoder module combines self-attention and cross-attention structures, utilizing the global context representation provided by the Transformer encoder module to achieve a more stable denoising process. The output of the Transformer decoder module is mapped back to the original data form by the reconstruction module, ultimately outputting the first noise-free estimate. .

[0079] like Figure 4 As shown, in the sampling phase of the flow matrix estimation model,

[0080] The diffusion model module also uses a Transformer denoising network as the basic neural network. This Transformer denoising network mainly includes a feature embedding module, a Transformer encoder module, a Transformer decoder module, and a reconstruction module. The feature embedding module consists of a 1D convolutional layer, which is responsible for embedding the feature into a 1D convolutional layer. Sampled flux matrix estimated by the diffusion model module at each time step Mapping to the feature space, extracting the first... Sampled flux matrix estimated by the diffusion model module at each time step The Transformer encoder module, composed of a multi-layer self-attention Transformer structure, captures the dependencies and short-term change patterns between adjacent time points. Sampled flux matrix estimated by the diffusion model module at each time step The Transformer decoder module utilizes the global context representation provided by the Transformer encoder module to achieve the denoising process. The output of the Transformer decoder module is mapped back to the original data form by the reconstruction module, outputting an initial flow matrix estimate. Then, guided by known traffic and link load. Guided, repeated T times, the final output flow matrix is .

[0081] The post-processing module uses the Expectation Maximization (EM) algorithm, which is a parameter-free mathematical optimization algorithm. It is used to further optimize the results after the diffusion model module generates the model, so that the optimization results satisfy the link constraints.

[0082] This application presents a traffic matrix estimation method that combines observed traffic with link load constraints. The method specifically includes the following steps:

[0083] Step 1: Obtain the traffic matrix dataset and the routing matrix dataset. Divide the obtained traffic matrix dataset into training and test sets proportionally. Perform data preprocessing on the training and test sets of the traffic matrix dataset to obtain the 0-1 observation mask for training. Training flow matrix 01 observation mask used for testing With the observed flow matrix The routing matrix dataset is assumed to be known and divided into training and test sets in equal proportions. The test set of the routing matrix dataset... Perform matrix multiplication with the test set of the traffic matrix dataset to obtain link load. Link load The default is known; the 0-1 observation mask used for training. 01 observation mask used for testing In the model, 0 indicates that the value at that position is missing, and 1 indicates that the value at that position can be observed. The model needs to be trained before it can be validated on the test set data.

[0084] like Figure 1 As shown, steps 2 and 3 are the training process, specifically:

[0085] Step 2: Use the preprocessing module to complete and coarse-grained estimate the missing training data. Training flow matrix. 01 observation mask used for training The input is fed into the preprocessing module, ultimately yielding a complete flow matrix sample for training the diffusion model module. Specifically, the steps include the following:

[0086] Step 2.1: The training flow matrix with missing values... The input is fed into the encoder, which, leveraging the non-linear processing capability of the ReLU activation function, processes the flow matrix used for training. Compression and refinement are performed on the traffic matrix used for training. The process of transforming the raw data into a core feature representation with lower dimensions but more information-dense features removes redundant information and retains key patterns.

[0087] Step 2.2: Use a non-causal temporal convolutional network to deeply analyze the patterns of change of the core feature representations in Step 2.1 over time. Non-causal temporal convolutional networks can simultaneously capture short-term, medium-term, and long-term temporal dependency patterns and output feature representations.

[0088] Step 2.3: After receiving the feature representation output from Step 2.2, the decoder expands the core feature representation and maps it back to the original data form, outputting the fully reconstructed traffic data. This is achieved by minimizing the reconstructed traffic matrix output by the preprocessing module compared to the training traffic matrix. 0-1 observation mask used in training The differences between known positions on the network are used to update the neural network parameters through the loss function of the preprocessing module until the loss function of the preprocessing module converges. The loss function of the preprocessing module is as follows:

[0089]

[0090] in, This indicates the overall processing flow of the preprocessing module. During training, reconstruction error is only calculated at observed locations, thereby guiding the network to focus on learning the statistical structure near the observed values ​​and mitigating the impact of missing terms on training stability.

[0091] Step 2.4, Loss function of the preprocessing module After updating the neural network parameters, the flow matrices of all locations in the flow matrix training set are estimated to obtain preliminary estimation results. For these preliminary estimates, the entries corresponding to the actual observation locations are replaced with real data to ensure consistency with the measured data. Finally, a complete flow matrix sample is obtained for training the diffusion model module. .

[0092] Step 3: Complete flow matrix sample output by the preprocessing module Based on this, the diffusion model module is trained. The diffusion model module first undergoes a forward process from time step... Calculate the training flow matrix samples at any time step. Then the diffusion model module is based on the first Training traffic matrix samples at each time step , estimate the first The noise at the nth time step is calculated simultaneously. The first noise-free estimate of the output at each time step The parameters of the Transformer denoising network in the diffusion model module are updated by minimizing the loss function. This process continues until convergence. Specifically, it includes the following steps:

[0093] Step 3.1, Complete Traffic Matrix Sample Using the training samples of the diffusion model module, the forward noise addition formula of the diffusion model module is reparameterized to calculate the first... Training traffic matrix samples corresponding to each time step :

[0094]

[0095] in, For the first The training traffic matrix samples obtained after adding noise at each time step and The hyperparameters of the Transformer denoising network for the diffusion model module are: It is random noise. It follows a Gaussian distribution with a mean of 0 and a variance of 1. Represents the identity matrix.

[0096] Step 3.2: Sample the training traffic matrix The Transformer denoising network of the input diffusion model module begins to gradually remove the noise added in the forward process. In the inverse denoising stage... At each time step, the Transformer denoising network estimates the noise. And according to noise Calculate the first The first noise-free estimate at each time step :

[0097]

[0098] in, The first denoising network estimated by the Transformer Noise added at each time step;

[0099] Step 3.3: Use the 0-1 observation mask for training. Only for the first noise-free estimate at the observed locations Supervised updates are performed, using the loss function to update the parameters of the Transformer denoising network in the diffusion model module. The loss function of the diffusion model module converges until the loss function of the diffusion model module is defined as:

[0100]

[0101] By minimizing the loss, while ensuring consistency with the actual observed data, the parameters of the Transformer denoising network in the diffusion model module are adjusted. Optimization was performed so that the diffusion model module generates a flow matrix that conforms to the statistical laws of the real network during sampling in step 4.2.

[0102] like Figure 3 As shown, steps 4, 5, and 6 constitute the sampling process. Specifically:

[0103] Step 4: After the diffusion model module has been trained, the observed flow matrix is ​​input into the diffusion model module. 01 observation mask used for testing Sampling from standard Gaussian noise Begin, according to time steps The reverse denoising process is executed sequentially, at the first... At the first time step, the Transformer denoising network of the diffusion model module first passes through the first... Sample flow matrix at each time step Generate initial flow matrix estimation And estimate based on the obtained initial flow matrix Calculate the flow matrix of the previous time step Then, based on the observed flow matrix 01 observation mask used for testing Calculate additional likelihood terms and use these likelihood terms to guide the flow matrix of the previous time step. To ensure that the generated flow matrix satisfies the known flow constraints, the final flow matrix with known flow constraints is obtained. Specifically, the steps include the following:

[0104] Step 4.1: First, utilize the known observed flow matrix. Calculate the first step based on the forward process. Sample state at each time step Then the first Sample state at each time step Replace the current number Sampled flux matrix estimated by the diffusion model module at each time step Obtaining noisy samples :

[0105]

[0106] Step 4.2: Use the Transformer denoising network to denoise the noisy samples obtained in Step 4.1. Perform noise prediction and recover the initial flow matrix estimation Then the noisy samples and initial flow matrix estimation Substituting into the following formula, we can calculate the mean of the previous time step. variance from the previous time step The unconditional flow matrix of the previous time step is obtained through the reparameterization formula. :

[0107]

[0108]

[0109]

[0110] in, Let be the hyperparameter of the diffusion model module, representing the th The variance scheduling of adding noise at each time step is mainly adjusted manually. Represents the identity matrix;

[0111] Step 4.3: In order to guide each step of the reverse process using observed flow rates, this application introduces an additional likelihood term. Guiding unconditional flow matrix :

[0112]

[0113] in, For the 01 observation mask used in the test The masking operator, For known noisy samples At that time, the observed flow matrix The likelihood distribution, For covariance:

[0114]

[0115] in, This represents the Jacobian matrix predicted by the Transformer denoising network;

[0116] Step 4.4: Considering the high dimension of the covariance matrix, this application is based on covariance... The conjugate gradient method is used to solve the linear equations. For linear systems, the solution method is as follows:

[0117]

[0118] Finally, we obtain the guided flow matrix with known flow constraints. :

[0119]

[0120] in, This indicates the guiding strength, which can dynamically adjust the contribution between the two factors to obtain better sampling results. This is a transpose.

[0121] Step 5: Based solely on the known flow constraints, use the range-null space decomposition method to ensure that the generated flow matrix complies with the known flow constraints. Satisfying the link load equation Repeat steps 4-5 T times to finally obtain the output flow matrix. Specifically, the steps include the following:

[0122] Step 5.1: The flow matrix, after passing through known flow constraints... The diffusion model module is input again, and it predicts the first... The second noiseless estimate at each time step At this point, the second noiseless estimate The information already includes known flow rates, and the range-null space decomposition method formula is used to obtain the desired result. corrected solution :

[0123]

[0124] in, Test set for the routing matrix dataset The false reversal;

[0125] In the diffusion model module During the reverse process at each time step, the corrected solution is used at each time step. Replace the second noiseless estimate This constrains the sampling trajectory to satisfy Within the solution space, a flow matrix is ​​finally generated that satisfies both the observed flow and the link load constraints. The final sampling results at each step... for:

[0126] ;

[0127] Output sampling results The sampled flow matrix from step 4.1 is then input into the diffusion model module again. Steps 4 and 5 are executed T times, and the final output is a flow matrix. .

[0128] Step 6: Introduce the Expectation Maximization (EM) algorithm to process the flow matrix output in Step 5. Post-processing optimization is performed to finally output a complete estimated flow matrix. .

[0129] In this step, the Expectation Maximization (EM) algorithm takes the flow matrix output from step 5 as input. Iteratively update the traffic vector on the test set of a given routing matrix dataset. and link load Approaching satisfaction gradually under the circumstances The maximum likelihood solution is calculated iteratively as follows:

[0130]

[0131] in, and These are the flow matrices. With link load The The, the One portion, and These are the flow matrices in the flow matrix dataset. With link load Dimensions This represents the test set located in the routing matrix dataset. No. row and number The value of the column, Test set for the routing matrix dataset No. row and number The value of the column, The flow matrix in the flow matrix dataset The Each component, as the iteration progresses, results in a flow matrix in the flow matrix dataset. The solution will gradually converge to a feasible solution under link constraints, thereby further refining and correcting the estimation based on the diffusion generation results, and finally outputting a complete estimated flow matrix. .

[0132] like Figures 5-6 The proposed method was compared with four other algorithms in terms of Normalized Mean Absolute Error (NMAE) using two datasets: Abilene and Geant. During training, the proportion of known data was set at 20%, 30%, 50%, 75%, and 100%, while during testing, the proportion of known traffic data was fixed at 20%. As the proportion of training data increased, the NMAE of all algorithms decreased. However, even with significant missing training data, the proposed method still achieved better NMAE results. Furthermore, because it utilizes known traffic data, its estimation performance is superior to algorithms that directly estimate the traffic matrix.

[0133] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A flow matrix estimation method combining observed flow and link load constraints, wherein the flow matrix estimation method is implemented through a flow matrix estimation model, the flow matrix estimation model comprising a preprocessing module, a diffusion model module, and a post-processing module, characterized in that: The flow matrix estimation method specifically includes the following steps: Step 1: Obtain the traffic matrix dataset and the routing matrix dataset. Divide the obtained traffic matrix dataset into training and test sets proportionally. Perform data preprocessing on the training and test sets of the traffic matrix dataset to obtain the 0-1 observation mask for training. Training flow matrix 01 observation mask used for testing With the observed flow matrix The routing matrix dataset is assumed to be known and divided into training and test sets in equal proportions. The test set of the routing matrix dataset... Perform matrix multiplication with the test set of the traffic matrix dataset to obtain link load. Link load It is assumed to be known. Step 2: Training the flow matrix 01 observation mask used for training The input is fed into the preprocessing module, ultimately yielding a complete flow matrix sample for training the diffusion model module. ; Step 3: Complete flow matrix sample output by the preprocessing module Based on this, the diffusion model module is trained. The diffusion model module first undergoes a forward process from time step... Calculate the training flow matrix samples at any time step. Then the diffusion model module is based on the first Training traffic matrix samples at each time step , estimate the first The noise at the nth time step is calculated simultaneously. The first noise-free estimate of the output at each time step The parameters of the Transformer denoising network in the diffusion model module are updated by minimizing the loss function. until convergence; Step 4: After the diffusion model module has been trained, the observed flow matrix is ​​input into the diffusion model module. 01 observation mask used for testing Sampling from standard Gaussian noise Begin, according to time steps The reverse denoising process is executed sequentially, at the first... At the nth time step, the Transformer denoising network of the diffusion model first passes through the first... Sample flow matrix at each time step Generate initial flow matrix estimation And estimate based on the obtained initial flow matrix Calculate the flow matrix of the previous time step Then, based on the observed flow matrix 01 observation mask used for testing Calculate additional likelihood terms and use these likelihood terms to guide the flow matrix of the previous time step. To ensure that the generated flow matrix satisfies the known flow constraints, the final flow matrix with known flow constraints is obtained. ; Step 5: Based solely on the known flow constraints, use the range-null space decomposition method to ensure that the generated flow matrix complies with the known flow constraints. Satisfying the link load equation Repeat steps 4-5 T times to finally obtain the output flow matrix. ; Step 6: Introduce the expectation-maximization algorithm to process the flow matrix output in Step 5. Post-processing optimization is performed to finally output a complete estimated flow matrix. .

2. The traffic matrix estimation method combining observed traffic and link load constraints according to claim 1, characterized in that: During the training phase of the flow matrix estimation model, The preprocessing module includes an autoencoder, which consists of an encoder and a decoder. The encoder uses a multi-layer fully connected network, and a non-causal temporal convolutional network is introduced between the encoder and the decoder. The diffusion model module uses a Transformer denoising network as the basic neural network. This Transformer denoising network mainly includes a feature embedding module, a Transformer encoder module, a Transformer decoder module, and a reconstruction module. The feature embedding module consists of a 1D convolutional layer, which is responsible for embedding the feature into a 1D convolutional layer. Training traffic matrix samples at each time step Mapping to the feature space, extracting the first... Training traffic matrix samples at each time step The Transformer encoder module, composed of a multi-layer self-attention Transformer structure, captures the dependencies and short-term change patterns between adjacent time points in the data. Training traffic matrix samples at each time step The global context representation provided by the Transformer encoder module is used to complete the denoising process. The output of the Transformer decoder module is then mapped back to the original data form by the reconstruction module, finally outputting the first noise-free estimate. ; During the sampling phase of the flow matrix estimation model, The diffusion model module also uses a Transformer denoising network as the basic neural network. This Transformer denoising network mainly includes a feature embedding module, a Transformer encoder module, a Transformer decoder module, and a reconstruction module. The feature embedding module consists of a 1D convolutional layer, which is responsible for embedding the feature into a 1D convolutional layer. Sampled flux matrix estimated by the diffusion model module at each time step Mapping to the feature space, extracting the first... Sampled flux matrix estimated by the diffusion model module at each time step The Transformer encoder module, composed of a multi-layer self-attention Transformer structure, captures the dependencies and short-term change patterns between adjacent time points. Sampled flux matrix estimated by the diffusion model module at each time step The Transformer decoder module utilizes the global context representation provided by the Transformer encoder module to achieve the denoising process. The output of the Transformer decoder module is mapped back to the original data form by the reconstruction module, outputting an initial flow matrix estimate. Then, guided by known traffic and link load. Guided, repeated T times, the final output flow matrix is ; The post-processing module uses the expectation-maximization algorithm, which is a parameter-free mathematical optimization algorithm. It is used to optimize the results after the diffusion model module generates the results, so that the optimization results satisfy the link constraints.

3. The traffic matrix estimation method combining observed traffic and link load constraints according to claim 1, characterized in that: Step 2 specifically includes the following steps: Step 2.1: The training flow matrix with missing values... The input is fed into the encoder, which processes the flow matrix used for training. Compression and refinement are performed on the training flow matrix. Transforming raw data into core feature representations; Step 2.2: Use a non-causal temporal convolutional network to analyze the time-varying pattern of the core feature representations from Step 2.1 and output the feature representations; Step 2.3: After receiving the feature representation output from Step 2.2, the decoder expands the core feature representation and maps it back to the original data form, outputting the fully reconstructed traffic data. This is achieved by minimizing the reconstructed traffic matrix output by the preprocessing module compared to the training traffic matrix. 0-1 observation mask used in training The differences between known positions on the network are used to update the neural network parameters through the loss function of the preprocessing module until the loss function of the preprocessing module converges. The loss function of the preprocessing module is as follows: in, This indicates the overall processing flow of the preprocessing module; Step 2.4, Loss function of the preprocessing module After updating the neural network parameters, the flow matrices of all locations in the flow matrix training set are estimated to obtain preliminary estimation results. For these preliminary estimates, the entries corresponding to the actual observation locations are replaced with real data to ensure consistency with the measured data. Finally, a complete flow matrix sample is obtained for training the diffusion model module. .

4. The traffic matrix estimation method combining observed traffic and link load constraints according to claim 1, characterized in that: Step 3 specifically includes the following steps: Step 3.1, Complete Traffic Matrix Sample Using the training samples of the diffusion model module, the forward noise addition formula of the diffusion model module is reparameterized to calculate the first... Training traffic matrix samples corresponding to each time step : in, For the first The training traffic matrix samples obtained after adding noise at each time step and The hyperparameters of the Transformer denoising network for the diffusion model module are: It is random noise. It follows a Gaussian distribution with a mean of 0 and a variance of 1. Represents the identity matrix; Step 3.2: Sample the training traffic matrix The Transformer denoising network of the input diffusion model module begins to gradually remove the noise added in the forward process. In the inverse denoising stage... At each time step, the Transformer denoising network estimates the noise. And according to noise Calculate the first The first noise-free estimate at each time step : in, The first denoising network estimated by the Transformer Noise added at each time step; Step 3.3: Use the 0-1 observation mask for training. Only for the first noise-free estimate at the observed locations Supervised updates are performed, using the loss function to update the parameters of the Transformer denoising network in the diffusion model module. The loss function of the diffusion model module converges until the loss function of the diffusion model module is defined as: By minimizing the loss, while ensuring consistency with the actual observed data, the parameters of the Transformer denoising network in the diffusion model module are adjusted. Optimization was performed so that the diffusion model module generates a flow matrix that conforms to the statistical laws of the real network during sampling in step 4.

2.

5. The traffic matrix estimation method combining observed traffic and link load constraints according to claim 4, characterized in that: Step 4 specifically includes the following steps: Step 4.1: First, utilize the known observed flow matrix. Calculate the first step based on the forward process. Sample state at each time step Then the first Sample state at each time step Replace the current number Sampled flux matrix estimated by the diffusion model module at each time step Obtaining noisy samples : Step 4.2: Use the Transformer denoising network to denoise the noisy samples obtained in Step 4.

1. Perform noise prediction and recover the initial flow matrix estimation Then the noisy samples and initial flow matrix estimation Substituting into the following formula, we can calculate the mean of the previous time step. variance from the previous time step The unconditional flow matrix of the previous time step is obtained through the reparameterization formula. : in, Let be the hyperparameter of the diffusion model module, representing the th Variance scheduling with added noise at each time step. Represents the identity matrix; Step 4.3: Introduce an additional likelihood term. Guiding unconditional flow matrix : in, For the 01 observation mask used in the test The masking operator, For known noisy samples At that time, the observed flow matrix The likelihood distribution, For covariance: in, This represents the Jacobian matrix predicted by the Transformer denoising network; Step 4.4, based on covariance The conjugate gradient method is used to solve the linear equations. For linear systems, the solution method is as follows: Finally, we obtain the guided flow matrix with known flow constraints. : in, Indicates the guiding strength. This is a transpose.

6. The traffic matrix estimation method combining observed traffic and link load constraints according to claim 5, characterized in that: Step 5 includes the following steps: Step 5.1: Obtain the flow matrix subject to known flow constraints. The diffusion model module is input again, and it predicts the first... The second noiseless estimate at each time step At this point, the second noiseless estimate The information already includes known flow rates, and the range-null space decomposition method formula is used to obtain the desired result. corrected solution : in, Test set for the routing matrix dataset The false reversal; In the diffusion model module During the reverse process at each time step, the corrected solution is used at each time step. Replace the second noiseless estimate This constrains the sampling trajectory to satisfy Within the solution space, a flow matrix is ​​finally generated that satisfies both the observed flow and the link load constraints. The final sampling results at each step... for: ； Output sampling results The sampled flow matrix from step 4.1 is then input into the diffusion model module again. Steps 4 and 5 are executed T times, and the final output is a flow matrix. .

7. The traffic matrix estimation method combining observed traffic and link load constraints according to claim 6, characterized in that: In step 6, the expectation-maximization algorithm takes the flow matrix output in step 5 as input. Iteratively update the traffic vector on the test set of a given routing matrix dataset. and link load Approaching satisfaction gradually under the circumstances The maximum likelihood solution is calculated iteratively as follows: in, and The final output flow matrix With link load The The, the One portion, and These are the flow matrices in the flow matrix dataset. With link load Dimensions This represents the test set located in the routing matrix dataset. No. row and number The value of the column, Test set for the routing matrix dataset No. row and number The value of the column, The flow matrix in the flow matrix dataset The Each component, as the iteration progresses, results in a flow matrix in the flow matrix dataset. The estimation will gradually converge to a feasible solution under link constraints. Based on the diffusion generation results, the estimation will be further refined and corrected, and finally a complete estimated flow matrix will be output. .

Images