A method for estimating parameters of mixture probability distribution based on deep learning

Abstract

The present invention discloses a method for estimating parameters of a mixed probability distribution based on deep learning, which includes calculating the mean and variance of all data samples in a given data set X, and then normalizing each data sample to obtain a normalized sample X. (i) ,i=[1,2,...,N],N is the total number of samples; establish the probability density function of the mixed probability distribution#imgabs0#where p k (X (i) ,θ k ) is the probability density function of the kth component distribution in the mixed probability distribution, w k is the weight of the kth probability distribution in the overall distribution, θ k The parameter of the probability density function of the kth component, k = [1, 2, ..., K], where K is the total number of components in the mixture distribution, is used. An objective function and a lower bound function for estimating the parameters of the mixture probability distribution are established. A deep learning method is used to iteratively optimize the lower bound function and solve for the parameter θ. This invention is capable of processing large amounts of data with high processing speed and high estimation accuracy.

Description

Technical Field

[0001] The present invention relates to a method for estimating parameters of a mixed probability distribution, and in particular to a method for estimating parameters of a mixed probability distribution based on deep learning. Background Art

[0002] A mixed probability distribution is a probability distribution composed of several independent parameterized probability distribution components, combined according to certain weights. Parameter estimation for a mixed probability distribution is the process of inferring or estimating the unknown parameters and weights of the probability distribution components. It is widely used in fields such as engineering simulation, anomaly detection, and signal compression. Existing probability distribution parameter estimation methods include the method of moments, maximum likelihood estimation, Bayesian estimation, least squares method, maximum margin estimation, quantile estimation, and the EM algorithm, among other statistical methods.

[0003] Since mixed probability distribution is more complex than ordinary probability distribution, existing probability distribution parameter estimation methods will encounter many problems in the practical application of mixed probability distribution estimation. For example, the methods that use optimization methods such as maximum likelihood estimation and EM algorithm to find the optimal parameters require calculating the partial derivatives of the parameters for the specific probability distribution of each component in each mixed distribution. The calculation method is complex, difficult to solve directly, and prone to errors.

[0004] However, if deterministic iterative methods such as the Newton iteration method are used to calculate the roots of the partial derivative formulas obtained in the maximum likelihood estimation method and the EM algorithm, all data must be traversed during each iteration, which takes a long time to process and has limited applications and cannot handle large amounts of data.

[0005] Therefore, in the prior art, there is no method with fast processing speed and high estimation accuracy to estimate the parameters of complex mixed probability distributions with large amounts of data. Summary of the Invention

[0006] The present invention aims to provide a method for estimating parameters of a mixed probability distribution based on deep learning. The present invention has the characteristics of being able to process large amounts of data with high processing speed and high estimation accuracy.

[0007] The technical solution of the present invention is a method for estimating parameters of a mixed probability distribution based on deep learning, comprising the following steps:

[0008] 1. Data preprocessing:

[0009] Given a set of data sets X, calculate the mean and variance of all data samples in the data set, and then normalize each data sample to obtain a normalized sample X (i) , i=[1,2,...,N], N is the total number of samples;

[0010] 2. Establishment of probability density function of mixed probability distribution:

[0011] Probability density function of a mixture probability distribution ;in is the probability density function of the kth component probability distribution in the mixed probability distribution, w k is the weight of the kth probability distribution in the overall distribution, is the parameter of the probability density function of the kth component, k=[1,2,...,K], K is the total number of components of the mixed distribution; θ=<θ1,θ2,...,θ K > is the total parameter vector consisting of the probability density function parameters of all components;

[0012] 3. Establishment of the objective function for mixed probability distribution parameter estimation:

[0013] Establish the objective function and lower bound function for parameter estimation of mixed probability distribution;

[0014] 4. Use deep learning methods to iteratively optimize the lower bound function and solve the parameter θ:

[0015] 4.1) Sample the parameter θ from a standard normal distribution with a mean of 0 and a variance of 1 as the initial value of the parameter;

[0016] 4.2) Initialization w k =1 / K, initialize the error count en to 0, and the number of steps step=1;

[0017] 4.3) Define the latent variable Z as a two-dimensional matrix with N rows and K columns, and calculate the value of Z;

[0018] 4.4) Calculate w based on Z value k , according to the Z value and w k Calculate the Q function and obtain θ' based on the Q function;

[0019] 4.5) Let the error e = |θ - θ'|. If the error e is less than 0.001, then let en = en + 1; otherwise, let en = 0.

[0020] 4.6) Let θ = θ'. If en > the error termination condition, output the parameter θ; otherwise, execute step 4.7).

[0021] 4.7) Let step = step + 1. If step > the maximum number of steps, output the parameter θ; otherwise, execute step 4.3).

[0022] In the aforementioned method for estimating parameters of a mixed probability distribution based on deep learning, in step 1, the dataset X={x (1) ,x(2) ,......,x (N)}; where x (i) is the i-th data sample, i=[1,2,...,N], N is the total number of samples, each x (i) Represented as a P dimensional vector, i.e. x (i) =< x i1 , x i2 ,.... x iP >, P is the dimension of the vector.

[0023] In the aforementioned method for estimating parameters of a mixed probability distribution based on deep learning, in step 1, the mean of the data sample is ; Standard deviation of the data sample .

[0024] In the aforementioned method for estimating parameters of a mixed probability distribution based on deep learning, in step 1, the normalized sample .

[0025] In the aforementioned method for estimating parameters of a mixed probability distribution based on deep learning, in step 3, the objective function ; Lower bound function .

[0026] In the aforementioned method for estimating parameters of a mixed probability distribution based on deep learning, before solving the parameter θ in step 4, predefined hyperparameters are also included. The hyperparameters include: the number of data batches batch_size, the number of execution rounds num_epochs, the learning rate lr, the error termination condition em, and the maximum number of steps maxstep.

[0027] In the aforementioned deep learning-based mixed probability distribution parameter estimation method, batch_size is 512-4096, with a default value of 2048; num_epochs is 20-100, with a default value of 30; lr is 1e-3; em is 5, and maxstep is 100.

[0028] In the aforementioned method for estimating parameters of a mixed probability distribution based on deep learning, 4.3) is specifically as follows:

[0029] 4.3.1) Input the entire dataset X;

[0030] 4.3.2) All elements of the latent variable Z are initialized to 0;

[0031] 4.3.3) Let i = 1;

[0032] 4.3.4) Get the i-th data in the dataset X X (i) ;

[0033] 4.3.5) Let sum_Zi = 0;

[0034] 4.3.6) Let k = 1;

[0035] 4.3.7) Z[i,k] = w k * p k (X (i) ,θ k );

[0036] 4.3.8) sum_Zi = sum_Zi + Z[i,k];

[0037] 4.3.9) Let k = k + 1. If k > K, proceed to step 5.3.10. Otherwise, proceed to step 5.3.7.

[0038] 4.3.10) Let k = 1;

[0039] 4.3.11) Let Z[i,k] = Z[i,k] / sum_Zi;

[0040] 4.3.12) Let k = k + 1. If k > K, proceed to step 4.3.13). Otherwise, proceed to step 4.3.11).

[0041] 4.3.13) Let i = i + 1. If i > N, then Z[i, k] is calculated and step 4.4 is executed. Otherwise, step 4.3.4 is executed.

[0042] In the aforementioned method for estimating parameters of a mixed probability distribution based on deep learning, 4.4) is specifically as follows:

[0043] 4.4.1) Define θ' = θ, set B = N / batch_size, divide the dataset into B batches, with the amount of data in each batch being batch_size, and randomly divide each data item in the dataset into batches;

[0044] 4.4.2) Set epoch = 1;

[0045] 4.4.3) Let b = 1, let X[b] be the bth batch of data, X[b,i] be the i-th data in the bth batch of data, and the serial number of this data in the original data set is idx i ;

[0046] 4.4.4) Calculate the lower bound function Q function;

[0047] 4.4.5) Calculate the gradient of the parameter θ' on the negative function -Q of the Q function and perform backpropagation. The backpropagation optimizer uses the Nadam method with a learning rate of 1e-3. After backpropagation, θ' is updated by the Nadam optimizer to obtain θ';

[0048] 4.4.6) Let b = b + 1. If b > B, proceed to step 4.4.7). Otherwise, proceed to step 4.4.4.

[0049] 4.4.7) Set epoch = epoch + 1. If epoch > num_epochs, proceed to step 4.4.8). Otherwise, skip to step 4.4.3).

[0050] 4.4.8) Order ,k=[1,2,....,K].

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] The present invention adopts the expectation maximization process to achieve approximate optimization, and uses the random optimization method in the approximate maximization calculation process, so that each iteration does not need to use all the data, but instead selects a batch of representative data for parameter estimation, thereby being able to process probability distribution parameters of large amounts of data, greatly improving the processing speed and convergence speed of probability distribution parameter estimation, and the parameter estimation accuracy is high. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a set of actual signals.

[0054] Figure 2 It is a density map given by existing statistical non-parametric estimation methods.

[0055] Figure 3 It is a probability distribution diagram obtained by calculating the parameters for the 10th iteration according to the present invention.

[0056] Figure 4 It is a probability distribution diagram obtained by calculating the parameters for the 50th iteration according to the present invention.

[0057] Figure 5 It is a probability distribution diagram obtained by calculating the parameters for the 90th iteration according to the present invention. DETAILED DESCRIPTION

[0058] The present invention will be further described below with reference to the examples, but they are not intended to limit the present invention.

[0059] Example 1:

[0060] A method for estimating parameters of a mixture probability distribution based on deep learning, comprising the following steps:

[0061] 1. Data preprocessing:

[0062] 1.1) Given a set of data sets X: X={x (1) ,x (2) ,......,x (N)}; where x (i) is the i-th data sample, i=[1,2,...,N], N is the total number of samples, each x (i) Represented as a P dimensional vector, i.e. x (i) =< x i1 , x i2 ,.... x iP >, P is the dimension of the vector.

[0063] 1.2) Calculate the mean and variance of all data samples, where the mean of each dimension is ;Standard deviation of each dimension .

[0064] 1.3) Normalize each data sample to obtain a normalized sample X (i) : .

[0065] 2. Establishment of probability density function of mixed probability distribution:

[0066] Probability density function of a mixture probability distribution pdf : ;in is the probability density function of the kth component probability distribution in the mixed probability distribution, w k is the weight of the kth probability distribution in the overall distribution, is the parameter of the probability density function of the kth component, k=[1,2,...,K], K is the total number of components of the mixed distribution; θ=<θ1,θ2,...,θ K > is the total parameter vector composed of the probability density function parameters of all components, which is the parameter to be estimated.

[0067] 3. Establishment of the objective function for mixed probability distribution parameter estimation:

[0068] Objective function L for parameter estimation of mixed probability distribution MLE : .

[0069] In practical applications, the objective function is often complex and difficult to calculate directly. Therefore, the present invention employs deep learning methods to iteratively approximate the objective function. The objective function LMLE is complex in practical applications and difficult to optimize directly for precise calculation. Therefore, the present invention establishes a lower bound function for the objective function and iteratively optimizes the lower bound function using deep learning methods to approximate the objective function.

[0070] 4. Predefined hyperparameters:

[0071] 4.1) The batch size of data is 512-4096, and the default value is 2048;

[0072] 4.2) The number of execution rounds num_epochs is 20-100, and the default value is 30;

[0073] 4.3) The learning rate lr is 1e-3;

[0074] 4.4) The error termination condition em is 5, and the maximum number of steps maxstep is 100.

[0075] 5. Use deep learning methods to iteratively optimize the Q function and solve the parameter θ:

[0076] 5.1) Sample the parameter θ from a standard normal distribution with mean 0 and variance 1 as the initial value of the parameter.

[0077] 5.2) Initialization w k =1 / K, initialize the error count en to 0, and the number of steps step=1.

[0078] 5.3) Define the latent variable Z as a two-dimensional matrix with N rows and K columns, representing X (i) The posterior probability of the kth component from the mixed probability distribution is calculated by performing the E step to calculate the Z value. The specific process is as follows:

[0079] 5.3.1) Input the entire dataset X;

[0080] 5.3.2) All elements of the latent variable Z are initialized to 0;

[0081] 5.3.3) Let i = 1;

[0082] 5.3.4) Get the i-th data in the dataset X X (i) ;

[0083] 5.3.5) Let sum_Zi = 0;

[0084] 5.3.6) Let k = 1;

[0085] 5.3.7) Z[i,k] = w k * p k ( X (i) ,θ k );

[0086] 5.3.8) sum_Zi = sum_Zi + Z[i,k];

[0087] 5.3.9) Let k = k + 1. If k > K, proceed to step 5.3.10. Otherwise, proceed to step 5.3.7.

[0088] 5.3.10) Let k = 1;

[0089] 5.3.11) Let Z[i,k] = Z[i,k] / sum_Zi;

[0090] 5.3.12) Let k = k + 1. If k > K, proceed to step 5.3.13. Otherwise, proceed to step 5.3.11.

[0091] 5.3.13) Let i = i + 1. If i > N, then Z[i, k] is calculated and proceed to step 5.4. Otherwise, proceed to step 5.3.4.

[0092] 5.4) Perform M random steps to calculate θ'. The specific process is as follows:

[0093] 5.4.1) Define θ' = θ, set B = N / batch_size, divide the dataset into B batches, with the amount of data in each batch being batch_size, and randomly divide each data item in the dataset into batches;

[0094] 5.4.2) Set epoch = 1;

[0095] 5.4.3) Let b = 1, let X[b] be the bth batch of data, X[b,i] be the i-th data in the bth batch of data, and the serial number of this data in the original data set is idx i ;

[0096] 5.4.4) Calculate the Q function, which is expressed as the expectation of the joint probability distribution of the data and the latent variable Z on the posterior of the latent variable under the current parameters:

[0097] ;

[0098] 5.4.5) Calculate the gradient of the parameter θ' on the negative function -Q of the Q function and perform backpropagation. The backpropagation optimizer uses the Nadam method with a learning rate of 1e-3. After backpropagation, θ' is updated by the Nadam optimizer to obtain θ'.

[0099] 5.4.6) Let b = b + 1. If b > B, proceed to step 5.4.7). Otherwise, proceed to step 5.4.4.

[0100] 5.4.7) Set epoch = epoch + 1. If epoch > num_epochs, proceed to step 5.4.8). Otherwise, skip to step 5.4.3).

[0101] 5.4.8) Order ,k=[1,2,....,K].

[0102] 5.5) Let the error e = |θ - θ'|. If the error e is less than 0.001, then let en = en + 1; otherwise, let en = 0.

[0103] 5.6) Let θ = θ'. If en > em, the parameter learning process ends and the learned parameter θ is output; otherwise, proceed to step 5.7).

[0104] 5.7) Let step = step + 1. If step > maxstep, the parameter learning process ends and the learned parameter θ is output; otherwise, proceed to step 5.3).

[0105] Example 2:

[0106] In communication systems, signals are represented as two-dimensional vectors X={ X (i)}={ X i1 , X i2}, i=[1,2,...,N]. According to known experience, the signal in this communication system follows an asymmetric binary mixed generalized normal probability distribution, and its probability density function is as follows:

[0107] ;

[0108] in ,

[0109] where θ=<θ1,θ2,...θ K >,θ k =< α k,d , u k,d , σk,d,1 , σ k,d,2 >,k=[1,2,....,K],d=[1,2].

[0110] In the probability density function above, θ is an unknown parameter, meaning the parameter to be estimated is a four-dimensional vector. To analyze these signals, it is often necessary to first determine the probability distribution parameters of these signals before proceeding to the next step of signal processing.

[0111] Figure 1 A set of actual signal samples is given, where the blue points are two-dimensional signal points; Figure 2 It is a density map given by the kernel density estimation method (KDE), which can be regarded as an approximation of the true distribution; Figure 3-Figure 5 The probability distribution plots for the parameters calculated using this method at the 10th, 50th, and 90th iterations show that the probability distributions increasingly approximate the true distribution as the number of iterations increases. By obtaining a mathematical representation of the mixed probability distribution—that is, its mathematical form and parameters—it can be used for further signal compression, simulation, and other processing in practical engineering applications.

[0112] It should be understood that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Those skilled in the art may modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein; and all these modifications and replacements should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for estimating parameters of a mixture probability distribution based on deep learning, characterized by: The following steps are involved:

1. Data preprocessing: Given a set of signal data sets X in a communication system, calculate the mean and variance of all signals in the data set, and then normalize each data sample to obtain a normalized sample X (i) , i=[1,2,...,N], N is the total number of samples; 2. Establishment of signal probability density function: Probability density function of a mixture probability distribution ;in is the probability density function of the kth component probability distribution in the mixed probability distribution, w k is the weight of the kth probability distribution in the overall distribution, is the parameter of the probability density function of the kth component, k=[1,2,...,K], K is the total number of components of the mixed distribution; θ=<θ1,θ2,...,θ K > is the total parameter vector consisting of the probability density function parameters of all components; 3. Establishment of the objective function for mixed probability distribution parameter estimation: Establish the objective function and lower bound function for parameter estimation of mixed probability distribution; 4. Use deep learning methods to iteratively optimize the lower bound function and solve the parameter θ: 4.1) Sample the parameter θ from a standard normal distribution with a mean of 0 and a variance of 1 as the initial value of the parameter; 4.2) Initialization w k =1 / K, initialize the error count en to 0, and the number of steps step=1; 4.3) Define the latent variable Z as a two-dimensional matrix with N rows and K columns, and calculate the value of Z; 4.4) Calculate w based on Z value k , according to the Z value and w k Calculate the Q function and obtain θ' based on the Q function; 4.5) Let the error e = |θ - θ'|. If the error e is less than 0.001, then let en = en + 1; otherwise, let en = 0. 4.6) Let θ = θ'. If en > the error termination condition, output the parameter θ; otherwise, execute step 4.7). 4.7) Let step = step + 1. If step > the maximum number of steps, output the parameter θ; otherwise, execute step 4.3).

2. The method for estimating parameters of a mixed probability distribution based on deep learning according to claim 1, wherein: In step 1, the dataset X={x (1) ,x (2) ,......,x (N) }; where x (i) is the i-th data sample, i=[1,2,...,N], N is the total number of samples, each x (i) Represented as a P dimensional vector, i.e. x (i) =< x i1 , x i2 ,.... x iP >, P is the dimension of the vector.

3. The method for estimating parameters of a mixed probability distribution based on deep learning according to claim 2, characterized in that: In step 1, the mean of the data sample ; Standard deviation of the data sample .

4. The method for estimating parameters of a mixed probability distribution based on deep learning according to claim 3, wherein: In step 1, normalize the samples .

5. The method for estimating parameters of a mixed probability distribution based on deep learning according to claim 1, wherein: In step 3, the objective function ; Lower bound function .

6. The method for estimating parameters of a mixed probability distribution based on deep learning according to claim 1, characterized in that: Before solving the parameter θ in step 4, predefined hyperparameters are also included. The hyperparameters include: batch size of data, number of execution rounds num_epochs, learning rate lr, error termination condition em and maximum number of steps maxstep.

7. The method for estimating parameters of a mixed probability distribution based on deep learning according to claim 6, characterized in that: batch_size is 512-4096, the default value is 2048; num_epochs is 20-100, the default value is 30; lr is 1e-3; em is 5, and maxstep is 100.

8. The method for estimating parameters of a mixed probability distribution based on deep learning according to claim 5, characterized in that: 4.3) Specifically: 4.3.1) Input the entire dataset X; 4.3.2) All elements of the latent variable Z are initialized to 0; 4.3.3) Let i = 1; 4.3.4) Get the i-th data in the dataset X X (i) ; 4.3.5) Let sum_Zi = 0; 4.3.6) Let k = 1; 4.3.7)Z[i,k] = w k * p k (X (i) ,θ k ); 4.3.8) sum_Zi = sum_Zi + Z[i,k]; 4.3.9) Let k = k + 1. If k > K, proceed to step 5.3.

10. Otherwise, proceed to step 5.3.

7. 4.3.10) Let k = 1; 4.3.11) Let Z[i,k] = Z[i,k] / sum_Zi; 4.3.12) Let k = k + 1. If k > K, proceed to step 4.3.13). Otherwise, proceed to step 4.3.11). 4.3.13) Let i = i + 1. If i > N, then Z[i, k] is calculated and step 4.4 is executed. Otherwise, step 4.3.4 is executed.

9. The method for estimating parameters of a mixed probability distribution based on deep learning according to claim 5, characterized in that: 4.4) Specifically: 4.4.1) Define θ' = θ, set B = N / batch_size, divide the dataset into B batches, with the amount of data in each batch being batch_size, and randomly divide each data item in the dataset into batches; 4.4.2) Set epoch = 1; 4.4.3) Let b = 1, let X[b] be the bth batch of data, X[b,i] be the i-th data in the bth batch of data, and the serial number of this data in the original data set is idx i ; 4.4.4) Calculate the lower bound function Q function; 4.4.5) Calculate the gradient of the parameter θ' on the negative function -Q of the Q function and perform backpropagation. The backpropagation optimizer uses the Nadam method with a learning rate of 1e-3. After backpropagation, θ' is updated by the Nadam optimizer to obtain θ'; 4.4.6) Let b = b + 1. If b > B, proceed to step 4.4.7). Otherwise, proceed to step 4.4.

4. 4.4.7) Set epoch = epoch + 1. If epoch > num_epochs, proceed to step 4.4.8). Otherwise, skip to step 4.4.

3. 4.4.8) Order ,k=[1,2,....,K].

Images