Training data robust screening method for automatically eliminating abnormal data
By using normalized covariance matrix and iterative estimation technology in the radar system, multiple assumptions of abnormal data sets are constructed, and the screening performance degradation problem is solved in the case of unknown number of abnormal data, and the accuracy of abnormal data in the training data is achieved. Improved screening and detection performance.
Patent Information
- Application Number
- CN202410920191.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-10
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-10
AI Technical Summary
In the scenario where the number of abnormal data is unknown, the performance of abnormal data screening methods deteriorates or fails, making it difficult to accurately screen out abnormal data in training data, affecting radar detection performance.
Using normalized covariance matrix and iterative estimation method, multiple assumptions of the abnormal data set are constructed through the maximum likelihood function and fast maximum likelihood method, iterative sorting and eigenvalue decomposition, and gradually filter out abnormal data to improve anti-exception data. ability.
It realizes the improvement of robust screening performance when the number of abnormal data is unknown, reduces the adverse impact of abnormal data on detection performance, and improves the accurate screening ability of abnormal data in training data.
Smart Images

Figure CN118939950A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a robust screening method for training data for automatically eliminating abnormal data, and belongs to the technical field of radar signal processing. Background Art
[0002] In radar target multi-channel adaptive detection, training data is often used to estimate the unknown noise covariance matrix, and it is assumed that the training data only contains independent and identically distributed pure noise components, but not target components. However, the detection background of the actual detection environment is often non-uniform, and the training data often produces power changes due to the influence of the transmission distance. Unwanted abnormal signals will also change the structure of the covariance matrix. At this time, if the non-uniform training data is directly used to estimate the unknown covariance matrix, its estimation accuracy is difficult to guarantee, resulting in insufficient clutter suppression and target signal cancellation, which greatly affects the detection performance and even causes the detector to fail completely.
[0003] In many cases such as strong multipath effects, sudden terrain conditions, dynamic electromagnetic interference, and other target echo interference around the unit to be detected, there are abnormal data mixed in the training data, and the number of abnormal data is often unknown and changing. Among the training data screening methods with a fixed number of abnormal data, the generalized inner product (GIP) is expressed as the inner product of the training data after covariance matrix whitening, which can be intuitively used as a reasonable means to eliminate abnormal data. There are various methods for estimating the covariance matrix used to whiten the training data. Using sampling matrix inversion (SMI) and fast maximum likelihood (FML), we can obtain the SMI method for non-iterative censored-SMI (NC-SMI) and the FML method for non-iterative censored-FML (NC-FML) respectively. However, the GIP statistic is heavily dependent on the accuracy of the background covariance matrix estimation. In actual scenarios, the covariance matrix in GIP usually needs to be estimated using training data. The abnormal data will lead to inaccurate covariance matrix estimation, especially when the power of abnormal data is greater than the background clutter power, the covariance matrix can no longer accurately describe the background, affecting the screening of abnormal data. It is proposed to consider using the normalized sample covariance matrix (NSCM) to replace the sample covariance matrix (SCM) in GIP, and obtain the normalized generalized inner product (NGIP) to screen the training data, so as to minimize the adverse effects caused by the excessive power of abnormal data. In addition, on the basis of using the normalized covariance matrix, the abnormal data set is estimated in combination with the iterative estimation method, so as to further avoid the adverse effects of abnormal data on the screening performance and improve the ability to resist abnormal data.
[0004] At present, most methods for abnormal data screening do not consider the scenario where the number of abnormal data is unknown, which leads to the degradation or even failure of screening performance when the number of abnormal data is unknown. In the scenario where the number of abnormal data is unknown, how to reasonably select the covariance matrix estimation method to minimize the adverse effects caused by abnormal data, combine iterative estimation methods to further avoid the adverse effects of abnormal data on screening performance, improve the ability to resist abnormal data, and at the same time accurately screen out abnormal data in training data is the key to improving the robust screening method of training data for automatic removal of abnormal data, and it is also one of the problems that need to be solved urgently. Summary of the invention
[0005] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a robust screening method for training data with automatic removal of abnormal data. In the scenario where the number of abnormal data is unknown, the present invention addresses the challenge of degradation or even complete failure of existing abnormal data screening methods, and how to reasonably select a covariance matrix estimation method to minimize the adverse effects caused by abnormal data, and combine iterative estimation methods to further avoid the impact of abnormal data on screening performance and improve the ability to resist abnormal data. While taking into account the accurate screening of abnormal data in training data, the robust screening capability of training data with automatic removal of abnormal data is improved.
[0006] The robust screening method for training data for automatically eliminating abnormal data of the present invention is special in that it comprises the following steps:
[0007] Step 1) The radar system obtains N×R dimensional training data Z from the distance cells around the unit to be detected; establishes multiple hypotheses H of the unknown abnormal data set p ,p=1,2,...,l, under the condition that the clutter covariance matrix M1 and the abnormal data covariance matrix M2 are known, the maximum value of l likelihood functions under the unknown abnormal data set is optimized to construct a preliminary estimate of the abnormal data set
[0008] Step 2) Combine the iterative estimation method to estimate the abnormal data set. In the iterative startup phase, the radar system calculates the normalized sampling covariance matrix of all training data. Based on the first iteration NGIP of each distance unit, all training data are sorted from large to small to obtain the first sorting index sequence J of the training data. (1) , J (1) The top elements in the list are identified as potential abnormal data indices. Therefore, under the condition that the covariance matrix is known, The solution of is simplified to solving the normalized iterative estimate of the number of outliers;
[0009] Step 3) In the first iteration estimation, the radar system establishes H p ML estimates of M1 and M2 under the assumption and Under the noisy additive structure, the ML estimation of the unknown matrices M1 and M2 can be expressed as FML estimation form. To this end, the SCM estimation of M1 and M2 is subjected to eigenvalue decomposition, the eigenvalue matrix is reconstructed, and the eigenvalues smaller than the noise power are loaded to the same level as the noise power;
[0010] Step 4) and Substitute the likelihood function under the condition of unknown abnormal data to obtain the generalized likelihood function under the condition of unknown abnormal data set, maximize the generalized likelihood function under the condition of unknown abnormal data set, and obtain the normalized iterative adaptive estimate of the number of outliers in the first iteration Finally, combined with the sorting results, the adaptive normalized iterative estimation of the first iteration abnormal data set is obtained.
[0011] Step 5) Combined with the iterative estimation method, the covariance matrix in NGIP is re-estimated to obtain the removal The sampling covariance matrix estimate of Thus, all training data are reordered to obtain the second sorting index sequence J of training data (2) , based on step 3) and step 4), the second normalized iterative adaptive estimate of the number of outliers is obtained Will With J (2) Combine and select the top order after sorting index, constructed as the adaptive normalized iterative estimation of the abnormal data set for the second iteration Repeat until the maximum number of iterations T is reached or the results of two iterations are consistent. Output abnormal data set adaptive normalization iterative estimation
[0012] Preferably, the specific steps of step 1) include:
[0013] For a given N×R dimensional training data Z, where N represents the number of array elements and R represents the number of training data, z i ,i=1,2,...,R is an N×1 dimensional complex vector, representing the i-th training data,
[0014]
[0015] Ω={1,2,...,R} represents the set of R training data indices, that is, the entire training data set, Ω0={i1,i2,...,i m} is a subset of Ω, representing a set of m training data indexes disturbed by outliers, hereinafter referred to as an outlier data set. The present invention assumes that the number of outliers m is unknown and Rm ≥ N. Ω-Ω0 is the complement of Ω0, representing a set of training data indexes without outliers, that is, a uniform training data set.
[0016] For the uniform training data set Ω-Ω0, which only contains noise components and no outliers, assuming z i , i∈Ω-Ω0 obeys the zero-mean complex circular Gaussian distribution with unknown covariance matrix M1, that is
[0017] For the abnormal data set Ω0 containing outliers, the existence of outliers makes z i The covariance matrix structure of changes, when i∈Ω0, z i It obeys the zero-mean complex circular Gaussian distribution with unknown covariance matrix M2, that is
[0018] When M1 and M2 are known, in order to filter out the abnormal data set Ω0, multiple hypotheses of unknown abnormal data sets are considered.
[0019]
[0020] Among them, p = 1, 2, ..., l, l represents the total number of all possible sets corresponding to Ω0. According to the combination rule, we know (·)! represents factorial; the pth set Ω p Contains m p outliers, H p The abnormal data set is Ω p Assumptions; Indicated by Ω p The corresponding m p N×m training data p dimensional matrix, Indicated by Ω-Ω p The corresponding Rm p N×(Rm p )-dimensional matrix.
[0021] H p The joint conditional probability density function of Z under the assumption is expressed as
[0022]
[0023] Where ∏(·) represents the multiplication symbol, (·) H denotes the conjugate transpose of a matrix, det(·) denotes the determinant of a matrix, exp(·) denotes the exponential function with the natural coefficient e as the base, and Tr(·) denotes the trace of a matrix; and Corresponding to the set Ω-Ω p and Ω p , ∑(·) represents the summation symbol.
[0024] Preliminary estimation of abnormal data sets To maximize the value of formula (3), p , It can be expressed as
[0025]
[0026] in Indicates the Ω when (·) takes the maximum value p , simplify the right side of equation (4) and remove the p Irrelevant term π -RN , and take the negative logarithm,
[0027]
[0028] in Indicates the Ω when (·) takes the minimum value p , ln(·) refers to the natural coefficient e is a low logarithmic function.
[0029] Preferably, the specific steps of step 2) include:
[0030] At the start of the iteration, the normalized sampling covariance matrix of all training data It can be expressed as,
[0031]
[0032] where ||·|| represents the Euclidean norm.
[0033] The first iteration normalized generalized inner product of the i-th training data It can be expressed as
[0034]
[0035] Sort the first iteration NGIP of all training data from large to small to obtain the first sorting index sequence J of training data (1) ={(1),(2),...,(R)}, where (j) represents the jth training data index after the initial GIP sorting from large to small, H p Under the assumption, the first iteration is based on the simplified estimation of the abnormal data set sorted by NGIP
[0036] At this time, Σ p1 and Σ p2 A simplified estimate of and Can be expressed as
[0037]
[0038] When the covariance matrix is known, The solution process is simplified to the normalized iterative estimation of the number of outliers in the first iteration The solution of formula (5) can be further simplified to
[0039]
[0040] Preferably, the specific steps of step 3) include:
[0041] In the context of clutter and noise coexisting, the uniform training data covariance matrix M1 = I N +M c , abnormal data covariance matrix M2 = M1 + M p , where the N×N dimensional identity matrix I N represents the noise covariance matrix, assuming the noise power is 1, M c is the clutter covariance matrix, M p represents the difference in structure between the two covariance matrices.
[0042] For the covariance matrix of the above-mentioned noisy additive structure, the maximum likelihood estimation of the unknown matrices M1 and M2 can be expressed as the FML estimation form, that is, firstly reconstruct the eigenvalue matrix after SCM eigenvalue decomposition, and load the eigenvalues smaller than the noise power to a level equal to the noise power.
[0043] In H p Under the assumption, a simplified estimate based on the uniform training data set sorted by NGIP in the first iteration is The corresponding SCM is Simplified estimation of abnormal data sets based on NGIP sorting The corresponding SCM is
[0044]
[0045] Further, and Perform eigenvalue decomposition and we can get
[0046]
[0047] Among them, Φ1 and Φ2 represent M S1 and M S2 The matrix composed of the eigenvectors of M; Λ1 and Λ2 represent S1 and M S2 The diagonal matrix composed of the eigenvalues of N ) and Λ2=Diag(ν1,ν2,...,ν N ), μ n ,n=1,2,...,N and ν n ,n=1,2,...,N represent M respectively S1 and M S2 The nth eigenvalue of , Diag(·) represents a diagonal matrix consisting of the specified elements and order.
[0048] Load the eigenvalues smaller than the noise power to a size equal to the noise power, reconstruct the eigenvalue matrix, and then replace the original eigenvalue matrices Λ1 and Λ2 in equation (11), and we can get H p Assume and FML estimation and
[0049]
[0050] Where max(a,b) represents the larger value of a and b.
[0051] It is worth noting that the use of the fast maximum likelihood method to estimate the noise covariance matrix M1 and the abnormal data covariance matrix M2 makes full use of the structural prior information of the noise covariance matrix, improves the estimation accuracy of the unknown covariance matrix structure, and under different assumptions, the amount of abnormal data training data may be less than the number of array elements N, ensuring and It is a non-singular matrix, which is convenient for subsequent calculations.
[0052] Preferably, the specific steps of step 4) include:
[0053] The FML estimate in equation (12) is and Substituting the true values M1 and M2 in equation (9), the likelihood function is reconstructed to obtain the generalized likelihood function under the condition of unknown abnormal data set. By maximizing the generalized likelihood function under the condition of unknown abnormal data set, the adaptive normalized estimate of the number of abnormal data is obtained:
[0054]
[0055] Combined with the first iteration NGIP sorting result J in step 2 (1) , select the first one after sorting training data, constructed as the first iteration abnormal data set adaptive normalization iterative estimation
[0056]
[0057] Preferably, the specific steps of step 5) include:
[0058] Normalized generalized inner product of the second iteration of the i-th training data It can be expressed as
[0059]
[0060] in Represents the NGIP covariance matrix of the second iteration, which is obtained by removing the abnormal data set estimated in the first iteration and calculating the remaining training data NSCM:
[0061]
[0062] Sort the NGIPs of all training data from large to small to obtain the second training data sorting index sequence J (2) ={(1),(2), ... ,(R)}. Based on steps 3 and 4, the second normalized iterative adaptive estimate of the number of outliers is obtained
[0063]
[0064] Combined with J (2) , before selecting The indexes constitute the second iteration abnormal data set adaptive normalization iterative estimation
[0065]
[0066] This cycle iterates until the maximum number of iterations T is reached or the results of two iterations are consistent. The output is an adaptive normalized iterative estimate of the abnormal data set In summary, the method can be summarized as a robust screening method for training data with automatic removal of abnormal data.
[0067] The beneficial effects of the present invention are:
[0068] 1) A robust screening method for training data that automatically removes abnormal data is constructed, which has low computational complexity and is easy to implement in engineering. Its screening performance is better than the existing screening method with a fixed upper limit on the number of abnormal data.
[0069] 2) By replacing SCM with NSCM and utilizing the NGIP sorting method, the adverse effects of strong abnormal data power on screening performance are reduced and sorting accuracy is improved.
[0070] 3) Combine iterative estimation method to estimate abnormal data sets, further avoid the adverse effects of outliers on screening performance, and improve the ability to resist outliers.
[0071] 4) The method of the present invention is applicable to situations where the number of abnormal data is unknown, for example, it has good application prospects in adaptive detection under dynamic electromagnetic interference. BRIEF DESCRIPTION OF THE DRAWINGS
[0072] Figure 1 It is a functional module diagram of the robust screening method for training data for automatic elimination of abnormal data of the present invention;
[0073] Figure 2 It is a performance comparison chart of the normalized signal-to-interference ratio (NSIR) between the method of the present invention and the existing training data screening method. Figure 2 The outlier steering vector is consistent with the target steering vector, the outlier power to noise power ratio ONR = 20dB, the clutter power to noise power ratio CNR = 10dB, N = 8, m = 4, the one-way correlation coefficient ρ = 0.9, and the maximum number of iterations T = 10. DETAILED DESCRIPTION
[0074] The present invention is further described below in conjunction with the accompanying drawings. The embodiments of the present invention are used to explain the present invention rather than to limit the present invention. Any modifications and changes made to the present invention within the spirit of the present invention and the protection scope of the claims fall within the protection scope of the present invention.
[0075] Example 1
[0076] The robust screening method for training data for automatically eliminating abnormal data in this embodiment includes the following steps:
[0077] Step 1) The radar system obtains N×R dimensional training data Z from the distance cells around the unit to be detected; establishes multiple hypotheses H of the unknown abnormal data set p ,p=1,2,...,l, under the condition that the clutter covariance matrix M1 and the abnormal data covariance matrix M2 are known, the maximum value of l likelihood functions under the unknown abnormal data set is optimized to construct a preliminary estimate of the abnormal data set
[0078] Step 2) Combine the iterative estimation method to estimate the abnormal data set. In the iterative startup phase, the radar system calculates the normalized sampling covariance matrix of all training data. Based on the first iteration NGIP of each distance unit, all training data are sorted from large to small to obtain the first sorting index sequence J of the training data. (1) , J (1) The top elements in the list are identified as potential abnormal data indices. Therefore, under the condition that the covariance matrix is known, The solution of is simplified to solving the normalized iterative estimate of the number of outliers;
[0079] Step 3) In the first iteration estimation, the radar system establishes H p ML estimates of M1 and M2 under the assumption and Under the noisy additive structure, the ML estimation of the unknown matrices M1 and M2 can be expressed as FML estimation form. To this end, the SCM estimation of M1 and M2 is subjected to eigenvalue decomposition, the eigenvalue matrix is reconstructed, and the eigenvalues smaller than the noise power are loaded to the same level as the noise power;
[0080] Step 4) and Substitute the likelihood function under the condition of unknown abnormal data to obtain the generalized likelihood function under the condition of unknown abnormal data set, maximize the generalized likelihood function under the condition of unknown abnormal data set, and obtain the normalized iterative adaptive estimate of the number of outliers in the first iteration Finally, combined with the sorting results, the adaptive normalized iterative estimation of the first iteration abnormal data set is obtained.
[0081] Step 5) Combined with the iterative estimation method, the covariance matrix in NGIP is re-estimated to obtain the removal The sampling covariance matrix estimate of Thus, all training data are reordered to obtain the second sorting index sequence J of training data (2) , based on step 3) and step 4), the second normalized iterative adaptive estimate of the number of outliers is obtained Will With J (2) Combine and select the top order after sorting index, constructed as the adaptive normalized iterative estimation of the abnormal data set for the second iteration Repeat until the maximum number of iterations T is reached or the results of two iterations are consistent. Output abnormal data set adaptive normalization iterative estimation
[0082] Preferably, the specific steps of step 1) include:
[0083] For a given N×R dimensional training data Z, where N represents the number of array elements and R represents the number of training data, z i ,i=1,2,...,R is an N×1 dimensional complex vector, representing the i-th training data,
[0084]
[0085] Ω={1,2,...,R} represents the set of R training data indices, that is, the entire training data set, Ω0={i1,i2,...,i m} is a subset of Ω, representing a set of m training data indexes disturbed by outliers, hereinafter referred to as an outlier data set. The present invention assumes that the number of outliers m is unknown and Rm ≥ N. Ω-Ω0 is the complement of Ω0, representing a set of training data indexes without outliers, that is, a uniform training data set.
[0086] For the uniform training data set Ω-Ω0, which only contains noise components and no outliers, assuming z i , i∈Ω-Ω0 obeys the zero-mean complex circular Gaussian distribution with unknown covariance matrix M1, that is
[0087] For the abnormal data set Ω0 containing outliers, the existence of outliers makes z i The covariance matrix structure of changes, when i∈Ω0, z i It obeys the zero-mean complex circular Gaussian distribution with unknown covariance matrix M2, that is
[0088] When M1 and M2 are known, in order to filter out the abnormal data set Ω0, multiple hypotheses of unknown abnormal data sets are considered.
[0089]
[0090] Among them, p = 1, 2, ..., l, l represents the total number of all possible sets corresponding to Ω0. According to the combination rule, we know (·)! represents factorial; the pth set Ω p Contains m p outliers, H p The abnormal data set is Ω p Assumptions; Indicated by Ω p The corresponding m p N×m training data p dimensional matrix, Indicated by Ω-Ω p The corresponding Rm p N×(Rm p )-dimensional matrix.
[0091] H p The joint conditional probability density function of Z under the assumption is expressed as
[0092]
[0093] Where ∏(·) represents the multiplication symbol, (·) H denotes the conjugate transpose of a matrix, det(·) denotes the determinant of a matrix, exp(·) denotes the exponential function with the natural coefficient e as the base, and Tr(·) denotes the trace of a matrix; and Corresponding to the set Ω-Ω p and Ω p , ∑(·) represents the summation symbol.
[0094] Preliminary estimation of abnormal data sets To maximize the value of formula (3), p , It can be expressed as
[0095]
[0096] in Indicates the Ω when (·) takes the maximum value p , simplify the right side of equation (4) and remove the p Irrelevant term π -RN , and take the negative logarithm,
[0097]
[0098] in Indicates the Ω when (·) takes the minimum value p , ln(·) refers to the logarithmic function with the natural coefficient e as the low.
[0099] Preferably, the specific steps of step 2) include:
[0100] At the start of the iteration, the normalized sampling covariance matrix of all training data It can be expressed as,
[0101]
[0102] where ||·|| represents the Euclidean norm.
[0103] The first iteration normalized generalized inner product of the i-th training data It can be expressed as
[0104]
[0105] Sort the first iteration NGIP of all training data from large to small to obtain the first sorting index sequence J of training data (1) ={(1),(2),...,(R)}, where (j) represents the jth training data index after the initial GIP sorting from large to small, Under the Hp assumption, the first iteration is based on the simplified estimation of the abnormal data set sorted by NGIP
[0106] At this time, Σ p1 and Σ p2 A simplified estimate of and Can be expressed as
[0107]
[0108] When the covariance matrix is known, The solution process is simplified to the normalized iterative estimation of the number of outliers in the first iteration The solution of formula (5) can be further simplified to
[0109]
[0110] Preferably, the specific steps of step 3) include:
[0111] In the context of clutter and noise coexisting, the uniform training data covariance matrix M1 = I N +M c , abnormal data covariance matrix M2 = M1 + M p , where the N×N dimensional identity matrix I N represents the noise covariance matrix, assuming the noise power is 1, M c is the clutter covariance matrix, M p represents the difference in structure between the two covariance matrices.
[0112] For the covariance matrix of the above-mentioned noisy additive structure, the maximum likelihood estimation of the unknown matrices M1 and M2 can be expressed as the FML estimation form, that is, firstly reconstruct the eigenvalue matrix after SCM eigenvalue decomposition, and load the eigenvalues smaller than the noise power to a level equal to the noise power.
[0113] In H p Under the assumption, a simplified estimate based on the uniform training data set sorted by NGIP in the first iteration is The corresponding SCM is Simplified estimation of abnormal data sets based on NGIP sorting The corresponding SCM is
[0114]
[0115] Further, and Perform eigenvalue decomposition and we can get
[0116]
[0117] Among them, Φ1 and Φ2 represent M S1 and M S2 The matrix composed of the eigenvectors of M; Λ1 and Λ2 represent S1 and M S2The diagonal matrix composed of the eigenvalues of N ) and Λ2=Diag(ν1,ν2,...,ν N ), μ n ,n=1,2,...,N and ν n ,n=1,2,...,N represent M respectively S1 and M S2 The nth eigenvalue of , Diag(·) represents a diagonal matrix consisting of the specified elements and order.
[0118] Load the eigenvalues smaller than the noise power to a size equal to the noise power, reconstruct the eigenvalue matrix, and then replace the original eigenvalue matrices Λ1 and Λ2 in equation (11), and we can get H p Assume and FML estimation and
[0119]
[0120] Where max(a,b) represents the larger value of a and b.
[0121] It is worth noting that the use of the fast maximum likelihood method to estimate the noise covariance matrix M1 and the abnormal data covariance matrix M2 makes full use of the structural prior information of the noise covariance matrix, improves the estimation accuracy of the unknown covariance matrix structure, and under different assumptions, the amount of abnormal data training data may be less than the number of array elements N, ensuring and It is a non-singular matrix, which is convenient for subsequent calculations.
[0122] Preferably, the specific steps of step 4) include:
[0123] The FML estimate in equation (12) is and Substituting the true values M1 and M2 in equation (9), the likelihood function is reconstructed to obtain the generalized likelihood function under the condition of unknown abnormal data set. By maximizing the generalized likelihood function under the condition of unknown abnormal data set, the adaptive normalized estimate of the number of abnormal data is obtained:
[0124]
[0125] Combined with the first iteration NGIP sorting result J in step 2 (1) , select the first one after sorting The training data is constructed as the first iteration abnormal data set adaptive normalization iterative estimation
[0126]
[0127] Preferably, the specific steps of step 5) include:
[0128] Normalized generalized inner product of the second iteration of the i-th training data It can be expressed as
[0129]
[0130] in Represents the NGIP covariance matrix of the second iteration, which is obtained by removing the abnormal data set estimated in the first iteration and calculating the remaining training data NSCM:
[0131]
[0132] Sort the NGIPs of all training data from large to small to obtain the second training data sorting index sequence J (2) ={(1),(2),...,(R)}. Based on steps 3 and 4, the second normalized iterative adaptive estimate of the number of outliers is obtained.
[0133]
[0134] Combined with J (2) , before selecting The indexes constitute the second iteration abnormal data set adaptive normalization iterative estimation
[0135]
[0136] This cycle iterates until the maximum number of iterations T is reached or the results of two iterations are consistent. The output is an adaptive normalized iterative estimate of the abnormal data set In summary, the method can be summarized as a robust screening method for training data with automatic removal of abnormal data.
[0137] In order to verify the effectiveness of the method described in the present invention, this specific implementation mode provides two embodiments, the first embodiment is aimed at the sea detection environment, and the second embodiment is aimed at the ground detection environment.
[0138] Example 2
[0139] Refer to the instruction manual Figure 1 The specific implementation of Example 2 is divided into the following steps:
[0140] Step A1: Use the ground detection radar to illuminate the target-free range around the area to be detected, and obtain training data of R reference distance units. The training data Z is sent to the normalized generalized inner product sorting module, and the training data is sorted from large to small according to the generalized inner product; the sorted training data is sent to the multiple hypothesis establishment module, assuming that the number of abnormal data is m p ∈[1,RN], identify the first m after sorting p The training data of RN hypotheses are abnormal data; the training data of different hypotheses are sent to the likelihood function construction module under the condition of known covariance matrix, and the likelihood function under RN hypotheses is constructed according to formula (9) and sent to the generalized likelihood function construction module.
[0141] It is worth noting that in step A1, for dynamic electromagnetic interference, a model with an unknown number of abnormal data is established, which better fits the structure of the training data under real conditions. The proposed robust training data screening method with automatic removal of abnormal data can improve the screening performance under the condition of unknown number of abnormal data. The performance comparison results of the proposed method and the existing training data screening method are shown in the attached Figure 2 . The results show that compared with the existing NC-SMI and NC-FML methods, the method of the present invention has better screening performance. By normalizing the generalized inner product sorting method, the estimation of the abnormal data set can be associated with the estimation of the number of abnormal data, so that the maximum likelihood estimation process of the abnormal data set is simplified to the estimation of the number of abnormal data, and the computational efficiency is greatly improved. Compared with the generalized inner product sorting method, the adverse effects caused by excessive power of abnormal data can be minimized.
[0142] Step A2: Under different assumptions, the training data is sent to the covariance matrix fast maximum likelihood estimation module to estimate M1 and M2, and H is obtained according to formula (12): p Assume that the tth iteration and Will and The generalized likelihood function is sent to the generalized likelihood function construction module, and the likelihood function obtained in step A1 is substituted to replace the unknown covariance matrix to obtain the generalized likelihood function under different assumptions. The values of the RN generalized likelihood functions are sent to the maximum value selection module to determine under which assumption the value of the generalized likelihood function is the largest. According to formula (17), the normalized iterative adaptive estimate of the number of outliers in the tth iteration is obtained:
[0143] It is worth noting that in step A2, the robust screening method for training data with automatic removal of abnormal data is constructed, and the noise covariance matrix M1 and the abnormal data covariance matrix M2 are estimated using the fast maximum likelihood method, which makes full use of the structural prior information of the noise covariance matrix and improves the estimation accuracy of the unknown covariance matrix structure. In addition, the abnormal data only accounts for a small part of the training data. Under different assumptions, the amount of abnormal data training data may be less than the number of array elements N. The fast maximum likelihood method ensures that the estimated covariance matrix is a non-singular matrix, which is convenient for engineering implementation.
[0144] Step A3 sends the sorting result of the normalized generalized inner product sorting module to the abnormal data set construction module, and combines the normalized iterative adaptive estimation of the number of abnormal values in the t-th iteration process obtained in step A2 According to formula (18), we can estimate the abnormal data set Will It is sent to the iteration module to determine whether the iteration stop condition is met. If t<T, the iteration continues. Send it to the generalized inner product sorting module, re-estimate the covariance matrix used to whiten the training data according to formula (16), and repeat steps A1, A2, and A3; if t = T, then Send to the screening results and end the screening process.
[0145] It is worth noting that in step A3, It is sent to the iteration module to further avoid the adverse effects of abnormal data on screening performance and improve the ability to resist abnormal data by optimizing the method flow.
[0146] Example 3
[0147] Refer to the instruction manual Figure 1 The specific implementation of Example 3 is divided into the following steps:
[0148] Step B1 uses ground detection radar to illuminate the target-free range around the area to be detected, and obtains training data of R reference distance units. The training data Z is sent to the normalized generalized inner product sorting module, and the training data is sorted from large to small according to the generalized inner product; the sorted training data is sent to the multiple hypothesis establishment module, assuming that the number of abnormal data is m p ∈[1,RN], identify the first m after sorting p The training data of RN hypotheses are abnormal data; the training data of different hypotheses are sent to the likelihood function construction module under the condition of known covariance matrix, and the likelihood function under RN hypotheses is constructed according to formula (9) and sent to the generalized likelihood function construction module.
[0149] It is worth noting that in step B1, for dynamic electromagnetic interference, a model with an unknown number of abnormal data is established, which better fits the structure of the training data under real conditions. The proposed robust screening method for training data with automatic removal of abnormal data can improve the screening performance under conditions with an unknown number of abnormal data. The performance comparison results of the proposed method and the existing training data screening method are shown in the attached Figure 2 . The results show that compared with the existing NC-SMI and NC-FML methods, the method of the present invention has better screening performance. Through the normalized generalized inner product sorting method, the estimation of the abnormal data set can be associated with the estimation of the number of abnormal data, so that the maximum likelihood estimation process of the abnormal data set is simplified to the normalized iterative estimation of the number of outliers, and the computational efficiency is greatly improved. Compared with the generalized inner product sorting method, the adverse effects caused by excessive power of abnormal data can be minimized.
[0150] Step B2: Under different assumptions, the training data is sent to the covariance matrix fast maximum likelihood estimation module to estimate M1 and M2, and H is obtained according to formula (12): p Assume that the tth iteration and Will and Send it to the generalized likelihood function construction module, substitute the likelihood function obtained in step B1, replace the unknown covariance matrix, and obtain the generalized likelihood function under different assumptions; send the values of RN generalized likelihood functions to the maximum value selection module to determine under which assumption the value of the generalized likelihood function is the largest, and obtain the normalized iterative adaptive estimate of the number of outliers in the tth iteration according to formula (17):
[0151] It is worth noting that in step B2, the robust screening method for training data with automatic removal of abnormal data is constructed, and the noise covariance matrix M1 and the abnormal data covariance matrix M2 are estimated using the fast maximum likelihood method, which makes full use of the structural prior information of the noise covariance matrix and improves the estimation accuracy of the unknown covariance matrix structure. In addition, the abnormal data only accounts for a small part of the training data. Under different assumptions, the amount of abnormal data training data may be less than the number of array elements N. The fast maximum likelihood method ensures that the estimated covariance matrix is a non-singular matrix, which is convenient for engineering implementation.
[0152] Step B3 sends the sorting result of the normalized generalized inner product sorting module to the abnormal data set construction module, and combines the normalized iterative adaptive estimation of the number of abnormal values in the t-th iteration process obtained in step B2 According to formula (18), we can estimate the abnormal data set Will It is sent to the iteration module to determine whether the iteration stop condition is met. If t<T, the iteration continues. Send it to the generalized inner product sorting module, re-estimate the covariance matrix used to whiten the training data according to formula (16), and repeat steps B1, B2, and B3; if t = T, then Send to the screening results and end the screening process.
[0153] It is worth noting that in step B3, It is sent to the iteration module to further avoid the adverse effects of abnormal data on screening performance and improve the ability to resist abnormal data by optimizing the method flow.
Claims
1. A robust screening method for training data with automatic removal of abnormal data, characterized by The following steps are involved: Step 1) The radar system obtains N×R dimensional training data Z from the distance cells around the unit to be detected; establishes multiple hypotheses H of the unknown abnormal data set p ,p=1,2,...,l, under the condition that the clutter covariance matrix M1 and the abnormal data covariance matrix M2 are known, the maximum value of l likelihood functions under the unknown abnormal data set is optimized to construct a preliminary estimate of the abnormal data set Step 2) Combine the iterative estimation method to estimate the abnormal data set. In the iterative startup phase, the radar system calculates the normalized sampling covariance matrix of all training data. Based on the first iteration NGIP of each distance unit, all training data are sorted from large to small to obtain the first sorting index sequence J of the training data. (1) , J (1) The top elements in the list are identified as potential abnormal data indices. Therefore, under the condition that the covariance matrix is known, The solution of is simplified to solving the normalized iterative estimate of the number of outliers; Step 3) In the first iteration estimation, the radar system establishes H p ML estimates of M1 and M2 under the assumption and Under the noisy additive structure, the ML estimation of the unknown matrices M1 and M2 can be expressed as FML estimation form. To this end, the SCM estimation of M1 and M2 is subjected to eigenvalue decomposition, the eigenvalue matrix is reconstructed, and the eigenvalues smaller than the noise power are loaded to the same level as the noise power; Step 4) and Substitute the likelihood function under the condition of unknown abnormal data to obtain the generalized likelihood function under the condition of unknown abnormal data set, maximize the generalized likelihood function under the condition of unknown abnormal data set, and obtain the normalized iterative adaptive estimate of the number of outliers in the first iteration Finally, combined with the sorting results, the adaptive normalized iterative estimation of the first iteration abnormal data set is obtained. Step 5) Combined with the iterative estimation method, the covariance matrix in NGIP is re-estimated to obtain the removal The sampling covariance matrix estimate of Thus, all training data are reordered to obtain the second sorting index sequence J of training data (2) , based on step 3) and step 4), the second normalized iterative adaptive estimate of the number of outliers is obtained Will With J (2) Combine and select the top order after sorting index, constructed as the adaptive normalized iterative estimation of the abnormal data set for the second iteration Repeat until the maximum number of iterations T is reached or the results of two iterations are consistent. Output abnormal data set adaptive normalization iterative estimation 2. The method for robustly screening training data for automatically eliminating abnormal data according to claim 1 is characterized in that The specific steps of step 1) include: For a given N×R dimensional training data Z, where N represents the number of array elements and R represents the number of training data, z i ,i=1,2,...,R is an N×1 dimensional complex vector, representing the i-th training data, Ω={1,2,...,R} represents the set of R training data indices, that is, the entire training data set, Ω0={i1,i2,...,i m } is a subset of Ω, representing a set of m training data indexes disturbed by outliers, hereinafter also referred to as an outlier data set. The present invention assumes that the number of outliers m is unknown and Rm≥N; Ω-Ω0 is the complement of Ω0, representing a set of training data indexes without outliers, that is, a uniform training data set; For the uniform training data set Ω-Ω0, which only contains noise components and no outliers, assuming z i , i∈Ω-Ω0 obeys the zero-mean complex circular Gaussian distribution with unknown covariance matrix M1, that is, z i ~CN(0,M1); For the abnormal data set Ω0 containing outliers, the existence of outliers makes z i The covariance matrix structure of changes, when i∈Ω0, z i It obeys the zero-mean complex circular Gaussian distribution with unknown covariance matrix M2, that is, z i ~CN(0,M2); When M1 and M2 are known, in order to filter out the abnormal data set Ω0, multiple hypotheses of unknown abnormal data sets are considered. Among them, p = 1, 2, ..., l, l represents the total number of all possible sets corresponding to Ω0. According to the combination rule, we know represents factorial; the pth set Ω p Contains m p outliers, H p The abnormal data set is Ω p Assumptions; Indicated by Ω p The corresponding m p N×m training data p dimensional matrix, Indicated by Ω-Ω p The corresponding Rm p N×(Rm p )-dimensional matrix; H p The joint conditional probability density function of Z under the assumption is expressed as Where Π(·) represents the continuous multiplication symbol, (·)H represents the conjugate transpose of the matrix, det(·) represents the determinant of the matrix, exp(·) represents the exponential function with the natural coefficient e as the base, and Tr(·) represents the trace of the matrix; and Corresponding to the set Ω-Ω p and Ω p , ∑(·) represents the summation symbol; Preliminary estimation of abnormal data sets To maximize the value of formula (3), p , It can be expressed as in Indicates the Ω when (·) takes the maximum value p , simplify the right side of equation (4) and remove the p Irrelevant term π -RN , and take the negative logarithm at the same time, in Indicates the Ω when (·) takes the minimum value p , ln(·) refers to the logarithmic function with the natural coefficient e as the low.
3. The method for robustly screening training data for automatically eliminating abnormal data according to claim 2 is characterized in that The specific steps of step 2) include: At the start of the iteration, the normalized sampling covariance matrix of all training data It can be expressed as, where ||·|| represents the Euclidean norm; The first iteration normalized generalized inner product of the i-th training data It can be expressed as Sort the first iteration NGIP of all training data from large to small to obtain the first sorting index sequence J of training data (1) ={(1),(2),...,(R)}, where (j) represents the jth training data index after the initial GIP sorting from large to small, H p Under the assumption, the first iteration is based on the simplified estimation of the abnormal data set sorted by NGIP At this time, Σ p1 and Σ p2 A simplified estimate of and Can be expressed as When the covariance matrix is known, The solution process is simplified to the normalized iterative estimation of the number of outliers in the first iteration The solution of formula (5) can be further simplified to 4. The method for robustly screening training data for automatically eliminating abnormal data according to claim 1 is characterized in that The specific steps of step 3) include: In the context of clutter and noise coexisting, the uniform training data covariance matrix M1 = I N +M c , abnormal data covariance matrix M2 = M1 + M p , where the N×N dimensional identity matrix I N represents the noise covariance matrix, assuming the noise power is 1, M c is the clutter covariance matrix, M p Represents the difference in structure between two covariance matrices; For the covariance matrix of the above-mentioned noise additive structure, the maximum likelihood estimation of the unknown matrices M1 and M2 can be expressed as the FML estimation form, that is, firstly reconstruct the eigenvalue matrix after SCM eigenvalue decomposition, and load the eigenvalues smaller than the noise power to the same level as the noise power; In H p Under the assumption, a simplified estimate based on the uniform training data set sorted by NGIP in the first iteration is The corresponding SCM is Simplified estimation of abnormal data sets based on NGIP sorting The corresponding SCM is Further, and Perform eigenvalue decomposition and we can get Among them, Φ1 and Φ2 represent M S1 and M S2 The matrix composed of the eigenvectors of M; Λ1 and Λ2 represent S1 and M S2 The diagonal matrix composed of the eigenvalues of N ) and Λ2=Diag(ν1,ν2,...,ν N ), μ n ,n=1,2,...,N and ν n ,n=1,2,...,N represent M respectively S1 and M S2 The nth eigenvalue of , Diag(·) represents a diagonal matrix consisting of the specified elements and order; Load the eigenvalues smaller than the noise power to a size equal to the noise power, reconstruct the eigenvalue matrix, and then replace the original eigenvalue matrices Λ1 and Λ2 in equation (11), and we can get H p Assume and FML estimation and Where max(a,b) represents the larger value of a and b.
5. The method for robustly screening training data for automatically eliminating abnormal data according to claim 3 is characterized in that The specific steps of step 4) include: The FML estimate in equation (12) is and Substituting the true values M1 and M2 in equation (9), the likelihood function is reconstructed to obtain the generalized likelihood function under the condition of unknown abnormal data set. By maximizing the generalized likelihood function under the condition of unknown abnormal data set, the adaptive normalized estimate of the number of abnormal data is obtained: Combined with the first iteration NGIP sorting result J in step 2 (1) , select the first one after sorting training data, constructed as the first iteration abnormal data set adaptive normalization iterative estimation 6. The method for robustly screening training data for automatically eliminating abnormal data according to claim 1, characterized in that The specific steps of step 5) include: Normalized generalized inner product of the second iteration of the i-th training data Expressed as in Represents the NGIP covariance matrix of the second iteration, which is obtained by removing the abnormal data set estimated in the first iteration and calculating the remaining training data NSCM: Sort the NGIPs of all training data from large to small to obtain the second training data sorting index sequence J (2) ={(1),(2),...,(R)}; Based on steps 3 and 4, the second normalized iterative adaptive estimate of the number of outliers is obtained Combined with J (2) , before selecting The indexes constitute the second iteration abnormal data set adaptive normalization iterative estimation This cycle iterates until the maximum number of iterations T is reached or the results of two iterations are consistent. The output is an adaptive normalized iterative estimate of the abnormal data set
Citation Information
Patent Citations
An outlier detection method based on agricultural big data
CN109345137A
Abnormal value interference resistant complex curved surface reconstruction method and system
CN118097025A
Method for estimating covariance matrices and use thereof
US20150131758A1
System and method for characterizing and tracking aging, resilience, cognitive decline, and disorders using brain dynamic biomarkers
WO2024092277A1
