Near infrared spectrum abnormal sample identification method based on high-dimensional subspace

Abstract

The invention discloses a near infrared spectrum abnormal sample identification method based on a high-dimensional subspace, and belongs to the technical field of near infrared spectrum analys.The technical scheme includes the steps that original spectrum data containing normal samples and abnormal samples are collected, and spectrums of a group of samples form a spectrum matrix A; preprocessing the original spectral data to obtain a preprocessed spectral matrix D; selecting a sparse subspace according to the processed spectrum matrix D; and calculating an abnormal sample according to the obtained subspace. The method has the advantages that the anomaly detection accuracy is effectively improved through the anomaly detection method based on the subspace, the related subspace is selected according to the local distribution non-uniformity, the one-dimensional variance, the local sparsity and other standards in the high-dimensional near infrared spectrum data, and the influence of the dimensionality on anomaly recognition is reduced. The abnormal value is determined in combination with the adaptive neighborhood and the threshold value, and the influence of neighborhood parameter selection on abnormal value identification is reduced.

Description

Technical Field

[0001] The invention belongs to the technical field of near-infrared spectrum analysis, and in particular relates to a near-infrared spectrum abnormal sample recognition method based on high-dimensional subspace. Background Art

[0002] Near infrared spectroscopy (NIRS) is a non-invasive optical sensing technology that uses light in the near infrared (NIR) range (700 to 2500nm) to measure changes in the optical properties of an object. It has the advantages of fast analysis speed, non-invasiveness, high penetration of the detection radiation beam, and low analysis cost. Therefore, this technology is widely used in pharmaceutical manufacturing, food and beverage industry, agriculture, biomedicine and other fields for quality control, process monitoring, component analysis, etc. However, in the process of NIR spectroscopy acquisition, outliers usually appear due to factors such as uneven sample distribution, environmental changes, measurement instrument deviation and improper operation. These outliers will deviate from the direction predicted by the model, making the model prediction results unreliable. Therefore, it is necessary to eliminate outliers in the NIR modeling process to improve the accuracy of the model. In addition, NIR spectral data usually covers multiple wavelengths, each of which can provide important information about the characteristics of the sample, so its dimensionality is usually high. In high-dimensional data sets, the distribution of each dimension is usually different, with obvious differences in some dimensions and no significant differences in other dimensions, which are irrelevant to anomaly detection. Therefore, how to eliminate the impact of dimensionality and accurately remove outliers in high-dimensional data sets is a key issue.

[0003] At present, in the field of near-infrared spectroscopy analysis, the commonly used abnormal sample detection methods mainly include Mahalanobis distance method, spectral residual ratio, leverage value, COOK distance and principal component analysis, etc. These methods rely on the center of gravity of the data and are sensitive to the data distribution conditions. They are effective in identifying only a small number of abnormal samples, but when the sample set contains a large number or multiple types of abnormal samples, it will change the mean and dispersion of the overall sample data, resulting in poor recognition effect, serious misjudgment and misjudgment, and thus affecting the stability and accuracy of the model. In addition, the above-mentioned abnormal sample identification methods are mostly empirical values ​​for the selection of abnormal sample discrimination detection limits or are subjectively set by the experimenter according to the experimental situation, which are highly random and lack unified standards and specifications. In addition, traditional outlier detection technology usually uses full-dimensional space and cannot effectively detect outliers in high-dimensional data. The existence of many irrelevant feature dimensions in high-dimensional data masks relevant information, so outliers are easily masked.

[0004] In order to solve the above problems, the present invention proposes a near-infrared spectrum abnormal sample recognition method based on high-dimensional subspace. Summary of the invention

[0005] Abnormal samples may appear in the process of near-infrared data acquisition due to improper instruments and operations. In addition, the high dimensionality and nonlinearity of near-infrared (NIR) spectral data make it difficult to measure outliers. The existing outlier measurement methods for near-infrared spectroscopy do not take into account the high-dimensional characteristics of near-infrared spectroscopy data, which will inevitably lead to a loss of sample measurement accuracy. Therefore, a near-infrared spectroscopy abnormal sample identification method based on high-dimensional subspace is proposed according to the high-dimensional characteristics of near-infrared spectroscopy.

[0006] The present invention is achieved by the following measures: a near infrared spectrum abnormal sample identification method based on high-dimensional subspace, characterized by comprising:

[0007] Collect the original spectral data of normal samples and abnormal samples, and combine the spectra of a group of samples into a spectral matrix A; select n samples including normal samples and abnormal samples as the modeling sample set, and collect their original spectral data, and directly irradiate the material with near-infrared light. After the light wave passes through the object, due to the vibration of the molecules inside the object itself, part of the light wave will be absorbed, and another part of the light wave will be reflected back or penetrate the object. The light wave reflected or penetrated by the object will change due to different internal components, so it can show the internal characteristic information of the object and obtain the characteristic spectrum containing various information of the substance. A spectrum of a sample can be represented by a vector, and the spectra of multiple samples can be represented by a matrix. Usually, the spectra of a group of samples are combined into a spectral matrix A of n*m dimensions (n ​​is the number of samples, m is the wavelength variable).

[0008] The raw spectral data is preprocessed to obtain a preprocessed spectral matrix D; the purpose of spectral preprocessing is to eliminate irrelevant information and noise in the spectral data, extract useful signals, and improve the correlation between spectral data and chemical components. Common preprocessing methods include mean centering, standardization, normalization, smoothing, derivative, multivariate scattering correction, etc. It is preferred to use multivariate scattering correction (MSC) and Savitzky-Golay convolution smoothing to preprocess the spectrum to preprocess the sample spectral matrix A to obtain the spectral matrix D.

[0009] Select a sparse subspace based on the processed spectral matrix D;

[0010] The outlier samples are calculated based on the obtained subspace.

[0011] In an optional implementation, let D = {x 1 ,x 2 ,…,x N} is a set of N data points in M-dimensional feature space, where x i is the i-th data point, each data point is represented as a normalized feature M-dimensional vector; the outliers are data points in the sparse area, so the data point xi The relevant subspace of is defined as the data point x in this subspace i The sparsest subspace of the k nearest neighbors of .

[0012] Then select the sparse subspace according to the processed spectral matrix D, including:

[0013] Determine the k value of the optimal neighborhood of the data point;

[0014] Calculate the local sparsity of data points;

[0015] Enter the feature dimension for calculation;

[0016] Iterate to get the optimal subspace.

[0017] In an optional implementation, determining the k value of the optimal neighborhood of the data point includes:

[0018] First, according to the formula Construct the matrix M of point P;

[0019] Construct a local neighborhood dimension eigenmode:

[0020] Determine the value of k so that E f =-L λ ln(L λ )-P λ ln(P λ )-S λ ln(S λ )The local entropy function calculated is the minimum;

[0021] The optimal neighborhood is obtained by the minimum entropy function criterion: k optimized =argmin(E f );

[0022] where k is the number of neighbors of P, is the center of the set of neighboring points, let λ1≥λ2≥λ3 be the three eigenvalues ​​of M, where the corresponding eigenvector of λ3 can be used as an approximation of the normal vector of point P.

[0023] In an optional implementation, according to the formula Determine the data point x i The local sparsity of is the data point x i The projection of S on the subspace, Represents data point x i The local subspace neighborhood of , which contains x in the subspace S i The k nearest neighbors of Is a collection The center of the symbol ||·||2 Represents the square of the Euclidean distance.

[0024] In an alternative implementation, the possible subspaces are restricted to the exact number of subspaces λ with relevant features, reformulating the objective function:

[0025] Calculate the optimal subspace;

[0026] in, shows the M different features along Data point x j and center c i The M-dimensional distance vector between them.

[0027] In an optional implementation, the optimal subspace is iterated, specifically: initializing lspmax i = -∞, t = 1, and from size M: f i Select λ features from the complete feature set;

[0028] when t≤Iter max , given In the case of , calculate the local subspace neighborhood of the i-th data point, and then Calculate the optimal subspace of the i-th data point and its corresponding local sparse value, and update the best subspace found so far: f * ←f i and subspace f *i The local sparsity value of is:

[0029] In an optional implementation, calculating the abnormal samples according to the obtained subspace includes sorting the abnormal scores V in ascending order as (v1<v2<v3<…<v n ), and then use Calculate the jump degree of each sample: T k (1≤k≤n-1);

[0030] If T k Maximum, then the value is v k+1 ,v k+2 ,...,v n The corresponding samples of are most likely to be outliers, and these samples are added to the outlier set U, and the remaining samples are added to the sample set N;

[0031] If the maximum jump degree T k T n-1 , then it will be combined with v nThe corresponding samples are added to the anomaly set U, and the jump degree is recalculated with the remaining n-1 samples, and then the singular samples are determined according to the above steps.

[0032] The beneficial effects of the technical solution provided by the embodiment of the present invention are as follows: the subspace-based anomaly detection method effectively improves the accuracy of anomaly detection. The method selects relevant subspaces according to the local distribution inhomogeneity, one-dimensional variance, and local sparsity in high-dimensional near-infrared spectral data, thereby reducing the influence of dimension on anomaly recognition. In addition, the method combines adaptive neighborhoods and thresholds to determine outliers, thereby reducing the influence of neighborhood parameter selection on outlier recognition. The method is much better than other subspace methods in terms of recognition accuracy and model performance. Of course, due to the sensitivity of near-infrared data to noise, the method can be further combined with feature selection and practical applications, which is of great significance to the construction of calibration models. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solution of the present invention, a brief introduction will be given to the drawings used in the embodiments below. Obviously, the drawings listed below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 It is a flow chart of a method for identifying abnormal samples of near infrared spectrum based on high-dimensional subspace in an embodiment of the present invention;

[0035] Figure 2 It is the projection of the sample on the optimal subspace in the embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0037] Embodiment 1:

[0038] See also Figure 1-Figure 2 A near infrared spectrum abnormal sample recognition method based on high-dimensional subspace, characterized by comprising:

[0039] Step S1, collect the original spectral data of normal samples and abnormal samples, and form a spectral matrix A with the spectra of a group of samples; select n samples including normal samples and abnormal samples as the modeling sample set, and collect their original spectral data, and directly irradiate the material with near-infrared light. After the light wave passes through the inside of the object, due to the vibration of the molecules inside the object itself, part of the light wave will be absorbed, and another part of the light wave will be reflected back or penetrate the object. The light wave reflected or penetrated by the object changes due to different internal components, so it can display the internal characteristic information of the object, and obtain the characteristic spectrum containing various information of the material. A spectrum of a sample can be represented by a vector, and the spectra of multiple samples can be represented by a matrix. Usually, the spectra of a group of samples are formed into a spectral matrix A of n*m dimensions (n ​​is the number of samples, and m is the wavelength variable).

[0040] Step S2, preprocessing the original spectral data to obtain a preprocessed spectral matrix D; the purpose of spectral preprocessing is to eliminate irrelevant information and noise of spectral data, extract useful signals, and improve the correlation between spectral data and chemical components. Commonly used preprocessing methods include mean centering, standardization, normalization, smoothing, derivative, multivariate scattering correction, etc. It is preferred to use multivariate scattering correction (MSC) and Savitzky-Golay convolution smoothing to preprocess the spectrum and preprocess the sample spectral matrix A to obtain the spectral matrix D.

[0041] Step S3, selecting a sparse subspace according to the processed spectral matrix D;

[0042] Let D = {x 1 ,x 2 ,…,x N} is a set of N data points in M-dimensional feature space, where x i is the i-th data point. Each data point can be represented as a normalized feature An outlier is a data point in a sparse area, so the data point x i The relevant subspace of is defined as the data point x in this subspace i The sparsest subspace of the k nearest neighbors of .

[0043] Step S31, determining the k value of the optimal neighborhood of the data point;

[0044] In the subspace nearest neighbor search process, all data points are projected into the given subspace f i Then use the projection distance to find the k nearest neighbors of the ith data point. Since the choice of k value often affects the validity of the subspace, in order to determine the optimal neighborhood size, we introduce a method to determine the k value. Use the PCA algorithm to estimate the normal vector of a point based on its neighborhood coordinates. Let point P i =[xi ,y i ,z i ] The matrix M of the neighborhood point P is expressed as follows:

[0045] where k is the number of neighbors of P, is the center of the set of neighboring points. Since the matrix M is a symmetric semi-positive definite matrix, all three eigenvalues ​​are non-negative. Let λ1≥λ2≥λ3 be the three eigenvalues ​​of M, where the corresponding eigenvector of λ3 can be used as an approximation of the normal vector of point P. Then, we construct a local neighborhood dimensionality feature model based on the dimensionality features of the eigenvalues ​​of the local covariance matrix M:

[0046]

[0047] According to Shannon's information entropy theory, the local entropy function can be defined according to the equation:

[0048] E f =-L λ ln(L λ )-P λ ln(P λ )-S λ ln(S λ )

[0049] From the above formula, we can see that the smaller the entropy value of the local neighborhood information, the closer the spatial distribution characteristics of the local data points are under the neighborhood size, and the size of the neighborhood tends to be optimal. Therefore, the size of the optimal neighborhood can be obtained by the minimum entropy function criterion: optimized =argmin(E f ).

[0050] Step S32: Calculate the local sparsity of the data points.

[0051] Generate a data point x in the optimally chosen subspace with maximum local sparsity i In order to select the local optimal subspace, according to its S(lsp i,S ), defining the variance of the k nearest neighbors of a data point x i The local sparsity measure S in the subspace is as follows:

[0052] in is the data point x i The projection of S on the subspace, Represents data point x i The local subspace neighborhood of , which contains x in the subspace S i The k nearest neighbors of Is a collection The center of the symbol ||·|| 2 Represents the square of the Euclidean distance.

[0053] Our goal is to solve an optimization problem to find for each data point x i (i=1,2,...N) such that the local sparsity of the data point in its relevant subspace is maximized. The proposed RSS algorithm calculates the optimal subspace of the i-th data point as an M-dimensional vector The Mth element Indicates whether the i-th feature is selected The relevant subspace (f i ) to maximize its local sparsity in this subspace. Assume is the lth data point (x l ) in the subspace f i The projection on is expressed as follows: where ⊙ represents two M-dimensional vectors x l and f i Element-wise multiplication between .

[0054] Step S33: input feature dimension for calculation.

[0055] In the process of data collection, outliers are usually caused by various reasons and hidden in different features. As the dimension increases, the data becomes more sparse. Therefore, it is usually preferred to choose a higher-dimensional subspace for outlier identification. This problem can be avoided by limiting the possible subspaces to the exact number of subspaces λ with relevant features. Assume that the M-dimensional vector Each element of can only be 0 or 1, then the objective function is reformulated by adding the following two constraints:

[0056]

[0057]

[0058] in,

[0059] Considering the projection definition of the data point on its relevant subspace in the above formula, and For binary variables m = 1, 2, ... M, the objective function can be restated as follows:

[0060]

[0061] in, shows the M different features along Data point x j and center ci The M-dimensional distance vector between them.

[0062] Local subspace neighborhood set The final sum of the distances between each data point in and the center of the set can be expressed as:

[0063]

[0064] in,

[0065] Step S34, iteratively obtain the optimal subspace.

[0066] Specifies initialization of lspmax i = -∞, t = 1, and from size M: f i Select λ features from the complete feature set. t≤Iter max , given In the case of , calculate the local subspace neighborhood of the i-th data point, and then Calculate the optimal subspace of the i-th data point and its corresponding local sparse value, and update the best subspace found so far: f * ←f i and subspace f *i The local sparsity value of is:

[0067] The optimal subspace calculated based on the feature number and sparsity and the projection of all tobacco leaf near-infrared data samples are as follows: Figure 2 shown.

[0068] Step S4: Calculate abnormal samples according to the obtained subspace.

[0069] Assume that variable X (1) ,X (2) ,...,X (n-1) ,X (n) is the statistic of the sample size n and the population distribution function F(x,θ), is dependent on the variable X (1) ,X (2) ,...,X (k) The point estimate of the population mean u is then is the degree of jump of the mean u at point k. The steps to determine the singular sample are as follows:

[0070] First, sort the scores V in ascending order as (v1<v2<v3<...<v n ), and then use Calculate the jump degree T of each sample k (1≤k≤n-1). If Tk Maximum, then the value is v k+1 ,v k+2 ,...,v n The corresponding samples of are most likely to be outliers. These samples are added to the outlier set U and the remaining samples are added to the sample set N. If the maximum jump degree T k T n-1 , then it will be combined with v n The corresponding samples are added to the data set U, and the jump degree is recalculated with the remaining n-1 samples, and then the singular samples are determined according to the above steps. Finally, we can get an outlier cluster U.

[0071] Embodiment 2:

[0072] See also Figure 1-Figure 2 In this embodiment, tobacco leaves are taken as an example, and a specific implementation method of a near infrared spectrum abnormal sample identification method based on a high-dimensional subspace is as follows:

[0073] Step S1: Collect sample spectral data

[0074] 1000 tobacco leaf samples (all normal samples) and 65 abnormal samples (including 45 tobacco stem samples and 20 abnormal samples due to experimental operation errors and other factors) from different producing areas (Shandong, Sichuan, Yunnan, Guizhou, etc.) were selected, i.e., n=1065, and spectra were collected using a near-infrared spectrometer.

[0075] Step S2: Preprocessing of spectral data

[0076] In order to eliminate the influence of high-frequency noise and baseline drift on the spectrum, multivariate scattering correction (MSC) and Savitzky-Golay convolution smoothing are used to preprocess the spectrum in this example. Table 1 shows the spectral data of 1000 normal samples and 65 abnormal samples after preprocessing.

[0077] Table 1

[0078]

[0079]

[0080] Step S3: Select a sparse subspace based on the processed sample spectrum matrix D

[0081] S31. Determine the k value of the optimal neighborhood

[0082] In order to obtain the optimal neighborhood k, all data points are projected into the given subspace f i Then use the projection distance to find the k nearest neighbors of the ith data point. Construct the matrix M of point P to construct a local neighborhood dimension feature model: And determine the value of k so that E f =-L λ ln(L λ )-P λ ln(P λ )-S λ ln(S λ ) The local entropy function calculated is the smallest. Taking the normal sample 10 as an example, the calculated k optimized =36, k calculated based on the abnormal sample 980 optimized =29. To facilitate the subsequent calculation of local subspace sparseness, the average value of k of all sample points is taken as 32.

[0083] S32. Calculate local sparsity

[0084] According to the formula Determine the data point x i The local sparsity of each sample point in the local neighborhood is shown in Table 2.

[0085] Table 2

[0086] Sample points 1 2 3 4 … 1062 1063 1064 1065 Sparsity 0.236 0.301 0.241 0.681 0.562 0.521 0.489 0.446

[0087] S33. Input feature dimension for calculation

[0088] In order to more accurately extract the characteristic information of high-dimensional near-infrared spectral data, we restrict the possible subspaces to the exact number of subspaces λ with relevant features. Here we select 2500 wavelength points with key spectral information as input and re-calculate the objective function Compute the optimal subspace.

[0089] S34. Iterate to get the optimal subspace

[0090] Repeat the above steps to calculate the local subspace neighborhood of the i-th data point, and then Calculate the optimal subspace of the i-th data point and its corresponding local sparse value, and update the best subspace found so far: f * ←f i and subspace f *i The local sparsity value of is:

[0091] The optimal subspace calculated based on the feature number and sparsity and the projection of all tobacco leaf near-infrared data samples are as follows: Figure 2 As shown:

[0092] Step S4: Calculate abnormal samples based on the obtained subspace

[0093] After the dataset is projected into the global subspace, the jump degree is used to calculate the outliers and obtain the outlier cluster U.

[0094] S41. Sort the anomaly scores V in ascending order as (v1<v2<v3<...<v n ), and then use Calculate the jump degree T of each sample k (1≤k≤n-1).

[0095] S42. If T k Maximum, then the value is v k+1 ,v k+2 ,...,v n The corresponding samples of are most likely to be outliers. These samples are added to the outlier set U and the remaining samples are added to the sample set N.

[0096] S43. If the maximum jump degree T k T n-1 , then it will be combined with v n The corresponding samples are added to the anomaly set U, and the jump degree is recalculated with the remaining n-1 samples, and then the singular samples are determined according to the above steps. The obtained anomaly sample set is shown in Table 3.

[0097] Table 3

[0098]

[0099] S5: Remove the abnormal samples

[0100] According to the calculated abnormal sample set, the identified outliers are removed, and the results are shown in Table 4. The accuracy of abnormal sample identification is 95.3%.

[0101] Table 4

[0102]

[0103] As shown in the table, the method proposed in the present invention can effectively distinguish normal and abnormal tobacco samples in the subspace, and can reduce the impact of abnormal samples on the overall modeling in the analysis. In order to more intuitively and effectively compare the model performance, the PLS model of the total sugar index is established using the data set before and after the singular samples are removed, and the model is predicted using a test set of 300 samples. Table 5 shows the comparison of the model accuracy of the PLS model of the total sugar index before and after the singular samples are removed.

[0104] Table 5

[0105]

[0106] It can be seen from Table 5 that the method proposed in the present invention performs better in identifying outliers. After removing outliers using this method, the accuracy of the model is higher.

[0107] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A near infrared spectrum abnormal sample identification method based on high-dimensional subspace, characterized in that: include: Collecting raw spectral data including normal samples and abnormal samples, and composing a spectral matrix A of a group of samples; Preprocess the original spectral data to obtain a preprocessed spectral matrix D; Select a sparse subspace based on the processed spectral matrix D; The outlier samples are calculated based on the obtained subspace.

2. The near infrared spectrum abnormal sample identification method according to claim 1 is characterized in that: The raw spectral data were preprocessed using multivariate scattering correction and Savitzky-Golay convolution smoothing.

3. The near infrared spectrum abnormal sample identification method according to claim 1 is characterized in that: Let D = {x 1 ,x 2 ,…,x N } is a set of N data points in M-dimensional feature space, where x i is the i-th data point, each data point is represented as a normalized feature M-dimensional vector of; Then select the sparse subspace according to the processed spectral matrix D, including: Determine the k value of the optimal neighborhood of the data point; Calculate the local sparsity of data points; Enter the feature dimension for calculation; Iterate to get the optimal subspace.

4. The near infrared spectrum abnormal sample identification method according to claim 3 is characterized in that: Determine the k value of the optimal neighborhood of the data point, including: First, according to the formula Construct the matrix M of point P; Construct a local neighborhood dimension eigenmode: Determine the value of k so that E f =-L λ ln(L λ )-P λ ln(P λ )-S λ ln(S λ )The local entropy function calculated is the minimum; The optimal neighborhood is obtained by the minimum entropy function criterion: k optimized =argmin(E f ); where k is the number of neighbors of P, is the center of the set of neighboring points, let λ1≥λ2≥λ3 be the three eigenvalues ​​of M, where the corresponding eigenvector of λ3 can be used as an approximation of the normal vector of point P.

5. The near infrared spectrum abnormal sample identification method according to claim 3 is characterized in that: According to the formula Determine the data point x i The local sparsity of . in, is the data point x i The projection of S on the subspace, Represents data point x i The local subspace neighborhood of , which contains x in the subspace S i The k nearest neighbors of Is a collection The center of the symbol ||·|| 2 Represents the square of the Euclidean distance.

6. The near infrared spectrum abnormal sample identification method according to claim 3 is characterized in that: Restricting the possible subspaces to the exact number of subspaces λ with relevant features, reformulating the objective function: Calculate the optimal subspace; in, shows the M different features along Data point x j and center c i The M-dimensional distance vector between them.

7. The near infrared spectrum abnormal sample identification method according to claim 3 is characterized in that: Iterate to get the optimal subspace, specifically: specify initialization lspmax i = -∞, t = 1, and from size M: f i Select λ features from the complete feature set; when t≤Iter max , given In the case of , calculate the local subspace neighborhood of the i-th data point, and then Calculate the optimal subspace of the i-th data point and its corresponding local sparse value, and update the best subspace found so far: f * ←f i and subspace f *i The local sparsity value of is:

8. The near infrared spectrum abnormal sample identification method according to claim 1 is characterized in that: Calculate the abnormal samples according to the obtained subspace, including sorting the abnormal scores V in ascending order as (v1<v2<v3<…<v n ), and then use Calculate the jump degree of each sample: T k (1≤k≤n-1); If T k Maximum, then the value is v k+1 ,v k+2 ,…,v n The corresponding samples of are most likely to be outliers, and these samples are added to the outlier set U, and the remaining samples are added to the sample set N; If the maximum jump degree T k T n-1 , then it will be combined with v n The corresponding samples are added to the anomaly set U, and the jump degree is recalculated with the remaining n-1 samples, and then the singular samples are determined according to the above steps.

Images