Anomaly detection method based on Manhattan distance and tensor recovery

Abstract

The invention discloses an anomaly detection method based on Manhattan distance and tensor recovery, and the method comprises the steps: firstly representing input data in a three-order tensor form through tensor, and maintaining a multi-dimensional structure of the data; then, a tensor robust principal component analysis decomposition model is used for separating the tensor into a low-rank part and a sparse part, the low-rank part represents background information, and the sparse part reflects abnormal features; in order to optimize the decomposition process, an augmented Lagrangian function and t-product operation are introduced, and efficient decomposition is achieved. And the separated sparse part determines an abnormal point by calculating a Manhattan distance and a K nearest neighbor distance and combining a set threshold value. The method solves the problems that a traditional anomaly detection method is insufficient in support for high-dimensional and structured data, sensitive to noise and low in detection precision, can efficiently and accurately recognize abnormal points in a complex background, and has high robustness and applicability. The method can be widely applied to the fields of hyperspectral data analysis, image processing, industrial quality detection and network security.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an anomaly detection method based on Manhattan distance and tensor recovery. Background Art

[0002] With the continuous advancement of information technology and data collection capabilities, anomaly detection has gradually become a key technical means in many fields, such as financial fraud detection, industrial quality control, network intrusion detection, and medical image analysis. However, due to the rapid growth of data size and complexity, traditional anomaly detection methods have gradually exposed their shortcomings when dealing with high-dimensional, nonlinear, and structured data. How to efficiently and accurately detect anomalies in complex backgrounds is an important issue that needs to be solved in the current anomaly detection field.

[0003] Anomaly detection methods based on matrix decomposition Traditional anomaly detection methods are usually based on matrix decomposition techniques, such as principal component analysis (PCA) or low-rank matrix decomposition. These methods assume that the data can be decomposed into a low-rank background part and a sparse anomaly part, and use low-rank constraints to model the data. However, these methods face two main problems:

[0004] Data structure loss problem: Many real data have high-order structures (such as multi-dimensional tensor forms in images, videos, and hyperspectral data). When using PCA or low-rank matrix decomposition, the data often needs to be expanded into a matrix form, which destroys the original multi-dimensional spatial association of the data. This dimensionality reduction operation may lose important information and reduce the accuracy of anomaly detection.

[0005] Insufficient robustness to noise: Traditional matrix decomposition methods are sensitive to noise and outliers, especially when there is a large amount of noise or outliers in the data, which can easily lead to unsatisfactory separation of background and abnormal parts, thus affecting detection performance.

[0006] Another commonly used anomaly detection method is based on distance models (such as Euclidean distance or Mahalanobis distance). These methods use the distance relationship between samples to determine anomalies. The shortcomings of this type of method are mainly reflected in the following aspects:

[0007] Limitations of distance metrics: Euclidean distance is prone to failure in high-dimensional data (i.e., the "curse of dimensionality" problem), and is less sensitive to sparse data or non-uniformly distributed data, and cannot effectively capture the characteristics of outliers.

[0008] High computational complexity: As the size of data increases, the computational complexity of distance-based methods increases rapidly, especially when processing large-scale data sets, which makes it difficult to meet the real-time requirements of practical applications.

[0009] Insufficiency of traditional tensor decomposition methods Tensor decomposition methods have been widely used in recent years to process high-dimensional data, but traditional tensor decomposition methods (such as CANDECOMP / PARAFAC decomposition and Tucker decomposition) still have some significant shortcomings:

[0010] High algorithmic complexity: These methods usually require complex iterative optimization, and their computational cost grows exponentially with the increase of tensor dimension and size.

[0011] Limited ability to detect outliers: The goal of traditional tensor decomposition methods is mainly to reduce dimensions and compress data. They have weak modeling capabilities for sparse anomalies and find it difficult to effectively separate low-rank background from anomalies.

[0012] Manual rule method: Some anomaly detection tasks rely on manually designed rule sets. Although they can be deployed quickly, the rule design often lacks versatility and robustness, and is difficult to cope with complex or unknown scenarios.

[0013] Deep learning methods: In recent years, deep learning methods have shown certain advantages in anomaly detection, especially in anomaly detection of image and video data. However, such methods usually require a large amount of labeled data for model training, which is costly and time-consuming. In addition, the "black box" characteristics of deep learning models also make them insufficient in terms of result interpretability.

[0014] In general, the existing technologies have the following problems in anomaly detection:

[0015] Limited support for multi-dimensional structured data: Many methods cannot directly process data in the form of multi-dimensional tensors and need to reduce the dimensionality before processing, resulting in loss of data structure information.

[0016] Insufficient robustness and reliability: In scenarios with high noise or unclear anomalies, the detection performance of existing methods decreases significantly.

[0017] Computational efficiency issues: As data dimensions and scale increase, the computational complexity of many methods increases rapidly and cannot meet the needs of real-time applications.

[0018] Lack of versatility: Some methods rely on prior knowledge in specific fields and are difficult to apply to anomaly detection tasks in multiple scenarios and types.

[0019] To this end, we urgently need to design an anomaly detection method based on Manhattan distance and tensor recovery to solve the above problems. Summary of the invention

[0020] The purpose of the present invention is to solve the prior art problems raised in the above background technology and to provide an anomaly detection method based on Manhattan distance and tensor recovery.

[0021] The technical solution of the present invention to achieve the above purpose is:

[0022] An anomaly detection method based on Manhattan distance and tensor recovery, the method comprising the following steps:

[0023] Data tensorization: Input data Represented as a third-order tensor in:

[0024] represents the input tensor;

[0025] represents the set of real numbers;

[0026] I 1 ,I 2 ,I 3 Represents the size of the first, second, and third dimensions of the tensor respectively;

[0027] Tensor robust principal component analysis: Separate the low-rank part of the tensor by solving the following optimization problem And the sparse part S:

[0028]

[0029] in: Representing a tensor The nuclear norm of Representing a tensor 1 norm of ; λ is the regularization parameter;

[0030] In order to optimize the above problems, the augmented Lagrangian function is introduced:

[0031]

[0032] Where: Y is the Lagrange multiplier; μ is the step size parameter; ∥·∥ F represents the F-norm, which is the square root of the sum of the squares of all elements of the tensor; <·,·〉 represents the inner product of the tensor;

[0033] Calculation of Manhattan distance d(x,y): Calculate the Manhattan distance for each location point in the sparse part S;

[0034] K nearest neighbor detection: Calculate the K nearest neighbor distance of each point based on Manhattan distance, and set the threshold θ to determine the abnormal point.

[0035] As a preferred technical solution of the present invention, the nuclear norm of the tensor The definition is as follows:

[0036]

[0037] Where: L(i) For tensors The i-th front slice matrix obtained by Fourier transform under mode 3; ∥L (i) ∥ * Represents the matrix L (i) The nuclear norm of I is the sum of its singular values; 3 represents the size of the tensor modulus 3, and, since it is not enough to use the nuclear norm of the matrix to approximate the rank function of the matrix in practical problems, we use the weighted schatten-p norm of the tensor

[0038] in Express Perform Fourier transform, express The jth singular value of j represents the jth component of the weight vector w, and this method is called enhanced robust principal component analysis;

[0039] At the same time, we can improve the original TRPCA method by introducing a dictionary:

[0040]

[0041] The dictionary The construction method is

[0042]

[0043]

[0044] in

[0045] As a preferred technical solution of the present invention, the calculation formula of the Manhattan distance d(x, y) is:

[0046]

[0047] in: are m-dimensional vectors; x j ,y j Respectively represent the components of vector x and y in the jth dimension; |x j -y j | represents the absolute difference between two components; m represents the dimension of the vector.

[0048] As a preferred technical solution of the present invention, the K nearest neighbor distance d K (x) is defined as follows:

[0049]

[0050] Among them: x is the point to be detected; x (j) represents the jth sample closest to the Manhattan distance of point x; K is the neighborhood size.

[0051] As a preferred technical solution of the present invention, the tensor nuclear norm calculation in the tensor robust principal component analysis decomposition adopts the t-product operation, which is defined as follows:

[0052] For two rank 3 tensors and The t-product calculation formula is:

[0053]

[0054] Among them: and Performing discrete Fourier transform (DFT) in mode 3 gives:

[0055]

[0056] Perform element-wise matrix multiplication of the corresponding front-slice matrix in the Fourier domain:

[0057]

[0058] Perform an inverse Fourier transform on the result to get the final result:

[0059]

[0060] in They are the tensors after Fourier transformation respectively; · represents matrix multiplication; * represents t-product operation.

[0061] As a preferred technical solution of the present invention, the low-rank part The solution to the subproblem is:

[0062]

[0063] The solution is:

[0064]

[0065] Where: U∑V T is the matrix singular value decomposition; is the singular value threshold shrinkage operator; is the threshold parameter.

[0066] As a preferred technical solution of the present invention, the sub-problem solution of the sparse part S is:

[0067]

[0068] The solution is:

[0069]

[0070] in: represents the soft threshold operator; is the threshold parameter.

[0071] As a preferred technical solution of the present invention, the detection formula of the outlier point is:

[0072]

[0073] Where: θ represents the detection threshold.

[0074] As a preferred technical solution of the present invention, the detection performance is measured by AUC value, precision, recall and F1 value, and the formula is as follows:

[0075] Accuracy:

[0076] Recall:

[0077] F1 value:

[0078] AUC value: the area enclosed by the curve drawn with TPR as the ordinate and FPR as the abscissa and the horizontal axis

[0079] Among them: TP is a true anomaly number; TN is a true constant; FP is a false anomaly number; FN is a false normal number,

[0080] Compared with the prior art, the present invention has the following beneficial effects:

[0081] This technical solution successfully solves the problem of performance degradation of traditional principal component analysis methods due to noise and outliers when processing multi-dimensional tensor data by introducing tensor robust principal component analysis (TRPCA). Compared with the traditional method that requires reconstructing tensors into matrices, this solution directly operates on third-order tensors, using the tensor nuclear norm and the separation ability of sparse components to effectively retain the structured information of the data and improve the detection accuracy of outliers. In the hyperspectral data and PCB anomaly detection experiments, this method can accurately locate the abnormal area, reflecting its wide applicability in multiple fields.

[0082] By calculating the Manhattan distance and K nearest neighbor distance in the sparse part, this scheme can distinguish between noise and real anomalies. Compared with the traditional detection method based on Euclidean distance, Manhattan distance is more sensitive to the value changes of sparse matrices, and the K nearest neighbor method further reduces the misjudgment caused by noise. This method is particularly suitable for data scenarios with small abnormal areas and large noise impact, such as tiny defects in PCB images. Experimental results show that the method has high accuracy and F1 value.

[0083] This scheme constructs an efficient optimization algorithm by introducing augmented Lagrangian functions and t-product operations, so that the decomposition of low-rank and sparse parts of tensors can be quickly implemented on multi-dimensional data. In addition, this method provides a strong theoretical recovery guarantee, that is, under the conditions of satisfying the tensor nuclear norm and sparsity constraints, it can accurately separate low-rank background and sparse abnormal areas with high probability. This theoretical support provides a solid foundation for the reliability of the method and ensures its robustness in complex application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0085] Figure 1 It is a demonstration of TRPCA effect;

[0086] Figure 2 It is a PCB image display;

[0087] Figure 3 For hyperspectral data display;

[0088] Figure 4 The anomaly location effects of the three models in hyperspectral data;

[0089] Figure 5 The abnormal location effects of the three models in PCB data;

[0090] Figure 6 is the AUC value of each type of PCB in the Missinghole anomaly;

[0091] Figure 7 is the recall rate of each type of PCB in the Missinghole anomaly;

[0092] Figure 8 It is the accuracy of each type of PCB in Missinghole anomaly;

[0093] Fig. 9 It is the F1 value of each type of PCB in the Missinghole anomaly. DETAILED DESCRIPTION

[0094] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0095] The following combination Figure 1-Figure 8 The specific embodiments of the present invention are described in detail with reference to the accompanying drawings and examples.

[0096] Example 1: The implementation scheme proposed in Example 1 of the present invention is an anomaly detection method based on Manhattan distance and tensor recovery. The technical process is described below in conjunction with the accompanying drawings. Figure 1 As shown in (a), the input data x is expressed as follows after tensorization: Where: I 1 ,I 2 are the number of rows and columns of data respectively; I 3 is the modal dimension of the data (such as the number of bands or channels).

[0097] For example, for color images I 1 Indicates the number of rows of the image, I 2 Indicates the number of columns, I 3 =3 corresponds to RGB channels.

[0098] TRPCA model decomposition: Figure 1 As shown, the input data is decomposed into low-rank parts using tensor robust principal component analysis (TRPCA) and the sparse part S.

[0099] The optimization objective function is:

[0100]

[0101] Figure 1 In (b), the low-rank part The background part corresponding to the input data, such as the background sky;

[0102] Sparse part corresponds to abnormal regions, such as the foreground. In actual implementation, the tensor nuclear norm The calculation formula is:

[0103]

[0104] Where L (i) Represents the slice matrix obtained by Fourier transform of the tensor in mode 3, ∥L (i) ∥ * Represents the nuclear norm of the matrix.

[0105] The above optimization problem is solved by the augmented Lagrangian method, where the augmented Lagrangian function is defined as:

[0106]

[0107] like Figure 2 (a)-(c) show: Figure 2 (a) is the original PCB image; Figure 2 (b) is the background part after low-rank decomposition; Figure 2 (c) is the abnormal area corresponding to the sparse part.

[0108] Manhattan distance and K nearest neighbor detection: for sparse parts Calculate the Manhattan distance for each location point (i, j):

[0109]

[0110] where x k ,y k Represent the value of the kth dimension respectively.

[0111] like Figure 2 As shown, the sparse value distribution based on Manhattan distance can preliminarily reflect the location of the abnormal area.

[0112] On this basis, the K nearest neighbor distance is used for accurate detection:

[0113]

[0114] where x (j) represents the jth neighbor closest to point x in Manhattan distance.

[0115] Combined with the detection results of the sparse part, the threshold θ is set to determine the abnormal points:

[0116]

[0117] Example 2: Anomaly detection of hyperspectral data, data description using Figure 3 The hyperspectral datasets shown are Sadiego, SalianImpl, and SalianSyn, and the data sizes are respectively Figure 3The three RGB channels of each sub-image are the 37th, 18th, and 8th front slices of the hyperspectral data respectively.

[0118] The results are analyzed as Figure 4 (a)-(d),In (a), we can see that the abnormal part is a small number of sparse areas. The abnormal areas detected by the three methods ( Figure 4 (d)) and the actual GroundTruth ( Figure 4 (b)) is highly consistent; based on the Manhattan distance and K nearest neighbor method, K is taken as 10, and the threshold θ is taken from the sparse minimum value to the maximum value. The average performance indicators of Sadiego data are obtained in the TLRSR method: AUC value: 0.98, precision rate 2.55%, recall rate 25.79%, F1 value 4.23%.

[0119] Example 3: PCB image defect detection, data description: The PCB defect dataset HRIPCBshuju is used. The dataset contains 10 types of PCBs and each type of PCB has 6 defects, such as missing hole, short circuit, mouse bite, etc. The input data is like Figure 5 As shown in (a), there are multiple defect types in the PCB image.

[0120] The decomposition results are as follows Figure 5 As shown in (c)-(e), the sparse part reflects the defect area, and the sparse part of the ETRPCA method is consistent with the actual GroundTruth ( Figure 5 (b) Highly consistent

[0121] Anomaly detection: Based on Manhattan distance and K nearest neighbor method, defect detection of ETRPCA method is performed with K = 10 and threshold θ ranging from the sparse minimum value to the maximum value.

[0122] Result analysis: Figure 5 As shown in Figure 2, the inspection results successfully marked a variety of defects; Figure 6 As shown in the figure, the AUC value of the ETRPCA method is greater than 0.85 on all types of PCBs, which indicates that the model has excellent performance. Figure 7 We can see that the recall rates of the two anomaly detection methods are relatively low in any model, which means that our model does not completely extract the anomalies in the data when extracting anomalies. Figure 8As shown in the figure, the accuracy of both ETRPCA and TRPCA methods is relatively high. The accuracy of the ETRPCA model on various types of PCBs is even above 0.8, which shows that the abnormal points located by our method are relatively accurate. In almost every type of PCB, the accuracy of the EETRPCA method is higher than that of the TRPCA method, which shows that the abnormal area predicted by the ETRPCA method is more accurate than the abnormal area predicted by the TRPCA method. Finally, as Fig. 9 As shown in the figure, from the F1 value, the F1 value of ETRPCA is significantly higher than that of the TRPCA model in almost all types of PCBs, which further shows that the ETRPCA model has better ability to locate anomalies in PCB data than the TRPCA model.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An anomaly detection method based on Manhattan distance and tensor recovery, characterized in that The tensor is separated into a low-rank part and a sparse part, wherein the low-rank part represents background information and the sparse part reflects abnormal features. The method comprises the following steps: Data tensorization: convert the input data Represented as a third-order tensor in: represents the input tensor; represents the set of real numbers; I1, I2, I3 represent the size of the first, second, and third dimensions of the tensor respectively; Tensor robust principal component analysis: Separate the low-rank part of a tensor by solving the following optimization problem And the sparse part S: in: Representing a tensor The nuclear norm of Representing a tensor 1 norm of ; λ is the regularization parameter; In order to optimize the above problems, the augmented Lagrangian function is introduced: Where: Y is the Lagrange multiplier; μ is the step size parameter; ∥·∥ F represents the F-norm, which is the square root of the sum of the squares of all elements of the tensor; <·,·> represents the inner product of the tensor; Calculation of Manhattan distance d(x,y): Calculate the Manhattan distance for each location point in the sparse part S; K nearest neighbor detection: Calculate the K nearest neighbor distance of each point based on Manhattan distance, and set the threshold θ to determine the abnormal point.

2. The anomaly detection method based on Manhattan distance and tensor recovery according to claim 1, characterized in that: The nuclear norm of the tensor The definition is as follows: Where: L (i) For tensors The i-th front slice matrix obtained by Fourier transform under mode 3; ∥L (i) ∥ * Represents the matrix L (i) The nuclear norm of , that is, the sum of its singular values; I3 represents the size of the tensor mode 3, and, since it is not enough to use the nuclear norm of the matrix to approximate the rank function of the matrix in practical problems, we use the weighted schatten-p norm of the tensor in Express Perform Fourier transform, express The jth singular value of j represents the jth component of the weight vector w, and this method is called enhanced robust principal component analysis; At the same time, we can improve the original TRPCA method by introducing a dictionary: The dictionary The construction method is in 3. The anomaly detection method based on Manhattan distance and tensor recovery according to claim 1, characterized in that: The calculation formula of the Manhattan distance d(x,y) is: in: are m-dimensional vectors; x j ,y j Respectively represent the components of vector x and y in the jth dimension; |x j -y j | represents the absolute difference between two components; m represents the dimension of the vector.

4. The anomaly detection method based on Manhattan distance and tensor recovery according to claim 1, characterized in that: The K nearest neighbor distance d K (x) is defined as follows: Among them: x is the point to be detected; x (j) represents the jth sample closest to the Manhattan distance of point x; K is the neighborhood size.

5. The anomaly detection method based on Manhattan distance and tensor recovery according to claim 1, characterized in that: The tensor nuclear norm calculation in the tensor robust principal component analysis decomposition adopts the t-product operation, which is defined as follows: For two rank 3 tensors and The t-product calculation formula is: Among them: and Performing discrete Fourier transform (DFT) in mode 3 gives: Perform element-wise matrix multiplication of the corresponding front-slice matrix in the Fourier domain: Perform an inverse Fourier transform on the result to get the final result: in They are the tensors after Fourier transformation respectively; · represents matrix multiplication; * represents t-product operation.

6. The anomaly detection method based on Manhattan distance and tensor recovery according to claim 1, characterized in that: The low-rank part The solution to the subproblem is: The solution is: Where: UΣV T is the matrix singular value decomposition; is the singular value threshold shrinkage operator; is the threshold parameter.

7. The anomaly detection method based on Manhattan distance and tensor recovery according to claim 1, characterized in that: The subproblem solution of the sparse part S is: The solution is: in: represents the soft threshold operator; is the threshold parameter.

8. The anomaly detection method based on Manhattan distance and tensor recovery according to claim 1, characterized in that: The detection formula for outliers is: Where: θ represents the detection threshold.

9. The anomaly detection method based on Manhattan distance and tensor recovery according to claim 1, characterized in that: The detection performance of the method is measured by AUC value, precision, recall and F1 value, and the formula is as follows: Accuracy: Recall: F1 value: AUC value: the area enclosed by the curve drawn with TPR as the ordinate and FPR as the abscissa. Among them: TP is a true anomaly number; TN is a true constant; FP is a false anomaly number; FN is a false normal number,

Images