A two-layer progressive-based meteorological equipment anomaly detection method

Abstract

The application relates to a two-layer progressive-based meteorological equipment anomaly detection method and belongs to the meteorological equipment detection field. First, a feature selection method is used to process high-dimensional data, irrelevant data for anomaly detection is filtered out, and the purpose of dimension reduction is achieved. Secondly, an improved isolated forest algorithm is used to quickly scan a data set, based on the result after application, a pruning threshold method based on data anomaly degree is designed according to the discrete condition of the anomaly score, the obviously normal data is subtracted, and an abnormal candidate set is generated. Through local anomaly factor, more accurate anomaly detection is carried out, and the accuracy requirement can be better met. The application is improved on the basis of the traditional isolated forest, the influence of the branch operation standard of the binary tree on the detection accuracy of the local anomaly point is solved, the local anomaly factor and the pruning operation are introduced, the time complexity is reduced on the basis of improving the detection accuracy, and both requirements are considered.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of meteorological equipment detection, and particularly relates to a meteorological equipment anomaly detection method. BACKGROUND

[0002] Currently, most of the meteorological equipment anomaly detection models are constructed in the following two ways: a mechanism-based construction method and a data-driven method. The mechanism-based construction method usually requires a deep understanding of the physical principles and behaviors of the system. This may be a challenge for complex systems, requiring specialized domain knowledge and professional skills. At the same time, it has high requirements for a large amount of training data and comprehensive parameter range. If the data is insufficient or the parameter range is too extensive, the performance of the model may be limited. The data-driven method does not require simulation modeling of the mechanism of meteorological equipment.

[0003] Meteorological equipment includes precipitation instruments, gravity waves, cloud meters, etc. The full name of the precipitation instrument is a three-dimensional imaging measurement instrument for precipitation particles. The system composition of the precipitation instrument includes an optical unit, an imaging unit, a collection and control unit, a data processing unit, and a whole machine protection unit. The imaging unit takes the CMOS image sensor of IMX304 type as the core, and its function is to expose the precipitation particles passing through the sampling area quickly. According to the working principle of the precipitation instrument, whether the equipment can work normally is closely related to the working state of the precipitation instrument pulse light source and the CMOS image sensor. Therefore, analyzing the fault health condition of the precipitation instrument pulse light source and the CMOS image sensor can become a direct basis for judging the fault health condition of the precipitation instrument.

[0004] The general anomaly detection algorithm mainly detects anomalies through the extracted electrical signals. The algorithm with high detection efficiency is still inaccurate for local anomaly point detection. Considering the distribution of local anomaly points will produce large time complexity, which cannot meet the needs of accuracy and real-time. SUMMARY

[0005] The technical problem to be solved by the present application is:

[0006] In order to avoid the shortcomings of the prior art, the present application provides a two-layer progressive meteorological equipment anomaly detection method, which can effectively detect the anomaly of meteorological equipment data, and at the same time meet the needs of efficiency and real-time while having high detection accuracy, and finally verify it on the precipitation instrument meteorological equipment.

[0007] In order to solve the above technical problems, the technical scheme adopted by the present application is:

[0008] A two-layer progressive meteorological equipment anomaly detection method, characterized in that it comprises:

[0009] characteristic data selected to measure whether the meteorological equipment is faulty is filtered out from the characteristic vector irrelevant to the anomaly;

[0010] The meteorological equipment characteristic data is standardized;

[0011] The improved isolated forest algorithm is used to construct multiple effective isolated trees for the standardized meteorological equipment data, form an isolated forest, and calculate an anomaly score value;

[0012] According to the obtained anomaly score, the dispersion degree of the data is calculated, the obviously normal data is pruned, and an anomaly detection candidate is obtained;

[0013] The local reachable density and local anomaly factor of the anomaly detection candidate are calculated to judge the relative anomaly degree of the data point, and an anomaly score is obtained;

[0014] According to the anomaly score, whether the meteorological equipment is faulty is judged.

[0015] The further technical scheme of the present application is that the characteristic selection of the data by the variance filtering method is specifically:

[0016] The multi-dimensional meteorological equipment characteristic data is obtained;

[0017] The variance of each characteristic is calculated, and the data lower than the set threshold is removed.

[0018] The further technical scheme of the present application is that the standardization processing of the meteorological equipment characteristic data is specifically:

[0019] The minimum value and the maximum value in the data are calculated;

[0020] The data is standardized by using a normalization formula, so that the standardized data conforms to the normal distribution between 0 and 1, and the order of magnitude difference between samples is eliminated.

[0021] The further technical scheme of the present application is that the method for constructing multiple effective isolated trees for the standardized meteorological equipment data by using the improved isolated forest algorithm to form an isolated forest and calculating an anomaly score value is specifically:

[0022] The to-be-detected data F dec is added to the multiple effective isolated tree set F i =(F1,F2,...F k ) to obtain an isolated forest F iForest =(F1,F2,...F k ,F dec ); F iForest is selected as training data, and FiForest The maximum value F in the set max , minimum value F min ;

[0023] Select the cutting point x∈(F min ,F max ); the cutting point satisfies The left set is not satisfied with the right set, where p is the intercept and n is the slope; let the number of left sets of the first split point be NUM left , the number of sets on the right is NUM right ,So If τ∈[1,10], the tree is considered to be an inferior tree, so it is skipped and the next tree is constructed recursively until 100 trees are constructed. The termination condition of the tree growth is: there is only one data in each node of the tree or the maximum growth height of the tree is reached. The maximum height limit of the tree h is the number of subsamples. related:

[0024] Calculate the average path length C(n′) of the constructed binary tree, where n′ is the number of samples when constructing the isolation tree;

[0025] Assume that the data to be detected is F dec The average height of each tree in the isolation forest is E(h(x)); then the data to be tested F dec The anomaly score is Where x = F dec .

[0026] A further technical solution of the present invention is as follows: the method of calculating the degree of dispersion of the data based on the obtained anomaly score, pruning obviously normal data, and obtaining anomaly detection candidates is specifically as follows:

[0027] Obtain a dataset D = {x1, x2, ..., x i ,…,x n}, where n is the number of samples in D, x i are the data values; the coefficient of dispersion for this attribute is defined as: in is the mean of D, fd i Used to measure the degree of dispersion;

[0028] Calculate the pruning threshold θ of the data set by the discrete coefficient D ,θ D Used to represent the proportion of outliers in the data set:

[0029]

[0030] Where Top m refers to the data point with the largest anomaly score after sorting, m and a are adjustment factors, which depend on the size and distribution of the data set, D f is the anomaly coefficient vector of the data set;

[0031] Data sets less than the threshold value θ D are considered normal data, and greater than the threshold value θ D are considered as a candidate set of abnormal data.

[0032] The further technical solution of the present application is that the local reachable density and the local anomaly factor of the abnormal detection candidate are calculated to judge the relative anomaly degree of the data point, and the method for obtaining the anomaly score is specifically:

[0033] Set point p as the center point, calculate the distance between point p and other data points q Calculate the k-distance of point p, that is, sort the distance from point p to other data points, and record the distance from point p to the kth

[0034] The distance between data points is recorded as k-distance.

[0035] Calculate the k-distance neighborhood of point p, that is, given the k-distance of p, the k-distance neighborhood of p contains each object whose distance to p is not greater than the k-distance; these objects q are called the k-nearest neighbors of p, which is expressed as

[0036] N k-distance(p) (p)={q∈D\{p}|d(p,q)≤k-distance(p)};

[0037] Calculate the reachable distance of point p, let k be a natural number, and the reachable distance of object p relative to object o is defined as

[0038] reach-disk k (p,o)=max{k-distance(o),d(p,o)};

[0039] Calculate the local reachability density of point p, the local reachability density of object p is based on the reciprocal of the average reachability distance of the MinPtst nearest neighbors of p

[0040]

[0041] Calculate the anomaly score value

[0042] The further technical solution of the present application is that the local reachable density and the local anomaly factor of the abnormal detection candidate are calculated to judge the relative anomaly degree of the data point, and the method for obtaining the anomaly score is specifically:

[0043] ​The abnormal score value is compared with a threshold value: when the abnormal score value is greater than the threshold value, it is determined that the meteorological equipment is abnormal, that is, the equipment is malfunctioning; when the abnormal score value is less than or equal to the threshold value, it is determined that the meteorological equipment is normal, that is, the equipment is in good condition.

[0044] A computer system includes one or more processors, a computer readable storage medium storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described above.

[0045] A computer readable storage medium stores computer executable instructions, which when executed implement the method described above.

[0046] The beneficial effects of the present application are:

[0047] The meteorological equipment anomaly detection method based on two-layer progressive provided by the present application introduces slope and intercept in the traditional isolated forest algorithm, cuts the data in the random direction, eliminates the possibility of incorrect anomaly detection results caused by cutting the data in the direction parallel to the coordinate axis in the traditional isolated forest algorithm, introduces the density between each data point and its adjacent point to determine the abnormal value, eliminates the local detection disadvantage of the tree structure algorithm, and strengthens the accuracy of local detection of anomaly detection. Meanwhile, the code is designed to prune the data by judging the dispersion degree, form the abnormal data candidate, simplify the anomaly detection process, improve the efficiency of algorithm operation, and eliminate the problem of too high time complexity caused by local optimization. BRIEF DESCRIPTION OF DRAWINGS

[0048] The accompanying drawings are included to provide a further understanding of the application, and are incorporated in and constitute a part of this specification, illustrate embodiments of the application, and together with the description serve to explain the principles of the application. In the drawings:

[0049] Figure 1 The meteorological equipment anomaly detection method flow chart. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0051] The application provides a two-layer progressive meteorological equipment anomaly detection method, which is improved on the basis of a traditional isolated forest, solves the influence of branch operation standards of a binary tree on the accuracy of local anomaly point detection, introduces a local anomaly factor and a pruning operation, reduces time complexity on the basis of improving detection accuracy, and meets both requirements. First, a feature selection method is used to process high-dimensional data, filter out data irrelevant to anomaly detection, and achieve dimension reduction. Second, an improved isolated forest algorithm is used to quickly scan a data set, based on the application result, a pruning threshold method based on data anomaly degree is designed according to the dispersion of the anomaly score, the obviously normal data is subtracted, and an anomaly candidate set is generated. The local anomaly factor is used for more accurate anomaly detection, so that the accuracy requirement can be met. Finally, the application takes a precipitation gauge as an example, tests the precipitation gauge in different low-temperature environments, and verifies that the model has certain advancement and effectiveness.

[0052] As Figure 1 shown, the application specifically includes the following steps:

[0053] Step S1: feature selection is performed on data, and irrelevant feature vectors are filtered out from CMOS image sensor data of a precipitation gauge meteorological equipment:

[0054] Step S1.1: feature selection is performed on the multi-dimensional X i =(x1,x2,…x m ) of the electric signal data of the meteorological equipment. First, a filtering threshold threshold is specified.

[0055] Step S1.2: the variance of each feature is calculated. Score represents the variance value of each feature, that is, wherein represents the average value of the square of the difference between the data value and the average value.

[0056] Step S1.3: for the variance value of each feature, if the variance is less than the set threshold, it is considered that the value change of the feature is small, and there is no potential failure, and the feature is judged as a low-variance feature.

[0057] Step S1.4: all data columns marked as low-variance features are removed.

[0058] Step S2: the CMOS feature data of the precipitation gauge meteorological equipment is subjected to standardization processing:

[0059] Step S2.1: the minimum value x min in the data and the maximum value x max in the data are calculated.

[0060] Step S2.2: the formula The data is standardized so that the standardized data obeys normal distribution between 0 and 1, and the order of magnitude difference between samples is eliminated.

[0061] Step S3: using the improved isolation forest algorithm to construct multiple effective isolation trees for the standardized CMOS data, form an isolation forest, and calculate the anomaly score value:

[0062] Step S3.1: add the data to be detected to the multiple effective isolation tree set F i =(F1,F2,...F k ) to obtain the isolation forest F iForest =(F1,F2,...F k ,F dec ). Select F iForest as the training data, determine the maximum value F iForest and the minimum value F max in the F min set.

[0063] Step S3.2: select a cutting point x∈(F min ,F max ). Put the cutting point satisfying as the left node (constitute the left set), and put the cutting point not satisfying as the right node (constitute the right set), where p is the intercept and n is the slope. Let the number of left set of the first split point be NUM left , and the number of right set be NUM right , then If τ∈[1,10], it is considered that the tree is a disadvantageous tree, then skip the tree, continue to construct the next tree in a recursive manner, until 100 trees are constructed. The termination condition of tree growth is that each node of the tree has only one data or reaches the maximum growth height of the tree. The maximum height limit h of the tree is related to the number of sub-samples :

[0064] Step S3.3: calculate the average path length of the constructed binary tree where n is the number of samples when constructing the isolation tree, and H(n-1)=ln(n-1)+0.5772156649.

[0065] Step S3.4: let the detected electric signal data F dec be the average height E(h(x)) on each tree in the isolation forest. Then the anomaly score value of the detected electric signal data F dec is where x=F dec .

[0066] Step S4: According to the obtained anomaly score, the dispersion degree of the CMOS data of the precipitation weather equipment is calculated, the obviously normal data is pruned, and the anomaly detection candidate is obtained.

[0067] Step S4.1: Obtain a data set D = {x1, x2, …, xn} composed of anomaly scores, where n is the sample number of D, and x is the data value. The dispersion coefficient of this attribute is defined as: i ,…,x n} where n is the sample number of D, and x is the data value. The dispersion coefficient of this attribute is defined as: i is the mean of D, and fd i is used to measure the dispersion degree.

[0068] Step S4.2: The pruning threshold θ D of the data set can be calculated by the dispersion coefficient, and θ D can represent the proportion of abnormal values in the data set, which is specifically expressed as:

[0069]

[0070] where Top_m refers to the maximum anomaly score after sorting, and m and a are adjustment factors depending on the size and distribution of the data set. f D

[0071] Step S4.3: The data set smaller than the threshold θ D is regarded as normal data, and the data set greater than the threshold θ D is regarded as the candidate set of abnormal data.

[0072] Step S5: The local reachable density and local anomaly factor of the anomaly detection candidate are calculated to judge the relative abnormal degree of the data point, and the anomaly score is obtained:

[0073] Step S5.1: Set point p as the center point, and calculate the distance d(p, q) between point p and other data points q.

[0074] Step S5.2: Calculate the k-distance of point p, i.e. sort the distance from point p to other data points, and record the distance from point p to the kth data point as k-distance.

[0075] Step S5.3: Calculate the k-distance neighborhood of point p, i.e. given the k-distance of p, the k-distance neighborhood of p contains each object whose distance to p is not greater than the k-distance. These objects q are called the k-neighbors of p, which is expressed as: k-distance(p) N(p) = {q ∈ D \ {p} | d(p, q) ≤ k-distance(p)}.

[0076] ​Step S5.4: calculating the reach distance of point p, let k be a natural number, the reachability distance of object p relative to object o is defined as reach-disk k (p, o) = max{k-distance(o), d(p, o)}.

[0077] Step S5.5: calculating the local reachability density of point p, the local reachability density of object p is based on the reciprocal of the average reachability distance of the nearest neighbors of p's MinPtst

[0078] Step S5.6: calculating the anomaly score value

[0079] Step S6: judging whether the precipitation weather equipment CMOS image sensor data is abnormal, i.e. whether the precipitation weather equipment is in failure:

[0080] Step S6.1: determining the score value LOF for judging the abnormality of the weather equipment data MinPts (p): when LOF MinPts (p) > S(LOF), it is determined that the weather equipment is abnormal, i.e. the equipment is in failure; when LOF MinPts (p) ≤ S(LOF), it is determined that the weather equipment is normal, i.e. the equipment is in good condition. Wherein, S(LOF) is the threshold value of the relative anomaly degree of the local outlier factor.

[0081] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited to this, any skilled person in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application.

Claims

1. A two-layer progressive meteorological equipment anomaly detection method, characterized in that: include: The variance filtering method is used to select features of meteorological equipment data, and the feature vectors irrelevant to the anomaly are filtered out from the feature data selected to measure whether the meteorological equipment has failed. Standardize the characteristic data of meteorological equipment; The improved isolation forest algorithm is used to construct multiple effective isolation trees on the standardized meteorological equipment data to form an isolation forest and calculate the anomaly score value; According to the obtained anomaly score, the degree of dispersion of the data is calculated, and the obviously normal data is trimmed to obtain the anomaly detection candidates; Calculate the local reachable density and local anomaly factor of the anomaly detection candidate to determine the relative degree of anomaly of the data point and obtain the anomaly score; Determine whether meteorological equipment has failed based on the anomaly score; The improved isolation forest algorithm is used to construct multiple effective isolation trees on the standardized meteorological equipment data to form an isolation forest. The calculation of the abnormality score value is as follows: Join to multiple valid isolated tree sets Isolation Forest ;choose As training data, determine Maximum value in a set , minimum ;Select cutting point ; The cutting point satisfies constitute the left set, but do not satisfy the right set, where is the intercept, is the slope; let the number of sets on the left side of the first split point be , the number of sets on the right is ,So ;if , then the tree is considered to be an inferior tree, so it is skipped and the next tree is constructed recursively until 100 trees are constructed; The method of calculating the degree of dispersion of the data based on the obtained anomaly scores and pruning the obviously normal data to obtain anomaly detection candidates is as follows: obtaining a data set consisting of data corresponding to the anomaly scores ,in n yes D The number of samples, are the data values; the coefficient of dispersion for this attribute is defined as: ;in is the mean of D, Used to measure the degree of dispersion; calculate the pruning threshold of the data set through the dispersion coefficient , Used to represent the proportion of outliers in the data set: ,in, refers to the one with the largest anomaly score after sorting, m and is an adjustment factor that depends on the size and distribution of the dataset, is the outlier coefficient vector of the data set; The data set is considered normal data and is greater than the threshold Candidate sets considered as abnormal data.

2. A two-layer progressive meteorological equipment anomaly detection method according to claim 1, characterized in that: The method of using variance filtering to select features of data is specifically as follows: Obtain multi-dimensional meteorological equipment characteristic data; Calculate the variance of each feature, compare it with the set threshold, and remove data below the set threshold.

3. The method for detecting abnormalities of meteorological equipment based on a two-layer progressive approach according to claim 2, characterized in that: The standardization process of the meteorological equipment characteristic data is specifically as follows: Calculate the minimum and maximum values ​​in the data; The normalization formula is used to standardize the data so that the standardized data obeys the normal distribution between 0 and 1, eliminating the order of magnitude differences between samples.

4. The method for detecting abnormalities in meteorological equipment based on a two-layer progressive approach according to claim 1, characterized in that: The method of using the improved isolation forest algorithm to construct multiple valid isolation trees on the standardized meteorological equipment data to form an isolation forest and calculating the anomaly score value also includes: The termination condition of tree growth is: each node of the tree has only one data or the maximum growth height of the tree is reached; the maximum height limit of the tree is and the number of subsamples related: ; Calculate the average path length of the constructed binary tree ,in is the number of samples when constructing the isolation tree; Assume the data to be tested The average height of each tree in an isolated forest is ; Then the data to be tested The anomaly score is , where .

5. The method for detecting abnormality of meteorological equipment based on a two-layer progressive approach according to claim 1, characterized in that: The method of calculating the local reachable density and local anomaly factor of the anomaly detection candidate to determine the relative degree of anomaly of the data point and obtaining the anomaly score is as follows: Set Point p As the center point, calculate the point p To other data points q The distance between ; Calculation Point p of k Distance, that is, from point p Sort by distance to other data points and sort from point p To k The distance between data points is recorded as k-distance; Calculation Point p of k Distance neighborhood, that is, given p of k distance, p of k The distance neighborhood contains it and p The distance is no greater than k distance to each object; these objects q Known as p of k Nearest neighbor, the formula is expressed as ; Calculation Point p The reachable distance is k is a natural number, object p Relative to the object o The reachability distance is defined as ; Calculation Point p The local reachability density of the object p The local reachability density is based on p MinPtst is the inverse of the average reachability distance to the nearest neighbor ; Calculate the anomaly score value .

6. A two-layer progressive meteorological equipment anomaly detection method according to claim 1, characterized in that: The method of judging whether a meteorological device fails according to the anomaly score is specifically as follows: The anomaly score value is compared with the threshold: when the anomaly score value is greater than the threshold, it is determined that the meteorological equipment has an anomaly, that is, the equipment has failed; when the anomaly score value is less than or equal to the threshold, it is determined that the meteorological equipment is normal, that is, the equipment is in good condition.

7. A computer system, characterized in that include: One or more processors, a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that Computer-executable instructions are stored, and when the instructions are executed, they are used to implement the method according to any one of claims 1 to 6.

Images