Distance-based electromagnetic spectrum monitoring abnormal data detection method

Abstract

The invention relates to the technical field of electromagnetic spectrum monitoring, and particularly discloses a distance-based electromagnetic spectrum monitoring abnormal data detection method, which comprises the following steps of: performing short-time Fourier transform and normalization processing on an acquired original signal to generate an energy density distribution characteristic graph; constructing a multivariate Gaussian distribution model based on non-abnormal historical data; during real-time monitoring, the mahalanobis distance between the collected data and the mean vector of the historical model is calculated after the collected data is preprocessed. And comparing the distance metric value with a preset threshold value to preliminarily judge abnormity, and calculating a distance fluctuation variance through a sliding window mechanism to perform secondary verification. And finally, processing and positioning anomalies by using image morphology, and dividing the degree of anomalies according to the relative deviation between the energy density and the mean value of the historical model. According to the method, the mahalanobis distance and the multivariate Gaussian distribution are introduced, secondary verification and abnormal positioning are combined, the limitation of a traditional method is overcome, the detection accuracy and reliability are effectively improved, the misjudgment and missing judgment rate is reduced, and the method does not depend on a large amount of labeled data and is high in practicability.

Description

Technical Field

[0001] The present invention relates to the technical field of electromagnetic spectrum monitoring, and specifically refers to a method for detecting abnormal data in electromagnetic spectrum monitoring based on distance. Background Art

[0002] In the context of the rapid development of modern communication and electronic technologies, the electromagnetic environment is becoming increasingly complex. As an important strategic resource, the effective monitoring and management of the electromagnetic spectrum are crucial. Accurately detecting abnormal data in the electromagnetic spectrum can timely discover problems such as interference signals and illegal transmissions, ensuring the normal operation of communication systems and information security.

[0003] Currently, existing methods for detecting abnormal data in the electromagnetic spectrum have certain limitations. Some methods based on threshold judgment, due to the dynamic changes in the electromagnetic environment, it is difficult for a fixed threshold to adapt to complex and changeable situations, and false positives and false negatives are likely to occur; although machine learning-based methods have a certain degree of adaptability, they require a large amount of labeled data for training, and the cost of model training and maintenance is relatively high.

[0004] Therefore, a method for detecting abnormal data in electromagnetic spectrum monitoring based on distance has become an urgent problem to be solved. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method for detecting abnormal data in electromagnetic spectrum monitoring based on distance. By introducing Mahalanobis distance metric and multivariate Gaussian distribution model, combined with a secondary verification and anomaly localization mechanism, the accuracy and reliability of electromagnetic spectrum abnormal data detection are improved, the false positive rate and false negative rate are reduced to adapt to the complex and changeable electromagnetic environment.

[0006] To solve the above technical problem, the technical solution provided by the present invention is: A method for detecting abnormal data in electromagnetic spectrum monitoring based on distance, including the following steps:

[0007] S1. Data preprocessing: Perform short-time Fourier transform on the original electromagnetic spectrum signal collected to generate a time-frequency distribution matrix; perform normalization processing on the matrix to obtain an energy density distribution feature map;

[0008] S2. Historical model construction: Based on the historical spectrum data without anomalies, calculate the energy mean vector and covariance matrix of each frequency point within the frequency band, and establish a multivariate Gaussian distribution model;

[0009] S3. Real-time monitoring:

[0010] Perform the same preprocessing process as the historical data on the real-time collected spectrum data to generate a feature matrix to be detected;

[0011] Calculate the Mahalanobis distance between the feature matrix to be detected and the mean vector of the historical model to obtain a distance metric value;

[0012] S4, Abnormality determination:

[0013] Compare the Mahalanobis distance metric value with a preset threshold. If it exceeds the threshold, it is determined as an abnormal signal;

[0014] Perform secondary verification on the abnormal signal. Use the sliding window mechanism to calculate the variance of the distance fluctuation within the window. If the variance exceeds the dynamic threshold, confirm the abnormality;

[0015] S5, Abnormality localization:

[0016] For the spectrum data with confirmed abnormality, extract its two-dimensional frequency-time feature map;

[0017] Adopt the image morphology processing method. Remove the noise interference through erosion and dilation operations, and locate the frequency band coordinates and time interval of the abnormal signal.

[0018] Furthermore, in step S1, the window function of the short-time Fourier transform uses a Hamming window, the window length is 512 sampling points, and the overlap rate is 75%;

[0019] The normalization processing adopts the maximum-minimum normalization method to map the energy density value to the interval [0, 1].

[0020] Furthermore, in step S2, the selection of the historical spectrum data needs to meet the following conditions:

[0021] The data acquisition period is not less than 30 days, and the daily acquisition time period covers the all-weather usage scenario of the target frequency band;

[0022] The proportion of abnormal samples in the historical data does not exceed 0.5%;

[0023] The parameter estimation of the multivariate Gaussian distribution model adopts the maximum likelihood estimation method, and a regularization term is added during the calculation of the covariance matrix to avoid numerical instability.

[0024] Furthermore, in step S3, the Mahalanobis distance is calculated using the following formula:

[0025] where x is the feature vector to be detected, μ is the historical mean vector, and Σ is the covariance matrix;

[0026] The preset threshold is determined by the kernel density estimation method and is set based on the 99.9% quantile of the historical data distance distribution.

[0027] Furthermore, in step S4, the window length of the sliding window is 10 seconds and the step size is 1 second;

[0028] The dynamic threshold is calculated using the exponentially weighted moving average method, and the weight coefficient is 0.2;

[0029] The secondary verification needs to satisfy simultaneously:

[0030] The distance metric value exceeds the static threshold; the variance of the distance fluctuation within the window exceeds the dynamic threshold.

[0031] Furthermore, for the signals confirmed as abnormal, calculate the relative deviation of their energy density from the historical model mean, and the formula is:

[0032]

[0033] Classify the degree of abnormality into three levels according to the deviation value:

[0034] Level 1 abnormality (δ≥50%);

[0035] Level 2 abnormality (20%≤δ<50%);

[0036] Level 3 abnormality (δ<20%).

[0037] The advantages of the present invention compared with the prior art are as follows:

[0038] By introducing the Mahalanobis distance metric and the multivariate Gaussian distribution model, the present invention fully considers the covariance structure of the data and can more accurately describe the distribution characteristics of electromagnetic spectrum data. Compared with the traditional method, the accuracy of anomaly detection is improved.

[0039] The present invention adopts a secondary verification mechanism, combines the sliding window and dynamic threshold calculation, effectively reduces the false positive rate, and enhances the adaptability and reliability of the method in a complex electromagnetic environment.

[0040] The present invention has the functions of anomaly location and anomaly degree classification. It can not only detect abnormal signals in time, but also provide detailed information about the abnormal signals, providing strong support for subsequent interference troubleshooting and spectrum management.

[0041] The present invention does not rely on a large amount of labeled data, the calculation process is relatively simple, reduces the detection cost and calculation complexity, and has high practicability and promotion value. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flowchart of a distance-based abnormal data detection method for electromagnetic spectrum monitoring according to the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0043] The following will refer to the drawings to describe various exemplary embodiments of the present invention in detail. It should be noted that: Unless otherwise specifically stated, the relative arrangements, numerical expressions, and numerical values of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0044] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way a limitation on the present invention, its application, or its use.

[0045] Technologies, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such technologies, methods, and devices should be regarded as part of the specification.

[0046] In all the examples shown and discussed here, any specific values should be construed as merely exemplary and not as a limitation. Thus, other examples of the exemplary embodiments may have different values.

[0047] The following further elaborates on a method for detecting abnormal data in electromagnetic spectrum monitoring based on distance in the present invention with reference to the accompanying drawings.

[0048] Combined with the attached Figure 1 , the present invention is introduced in detail.

[0049] A method for detecting abnormal data in electromagnetic spectrum monitoring based on distance specifically includes the following steps:

[0050] Data preprocessing: Perform short-time Fourier transform on the collected original electromagnetic spectrum signal to generate a time-frequency distribution matrix; perform normalization processing on the matrix to obtain an energy density distribution feature map. Among them, the window function of the short-time Fourier transform uses a Hamming window, the window length is 512 sampling points, and the overlap rate is 75%; the normalization processing uses the maximum-minimum normalization method to map the energy density value to the [0,1] interval. Through the short-time Fourier transform, the time-domain signal can be converted into a time-frequency domain signal, intuitively showing the energy distribution of the signal at different times and frequencies; the normalization processing makes the data have a unified scale, facilitating subsequent processing.

[0051] Historical model construction: Based on the historical spectrum data without anomalies, calculate the energy mean vector and covariance matrix of each frequency point within the frequency band, and establish a multivariate Gaussian distribution model. The selection of historical spectrum data needs to meet: the data collection period is not less than 30 days, and the daily collection period covers the all-weather usage scenario of the target frequency band; the proportion of abnormal samples in the historical data does not exceed 0.5%. The parameter estimation of the multivariate Gaussian distribution model uses the maximum likelihood estimation method, and a regularization term is added when calculating the covariance matrix to avoid numerical instability. Using a large amount of historical data without anomalies to construct the model can accurately reflect the statistical characteristics of normal electromagnetic spectrum signals and provide a reliable reference basis for subsequent anomaly detection.

[0052] Real-time monitoring: Perform the same preprocessing process on the real-time collected spectrum data as on the historical data to generate a feature matrix to be detected; calculate the Mahalanobis distance between the feature matrix to be detected and the mean vector of the historical model to obtain a distance metric value. The Mahalanobis distance calculation uses the formula:

[0053] Among them, \(x\) is the feature vector to be detected, \(\mu\) is the historical mean vector, and \(\sum\) is the covariance matrix; the preset threshold is determined by the kernel density estimation method and set based on the 99.9% quantile of the historical data distance distribution. By calculating the Mahalanobis distance, the degree of difference between the real-time data and the normal data model can be measured. The larger the distance, the more the data deviates from the normal range and the more likely it is abnormal data.

[0054] Abnormality determination: Compare the Mahalanobis distance measurement value with the preset threshold. If it exceeds the threshold, it is determined as an abnormal signal; perform secondary verification on the abnormal signal. Use the sliding window mechanism to calculate the variance of the distance fluctuation within the window. If the variance exceeds the dynamic threshold, the abnormality is confirmed. The window length of the sliding window is 10 seconds and the step size is 1 second; the dynamic threshold is calculated using the exponentially weighted moving average method with a weight coefficient of 0.2; the secondary verification needs to satisfy both: the distance measurement value exceeds the static threshold; the variance of the distance fluctuation within the window exceeds the dynamic threshold. The secondary verification mechanism can effectively exclude misjudgments caused by accidental factors and improve the accuracy of abnormality determination.

[0055] Abnormality localization: For the spectrum data confirmed to be abnormal, extract its two-dimensional frequency-time feature map; use the image morphological processing method to remove noise interference through erosion and dilation operations, and locate the frequency band coordinates and time interval of the abnormal signal. In addition, for the signal confirmed to be abnormal, calculate the relative deviation between its energy density and the historical model mean, and the formula is: The abnormality level is divided into three levels according to the deviation value: level 1 abnormality (\(\delta\geq50\%\)); level 2 abnormality (\(20\%\leq\delta\lt50\%\)); level 3 abnormality (\(\delta\lt20\%\)). Through abnormality localization and abnormality level classification, the characteristics of the abnormal signal can be understood in more detail, providing more valuable information for subsequent processing.

[0056] The specific implementation process of a distance-based electromagnetic spectrum monitoring abnormal data detection method of the present invention is as follows:

[0057] In the electromagnetic spectrum monitoring scenario of a certain communication base station, the method of the present invention is applied to detect abnormal data.

[0058] Data preprocessing: Collect the original electromagnetic spectrum signal around the base station, use the short-time Fourier transform (the window function is the Hamming window, the window length is 512 sampling points, and the overlap rate is 75%) to convert it into a time-frequency distribution matrix, and then normalize the matrix through the maximum-minimum normalization method to obtain the energy density distribution feature map.

[0059] Historical model construction: Collect the electromagnetic spectrum data of the base station in the past 40 days, ensuring that the daily acquisition period covers the all-weather usage scenarios of the target frequency band, and the proportion of abnormal samples is less than 0.5%. Based on this anomaly-free historical data, calculate the energy mean vector and covariance matrix of each frequency point within the frequency band, and use the maximum likelihood estimation method to estimate the parameters of the multivariate Gaussian distribution model. Add a regularization term during the calculation of the covariance matrix to establish a historical model.

[0060] Real-time monitoring: Collect the current spectrum data in real time, and generate the feature matrix to be detected according to the same preprocessing process as the historical data. According to the formula: Calculate the Mahalanobis distance between the feature matrix to be detected and the mean vector of the historical model. The preset threshold is set based on the 99.9% quantile of the historical data distance distribution using the kernel density estimation method.

[0061] Anomaly determination: Compare the calculated Mahalanobis distance metric value with the preset threshold. If it exceeds the threshold, it is initially determined as an abnormal signal. For the initially determined abnormal signals, use the sliding window mechanism (window length 10 seconds, step size 1 second) to calculate the variance of the distance fluctuation within the window. The dynamic threshold is calculated using the exponentially weighted moving average method (weight coefficient 0.2). When the distance metric value exceeds the static threshold and the variance of the distance fluctuation within the window exceeds the dynamic threshold, confirm that the signal is an abnormal signal.

[0062] Anomaly localization: For the confirmed abnormal signals, extract their two-dimensional frequency-time feature maps, and perform erosion and dilation operations using image morphological processing methods to remove noise interference, thereby localizing the frequency band coordinates and time intervals of the abnormal signals. At the same time, calculate the relative deviation of the energy density of the abnormal signal from the mean of the historical model Classify the degree of anomaly according to the deviation value.

[0063] The above describes the present invention and its implementation manners. This description is not restrictive. What is shown in the drawings is only one of the implementation manners of the present invention, and the actual structure is not limited thereto. Generally speaking, if those of ordinary skill in the art are inspired by it and design similar structural manners and embodiments to this technical solution without creative efforts without departing from the purpose of the present invention, they shall fall within the protection scope of the present invention.

Claims

1. A distance-based method for detecting abnormal data in electromagnetic spectrum monitoring, characterized in that, It includes the following steps: S1. Data preprocessing: Perform short-time Fourier transform on the collected original electromagnetic spectrum signals to generate a time-frequency distribution matrix; perform normalization processing on the matrix to obtain an energy density distribution feature map; S2. Historical model construction: Based on the historical spectrum data without anomalies, calculate the energy mean vector and covariance matrix of each frequency point within the frequency band, and establish a multivariate Gaussian distribution model; S3. Real-time monitoring: Perform the same preprocessing process on the real-time collected spectrum data as the historical data to generate a feature matrix to be detected; Calculate the Mahalanobis distance between the feature matrix to be detected and the historical model mean vector to obtain a distance metric value; S4. Anomaly determination: Compare the Mahalanobis distance metric value with a preset threshold. If it exceeds the threshold, it is determined as an abnormal signal; Perform secondary verification on the abnormal signal. Adopt a sliding window mechanism to calculate the distance fluctuation variance within the window. If the variance exceeds the dynamic threshold, the abnormality is confirmed; S5. Anomaly localization: For the spectrum data confirmed to be abnormal, extract its two-dimensional frequency-time feature map; Adopt an image morphological processing method, and remove noise interference through erosion and dilation operations to locate the frequency band coordinates and time interval of the abnormal signal.

2. The method for detecting abnormal data in electromagnetic spectrum monitoring based on distance according to claim 1, wherein: In step S1, the window function of the short-time Fourier transform uses a Hamming window, the window length is 512 sampling points, and the overlap rate is 75%; The normalization processing adopts the maximum-minimum normalization method to map the energy density value to the interval [0, 1].

3. The method for detecting abnormal data in electromagnetic spectrum monitoring based on distance according to claim 2, wherein: In step S2, the selection of the historical spectrum data needs to meet the following conditions: The data collection period is not less than 30 days, and the daily collection period covers the all-weather usage scenario of the target frequency band; The proportion of abnormal samples in the historical data does not exceed 0.5%; The parameter estimation of the multivariate Gaussian distribution model adopts the maximum likelihood estimation method, and a regularization term is added during the calculation of the covariance matrix to avoid numerical instability.

4. The method for detecting abnormal data in electromagnetic spectrum monitoring based on distance according to claim 3, characterized in that: In step S3, the Mahalanobis distance is calculated using the following formula: Among them, x is the feature vector to be detected, μ is the historical mean vector, and Σ is the covariance matrix; The preset threshold is determined by the kernel density estimation method and is set based on the 99.9% quantile of the historical data distance distribution.

5. The method for detecting abnormal data in electromagnetic spectrum monitoring based on distance according to claim 4, wherein: In step S4, the window length of the sliding window is 10 seconds, and the step size is 1 second; The dynamic threshold is calculated by the exponentially weighted moving average method, and the weight coefficient is 0.2; The secondary verification needs to meet both of the following conditions: The distance metric value exceeds the static threshold; the distance fluctuation variance within the window exceeds the dynamic threshold.

6. The method for detecting abnormal data in electromagnetic spectrum monitoring based on distance according to claim 5, characterized in that: For the signals with confirmed anomalies, calculate the relative deviation of their energy density from the historical model mean. The formula is as follows: The anomaly level is divided into three levels according to the deviation value: Level 1 anomaly (δ≥50%); Level 2 anomaly (20%≤δ<50%); Level 3 anomaly (δ<20%).

Images