Industrial control network threat trend detection method and system based on time sequence characteristics

Abstract

The invention discloses an industrial control network threat trend detection method and system based on time sequence characteristics, and particularly relates to the technical field of industrial control networks. The method comprises the following steps: dynamically adjusting an existing threat detection model to improve the adaptability by introducing time series data after new equipment or system upgrading, inputting new data into an original model, and evaluating the model performance according to a reaction speed; if the performance is reduced, judging the characteristic consistency of the model by analyzing the drifting of the time series data characteristics; then, the stability of the model on different data sets is evaluated through hybrid verification; according to the evaluation results, the adaptability of the model is judged, the model is divided into a high-adaptability model or a low-adaptability model, for the low-adaptability model, a threshold value is dynamically adjusted, false alarm frequency abnormity is further analyzed and false alarms are filtered through a secondary detection mechanism, the adaptability of the model in data change is effectively improved, the false alarm risk is reduced, and the accuracy of the data change is improved. And the safety and the reliability of the system are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial control networks, and in particular to a method and system for detecting industrial control network threat trends based on time series features. Background Art

[0002] The detection method of industrial control network threat trends based on time series features is of great significance in the context of new power. With the widespread application of new power technologies (such as renewable energy and advanced battery technology), the security challenges faced by industrial control systems are becoming increasingly complex. Traditional network security methods are difficult to effectively deal with these emerging threats, so the detection method based on time series features has become a forward-looking solution. By analyzing the time series data (such as sensor data, control signals, etc.) in the industrial control system, the statistical characteristics, frequency domain characteristics and time domain characteristics of the data are extracted, and the threat detection model is constructed using machine learning technology. These models can identify abnormal behaviors, including data traffic from unknown sources, abnormal frequencies or pattern changes, so as to timely detect potential network attacks or equipment failures and ensure the safety and stable operation of new power systems. It involves an in-depth understanding of the unique architecture, communication protocols and energy conversion processes of industrial control systems, as well as a comprehensive consideration of the characteristics and security requirements of new power technologies. This method not only meets the requirements of energy production and distribution efficiency in the new energy era, but also promotes the transformation of industrial control system security technology to a more intelligent and efficient one.

[0003] The prior art has the following deficiencies: When new equipment is introduced or the system is upgraded, the equipment's operating parameters, data generation mode, workload, etc. may be different from the original system, resulting in changes in the characteristics of the time series data. If the original threat detection model is not adjusted or updated in time according to these changes, the model will not be able to adapt to the new data pattern, which may lead to the inability to accurately identify potential equipment problems or failures, increase the risk of false alarms, and thus affect the security and reliability of the system. In addition, the false alarm rate may also increase, interfering with normal operations. Summary of the invention

[0004] On one hand, the present invention provides an industrial control network threat trend detection method based on time series characteristics to address the deficiencies in the background technology.

[0005] Another aspect of the present invention provides an industrial control network threat trend detection system based on time series characteristics.

[0006] In order to achieve the above object, the present invention provides the following technical solution: a method for detecting industrial control network threat trends based on time series features, comprising the following steps: Input the newly collected data into the original time series feature extraction and threat detection model, and determine whether the performance of the threat detection model has declined based on the reaction speed of the threat detection model when the new data is input; When the performance of the threat detection model degrades, compare the time series data of the new device with the data of the original device, judge the drift state of the time series data features, and determine the consistency between the time series data features and the threat detection model; Mix the new time series data with the original data for verification and evaluate the stability of the threat detection model on different data sets; According to the consistency between the time series data features and the threat detection model and the stability of the threat detection model on different data sets, the adaptability of the threat detection model when the time series data pattern changes is evaluated, and it is divided into high-adaptability models and low-adaptability models; For low-adaptation models, the threshold in the threat detection model is dynamically adjusted, and the abnormal degree of false alarm frequency of the threat detection model in the subsequent fixed time period is further analyzed. If the abnormal degree of false alarm frequency is high, the false alarm type is identified and filtered through the secondary detection mechanism.

[0007] Preferably, judging whether the performance of the threat detection model is degraded according to the reaction speed of the threat detection model when new data is input, specifically comprising: collecting inference time data from the threat detection model, establishing a baseline distribution of inference time, recorded as Thistory; when new data is input, recording the inference time of the model, generating a new inference time distribution, recorded as Tnew; A two-sample t-test is used to evaluate the difference in inference time between historical data and new data. The calculation expression is: ; In the formula, is the mean of historical reasoning time, is the mean inference time of new data, the square of the standard deviation of historical and new inference times; is the number of samples of historical inference time and new inference time, The square of the standard deviation of the inference time of the new data is set. The null hypothesis H0 is set, that is, there is no significant difference between the inference time of the new data and the inference time of the historical data. The t-test formula is used to calculate the t value. The significance level α=0.05 is selected. The critical value is found from the t-distribution table. If If it is greater than the critical value, the null hypothesis is rejected and there is a change in the inference time; Use KL divergence to detect changes in the inference time distribution. The expression is: ; Where P(x) is the historical reasoning time distribution, Q(x) is the new reasoning time distribution; If the inference time distribution of the calculated new data If the inference time distribution of the new data is greater than or equal to the inference time distribution of the historical data, the performance of the threat detection model will decrease; if the inference time distribution of the new data calculated is If the inference time distribution is smaller than the historical data, the performance of the threat detection model does not decrease.

[0008] Preferably, when the performance of the threat detection model decreases, the time series data of the new device is compared with the data of the original device, the drift state of the time series data characteristics is judged, and the consistency of the time series data characteristics and the threat detection model is determined, which specifically includes: comparing the time series data of the new device with the data of the original device, generating a feature distortion index according to the drift of the time series data characteristics, and determining the consistency of the time series data characteristics and the threat detection model. The method for obtaining the feature distortion index is: Extract the same type of time series features from the data of the new device and the old device, ; is the timing characteristics of the original equipment, with a length of n; is the timing feature of the new device, with a length of m. In the DTW algorithm, a distance matrix D is established between two feature sequences X and Y, where each element D(i,j) represents the local distance between Xi and Yj. The commonly used distance metric is the Euclidean distance, and the calculation formula is: ; The generated matrix D is of size n×m, where each element D(i,j) represents the distance between the i-th point of X and the j-th point of Y; the cumulative distance matrix C is calculated to find the optimal alignment path between X and Y. The element C(i,j) of the cumulative distance matrix is ​​calculated as: ; C(i,j) is the i,jth item of the cumulative distance matrix, is the corresponding element in the local distance matrix. By accumulating the distance matrix C, we find an optimal alignment path from C(1,1) to C(n,m), which can minimize the overall distance between the timing features of the original device and the new device, and is recorded as ; where k is the path length, the total distance of the path is expressed as: ; Calculate the characteristic distortion index, the expression is: ; In the formula, is the characteristic distortion index.

[0009] Preferably, the new time series data is mixed with the original data for verification, and the stability of the threat detection model on different data sets is evaluated, which specifically includes: generating a mixed data interference index after mixed verification of the time series data of the new device and the data of the original device, and evaluating the stability of the threat detection model on different data sets. The method for obtaining the mixed data interference index is: Mark the timing characteristics of the original equipment as ,in is the hth feature vector of the original device, with a data volume of h; the time series features of the new device are marked as ,in is the g-th feature vector of the new device, with a data volume of g. The kernel function is used to map the original data to a high-dimensional space, and the mean difference is calculated in the high-dimensional space. The expression is: ; where σ is the width parameter of the kernel function, using the selected kernel function The kernel matrix expression for calculating the original device data and the new device data is: ; s is the total number of time series features of the original equipment; the kernel matrix expression between the new equipment data sets is: ; w is the total number of timing characteristics of the new device; calculate the mixed data interference index, the expression is: ; In the formula, is the mixed data interference index.

[0010] Preferably, the adaptability of the threat detection model when the time series data pattern changes is evaluated according to the consistency between the time series data characteristics and the threat detection model and the stability of the threat detection model on different data sets, and the model is divided into a high-adaptability model and a low-adaptability model, specifically including: according to the consistency between the time series data characteristics and the threat detection model and the stability of the threat detection model on different data sets, the adaptability of the threat detection model when the time series data pattern changes is evaluated, and the model is divided into a high-adaptability model and a low-adaptability model; The feature distortion index and the mixed data interference index are converted into the first eigenvector, and the first eigenvector is used as the input of the machine learning model. The machine learning model takes the adaptive analysis value label of the threat detection model when the time series data pattern changes predicted by each group of first eigenvectors as the prediction target, and takes minimizing the sum of the prediction errors of the adaptive analysis value labels of the threat detection model when all time series data patterns change as the training target. The machine learning model is trained until the sum of the prediction errors converges and the model training is stopped. The adaptive analysis value of the threat detection model when the time series data pattern changes is determined according to the model output results, wherein the machine learning model is a polynomial regression model.

[0011] Preferably, the adaptive analysis value of the threat detection model when the acquired time series data pattern changes is compared with the standard adaptive analysis value of the historical data under normal conditions. If the adaptive analysis value of the threat detection model when the time series data pattern changes is greater than or equal to the standard adaptive analysis value, the adaptability of the threat detection model when the time series data pattern changes is high, and no warning signal is generated at this time, no further adjustment is required, and it is classified as a high-adaptability model; if the adaptive analysis value of the threat detection model when the time series data pattern changes is less than the standard adaptive analysis value, the adaptability of the threat detection model when the time series data pattern changes is low, and a warning signal is generated at this time, and it is classified as a low-adaptability model.

[0012] Preferably, for the low-adaptability model, the threshold in the threat detection model is dynamically adjusted, and the abnormal degree of false alarm frequency of the threat detection model in a subsequent fixed time period is further analyzed. If the abnormal degree of false alarm frequency is high, the false alarm type is identified and filtered through the secondary detection mechanism, which specifically includes: for the low-adaptability model, the threshold in the threat detection model is dynamically adjusted, and the abnormal degree of false alarm frequency of the threat detection model in a subsequent fixed time period is further analyzed; If the adaptability analysis value of the threat detection model is less than the standard adaptability analysis value when the time series data pattern changes, it is classified as a low adaptability model. For the low adaptability model, a sliding window of fixed length is used to monitor the performance of the model in real time, and the threshold T is adjusted dynamically. The adaptability analysis value of the threat detection model is calculated within the sliding window. If the adaptability analysis value increases, the threshold T is gradually increased. The expression is: ; If the adaptive analysis value decreases, the threshold T is gradually reduced, and the expression is: ; Each time the sliding window moves one unit, the adaptive analysis value of the threat detection model is recalculated and the threshold is updated. ΔT is the step size of the threshold adjustment.

[0013] Preferably, the abnormal degree of false alarm frequency of the threat detection model in the subsequent fixed time period is further analyzed, a fixed time period L is set, the number of false alarms FPR(t) of the model is counted in each time period, and a normal threshold Tfpr of the false alarm frequency is set as the normal upper limit of the false alarm frequency. If the false alarm frequency FPR(t) is greater than the normal threshold in the time period, it is considered that the false alarm frequency of the model in the time period is abnormal, and the abnormal degree of the false alarm frequency is detected. The expression is: ; In the formula, is the historical mean of false alarm frequency, is the historical standard deviation of false alarm frequency, is the abnormal degree of false alarm frequency, if If it is greater than Tfpr, it means that the false alarm frequency of the threat detection model in a fixed time period is abnormally high. At this time, an abnormal signal is generated. After detecting that the false alarm frequency is abnormally high, the secondary detection mechanism is activated to identify the type of false alarms and filter out the false alarms to improve the accuracy of the model. If it is less than Tfpr, it means that the false alarm frequency of the threat detection model in a fixed time period is low and no abnormal signal is generated.

[0014] Another aspect of the present invention provides an industrial control network threat trend detection system based on time series characteristics, comprising: The performance judgment module is used to input the newly collected data into the original time series feature extraction and threat detection model, and judge whether the performance of the threat detection model has decreased according to the reaction speed of the threat detection model when the new data is input; The consistency module is used to compare the time series data of the new device with the data of the original device when the performance of the threat detection model decreases, judge the drift state of the time series data characteristics, and determine the consistency of the time series data characteristics with the threat detection model; The evaluation module is used to mix the new time series data with the original data for verification and evaluate the stability of the threat detection model on different data sets; The adaptation classification module is used to evaluate the adaptability of the threat detection model when the time series data pattern changes, and classify it into high-adaptation models and low-adaptation models according to the consistency between the time series data characteristics and the threat detection model and the stability of the threat detection model on different data sets; The adjustment module is used to dynamically adjust the threshold in the threat detection model for the low-adaptation model, and further analyze the abnormal degree of false alarm frequency of the threat detection model in the subsequent fixed time period. If the abnormal degree of false alarm frequency is high, the false alarm type is identified and filtered through the secondary detection mechanism.

[0015] In the above technical solution, the technical effects and advantages provided by the present invention are: 1. The present invention introduces newly collected data to extract time series features and monitor performance, combines feature distortion index and mixed data interference index, evaluates the adaptability of the model and makes dynamic adjustments to ensure the high efficiency of the model in the new data environment. For low-adaptability models, the sliding window technology is used to dynamically adjust the detection threshold, monitor the model performance in real time, and analyze the abnormal degree of false alarm frequency, further identify and filter false alarms, and improve the accuracy and stability of the model.

[0016] 2. The present invention can not only effectively identify potential threats and equipment failures when new data patterns change, but also avoid false positives and false negatives caused by data drift, thereby improving system security and reliability. Through the introduction of adaptive analysis, dynamic adjustment and secondary detection mechanisms, it also reduces the interference of false positives on normal operations, ensuring the continuous accuracy and responsiveness of the industrial control network threat detection system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0018] Figure 1 is a flow chart of the method of the present invention; Figure 2 It is a system module diagram of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] For examples, see Figure 1 and Figure 2 As shown, the method for detecting industrial control network threat trends based on time series features described in this embodiment includes the following steps: S1: Input the newly collected data into the original time series feature extraction and threat detection model, and determine whether the performance of the threat detection model has decreased based on the reaction speed of the threat detection model when the new data is input; S2: When the performance of the threat detection model deteriorates, compare the time series data of the new device with the data of the original device, judge the drift state of the time series data features, and determine the consistency between the time series data features and the threat detection model; S3: Mix the new time series data with the original data for verification to evaluate the stability of the threat detection model on different data sets; S4: Based on the consistency between the time series data features and the threat detection model and the stability of the threat detection model on different data sets, the adaptability of the threat detection model when the time series data pattern changes is evaluated, and it is divided into a high-adaptability model and a low-adaptability model; S5: For low-adaptation models, dynamically adjust the threshold in the threat detection model, and further analyze the abnormal degree of false alarm frequency of the threat detection model in the subsequent fixed time period. If the abnormal degree of false alarm frequency is high, identify and filter the false alarm type through the secondary detection mechanism.

[0021] Among them, in S1, the newly collected data is input into the original time series feature extraction and threat detection model, and according to the reaction speed of the threat detection model when the new data is input, it is judged whether the performance of the threat detection model is reduced.

[0022] In the threat detection process, we must first ensure that the newly collected time series data can be accurately input into the existing feature extraction and detection models, including: Clear noise, duplicates, and missing values ​​from newly collected data to ensure the integrity and accuracy of input data. If the new data comes from different devices or multiple data sources, align timestamps to ensure consistency and synchronization of time series data. Use feature extraction methods consistent with the original system to extract time series features from newly collected data. Common features include: Statistical features: such as mean, variance, maximum, minimum, etc., which reflect the basic trend and distribution of time series data. Time domain features: such as autocorrelation, time series fluctuation amplitude, etc., to capture the time dependence of data. Frequency domain features: Extract frequency components through Fourier transform to analyze periodic changes in time series data.

[0023] Input the processed newly collected data into the threat detection model in batches, ensuring that the input dimensions and format are consistent with the model's training data. For example, for systems using deep learning models (such as LSTM), the input data should be a multi-dimensional time series feature matrix within a time window. In systems that require real-time monitoring, set a timer or trigger to input the latest time series data into the detection model at fixed time intervals (such as every minute, every hour).

[0024] Inference time data is collected from the threat detection model, and a baseline distribution of inference time is established, denoted as Thistory, which contains the inference time samples of the model on historical data. When new data is input, the inference time of the model is recorded to generate a new inference time distribution, denoted as Tnew.

[0025] The t-test is used to evaluate the difference in inference time. The t-test is a commonly used statistical test that is suitable for evaluating the significant difference between the means of two groups of samples. A two-sample t-test is used to evaluate the difference in inference time between historical data and new data. The calculation expression is: ; In the formula, is the mean of historical reasoning time, is the mean inference time of new data, the square of the standard deviation of historical and new inference times; is the number of samples of historical inference time and new inference time, The square of the standard deviation of the inference time of the new data is set. The null hypothesis H0 is set, that is, there is no significant difference between the inference time of the new data and the inference time of the historical data. The t-test formula is used to calculate the t value. The significance level α=0.05 is selected. The critical value is found from the t-distribution table. If If it is greater than the critical value, the null hypothesis is rejected and there is a change in the inference time; Use KL divergence to detect changes in the inference time distribution. The expression is: ; Where P(x) is the historical reasoning time distribution, Q(x) is the new reasoning time distribution; If the inference time distribution of the calculated new data If the inference time distribution of the new data is greater than or equal to the inference time distribution of the historical data, the performance of the threat detection model will decrease; if the inference time distribution of the new data calculated is If the inference time distribution is smaller than the historical data, the performance of the threat detection model does not decrease.

[0026] In S2, when the performance of the threat detection model degrades, the key is to compare the time series data of the new device with the data of the original device to determine whether the time series data features have drifted, so as to evaluate the consistency of the features with the threat detection model. Feature drift will cause the model to perform poorly for the new device data, so it is necessary to specifically evaluate the drift.

[0027] Compare the time series data of the new device with the data of the original device, generate a feature distortion index based on the drift of the time series data features, and determine the consistency of the time series data features with the threat detection model. The method for obtaining the feature distortion index is as follows: Extract the same type of time series features from the data of the new device and the old device, ; is the timing characteristics of the original equipment, with a length of n; is the timing feature of the new device, with a length of m. In the DTW algorithm, a distance matrix D is established between two feature sequences X and Y, where each element D(i,j) represents the local distance between Xi and Yj. The commonly used distance metric is the Euclidean distance, and the calculation formula is: ; The generated matrix D is of size n×m, where each element D(i,j) represents the distance between the i-th point of X and the j-th point of Y; the cumulative distance matrix C is calculated to find the optimal alignment path between X and Y. The element C(i,j) of the cumulative distance matrix is ​​calculated as: ; C(i,j) is the i,jth item of the cumulative distance matrix, is the corresponding element in the local distance matrix. By accumulating the distance matrix C, we find an optimal alignment path from C(1,1) to C(n,m), which can minimize the overall distance between the timing features of the original device and the new device, and is recorded as ; where k is the path length, the total distance of the path is expressed as: ; Calculate the characteristic distortion index, the expression is: ; In the formula, is the characteristic distortion index.

[0028] When the feature distortion index is large, it means that the time series data features of the new device are significantly different from those of the original device, indicating that the operating mode of the new device has changed significantly from the original system. This high distortion index means that the threat detection model is less adaptable to the data of the new device and may not be able to accurately identify the behavioral characteristics or potential threats of the new device. In this case, the model may generate more false positives or false negatives, or even fail to effectively capture the operating status of the new device. The model needs to be adjusted or retrained to improve its adaptability to the new device.

[0029] When the feature distortion index is small, it means that the time series data features of the new device are highly similar to the data of the original device, indicating that the operation mode of the new device is consistent with the original device or has only slight differences. This low distortion index shows that the threat detection model can adapt well to the data features of the new device and can still effectively identify potential threats and ensure the accuracy and reliability of detection. In this case, the model does not need to be adjusted significantly and can continue to run stably and maintain good detection results.

[0030] In S3, the stability of threat detection models on new time series data and original data sets can be evaluated by testing the models through hybrid verification. By combining the time series data of new devices with the data of original devices, the performance of the model in actual scenarios is simulated to evaluate its ability to adapt to new data and its overall stability on different data sets.

[0031] After hybrid verification of the time series data of the new device and the data of the original device, a hybrid data interference index is generated to evaluate the stability of the threat detection model on different data sets. The method for obtaining the hybrid data interference index is as follows: Mark the timing characteristics of the original equipment as ,in is the hth feature vector of the original device, with a data volume of h; the time series features of the new device are marked as ,in is the g-th feature vector of the new device, with a data volume of g. The kernel function is used to map the original data to a high-dimensional space, and the mean difference is calculated in the high-dimensional space. The expression is: ; where σ is the width parameter of the kernel function, using the selected kernel function The kernel matrix expression for calculating the original device data and the new device data is: ; s is the total number of time series features of the original equipment; the kernel matrix expression between the new equipment data sets is: ; w is the total number of timing characteristics of the new device; calculate the mixed data interference index, the expression is: ; In the formula, is the mixed data interference index.

[0032] When the mixed data interference index is larger, it means that the time series data characteristics of the new device are significantly different from the original device characteristics, and the model shows greater instability when facing different data sets. This indicates that the model may have a high false positive or false negative rate when processing new device data and cannot accurately capture the behavior pattern of the new device. In this case, the model's generalization ability is insufficient and its adaptability is poor. It may need to be retrained or adjusted to ensure that it can maintain a high threat detection capability in the new environment.

[0033] When the mixed data interference index is smaller, it means that the time series data characteristics of the new device are more consistent with the original device, and the model maintains good stability on different data sets. This shows that the model has strong adaptability to new device data, can still accurately identify potential threats when processing new device data, and maintain low false positive and false negative rates. This shows that the performance of the model remains stable in the new device data environment, without the need for major adjustments, and has strong generalization ability.

[0034] S4: Based on the consistency between the time series data features and the threat detection model and the stability of the threat detection model on different data sets, the adaptability of the threat detection model when the time series data pattern changes is evaluated and divided into high-adaptation models and low-adaptation models.

[0035] The feature distortion index and the mixed data interference index are converted into the first eigenvector, and the first eigenvector is used as the input of the machine learning model. The machine learning model takes the adaptive analysis value label of the threat detection model when the time series data pattern changes predicted by each group of first eigenvectors as the prediction target, and takes minimizing the sum of the prediction errors of the adaptive analysis value labels of the threat detection model when all time series data patterns change as the training target. The machine learning model is trained until the sum of the prediction errors converges and the model training is stopped. The adaptive analysis value of the threat detection model when the time series data pattern changes is determined according to the model output results, wherein the machine learning model is a polynomial regression model.

[0036] The method for obtaining the adaptive analysis value of the threat detection model when the time series data pattern changes is as follows: from the first feature vector training data of the trained machine learning model, obtain the corresponding function expression: ; In the formula, is the output function of the model, is the characteristic distortion index, is the mixed data interference index, and EK is the adaptive analysis value of the threat detection model when the time series data pattern changes.

[0037] The adaptive analysis value of the threat detection model when the acquired time series data pattern changes is compared with the standard adaptive analysis value under the normal state of historical data. If the adaptive analysis value of the threat detection model when the time series data pattern changes is greater than or equal to the standard adaptive analysis value, the adaptability of the threat detection model when the time series data pattern changes is high, and no warning signal is generated at this time, indicating that the model performs well and no further adjustment is required, and it is classified as a high-adaptability model; if the adaptive analysis value of the threat detection model when the time series data pattern changes is less than the standard adaptive analysis value, the adaptability of the threat detection model when the time series data pattern changes is low, and a warning signal is generated at this time, and it is classified as a low-adaptability model.

[0038] S5: For low-adaptation models, dynamically adjust the threshold in the threat detection model, and further analyze the abnormal degree of false alarm frequency of the threat detection model in the subsequent fixed time period. If the abnormal degree of false alarm frequency is high, identify and filter the false alarm type through the secondary detection mechanism.

[0039] If the adaptive analysis value of the threat detection model is less than the standard adaptive analysis value when the time series data pattern changes, it is classified as a low adaptive model. For the low adaptive model, a sliding window of fixed length (such as the most recent 100 or 500 data points) is used to monitor the performance of the model in real time and dynamically adjust the threshold T. The adaptive analysis value of the threat detection model is calculated within the sliding window. If the adaptive analysis value increases, the threshold T is gradually increased. The expression is: ; If the adaptive analysis value decreases, the threshold T is gradually reduced, and the expression is: Each time the sliding window moves one unit, the adaptive analysis value of the threat detection model is recalculated and the threshold is updated. ΔT is the step size of the threshold adjustment, which can be set to a smaller constant value (such as 0.01 or 0.05) according to the sensitivity of the model.

[0040] Further analyze the abnormal degree of false alarm frequency of the threat detection model in the subsequent fixed time period, set a fixed time period L (such as 1 hour, 1 day, etc.), count the number of false alarms FPR(t) of the model in each time period, and set a normal threshold Tfpr of false alarm frequency as the normal upper limit of the false alarm frequency. If the false alarm frequency FPR(t) is greater than the normal threshold in a certain time period, it is considered that the false alarm frequency of the model in this time period is abnormal. Detect the abnormal degree of false alarm frequency, for example, use Z-score to calculate the abnormal degree of false alarm frequency, the expression is: ; In the formula, is the historical mean of false alarm frequency, is the historical standard deviation of false alarm frequency, is the abnormal degree of false alarm frequency, if If it is greater than Tfpr, it means that the false alarm frequency of the threat detection model in a fixed time period is abnormally high, and an abnormal signal is generated at this time; If it is less than Tfpr, it means that the false alarm frequency of the threat detection model in a fixed time period is low and no abnormal signal is generated.

[0041] After detecting that the false alarm frequency is abnormally high, the secondary detection mechanism is activated to identify the type of false alarms and filter out possible false alarms to improve the accuracy of the model. The secondary detection mechanism is a process of further verifying the detection results after the first layer of detection. It can identify and filter out unnecessary false alarms through different algorithms or rules, thereby improving the overall detection accuracy.

[0042] Establish rules based on expert knowledge to filter out false positives that meet specific rules. For example, for certain specific system behaviors (such as short-term traffic spikes), even if the model considers it abnormal behavior, the rule engine can recognize that this is expected normal behavior and mark it as non-threatening. If a certain type of traffic appears repeatedly in a fixed time period and meets the preset behavior pattern (such as high-volume but non-persistent connections), the traffic can be marked as a false positive.

[0043] For detected false positive events, extract their feature values ​​(such as network traffic features, sensor data features, etc.) and match them with known normal behavior features. Use clustering algorithms (such as K-Means, DBSCAN) to cluster false positive data with normal data, identify events that are highly similar to normal data features, and mark these events as false positives. If false positive data has a high similarity with normal data in the feature space, it is marked as a false positive. Use an additional classification model to reclassify threat events marked by the first-level model. Train a lightweight secondary model to specifically identify which detection results are false positives. The model can be trained based on more complex features or historical false positive data to improve the model's false positive filtering capabilities. Results with low confidence in the model output can be directly marked as false positives.

[0044] Perform temporal correlation analysis on false alarms to check whether abnormal behaviors have temporal correlation. If similar abnormal behaviors occur frequently in a short period of time, it may be a false alarm. Aggregate false alarm events in a time window and check their temporal patterns. If some false alarms have similar temporal distribution characteristics, it may be that periodic or sporadic normal behaviors are misjudged as abnormal.

[0045] After the second level detection, the detection results that meet the false alarm characteristics are marked as false alarms to prevent them from triggering false alarms or further interfering with system operation. Based on the results of the second level detection, the weights or parameters of the model are dynamically adjusted to reduce the occurrence of specific types of false alarms. For example, the model can be further optimized by updating the characteristics of the false alarm type or dynamically adjusting the detection threshold.

[0046] After each false positive is filtered, the type and specific features of the false positive are fed back to the model training mechanism to optimize the model's training data set. This feedback mechanism can continuously enhance the model's anti-interference ability. The secondary detection model is further trained with false positive data to better adapt to future data changes. Based on the results of secondary detection and feedback, the threat detection model is retrained or fine-tuned. The updated model will reduce the occurrence of similar false positives and improve the ability to detect real threats.

[0047] In this embodiment, the newly collected data is input into the original time series feature extraction and threat detection model, and its performance is judged by analyzing the reaction speed of the model when the new data is input. If the performance is degraded, the time series data of the new device is further compared with the data of the original device to judge the drift state of the time series data features to determine the adaptability of the model to the new data. Next, the new data is mixed with the original data for verification to evaluate the stability of the model on different data sets. According to the consistency of the time series data features and the stability of the model, the adaptability of the model under the new data mode is evaluated, and it is divided into a high-adaptation model or a low-adaptation model. For the low-adaptation model, the detection threshold is dynamically adjusted, and the abnormal degree of the false alarm frequency is analyzed within a fixed time period. If the abnormal frequency is high, the false alarms are identified and filtered through the secondary detection mechanism to improve the accuracy of the model.

[0048] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0049] Another embodiment of the present invention provides an industrial control network threat trend detection system based on time series characteristics, including: The performance judgment module is used to input the newly collected data into the original time series feature extraction and threat detection model, and judge whether the performance of the threat detection model has decreased according to the reaction speed of the threat detection model when the new data is input; The consistency module is used to compare the time series data of the new device with the data of the original device when the performance of the threat detection model decreases, judge the drift state of the time series data characteristics, and determine the consistency of the time series data characteristics with the threat detection model; The evaluation module is used to mix the new time series data with the original data for verification and evaluate the stability of the threat detection model on different data sets; The adaptation classification module is used to evaluate the adaptability of the threat detection model when the time series data pattern changes, and classify it into high-adaptation models and low-adaptation models according to the consistency between the time series data characteristics and the threat detection model and the stability of the threat detection model on different data sets; The adjustment module is used to dynamically adjust the threshold in the threat detection model for the low-adaptation model, and further analyze the abnormal degree of false alarm frequency of the threat detection model in the subsequent fixed time period. If the abnormal degree of false alarm frequency is high, the false alarm type is identified and filtered through the secondary detection mechanism.

[0050] The performance judgment module is specifically used to judge whether the performance of the threat detection model has declined according to the reaction speed of the threat detection model when new data is input. Specifically, it includes: collecting inference time data from the threat detection model, establishing a baseline distribution of inference time, recorded as Thistory; when new data is input, recording the inference time of the model, generating a new inference time distribution, recorded as Tnew; A two-sample t-test is used to evaluate the difference in inference time between historical data and new data. The calculation expression is: ; In the formula, is the mean of historical reasoning time, is the mean inference time of new data, the square of the standard deviation of historical and new inference times; is the number of samples of historical inference time and new inference time, The square of the standard deviation of the inference time of the new data is set. The null hypothesis H0 is set, that is, there is no significant difference between the inference time of the new data and the inference time of the historical data. The t-test formula is used to calculate the t value. The significance level α=0.05 is selected. The critical value is found from the t-distribution table. If If it is greater than the critical value, the null hypothesis is rejected and there is a change in the inference time; Use KL divergence to detect changes in the inference time distribution. The expression is: ; Where P(x) is the historical reasoning time distribution, Q(x) is the new reasoning time distribution; If the inference time distribution of the calculated new data If the inference time distribution of the new data is greater than or equal to the inference time distribution of the historical data, the performance of the threat detection model will decrease; if the inference time distribution of the new data calculated is If the inference time distribution is smaller than the historical data, the performance of the threat detection model does not decrease.

[0051] The consistency module is specifically used to compare the time series data of the new device with the data of the original device, generate a feature distortion index according to the drift of the time series data characteristics, and determine the consistency of the time series data characteristics with the threat detection model. The method for obtaining the feature distortion index is: Extract the same type of time series features from the data of the new device and the old device, ; is the timing characteristics of the original equipment, with a length of n; is the timing feature of the new device, with a length of m. In the DTW algorithm, a distance matrix D is established between two feature sequences X and Y, where each element D(i,j) represents the local distance between Xi and Yj. The commonly used distance metric is the Euclidean distance, and the calculation formula is: ; The generated matrix D is of size n×m, where each element D(i,j) represents the distance between the i-th point of X and the j-th point of Y; the cumulative distance matrix C is calculated to find the optimal alignment path between X and Y. The element C(i,j) of the cumulative distance matrix is ​​calculated as: ; C(i,j) is the i,jth item of the cumulative distance matrix, is the corresponding element in the local distance matrix. By accumulating the distance matrix C, we find an optimal alignment path from C(1,1) to C(n,m), which can minimize the overall distance between the timing features of the original device and the new device, and is recorded as ; where k is the path length, the total distance of the path is expressed as: ; Calculate the characteristic distortion index, the expression is: ; In the formula, is the characteristic distortion index.

[0052] The evaluation module generates a mixed data interference index after hybrid verification of the time series data of the new device and the data of the original device, and evaluates the stability of the threat detection model on different data sets. The method for obtaining the mixed data interference index is: Mark the timing characteristics of the original equipment as ,in is the hth feature vector of the original device, with a data volume of h; the time series features of the new device are marked as ,in is the g-th feature vector of the new device, with a data volume of g. The kernel function is used to map the original data to a high-dimensional space, and the mean difference is calculated in the high-dimensional space. The expression is: ; where σ is the width parameter of the kernel function, using the selected kernel function The kernel matrix expression for calculating the original device data and the new device data is: ; s is the total number of time series features of the original equipment; the kernel matrix expression between the new equipment data sets is: ; w is the total number of timing characteristics of the new device; calculate the mixed data interference index, the expression is: ; In the formula, is the mixed data interference index.

[0053] The adaptation classification module evaluates the adaptability of the threat detection model when the time series data pattern changes, and divides it into high-adaptation models and low-adaptation models according to the consistency between the time series data features and the threat detection model and the stability of the threat detection model on different data sets; The feature distortion index and the mixed data interference index are converted into the first eigenvector, and the first eigenvector is used as the input of the machine learning model. The machine learning model takes the adaptive analysis value label of the threat detection model when the time series data pattern changes predicted by each group of first eigenvectors as the prediction target, and takes minimizing the sum of the prediction errors of the adaptive analysis value labels of the threat detection model when all time series data patterns change as the training target. The machine learning model is trained until the sum of the prediction errors converges and the model training is stopped. The adaptive analysis value of the threat detection model when the time series data pattern changes is determined according to the model output results, wherein the machine learning model is a polynomial regression model.

[0054] The adaptive analysis value of the threat detection model when the acquired time series data pattern changes is compared with the standard adaptive analysis value of the historical data under normal conditions. If the adaptive analysis value of the threat detection model when the time series data pattern changes is greater than or equal to the standard adaptive analysis value, the adaptability of the threat detection model when the time series data pattern changes is high, and no warning signal is generated at this time, no further adjustment is required, and it is classified as a high-adaptability model; if the adaptive analysis value of the threat detection model when the time series data pattern changes is less than the standard adaptive analysis value, the adaptability of the threat detection model when the time series data pattern changes is low, and a warning signal is generated at this time, and it is classified as a low-adaptability model.

[0055] The adjustment module dynamically adjusts the threshold in the threat detection model for low-adaptation models and further analyzes the abnormal degree of false alarm frequency of the threat detection model in a subsequent fixed time period; If the adaptability analysis value of the threat detection model is less than the standard adaptability analysis value when the time series data pattern changes, it is classified as a low adaptability model. For the low adaptability model, a sliding window of fixed length is used to monitor the performance of the model in real time, and the threshold T is adjusted dynamically. The adaptability analysis value of the threat detection model is calculated within the sliding window. If the adaptability analysis value increases, the threshold T is gradually increased. The expression is: ; If the adaptive analysis value decreases, the threshold T is gradually reduced, and the expression is: ; Each time the sliding window moves one unit, the adaptive analysis value of the threat detection model is recalculated and the threshold is updated. ΔT is the step size of the threshold adjustment.

[0056] Further analysis is performed on the abnormal degree of false alarm frequency of the threat detection model in the subsequent fixed time period. A fixed time period L is set. The number of false alarms FPR(t) of the model is counted in each time period. The normal threshold Tfpr of the false alarm frequency is set as the normal upper limit of the false alarm frequency. If the false alarm frequency FPR(t) is greater than the normal threshold in the time period, it is considered that the false alarm frequency of the model in the time period is abnormal. The abnormal degree of the false alarm frequency is detected. The expression is: ; In the formula, is the historical mean of false alarm frequency, is the historical standard deviation of false alarm frequency, is the abnormal degree of false alarm frequency, if If it is greater than Tfpr, it means that the false alarm frequency of the threat detection model in a fixed time period is abnormally high. At this time, an abnormal signal is generated. After detecting that the false alarm frequency is abnormally high, the secondary detection mechanism is activated to identify the type of false alarms and filter out the false alarms to improve the accuracy of the model. If it is less than Tfpr, it means that the false alarm frequency of the threat detection model in a fixed time period is low and no abnormal signal is generated.

[0057] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0058] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0059] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A method for detecting industrial control network threat trends based on time series features, characterized in that: The steps include: Input the newly collected data into the original time series feature extraction and threat detection model, and determine whether the performance of the threat detection model has declined based on the reaction speed of the threat detection model when the new data is input; When the performance of the threat detection model deteriorates, compare the time series data of the new device with the data of the original device, judge the drift state of the time series data features, and determine the consistency between the time series data features and the threat detection model; Mix the new time series data with the original data for verification and evaluate the stability of the threat detection model on different data sets; According to the consistency between the time series data features and the threat detection model and the stability of the threat detection model on different data sets, the adaptability of the threat detection model when the time series data pattern changes is evaluated, and it is divided into high-adaptability models and low-adaptability models; For low-adaptation models, the threshold in the threat detection model is dynamically adjusted, and the abnormal degree of false alarm frequency of the threat detection model in the subsequent fixed time period is further analyzed. If the abnormal degree of false alarm frequency is high, the false alarm type is identified and filtered through the secondary detection mechanism.

2. The method for detecting industrial control network threat trends based on time series features according to claim 1 is characterized in that: The newly collected data is input into the original time series feature extraction and threat detection model, and the performance of the threat detection model is judged to be decreased according to the reaction speed of the threat detection model when the new data is input, which specifically includes: collecting inference time data from the threat detection model, establishing a baseline distribution of inference time, recorded as Thistory; when new data is input, recording the inference time of the model, generating a new inference time distribution, recorded as Tnew; A two-sample t-test is used to evaluate the difference in inference time between historical data and new data. The calculation expression is: ; In the formula, is the mean of historical reasoning time, is the mean inference time of new data, the square of the standard deviation of historical and new inference times; is the number of samples of historical inference time and new inference time, The square of the standard deviation of the inference time of the new data is set. The null hypothesis H0 is set, that is, there is no significant difference between the inference time of the new data and the inference time of the historical data. The t-test formula is used to calculate the t value. The significance level α=0.05 is selected. The critical value is found from the t-distribution table. If If it is greater than the critical value, the null hypothesis is rejected and there is a change in the inference time; Use KL divergence to detect changes in the inference time distribution. The expression is: ; Where P(x) is the historical reasoning time distribution, Q(x) is the new reasoning time distribution; If the inference time distribution of the calculated new data If the inference time distribution of the new data is greater than or equal to the inference time distribution of the historical data, the performance of the threat detection model will decrease; if the inference time distribution of the new data calculated is If the inference time distribution is smaller than the historical data, the performance of the threat detection model does not decrease.

3. The method for detecting industrial control network threat trends based on time series features according to claim 1 is characterized in that: When the performance of the threat detection model decreases, the time series data of the new device is compared with the data of the original device, the drift state of the time series data features is judged, and the consistency of the time series data features and the threat detection model is determined, which specifically includes: comparing the time series data of the new device with the data of the original device, generating a feature distortion index according to the drift of the time series data features, and determining the consistency of the time series data features and the threat detection model. The method for obtaining the feature distortion index is: Extract the same type of time series features from the data of the new device and the old device, ; is the timing characteristics of the original equipment, with a length of n; is the timing feature of the new device, with a length of m. In the DTW algorithm, a distance matrix D is established between two feature sequences X and Y, where each element D(i,j) represents the local distance between Xi and Yj. The commonly used distance metric is the Euclidean distance, and the calculation formula is: ; The generated matrix D is of size n×m, where each element D(i,j) represents the distance between the i-th point of X and the j-th point of Y; the cumulative distance matrix C is calculated to find the optimal alignment path between X and Y. The element C(i,j) of the cumulative distance matrix is ​​calculated as: ; C(i,j) is the i,jth item of the cumulative distance matrix, is the corresponding element in the local distance matrix. By accumulating the distance matrix C, we find an optimal alignment path from C(1,1) to C(n,m), which can minimize the overall distance between the timing features of the original device and the new device, and is recorded as ; where k is the path length, the total distance of the path is expressed as: ; Calculate the characteristic distortion index, the expression is: ; In the formula, is the characteristic distortion index.

4. The method for detecting industrial control network threat trends based on time series features according to claim 3 is characterized in that: The new time series data is mixed with the original data for verification, and the stability of the threat detection model on different data sets is evaluated. Specifically, the mixed data interference index is generated after the time series data of the new device and the data of the original device are mixed for verification, and the stability of the threat detection model on different data sets is evaluated. The method for obtaining the mixed data interference index is: Mark the timing characteristics of the original equipment as ,in is the hth feature vector of the original device, with a data volume of h; the time series features of the new device are marked as ,in is the g-th feature vector of the new device, with a data volume of g. The kernel function is used to map the original data to a high-dimensional space, and the mean difference is calculated in the high-dimensional space. The expression is: ; where σ is the width parameter of the kernel function, using the selected kernel function The kernel matrix expression for calculating the original device data and the new device data is: ; s is the total number of time series features of the original equipment; the kernel matrix expression between the new equipment data sets is: ; w is the total number of timing characteristics of the new device; calculate the mixed data interference index, the expression is: ; In the formula, is the mixed data interference index.

5. The method for detecting industrial control network threat trends based on time series features according to claim 4 is characterized in that: The method of evaluating the adaptability of the threat detection model when the time series data pattern changes according to the consistency between the time series data features and the threat detection model and the stability of the threat detection model on different data sets, and dividing it into a high-adaptation model and a low-adaptation model, specifically includes: evaluating the adaptability of the threat detection model when the time series data pattern changes according to the consistency between the time series data features and the threat detection model and the stability of the threat detection model on different data sets, and dividing it into a high-adaptation model and a low-adaptation model; The feature distortion index and the mixed data interference index are converted into the first eigenvector, and the first eigenvector is used as the input of the machine learning model. The machine learning model takes the adaptive analysis value label of the threat detection model when the time series data pattern changes predicted by each group of first eigenvectors as the prediction target, and takes minimizing the sum of the prediction errors of the adaptive analysis value labels of the threat detection model when all time series data patterns change as the training target. The machine learning model is trained until the sum of the prediction errors converges and the model training is stopped. The adaptive analysis value of the threat detection model when the time series data pattern changes is determined according to the model output results, wherein the machine learning model is a polynomial regression model.

6. The method for detecting industrial control network threat trends based on time series features according to claim 5 is characterized in that: The adaptive analysis value of the threat detection model when the acquired time series data pattern changes is compared with the standard adaptive analysis value of the historical data under normal conditions. If the adaptive analysis value of the threat detection model when the time series data pattern changes is greater than or equal to the standard adaptive analysis value, the adaptability of the threat detection model when the time series data pattern changes is high, and no warning signal is generated at this time, no further adjustment is required, and it is classified as a high-adaptability model; if the adaptive analysis value of the threat detection model when the time series data pattern changes is less than the standard adaptive analysis value, the adaptability of the threat detection model when the time series data pattern changes is low, and a warning signal is generated at this time, and it is classified as a low-adaptability model.

7. The method for detecting industrial control network threat trends based on time series characteristics according to claim 1 is characterized in that: For the low-adaptability model, the threshold in the threat detection model is dynamically adjusted, and the abnormal degree of false alarm frequency of the threat detection model in a subsequent fixed time period is further analyzed. If the abnormal degree of false alarm frequency is high, the false alarm type is identified and filtered through the secondary detection mechanism, specifically including: for the low-adaptability model, the threshold in the threat detection model is dynamically adjusted, and the abnormal degree of false alarm frequency of the threat detection model in a subsequent fixed time period is further analyzed; If the adaptability analysis value of the threat detection model is less than the standard adaptability analysis value when the time series data pattern changes, it is classified as a low adaptability model. For the low adaptability model, a sliding window of fixed length is used to monitor the performance of the model in real time, and the threshold T is adjusted dynamically. The adaptability analysis value of the threat detection model is calculated within the sliding window. If the adaptability analysis value increases, the threshold T is gradually increased. The expression is: ; If the adaptive analysis value decreases, the threshold T is gradually reduced, and the expression is: ; Each time the sliding window moves one unit, the adaptive analysis value of the threat detection model is recalculated and the threshold is updated. ΔT is the step size of the threshold adjustment.

8. The method for detecting industrial control network threat trends based on time series features according to claim 7 is characterized in that: Further analysis is performed on the abnormal degree of false alarm frequency of the threat detection model in the subsequent fixed time period. A fixed time period L is set. The number of false alarms FPR(t) of the model is counted in each time period. The normal threshold Tfpr of the false alarm frequency is set as the normal upper limit of the false alarm frequency. If the false alarm frequency FPR(t) is greater than the normal threshold in the time period, it is considered that the false alarm frequency of the model in the time period is abnormal. The abnormal degree of the false alarm frequency is detected. The expression is: ; In the formula, is the historical mean of false alarm frequency, is the historical standard deviation of false alarm frequency, is the abnormal degree of false alarm frequency, if If it is greater than Tfpr, it means that the false alarm frequency of the threat detection model in a fixed time period is abnormally high. At this time, an abnormal signal is generated. After detecting that the false alarm frequency is abnormally high, the secondary detection mechanism is activated to identify the type of false alarms and filter out the false alarms to improve the accuracy of the model. If it is less than Tfpr, it means that the false alarm frequency of the threat detection model in a fixed time period is low and no abnormal signal is generated.

9. An industrial control network threat trend detection system based on time series characteristics, characterized by: include; The performance judgment module is used to input the newly collected data into the original time series feature extraction and threat detection model, and judge whether the performance of the threat detection model has decreased according to the reaction speed of the threat detection model when the new data is input; The consistency module is used to compare the time series data of the new device with the data of the original device when the performance of the threat detection model decreases, judge the drift state of the time series data characteristics, and determine the consistency of the time series data characteristics with the threat detection model; The evaluation module is used to mix the new time series data with the original data for verification and evaluate the stability of the threat detection model on different data sets; The adaptation classification module is used to evaluate the adaptability of the threat detection model when the time series data pattern changes, and classify it into high-adaptation models and low-adaptation models according to the consistency between the time series data characteristics and the threat detection model and the stability of the threat detection model on different data sets; The adjustment module is used to dynamically adjust the threshold in the threat detection model for the low-adaptation model, and further analyze the abnormal degree of false alarm frequency of the threat detection model in the subsequent fixed time period. If the abnormal degree of false alarm frequency is high, the false alarm type is identified and filtered through the secondary detection mechanism.

Images