Intelligent sensing system framework for environmental monitoring based on machine learning techniques

Abstract

The application discloses an intelligent sensing system framework for environment monitoring based on a machine learning technology, comprising a data collection layer, a data processing layer and an information layer, and through designing an abnormal event detection and clustering method, the machine learning technology is adopted to automatically process original sensor measurement values and convert the original sensor measurement values into organized information which is easy for final users to understand and access, so that the problems of narrow environment monitoring space, unreliability and non-intelligence are effectively solved, and an ecological system is better managed and protected.

Description

Technical fields:

[0001] This invention relates to the fields of environmental monitoring and machine learning, and designs an intelligent sensing system framework for environmental monitoring based on machine learning technology. Background technology:

[0002] Environmental monitoring refers to the long-term, systematic, scientific, qualitative and quantitative monitoring of various elements in the environment (such as air, water, soil, noise, electromagnetic radiation, etc.) using modern scientific and technological means such as chemistry, physics, biology, medicine, telemetry, remote sensing, and computers, in order to understand the changes in environmental quality, the causes of changes and the trends of changes, and to provide a scientific basis for environmental management, environmental protection and environmental planning.

[0003] Machine learning, a branch of artificial intelligence, refers to the use of computer algorithms and models to enable computer systems to automatically improve their ability to perform a specific task based on given data and objectives, without explicit programming. Simply put, machine learning allows computer systems to continuously learn and optimize to improve their performance and accuracy. In modern society, machine learning has been widely applied in various fields such as the Internet of Things, autonomous driving, medical diagnosis, financial risk assessment, and speech recognition, becoming an important driving force for technological development and social progress.

[0004] Existing technological shortcomings:

[0005] (a) Difficulty in widely deploying and collecting large spatial data. Due to issues such as cost, energy, maintenance and data transmission capabilities, especially in aquatic environments, deployment is often limited in number, making it difficult to widely deploy and collect large spatial data.

[0006] (b) Low reliability. Modern environmental sensing technology suffers from sensor unreliability due to environmental factors such as sensor drift and biofouling, which further leads to insufficient spatial coverage in most water areas. Summary of the Invention:

[0007] To address the aforementioned problems, this invention provides an intelligent sensing system framework for environmental monitoring based on machine learning technology, comprising a data collection layer, a data processing layer, and an information layer.

[0008] The front end of the data collection layer is connected to the observation station for environmental monitoring, while the back end is connected to the data processing layer. It is used to extract data from the in-situ sensors of each observation station, store the data in the data storage repository, and provide the data processing layer with data segments in a specific format.

[0009] The data processing layer extracts the required data segments from the data repository and, through machine learning, anomaly detection, and clustering, transforms in-situ sensor measurements into organized information that is easy for users to understand and access.

[0010] The information layer is used to provide a graphical user interface for end users, displaying abnormal events detected by the in-situ sensors and in-situ sensor readings for end users to query.

[0011] Furthermore, the workflow of the data collection layer includes:

[0012] Step 1.1: Predefine the data format and communication method of each in-situ sensor in the data collection layer interface;

[0013] Step 1.2: The data collection layer connects to multiple data sources from different observation stations and acquires data input;

[0014] Step 1.3: The data collection layer stores the acquired data in a centralized data repository using predefined communication methods.

[0015] Furthermore, the data processing layer implements abnormal event detection and clustering by executing abnormal event detection processes, abnormal feature extraction processes, event construction processes, event feature extraction processes, and event clustering processes.

[0016] Furthermore, the anomaly detection process is used to detect abnormal sensor values, and this process includes the following steps:

[0017] Step 2.1: Establish a trend model for measured values ​​and classify anomalies;

[0018] Define the trend model of the measured values ​​as B(t), where B(t) is composed of N measured values ​​B obtained from a data repository within a recent predetermined time period. i An array consisting of (t), i.e.:

[0019] B(t) = {B1(t),...,B i (t),...,B N (t)}

[0020] New measurement values ​​I(t) are retrieved from the data repository. The classification criteria for new measurement values ​​I(t) are as follows:

[0021]

[0022] Where, dist(I(t),B i (t) represents the new measured value I(t) and the known measured value B. i The distance between (t), min(dist(I(t),B) i(t))) represents the minimum distance between the new measurement value I(t) and all elements in B(t), and T(t) represents the distance threshold, I∈(1,2,...,N); if the new measurement value I(t) is 1, it is considered abnormal, and step 2.2 is executed;

[0023] Step 2.2: Calculate the distance Dist(), and classify anomalies based on the calculation results;

[0024]

[0025] Furthermore, in the anomaly detection process, if the new measured value I(t) is assigned a value of 0 and classified as a normal value, the process also includes updating the measured value trend model B(t) and updating the distance threshold T(t).

[0026] The method for updating the measured value trend model B(t) is as follows:

[0027] Step a. Randomly select B i (t)∈B(t)

[0028] Step b. Let B i (t) = I(t);

[0029] The method for updating the distance threshold T(t) is as follows:

[0030]

[0031] Among them, T inc / dec T is a static value that controls the threshold update rate; scale It is a fixed value; Represents the first N d min The average value of (t), d min (t)=min(dist(I(t),B i (t)));T lower T upper These are the upper and lower bounds of the threshold, respectively.

[0032] Furthermore, the anomaly feature extraction process is used to capture the similarity of detected anomalies and further cluster them into anomalous events. In this process, an anomaly feature set is set, defined as f:

[0033] f=[I(t-1)-I(t),I(t),I(t)-I(t+1),d min ,d min -T(t)]

[0034] The elements within f include: the difference between the previous sensor measurement I(t-1) and the current sensor measurement I(t), the current sensor measurement I(t), the current sensor measurement I(t) and the next sensor measurement I(t+1), and the minimum distance d between the sensor measurement and the measurement trend model. min minimum distance d min The distance between the distance threshold T(t) and the distance threshold T(t).

[0035] Furthermore, the event construction process groups the detected anomalies into events based on their time information using agglomerative hierarchical clustering, with the following grouping rules:

[0036] a. A series of anomalies will be combined into a single event;

[0037] b. If the time interval between a new exception and a previous exception is less than a preset fixed value, the new exception value will be merged into the same event; otherwise, a new event will be created.

[0038] Furthermore, the event feature extraction process employs the K-means clustering method.

[0039] Furthermore, the event clustering process is as follows:

[0040] Let the known set of events be E = {E1,...,E2}. num1}, where num1 represents the number of events, and the mapping function is: Let the input set X = E, and the mapping function... Set a threshold ε, N = 0, N max ;

[0041] Invoke the online robust agglomerative clustering method to obtain the event group set {y1,...,y}. num2}, where the subscript num2 represents the number of event groups.

[0042] Furthermore, the information layer provides a graphical user interface for end users, including a sensor reading display unit, a parameter setting unit, and an abnormal event display unit.

[0043] The beneficial effects of this invention are as follows: This invention proposes an intelligent sensing system framework for environmental monitoring based on machine learning technology, which has the following advantages compared with current environmental sensing technologies:

[0044] 1. High accuracy. The anomaly detection framework designed in this invention models the trends of known sensor measurements and can continuously adjust the decision threshold based on changes in the data stream, resulting in high accuracy.

[0045] 2. Suitable for large-scale deployment. The data repository designed in this invention can be a cloud data center, capable of addressing the increasing computational and storage resources required for data processing and storage as data volume grows. Furthermore, the anomaly detection framework designed in this invention is based on a modification of the MoPBAS unsupervised learning method. Compared to supervised learning methods, it has lower computational costs, consumes less power, and has lower maintenance costs, making it suitable for large-scale deployment. Attached image description:

[0046] Figure 1 This is a system framework diagram.

[0047] Figure 2 This is a diagram illustrating the implementation of the technical solution.

[0048] Figure 3 This is a diagram of the data layer framework.

[0049] Figure 4 This is a histogram of event features.

[0050] Figure 5 This is an example of a graphical user interface. Detailed implementation method:

[0051] The design concept of this invention is as follows: Based on machine learning and combined with current intelligent sensing technologies in various fields, this invention proposes an intelligent sensing system framework for environmental monitoring based on machine learning technology to address the problem of how to monitor the environment more broadly, reliably, and intelligently. By designing anomaly detection and clustering methods, machine learning technology is used to automatically process raw sensor measurements and convert them into organized information that is easy for end users to understand and access. This effectively solves the problems of limited scope, unreliability, and lack of intelligence in environmental monitoring, leading to a better understanding, management, and protection of ecosystems.

[0052] The following is combined with Figure 1-5 This embodiment will be described.

[0053] The system framework includes a data collection layer, a data processing layer, and an information layer.

[0054] I. Data Collection Layer

[0055] The data collection layer communicates with the observation stations at its front end and with the data processing layer at its back end. In this embodiment, the number of observation stations and the number of sensors equipped at each station can be dynamically expanded. The data collection layer extracts data from the in-situ sensors at each station using different communication methods (such as the Internet, mobile networks, radio waves, etc.), stores the data in a centralized data repository, and provides the data processing layer with specifically formatted data segments (such as JSON, CSV, XML, etc.).

[0056] The workflow of the data collection layer is as follows:

[0057] Step 1.1: Predefine the sensor's data format and communication method in the data collection layer interface. (Note: The data collection layer provides a standardized interface that can connect to various in-situ sensors. Each sensor has its own data and communication format for its sensing mode. For example, an in-situ sensor sends plain text measurement values ​​to the server via a GSM network.)

[0058] Step 1.2: The data collection layer connects to multiple data sources from different observation stations and obtains data input.

[0059] Step 1.3: The data collection layer stores the acquired data in a centralized data repository (such as a cloud data center) using predefined communication methods.

[0060] II. Data Processing Layer

[0061] The data processing layer extracts the required data segments from the data repository, processes them using advanced machine learning techniques, and transforms raw sensor measurements into organized information that is easy for end users to understand and access. This invention addresses the processing of data acquired from in-situ sensors, proposing an anomaly detection and clustering framework at the data processing layer.

[0062] The data processing layer's workflow includes anomaly detection, anomaly feature extraction, event construction, event feature extraction, and event clustering. These are explained below.

[0063] 1. Abnormal event detection process

[0064] Anomalies refer to unusual sensor measurements; anomalies are defined as sensor readings that differ significantly from recent observations. This embodiment uses trend modeling of known sensor measurements to detect anomalies.

[0065] The abnormal event detection process includes the following steps:

[0066] Step 2.1: Establish a trend model for measured values ​​and classify anomalies.

[0067] Define the trend model of the measured values ​​as B(t), where B(t) is composed of N measured values ​​B obtained from a data repository within a recent predetermined time period. i An array consisting of (t), i.e.:

[0068] B(t) = {B1(t),...,B i (t),...,B N (t)}

[0069] New measurement values ​​I(t) are retrieved from the data repository. The classification criteria for new measurement values ​​I(t) are as follows:

[0070]

[0071] Where, dist(I(t),B i (t) represents the new measured value I(t) and the known measured value B. i The distance between (t), min(dist(I(t),B) i (t))) represents the minimum distance between the new measurement value I(t) and all elements in B(t), T(t) represents the distance threshold, I∈(1,2,...,N), if the new measurement value I(t) is 1, it is considered abnormal and step 2.2 is executed;

[0072] If the new measurement I(t) is classified as a normal value (I(t) = 0), it can be used to update the measurement trend model B(t), with the update probability being the learning rate L(t).

[0073] The update process for the measured value trend model B(t) is as follows:

[0074] Step a. Randomly select B i (t)∈B(t)

[0075] Step b. Let B i (t)=I(t)

[0076] The process for updating the distance threshold T(t) is as follows:

[0077]

[0078] Among them, T inc / dec T is a static value that controls the threshold update rate. scale For a fixed value, it will Extends to the same range as T(t). l o wer T upper These are all fixed values, representing the upper and lower bounds of the threshold, respectively, so the threshold will not exceed the range. Represents the first N d min The average value of (t), d min (t)=min(dist(I(t),B i (t))).

[0079] The purpose of updating the distance threshold is to update the measurement trend model in order to achieve anomaly classification.

[0080] The process for updating the learning speed L(t) of the system framework is as follows:

[0081] The pixel-based adaptive segmenter (PBAS) is modified in this invention to obtain MOPBAS. The original learning rate is proposed as R(t), with an initial value generally in the range of (0,1). The learning rate L(t) defined in this embodiment is:

[0082]

[0083]

[0084] Where: If L(t) needs to be updated first, then L(t) needs to be calculated.

[0085] L inc A fixed value that increases at intervals;

[0086] L dec A fixed value that decreases at intervals;

[0087] Anomaly represents an anomaly. Anomaly = true is equivalent to I(t) = 1, and anomaly = false is equivalent to I(t) = 0.

[0088] R lower R(t) is a lower bound of R(t). upper R(t) is the upper bound of R(t). lower <R(t)<R upper

[0089] The purpose of updating the learning speed is to improve the performance of the trend measurement model. By adjusting the learning speed, the model's speed of processing data can be affected.

[0090] Step 2.2: Calculate the distance Dist() for anomaly classification;

[0091] In this embodiment, the distance Dist() is calculated as follows:

[0092]

[0093] 2. Anomaly Feature Extraction Process

[0094] The purpose of this process is to capture the similarity of detected anomalies and further cluster them into anomalous events. To this end, an anomaly feature set is proposed, defined as f:

[0095] f=[I(t-1)-I(t),I(t),I(t)-I(t+1),d min ,d min -T(t)]

[0096] f includes: the difference between the previous sensor measurement I(t-1) and the current sensor measurement I(t), the current sensor measurement I(t), the current sensor measurement I(t) and the next sensor measurement I(t+1), and the minimum distance d between the sensor measurement and the measurement trend model. min minimum distance d min The distance between the distance threshold T(t) and the distance threshold T(t).

[0097] 3. Event Construction Process

[0098] This process groups detected anomalies into events based on their time information using agglomerative hierarchical clustering, with the following grouping rules:

[0099] a. A series of anomalies will be combined into a single event.

[0100] b. The time interval between the new anomaly and the previous anomaly is less than T. gap (T gap If the value is a fixed value set during system initialization (the specific value is determined by the actual situation), then new abnormal values ​​will be merged into the same event; otherwise, a new event will be created.

[0101] 4. Event Feature Extraction Process

[0102] Before introducing the process, let's first explain the K-means clustering algorithm.

[0103] Data sample X = {X1,...,X} n}, where each object has m-dimensional attributes, namely X i ={a1,...,a m}. Randomly select k samples from X as initial cluster centers U = {μ1,...,μ k}, Let C = {C1,...,C} k The algorithm flow is as follows:

[0104] Repeat:

[0105] Step 0: Let

[0106] Step 4.1: Calculate X i ∈X and each cluster center μ j Distance ∈ U Where X it X represents i The t-th attribute, μ jt μ j Given the t-th attribute, we obtain the distance set Dis = {dis1, ..., dis...}k}

[0107] Step 4.2: When η = dis(X) i ,μλ) satisfies At that time, C λ =C λ ∪{X i} Imminent X i It should be classified into the corresponding cluster.

[0108] Step 4.3: Repeat Step 1 and Step 2 until all X are found. i All values ​​∈X are assigned to the corresponding cluster class.

[0109] Step 4.4: Update μ i : Where |C i | represents set C i The number of elements in, i.e., μ i ′ is cluster C i The average value of the elements in μ. i ′≠μ i Then let μ i For μ i ′, and vice versa.

[0110] Step 4.5: Repeat Step 4 until all μ are currently in use. i All ∈U are updated.

[0111] Until: Current U = {μ1,...,μ k All μ in} i The value remained unchanged.

[0112] The algorithm ultimately yields a cluster partition C = {C1,...,C}. k}, and cluster centers U={μ1,...,μ k}

[0113] Since each event may contain a different number of outliers, this invention aims to extract event features based on the K-means clustering method in order to quantify the similarity between events.

[0114] Let the abnormal feature set f = [I(t-1)-I(t),I(t),I(t)-I(t+1),d min ,d min -T(t)] is represented by the "sensor word" W, that is:

[0115] f = {W1, W2, W3, W4, W5}

[0116] W1 = I(t-1) - I(t)

[0117] W2=I(t)

[0118] W3 = I(t) - I(t+1)

[0119] W4=d min

[0120] W5=d min -T(t)

[0121] Given training data {f1,...,f n Event E i ={f q ,...,f k}, where f i Let n be an anomalous feature set, where the index n is equal to the number of anomalous feature sets.

[0122] The specific process is as follows:

[0123] a. For the training data, i.e. {f1,...,f n Perform K-means clustering and create a vocabulary List = {W} P ,...,W q}, which refers to the K-means cluster centers.

[0124] b. Regarding event E i Each set of anomalous features in f i ∈{f q ,...,f k Mapped to the word W in the vocabulary that is closest to it (i.e., the closest). j ∈List (value f) i Mapped to the word W in the nearest vocabulary. j ), that is, each exception f i Both are represented using code words, and the distance is calculated using dis(·,·).

[0125] c. Event E i The feature set is the code word {W} P ,...,W q The frequency of occurrence is used to create a histogram. For example... Figure 4 .

[0126] 5. Event Clustering Process

[0127] The kernel-induced distance will be explained below.

[0128] An input set X, a mapping function x i ∈X maps to a new space S with higher or infinite dimensions, i.e.

[0129] The kernel function is defined as an inner product in the new space S:

[0130]

[0131] Where x, y ∈ X, and <·, ·> are the inner product operations in the new space S, i.e.

[0132] Kernel-induced distance d(x,y) is defined as follows:

[0133]

[0134] The algorithm flow of the online robust agglomerative clustering method is as follows:

[0135] Step 1: Set a threshold ε to control the final number of clusters, and initialize the system with zero prototypes (usually a vector or a point, representing a set of data or a feature of the entire dataset): System prototype set prototypes = {y1,...,y...} N}, where N = 0 (initially, the system prototype set is empty), and the maximum number of prototypes in the predefined system is N. max .

[0136] Step 2: Input a new data point x, and define the prototype closest to x as... Updated to Winner's prototype y i =y winner and its weight c i =c winner , that is, y winner satisfy

[0137] c i =c winner =c winner +K(x,y winner )

[0138]

[0139] Step 3: If N < N max If N = N + 1, let γ = N, and proceed directly to Step 5. If N ≥ N max Then proceed to Step 4.

[0140] Step 4: Obtain the two closest prototypes Merge the two prototypes using the following formula, then proceed directly to Step 6:

[0141]

[0142] c γ =c γ +c δ

[0143] Step 5: Initialize prototype y γ =x, and its weight c γ =0

[0144] Step 6: If there is still data that needs to be clustered, go to Step 2;

[0145] Step 7: Delete all duplicates c i < ε's prototype y i .

[0146] The event clustering process is as follows:

[0147] This embodiment uses an online robust agglomerative clustering method to cluster and group events, given an event set E = {E1, ..., E...}. num1}, where the subscript num1 represents the number of events, and the mapping function is... The specific process is as follows:

[0148] Step 5.1: Let the input set X = E, and the mapping function... Set a threshold ε, N = 0, N max ;

[0149] Step 5.2: Call the online robust agglomerative clustering method to obtain the event group set {y1,…,y}. num2}, where the subscript num2 represents the number of event groups.

[0150] III. Information Layer

[0151] The information layer provides a graphical user interface (GUI) for end users, displaying abnormal events detected by the in-situ sensors and their readings. End users can query recent abnormal events. Based on the information provided, operators can send feedback to the deployed system, such as increasing or decreasing the sampling rate.

[0152] The graphical user interface of the present invention is as follows: Figure 5 As shown, it includes a sensor reading display unit, a parameter setting unit, and an abnormal event display unit. This embodiment does not limit the graphical user interface, allowing users to adjust and set components according to actual needs.

Claims

1. An intelligent sensing system for environmental monitoring based on machine learning technology, characterized in that, It includes a data collection layer, a data processing layer, and an information layer; The front end of the data collection layer is connected to the observation stations for environmental monitoring, while the back end is connected to the data processing layer. It is used to extract data from the in-situ sensors of each observation station, store the data in the data storage repository, and provide the data processing layer with specifically formatted data segments. The data processing layer extracts the required data segments from the data repository and, through machine learning, anomaly detection, and clustering, transforms in-situ sensor measurements into organized information that is easy for users to understand and access. The information layer is used to provide a graphical user interface for end users, displaying abnormal events detected by the in-situ sensors and in-situ sensor readings for end users to query. The data processing layer achieves anomaly event detection and clustering by executing anomaly event detection process, anomaly feature extraction process, event construction process, event feature extraction process, and event clustering process; The anomaly detection process is used to detect abnormal sensor values. This process includes the following steps: Step 2.1: Establish a trend model for measured values ​​and classify anomalies; Define the trend model of the measured values ​​as B(t), where B(t) is composed of N measured values ​​B obtained from a data repository within a recent predetermined time period. i An array consisting of (t), i.e.: B(t) = {B1(t),...,B i (t),...,B N The new measurement value I(t) is obtained from the data repository. The classification criteria for the new measurement value I(t) are as follows: ; Where, dist(I(t),B i (t) represents the new measured value I(t) and the known measured value B. i The distance between (t), min(dist(I(t),B) i (t))) represents the minimum distance between the new measurement value I(t) and all elements in B(t), and T(t) represents the distance threshold, I∈(1,2,...,N); if the new measurement value I(t) is 1, it is considered abnormal, and step 2.2 is executed; Step 2.2: Calculate the distance Dist(), and classify anomalies based on the calculation results; ； The anomaly feature extraction process is used to capture the similarity of detected anomalies and further cluster them into anomalous events. In this process, an anomaly feature set is set, defined as f: f=[I(t-1)-I(t),I(t),I(t)-I(t+1),d min ,d min -T(t)] The elements within f include: the difference between the previous sensor measurement I(t-1) and the current sensor measurement I(t), the current sensor measurement I(t), the current sensor measurement I(t) and the next sensor measurement I(t+1), and the minimum distance d between the sensor measurement and the measurement trend model. min minimum distance d min The distance between the distance and the distance threshold T(t); The event building process groups detected anomalies into events based on their time information using agglomerative hierarchical clustering, with the following grouping rules: a. A series of anomalies will be combined into a single event; b. If the time interval between a new exception and a previous exception is less than a preset fixed value, the new exception value will be merged into the same event; otherwise, a new event will be created. The event feature extraction process uses the K-means clustering method; The event clustering process is as follows: Let the known set of events be E = {E1,...,E2}. num1 }, where num1 represents the number of events, and the mapping function is: Let the input set X = E, and the mapping function... Set a threshold ε, N = 0, N max ; Invoke the online robust agglomerative clustering method to obtain the event group set {y1,...,y}. num2 }, where the subscript num2 represents the number of event groups.

2. The intelligent sensing system for environmental monitoring based on machine learning technology as described in claim 1, characterized in that, The workflow of the data collection layer includes: Step 1.1: Predefine the data format and communication method of each in-situ sensor in the data collection layer interface; Step 1.2: The data collection layer connects to multiple data sources from different observation stations and acquires data input; Step 1.3: The data collection layer stores the acquired data in a centralized data repository using predefined communication methods.

3. The intelligent sensing system for environmental monitoring based on machine learning technology as described in claim 1, characterized in that, In the anomaly detection process, if the new measured value I(t) is assigned a value of 0 and classified as a normal value, the process also includes updating the measured value trend model B(t) and updating the distance threshold T(t). The method for updating the measured value trend model B(t) is as follows: Step a. Randomly select B i (t)∈B(t) Step b. Let B i (t) = I(t); The method for updating the distance threshold T(t) is as follows: Among them, T inc / dec It is a static value that controls the threshold update rate; T scale It is a fixed value; Represents the first N d min The average value of (t), d min (t)=min(dist(I(t),B i (t)));T lower T upper These are the upper and lower bounds of the threshold, respectively.

4. The intelligent sensing system for environmental monitoring based on machine learning technology as described in claim 1, characterized in that, The information layer provides a graphical user interface for end users, including a sensor reading display unit, a parameter setting unit, and an abnormal event display unit.

Images