Water quality early warning and decision-making method based on machine learning and question and answer model

Abstract

The invention provides a water quality early warning and decision-making method based on machine learning and a question and answer model, and the method comprises the steps: obtaining water quality monitoring data and water quality knowledge text data, carrying out the preprocessing of the obtained data, and obtaining a monitoring data set and a text data set; using the monitoring data set to construct a water quality prediction model, and predicting the monitoring data of the next time period through the water quality prediction model; constructing an abnormal value detection model by using the monitoring data set and adopting an isolation forest algorithm, inputting the monitoring data of the next time period into the abnormal value detection model, and outputting an abnormal score; performing water quality early warning based on the abnormal score; and constructing a question-answering model by using the text data set, inputting the user question into the question-answering model, and outputting a question-answering result. By adopting the scheme, water quality early warning and decision making are realized.

Description

Technical Field

[0001] The present application relates to the field of water quality early warning technology, and in particular to a water quality early warning and decision-making method and device based on machine learning and question-answering models. Background Art

[0003] Existing water quality early warning and prediction models mainly rely on traditional monitoring data and mathematical statistical methods, with time series analysis, regression analysis and other methods as the core to predict trends and detect anomalies in water quality parameters. These models usually use historical monitoring data to establish mathematical models to predict the future state of water bodies and set thresholds to determine whether to trigger early warning signals.

[0004] In terms of operation, the model first needs to collect and process water quality monitoring data, including key indicators such as pH, dissolved oxygen, turbidity, organic matter content, etc. Then, statistical or machine learning methods are used to perform trend analysis and pattern recognition on these data in order to issue an early warning before a pollution incident occurs.

[0005] Although traditional models can effectively predict and warn of water quality problems, they usually lack sufficient flexibility and adaptability through simple fixed threshold settings, making it difficult to cope with the challenges of nonlinear and non-stationary water quality changes to water quality warnings, and often ignore the complex interactions within the water system and the impact of external environmental factors. In addition, these models still have a lot of room for improvement in prediction accuracy and warning timeliness, especially in the face of sudden water pollution incidents, and often find it difficult to provide timely and effective decision support.

[0006] Currently, intelligent water quality monitoring mainly adopts the following methods: (1) Using Internet of Things technology to realize real-time data collection of water quality monitoring points, including key indicators such as pH value, dissolved oxygen, turbidity, etc., and then setting thresholds according to standard specifications, and automatically alarming when the thresholds are exceeded; (2) Applying machine learning algorithms to conduct in-depth analysis of the collected data, predict water quality change trends through models, detect abnormal fluctuations in time, and trigger early warning mechanisms.

[0007] In addition, existing decision support methods are usually based on keyword retrieval, which has many limitations, such as the inability to understand knowledge information, lack of dynamic interaction, and lack of understanding of the text as a whole.

[0008] Current water pollution decision support systems usually rely on rule-driven and keyword-searching methods. Such systems are usually built on a fixed knowledge base, which contains predefined rules and logic for parsing input queries and retrieving relevant information. For example, an environmental monitoring system may trigger an alert when the concentration of a specific pollutant exceeds a threshold, and use keyword matching methods to find relevant treatment measures and policy guidelines in the knowledge base. Summary of the invention

[0009] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0010] To this end, the first purpose of this application is to propose a water quality early warning and decision-making method based on machine learning and question-answering model to achieve water quality early warning and decision-making.

[0011] The second purpose of this application is to propose a water quality warning and decision-making device based on machine learning and question-answering model.

[0012] To achieve the above objectives, the first embodiment of the present application proposes a water quality early warning and decision-making method based on machine learning and question-answering model, including:

[0013] Acquire water quality monitoring data and water quality knowledge text data, and preprocess the acquired data to obtain a monitoring data set and a text data set;

[0014] Use the monitoring data set to build a water quality prediction model, and use the water quality prediction model to predict the monitoring data for the next period;

[0015] Using the monitoring data set, the isolation forest algorithm is used to build an outlier detection model, the monitoring data of the next period is input into the outlier detection model, and the anomaly score is output;

[0016] Water quality warning based on anomaly scores;

[0017] Use text datasets to build a question-answering model, input user questions into the question-answering model, and output question-answering results.

[0018] The water quality warning and decision-making method based on machine learning and question-answering model in the embodiment of the present application pre-processes multimodal data including water quality time series data, spatial data, and text data, and uses the ARX regression model and isolation forest algorithm to build a comprehensive prediction and warning system to achieve comprehensive and accurate alarms. The question-answering model is then built through natural language processing technology to add an additional decision-making layer to the warning model. Once the warning model gives an alarm, the intelligent question-answering model automatically provides decision makers with relevant knowledge of water quality pollution problems in the form of questions and answers, which is of great significance to improving the scientificity, intelligence, and user-friendliness of decision-making.

[0019] Optionally, in one embodiment of the present application, preprocessing the water quality monitoring data includes:

[0020] Clean the water quality monitoring data, delete the unimportant data with missing values ​​greater than the threshold, and use the filling method to fill in the missing values ​​less than the threshold or important data;

[0021] Manually mark abnormal data in the cleaned data, and check or delete the marked data to ensure the data is authentic;

[0022] Standardize the data;

[0023] The water quality knowledge text data is preprocessed to obtain a text data set, including:

[0024] Segment text knowledge data and break sentences into separate words or phrases;

[0025] Clean the text knowledge data, remove useless data, and obtain a water quality text database;

[0026] Constructing question-answer pairs from the water quality text knowledge base through manual annotation, where the question-answer pairs include questions and answers;

[0027] The water quality text knowledge base and question-answer pairs are combined into a text dataset so that each question and answer corresponds one-to-one with the corresponding knowledge base entry.

[0028] Optionally, in one embodiment of the present application, the water quality prediction model is an ARX regression model, and the water quality prediction model is constructed using the monitoring data set, including:

[0029] Divide the monitoring data set according to time periods to determine the input period and prediction period of the water quality prediction model;

[0030] The ARX regression model is trained based on the data of the divided input time period and the data of the prediction time period. During training, the data of the input time period is input into the ARX regression model, and the monitoring data of the prediction time period is predicted. During training, the Bayesian information criterion is used to determine the order of the ARX regression model. Through training, the order of the ARX regression model is continuously adjusted to reach the optimal order, thereby obtaining the optimal parameters of the water quality prediction model.

[0031] Optionally, in one embodiment of the present application, the monitoring data of the next period is predicted by an ARX regression model, which is expressed as:

[0032] Y t =A1Y t-1 +A2Y t-2 +A p Y t-p +BX t +∈ t

[0033] Among them, Y t is the monitoring data at time t, Y t-1 is the monitoring data at time t-1, A p is the autoregressive coefficient, X t is an exogenous variable, A1, A2, and B are coefficients;

[0034] The order of the ARX regression model is determined using the Bayesian information criterion, expressed as:

[0035]

[0036] in, is the maximum value of the likelihood function of the estimated model M, is the parameter value that maximizes the likelihood function, x is the monitoring data, n is the number of data points in the monitoring data, k is the number of regression coefficients, and p is the optimal order of the ARX regression model when BIC is minimized.

[0037] Optionally, in one embodiment of the present application, a monitoring data set is used and an isolation forest algorithm is adopted to construct an outlier detection model, including:

[0038] Randomly select subsamples from the monitoring data set and build isolation trees. Each tree recursively divides the data by random selection until the set tree height limit is reached;

[0039] Adjust the parameters of the isolation tree to optimize the model’s anomaly detection performance and obtain an anomaly detection isolation forest model;

[0040] The monitoring data is input into the anomaly detection isolation forest model, and the anomaly score is output, including:

[0041] The anomaly detection isolation forest model is used to evaluate the samples in any period of the monitoring dataset, and the average path length of each sample in each isolation tree is calculated;

[0042] The anomaly score of each sample is calculated by the average path length of each sample in each isolation tree.

[0043] Optionally, in one embodiment of the present application, t isolation trees are constructed, which are expressed as:

[0044] iForest={t1,2,…, T}

[0045]

[0046] Among them, l is the tree height limit, j is the subsample size;

[0047] The average path length of each sample in each isolation tree is:

[0048]

[0049] Where T is the number of trees, h t (x) is the path length;

[0050] The anomaly score of sample x is expressed as:

[0051]

[0052] Among them, c(j) is the average path length of the binary search tree based on subsamples, and c(j) is expressed as:

[0053]

[0054] Where j is the subsample size and H(i) is the number of harmonics with sampling size i.

[0055] Optionally, in one embodiment of the present application, water quality early warning is performed based on the abnormality score, including:

[0056] When the average path length is close to 0 and the anomaly score is close to -0.5, the sample is considered abnormal;

[0057] When the average path length is close to the sample size n-1 and the anomaly score distance is close to 0.5, the sample is considered normal;

[0058] When the average path length is close to c(j) and the anomaly score is close to 0, the sample is considered to be suspected of being abnormal.

[0059] Optionally, in one embodiment of the present application, a question-answering model is constructed using a text dataset, including:

[0060] Convert water quality knowledge question-answer pairs into vectors;

[0061] Input user questions into the question-answering model and output the question-answering results, including:

[0062] Convert user questions into vectors;

[0063] Use a similarity function to calculate the similarity between the question vector and the water quality knowledge question-answer pair vector;

[0064] For user questions, relevant water quality information items are screened based on similarity scores and negative sampling contrast losses;

[0065] Use the generator to generate answers based on relevant water quality information items.

[0066] Optionally, in one embodiment of the present application, the question vector is expressed as:

[0067] h q =ool(E(q;θ1))

[0068] The vector of water quality question-answer pair is represented as:

[0069] h p =pool(E(p;θ2))

[0070] Among them, e(;θ1) is a model used to convert the input text into a vector representation, and pool is a pooling function that extracts a fixed-length vector from the output of the model;

[0071] The similarity s(q,p) between vectors is expressed as:

[0072] s(q,p)=sin(h q ,h p )

[0073] Relevant water quality information items are screened based on similarity scores and negative sampling contrast losses, expressed as:

[0074]

[0075] in, is a set of water quality information items that are not related to the problem, and τ is a temperature parameter used to adjust the smoothness of the softmax function;

[0076] The task of the generator is to maximize the conditional probability p(a|q,P + ), from the water quality information item P + The optimal result is selected from the generator, which minimizes the negative log-likelihood, expressed as:

[0077]

[0078] Among them, a is the answer sequence, a t is the tth word in the sequence, q is the question, and θ is the model parameter.

[0079] To achieve the above-mentioned purpose, the second embodiment of the present invention proposes a water quality early warning and decision-making device based on machine learning and question-answering model, comprising:

[0080] The data acquisition module is used to acquire water quality monitoring data and water quality knowledge text data, and pre-process the acquired data to obtain monitoring data sets and text data sets;

[0081] The first model building module is used to build a water quality prediction model using the monitoring data set, and predict the monitoring data of the next period through the water quality prediction model;

[0082] The second model building module is used to use the monitoring data set and the isolation forest algorithm to build an outlier detection model, input the monitoring data of the next period into the outlier detection model, and output an anomaly score;

[0083] Early warning module, used for water quality early warning based on abnormal scores;

[0084] The question-answering module is used to build a question-answering model using a text dataset, input user questions into the question-answering model, and output question-answering results.

[0085] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0087] Figure 1 A flow chart of a water quality early warning and decision-making method based on machine learning and question-answering model provided in Example 1 of the present application;

[0088] Figure 2 A flow chart of a water quality early warning and decision-making method based on machine learning and question-answering model provided in Example 2 of the present application;

[0089] Figure 3 This is a schematic diagram of an abnormal warning process based on prediction and real-time data according to the second embodiment of the present application;

[0090] Figure 4 This is a schematic diagram of the question-answering model assisted decision-making process of Embodiment 2 of the present application;

[0091] Figure 5 A schematic diagram of the structure of a water quality warning and decision-making device based on machine learning and question-answering models provided in an embodiment of the present application. DETAILED DESCRIPTION

[0092] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0093] The following describes the water quality warning and decision-making method and device based on machine learning and question-answering model in an embodiment of the present application with reference to the accompanying drawings.

[0094] Figure 1 A flow chart of a water quality warning and decision-making method based on machine learning and question-answering model provided in Example 1 of the present application.

[0095] like Figure 1 As shown, the water quality early warning and decision-making method based on machine learning and question-answering model includes the following steps:

[0096] Step 101, obtaining water quality monitoring data and water quality knowledge text data, and preprocessing the obtained data to obtain a monitoring data set and a text data set;

[0097] Step 102, constructing a water quality prediction model using the monitoring data set, and predicting the monitoring data for the next period of time through the water quality prediction model;

[0098] Step 103, using the monitoring data set and the isolation forest algorithm to build an outlier detection model, input the monitoring data of the next period into the outlier detection model, and output an anomaly score;

[0099] Step 104, issuing a water quality warning based on the abnormality score;

[0100] Step 105, use the text data set to build a question-answering model, input the user's question into the question-answering model, and output the question-answering result.

[0101] The water quality warning and decision-making method based on machine learning and question-answering model in the embodiment of the present application pre-processes multimodal data including water quality time series data, spatial data, and text data, and uses the ARX regression model and isolation forest algorithm to build a comprehensive prediction and warning system to achieve comprehensive and accurate alarms. The question-answering model is then built through natural language processing technology to add an additional decision-making layer to the warning model. Once the warning model gives an alarm, the intelligent question-answering model automatically provides decision makers with relevant knowledge of water quality pollution problems in the form of questions and answers, which is of great significance to improving the scientificity, intelligence, and user-friendliness of decision-making.

[0102] Figure 2 A flow chart of a water quality early warning and decision-making method based on machine learning and question-answering model provided in Example 2 of the present application is disclosed. Figure 3 This is a schematic diagram of an abnormal warning process based on prediction and real-time data in Example 2 of the present application. Figure 4 This is a schematic diagram of the question-and-answer model assisted decision-making process of Example 2 of the present application.

[0103] like Figure 2 , Figure 3 , Figure 4 As shown, the method includes:

[0104] S1, collect and organize water quality monitoring data, spatial data and water quality knowledge text data.

[0105] S2, based on the data set obtained in S1, performs data preprocessing, including data cleaning, outlier processing, manual labeling, etc., to ensure that the data quality meets the requirements of subsequent model training and construct monitoring data and text data sets.

[0106] S3, builds a water quality early warning model based on the data obtained in S2, uses the ARX regression model to build a multivariate time series prediction model for spatial water quality changes, and predicts the monitoring data of the next period through the monitoring values ​​of the historical period.

[0107] S4, based on the prediction model obtained in S3, uses the isolation forest algorithm to build an outlier detection model that can provide early warnings for current and future water quality.

[0108] S5, build a question-answering model, combining water pollution-related knowledge with natural language processing technology, so that the model can understand queries in natural language and provide accurate answers. The construction of the knowledge base can be based on expert experience, historical case analysis, etc.

[0109] Specifically, S1 includes the following steps:

[0110] S101: Collect and organize water quality monitoring data, including but not limited to time series data of key indicators such as pH value, dissolved oxygen, turbidity, and concentration of harmful substances, as well as spatial environmental data corresponding to the monitoring data, mainly including geographical location information of water bodies, watershed characteristics, rainfall, temperature, etc., to establish a comprehensive data set reflecting water quality monitoring data and pollution information in multiple sections in space and multiple time periods.

[0111] S102: Collect and organize text data for training the question-answering model, mainly including detailed records of water pollution events that have occurred in history, laws and regulations related to water quality protection, scientific papers and academic reports on solving water pollution problems, etc.

[0112] Specifically, S2 includes the following steps:

[0113] S201: Clean the water quality monitoring data and spatial environmental data obtained in S101, mainly processing blank missing values. For time periods or records with many missing values, directly delete these data; for time periods or records with few and important missing values, consider using interpolation methods or other reasonable filling methods to complete them.

[0114] S202: Manually mark the data obtained in S201 for outliers, manually mark the data with obvious abnormalities, and then conduct a unified inspection or delete them to ensure the authenticity of the data.

[0115] S203: Standardize the data obtained in S202 to obtain a standardized data set for training the prediction and early warning model, such as a standardized data set for predicting pH value and a standardized data set for predicting dissolved oxygen, etc. These data sets contain data from different time periods and different sections to accurately predict changes in water quality.

[0116] S204: Preprocess the text data obtained in S102 to obtain a text data set.

[0117] Further, the specific operation of step S204 includes the following steps:

[0118] S2041: Segment the text data organized in S102, and decompose the sentences into separate words or phrases.

[0119] S2042, clean the text obtained in S2041, remove HTML tags and some stop words that are not helpful for the model to understand the text topic, and obtain a water quality text knowledge base.

[0120] S2043, construct question-answer pairs from the knowledge base obtained from S2043 through manual annotation, for example: a) Question: How to quickly adjust the abnormal pH value? Answer: Methods to increase the pH value include adding sodium carbonate, lime, etc.; methods to lower the pH value include adding hydrochloric acid and using carbon dioxide; b) Question: What factors affect the dissolved oxygen level? Answer: The main factors are water temperature, water flow, organic matter load...; c) Question: For heavy metal pollution incidents, how far do pollutants spread in water bodies and how big is the impact? Answer: The spread and impact of heavy metal pollution depend on many factors, including the amount and type of pollution sources, the speed and direction of water flow, the degree of mixing of water bodies, and the adsorption capacity of bottom mud...; d) How does pesticide pollution affect groundwater systems and how big is its spread? Answer: Some chemicals in pesticides may have high solubility and low adsorption capacity, allowing them to move quickly in groundwater, thereby affecting a wide area. In addition, the half-life of pesticides is also an important factor...

[0121] S2044, combine the water quality text knowledge base obtained in S2043 and the question-answer pairs obtained in S2043 into a text dataset, ensuring that each question-answer pair corresponds one-to-one with the corresponding knowledge base entry. Save the question-answer pairs as a JSON format file, including the "question" and "answer" fields, for example:

[0122] [{"question":"How to quickly adjust abnormal pH value?","answer":"Adding sodium carbonate, lime, etc. are ways to increase pH value; adding hydrochloric acid and using carbon dioxide are ways to lower pH value."},{"question":"What factors affect the dissolved oxygen level?","answer":"The main factors are water temperature, water flow, organic matter load..."},{"question":"For heavy metal pollution incidents, how far do pollutants diffuse in water bodies and how big is their impact?","answer":"The diffusion range and impact of heavy metal pollution depend on many factors, including the amount and type of pollution source, the speed and direction of water flow, the mixing degree of water bodies, and the adsorption capacity of bottom mud..."},{"question":"How does pesticide pollution affect groundwater systems and how far is its diffusion range?","answer":"Some chemicals in pesticides may have high solubility and low adsorption capacity, allowing them to move quickly in groundwater, thus affecting a wide area. In addition, the half-life of pesticides is also an important factor..."}].

[0123] Specifically, S3 includes the following steps:

[0124] S301: Divide the monitoring data set obtained in S2 according to time periods, determine the input period and prediction period of the prediction model, and divide the training set and test set.

[0125] S302: Input the data set obtained in S301 into the ARX regression model to analyze the monitoring data Y in the future period. t predict:

[0126] Y t =1Y t-1 +2Y t-2 + p Y t-p +X t +∈ t

[0127] Among them, Y t is the target variable to be predicted, p is the order of the ARX model, and Y t-p is the historical monitoring value, A p is the autoregressive coefficient, X t is an exogenous variable, B is the coefficient of the exogenous variable, ∈ t is the error term.

[0128] S303: Determine the order p of the ARX autoregressive model using the Bayesian Information Criterion. When the BIC reaches the minimum value, p is the optimal order of the ARX model:

[0129]

[0130]

[0131] in, is the maximum value of the likelihood function of the estimated model M; is the parameter value that maximizes the likelihood function, x is the monitoring data, n is the number of data points in the monitoring data (sample size), and k is the number of regression coefficients.

[0132] S304: Input the historical data training set divided in S301 into the ARX model constructed in S303 for training, and repeat S302-S303 to obtain the optimal parameters of the prediction model.

[0133] Specifically, S4 includes the following steps:

[0134] S401: Randomly select subsamples or real-time monitoring data from the training set obtained in S301, and construct t isolation trees (iForest). Each tree recursively divides the data by random selection until the set tree height limit l is reached:

[0135] iForest={t1,2,…, T}

[0136]

[0137] Here, j is the subsample size.

[0138] S402: Adjust the parameters of the isolation tree in S401 to optimize the model anomaly detection performance.

[0139] S403: Use the anomaly detection isolation forest model obtained in S402 to evaluate the sample x in a certain period of time, and calculate the average path length E(h(x)) of each sample in each isolation tree:

[0140]

[0141] Where T is the number of trees, h t (x) is the path length.

[0142] S404: The abnormality level of each sample x can be obtained by calculating the abnormality score s(x,j):

[0143]

[0144] Where j is the subsample size, c(j) is the average path length of the binary search tree based on the subsample, and H(i) is the harmonic number with sampling size i (approximately H(i)≈ln(i)+0.5772156649).

[0145] S405: Based on the anomaly score s(x,j) of each sample obtained in S404, the following conditions are used to implement early warning of the sample: a) When the sample average distance to the isolation tree distance E(h(x)) is close to 0 and the anomaly score s(x,j) is close to -0.5, the sample is judged to be abnormal; b) When the sample average distance to the isolation tree distance E(h(x)) is close to the sample size n-1 and the anomaly score s(x,j) is close to 0.5, the sample is judged to be normal; c) When the sample average distance to the isolation tree distance E(h(x)) is close to the theoretical average path c(j) and the anomaly score s(x,j) is close to 0, the sample cannot be judged as abnormal.

[0146] S406: Based on the abnormal samples given in S405, further early warning forecasts are given.

[0147] Specifically, S5 includes the following steps:

[0148] S501: Convert the water quality knowledge question and answer pairs obtained in S2 into vector h q and h p :

[0149] h q =ool(E(q;θ1))

[0150] h p =pool(E(p;θ2))

[0151] Among them, E(;θ1) is a model used to convert the input text into a vector representation, and pool is a pooling function that extracts a fixed-length vector from the output of the model.

[0152] S502, using a similarity function to calculate the similarity s(q,p) between the question vector and the water quality information vector:

[0153] s(q,p)=sin(h q ,h p )

[0154] S503, for each question, screen relevant water quality information items based on similarity scores and negative sampling contrast losses to optimize the retrieval model and improve the accuracy of information screening:

[0155]

[0156] in, is a set of water quality information items that are not related to the problem, and τ is a temperature parameter used to adjust the smoothness of the softmax function.

[0157] S504, generate an answer based on the relevant water quality information retrieved in S503. The task of the generator is to maximize the conditional probability p(a|q,P + ), from the water quality information item P + Select the most suitable one as the final result. To achieve this goal, the method of minimizing negative log-likelihood is used:

[0158]

[0159] Among them, a is the answer sequence, a t is the tth word in the sequence, q is the question, P + is a set of related water quality information items, where θ is a model parameter.

[0160] S505, based on S504, a water quality information question-and-answer model is obtained. According to the pollution type given by the early warning model in S406, the question-and-answer model can provide a large number of questions and answers about pollution to assist scientific decision-making.

[0161] In order to implement the above embodiments, the present application also proposes a water quality warning and decision-making device based on machine learning and question-answering model.

[0162] Figure 5 A schematic diagram of the structure of a water quality warning and decision-making device based on machine learning and question-answering models provided in an embodiment of the present application.

[0163] like Figure 5 As shown, the water quality early warning and decision-making device based on machine learning and question-answering model includes:

[0164] The data acquisition module is used to acquire water quality monitoring data and water quality knowledge text data, and pre-process the acquired data to obtain monitoring data sets and text data sets;

[0165] The first model building module is used to build a water quality prediction model using the monitoring data set, and predict the monitoring data of the next period through the water quality prediction model;

[0166] The second model building module is used to use the monitoring data set and the isolation forest algorithm to build an outlier detection model, input the monitoring data of the next period into the outlier detection model, and output an anomaly score;

[0167] Early warning module, used for water quality early warning based on abnormal scores;

[0168] The question-answering module is used to build a question-answering model using a text dataset, input user questions into the question-answering model, and output question-answering results.

[0169] It should be noted that the aforementioned explanation of the embodiment of the water quality warning and decision-making method based on machine learning and question-answering model is also applicable to the water quality warning and decision-making device based on machine learning and question-answering model of this embodiment, and will not be repeated here.

[0170] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0171] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0172] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0173] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0174] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0175] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0176] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0177] The storage medium mentioned above may be a read-only memory, a disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A water quality early warning and decision-making method based on machine learning and question-answering model, characterized in that: include: Acquire water quality monitoring data and water quality knowledge text data, and preprocess the acquired data to obtain a monitoring data set and a text data set; Use the monitoring data set to build a water quality prediction model, and use the water quality prediction model to predict the monitoring data for the next period; Using the monitoring data set and the isolation forest algorithm, an outlier detection model is constructed, the monitoring data of the next period is input into the outlier detection model, and an anomaly score is output; Providing water quality warning based on the abnormal score; A question-answering model is constructed using the text data set, user questions are input into the question-answering model, and question-answering results are output.

2. The method according to claim 1, characterized in that Preprocess water quality monitoring data, including: Clean the water quality monitoring data, delete the unimportant data with missing values ​​greater than a threshold, and use a filling method to fill in the missing values ​​less than a threshold or important data; Manually mark abnormal data in the cleaned data, and check or delete the marked data to ensure the data is authentic; Standardize the data; The water quality knowledge text data is preprocessed to obtain a text data set, including: Segment text knowledge data and break sentences into separate words or phrases; Clean the text knowledge data, remove useless data, and obtain a water quality text database; Constructing a question-answer pair from the water quality text knowledge base through manual annotation, wherein the question-answer pair includes a question and an answer; The water quality text knowledge base and question-answer pairs are combined into a text data set so that each question and answer corresponds one-to-one with a corresponding knowledge base entry.

3. The method according to claim 2, characterized in that The water quality prediction model is an ARX regression model, which is constructed using the monitoring data set, including: Dividing the monitoring data set according to time periods to determine the input time period and prediction time period of the water quality prediction model; The ARX regression model is trained based on the data of the divided input time period and the data of the prediction time period. During training, the data of the input time period is input into the ARX regression model, and the monitoring data of the prediction time period is predicted. During training, the Bayesian information criterion is used to determine the order of the ARX regression model. Through training, the order of the ARX regression model is continuously adjusted to reach the optimal order, thereby obtaining the optimal parameters of the water quality prediction model.

4. The method according to claim 3, characterized in that The monitoring data for the next period is predicted by the ARX regression model, which is expressed as: Y t =A1Y t-1 +A2Y t-2 +A p Y t-p +BX t +∈ t Among them, Y t is the monitoring data at time t, Y t-1 is the monitoring data at time t-1, A p is the autoregressive coefficient, X t is an exogenous variable, A1, A2, and B are coefficients; The order of the ARX regression model is determined using the Bayesian information criterion, expressed as: in, is the maximum value of the likelihood function of the estimated model M, is the parameter value that maximizes the likelihood function, x is the monitoring data, n is the number of data points in the monitoring data, k is the number of regression coefficients, and p is the optimal order of the ARX regression model when BIC is minimized.

5. The method according to claim 1, characterized in that The monitoring data set is used to construct an outlier detection model using the isolation forest algorithm, including: Randomly select subsamples from the monitoring data set and build isolation trees. Each tree recursively divides the data by random selection until the set tree height limit is reached; Adjust the parameters of the isolation tree to optimize the model’s anomaly detection performance and obtain an anomaly detection isolation forest model; The monitoring data is input into the anomaly detection isolation forest model, and the anomaly score is output, including: The anomaly detection isolation forest model is used to evaluate the samples in any period of the monitoring dataset, and the average path length of each sample in each isolation tree is calculated; The anomaly score of each sample is calculated by the average path length of each sample in each isolation tree.

6. The method according to claim 5, characterized in that Construct t isolation trees, expressed as: iForest={t1,t2,…,t T } Among them, l is the tree height limit, j is the subsample size; The average path length of each sample in each isolation tree is: Where T is the number of trees, h t (x) is the path length; The anomaly score of sample x is expressed as: Among them, c(j) is the average path length of the binary search tree based on subsamples, and v(j) is expressed as: Where j is the subsample size and H(i) is the number of harmonics with sampling size i.

7. The method according to claim 6, characterized in that Water quality warnings based on abnormal scores, including: When the average path length is close to 0 and the anomaly score is close to -0.5, the sample is considered abnormal; When the average path length is close to the sample size n-1 and the anomaly score distance is close to 0.5, the sample is considered normal; When the average path length is close to c(j) and the anomaly score is close to 0, the sample is considered to be suspected of being abnormal.

8. The method according to claim 2, characterized in that The step of using the text dataset to construct a question-answering model includes: Convert water quality knowledge question-answer pairs into vectors; The step of inputting the user question into the question-answering model and outputting the question-answering result comprises: Convert user questions into vectors; Use a similarity function to calculate the similarity between the question vector and the water quality knowledge question-answer pair vector; For user questions, relevant water quality information items are screened based on similarity scores and negative sampling contrast losses; Use the generator to generate answers based on relevant water quality information items.

9. The method according to claim 8, characterized in that The problem vector is represented as: h q =pool(E(q;θ1)) The water quality knowledge question-answer pair vector is represented as: h p =pool(E(p;θ2)) Among them, E(;θ1) is a model used to convert the input text into a vector representation, and pool is a pooling function that extracts a fixed-length vector from the output of the model; The similarity s(q,p) between vectors is expressed as: s(q,p)=sin(h q ,h p ) Relevant water quality information items are screened based on similarity scores and negative sampling contrast losses, expressed as: in, is a set of water quality information items that are not related to the problem, and τ is a temperature parameter used to adjust the smoothness of the softmax function; The task of the generator is to maximize the conditional probability p(a|q,P + ), from the water quality information item P + The optimal result is selected from the generator, which adopts the method of minimizing the negative log-likelihood, which is expressed as: Among them, a is the answer sequence, a t is the tth word in the sequence, q is the question, and θ is the model parameter.

10. A water quality early warning and decision-making device based on machine learning and question-answering model, characterized in that: include: The data acquisition module is used to acquire water quality monitoring data and water quality knowledge text data, and pre-process the acquired data to obtain monitoring data sets and text data sets; The first model building module is used to build a water quality prediction model using the monitoring data set, and predict the monitoring data of the next period through the water quality prediction model; A second model building module is used to use the monitoring data set and the isolation forest algorithm to build an outlier detection model, input the monitoring data of the next period into the outlier detection model, and output an anomaly score; An early warning module, used for issuing a water quality early warning based on the abnormal score; The question-answering module is used to construct a question-answering model using the text data set, input user questions into the question-answering model, and output question-answering results.

Images