A new pollutant toxicity level assessment method and system based on big data analysis
Through big data analysis and deep learning algorithms, the new pollutant toxicity level evaluation model was constructed, which solved the problems of high labor costs and low accuracy in the toxicity assessment of new pollutant, and achieved efficient and accurate toxicity level evaluation.
Patent Information
- Application Number
- CN202311795129.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-25
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-12-25
AI Technical Summary
In the prior art, the toxicity assessment of new pollutants is high labor cost, high workload and low accuracy, so it is impossible to effectively regulate new pollutants.
Using a method based on big data analysis, the new pollutant toxicity level evaluation model is constructed by clustering the method of water quality detection in historical sea areas, and the deep learning algorithm is used for optimization training. Combined with FCM and IWOA algorithms, the rapid evaluation of the new pollutant toxicity level is achieved. .
It reduces labor cost investment, reduces workload, and improves the accuracy of the assessment of the toxicity level of new pollutants, simplifies the evaluation process, and improves the practicality of actual areas.
Smart Images

Figure CN117894400B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of new pollutants, and specifically relates to a method and system for evaluating the toxicity level of new pollutants based on big data analysis. Background Art
[0002] New pollutants refer to toxic and hazardous chemical substances that are discharged into the environment and have characteristics such as biological toxicity, environmental persistence, and bioaccumulation, which pose a great risk to the ecological environment or human health, but have not yet been included in management or the existing management measures are insufficient. New pollutants have characteristics such as biological toxicity, environmental persistence, and bioaccumulation, and have not yet been effectively regulated at this stage. Compared with traditional pollutants that have been regulated, they are relatively "new".
[0003] In the existing technology, the toxicity of new pollutants in the sea is mostly judged and analyzed through laboratory toxicity tests. This method requires high manpower costs and workload, and is prone to low accuracy of analysis results due to human errors. Summary of the invention
[0004] In order to solve the problems of high labor cost, heavy workload and low accuracy in the prior art, the present invention aims to provide a new pollutant toxicity level assessment method and system based on big data analysis.
[0005] The technical solution adopted by the present invention is:
[0006] A new pollutant toxicity level assessment method based on big data analysis includes the following steps:
[0007] According to the new pollutant definition rules, water quality detection rules and pollution toxicity assessment rules, set new pollutant types, new pollutant water quality detection indicators and corresponding pollution toxicity water quality detection indicators;
[0008] Perform clustering processing on the historical sea area water quality detection big data to obtain several cluster centers corresponding to the new pollutant types, and several historical sea area water quality detection data cluster clusters corresponding to the cluster centers;
[0009] According to the new pollutant water quality detection indicators and pollution toxicity water quality detection indicators of the new pollutant types, the indicator data of the corresponding historical sea water quality detection clusters are extracted to obtain several historical sea water quality detection key indicator data sets corresponding to the new pollutant types;
[0010] Based on the historical data set of key indicators of marine water quality testing, a deep learning algorithm is used to optimize training and build a new pollutant toxicity level assessment model;
[0011] Collect the real-time sea water quality detection data of the current sea area, and search and match the real-time sea water quality detection data with several cluster centers to obtain the corresponding real-time new pollutant types;
[0012] According to the new pollutant water quality detection indicators and pollution toxicity water quality detection indicators corresponding to the real-time new pollutant types, the real-time marine water quality detection data is extracted to obtain the corresponding real-time marine water quality detection key indicator data;
[0013] According to the real-time key indicator data of marine water quality detection, the toxicity level of new pollutants is assessed using the new pollutant toxicity level assessment model to obtain the new pollutant toxicity level assessment results.
[0014] Furthermore, according to the new pollutant definition rules, water quality detection rules and pollution toxicity assessment rules, new pollutant types, new pollutant water quality detection indicators and corresponding pollution toxicity water quality detection indicators are constructed, including the following steps:
[0015] According to the new pollutant definition rules, several new pollutant types are set;
[0016] According to the water quality testing rules, set a number of new pollutant water quality testing indicators corresponding to each new pollutant type;
[0017] According to the pollution toxicity assessment rules, a number of pollution toxicity water quality detection indicators corresponding to each new pollutant type are set.
[0018] Furthermore, the historical sea area water quality detection big data is clustered. Specifically, the FCM clustering algorithm is used to cluster the historical sea area water quality detection big data to obtain a number of cluster centers corresponding to the new pollutant type and a number of historical sea area water quality detection data clustering clusters corresponding to the cluster centers, including the following steps:
[0019] Performing data preprocessing on the historical sea area water quality detection big data to obtain a preprocessed historical sea area water quality detection data set; the historical sea area water quality detection big data includes a number of historical sea area water quality detection data, and each historical sea area water quality detection data includes a number of historical sea area water quality detection indicator data;
[0020] According to the number of new pollutant types, generate a number of corresponding initial cluster centers;
[0021] Obtaining the initial membership degree of each preprocessed historical sea area water quality detection data and several initial clustering centers in the preprocessed historical sea area water quality detection data set;
[0022] According to the initial cluster center and initial membership degree, the objective function of the FCM clustering algorithm is updated;
[0023] Obtain the merge function value and change value of the Lagrange multiplier method. If the merge function value is greater than the merge function value threshold, or the change value is less than the change value threshold, output the final cluster center. Otherwise, update the cluster center, obtain the updated cluster center, and proceed to the next step.
[0024] Obtain the updated membership of each pre-processed historical sea area water quality detection data and several updated cluster centers in the pre-processed historical sea area water quality detection data set, and re-update the merging function value;
[0025] According to the final membership degree between the preprocessed historical sea water quality detection data and the final cluster center, the preprocessed historical sea water quality detection data are divided into the corresponding historical sea water quality detection data clustering clusters, and the historical new pollutant type label corresponding to the final cluster center is spread to the corresponding historical sea water quality detection data clustering cluster.
[0026] Furthermore, according to the new pollutant water quality detection index and the pollution toxicity water quality detection index of the new pollutant type, the corresponding historical sea water quality detection cluster is extracted to obtain several historical sea water quality detection key indicator data sets corresponding to the new pollutant type, including the following steps:
[0027] According to the new pollutant types corresponding to the clusters of the current historical sea water quality detection clusters, the corresponding new pollutant water quality detection indicators and pollution toxicity water quality detection indicators are matched;
[0028] According to the matched water quality detection indicators of several new pollutants, several historical sea area water quality detection indicator data corresponding to the current historical sea area water quality detection data in the current historical sea area water quality detection cluster are extracted to obtain the corresponding several historical new pollutant water quality detection indicator data;
[0029] According to the matched several pollution toxicity water quality detection indicators, several historical sea area water quality detection indicator data corresponding to the current historical sea area water quality detection data in the current historical sea area water quality detection cluster are extracted to obtain the corresponding several historical pollution toxicity water quality detection indicator data;
[0030] Integrate the new pollutant type labels corresponding to the current historical sea area water quality detection data, several historical new pollutant water quality detection index data, and several historical pollution toxicity water quality detection index data to obtain the corresponding historical sea area water quality detection key index data;
[0031] Traverse the current historical sea water quality detection clusters to obtain the key indicator data set of historical sea water quality detection corresponding to the current new pollutant type;
[0032] Traverse all historical sea water quality detection clusters to obtain several historical sea water quality detection key indicator data sets corresponding to all new pollutant types.
[0033] Furthermore, a deep learning algorithm is used to construct a new pollutant toxicity level assessment model. Specifically, based on the historical sea area water quality detection key indicator data set, the Elman-IWOA algorithm is used for optimization training to construct a new pollutant toxicity level assessment model, including the following steps:
[0034] The indicator dimension features of each historical sea area water quality detection key indicator data set are fused and labels are added to obtain the corresponding training data of several new pollutant toxicity level assessments containing historical toxicity level labels;
[0035] The training data of toxicity level assessment of several new pollutants are divided into corresponding model training sample sets and model test sample sets in a ratio of 7:3;
[0036] The IWOA optimization algorithm is used to optimize the initial network parameters of the Elman network to obtain the optimal initial network parameters of the Elman network;
[0037] According to the optimal initial network parameters of the Elman network, the corresponding network structure is set, and the model training sample set is input for optimization training to construct the initial new pollutant toxicity level assessment model;
[0038] The model test sample set is input into the initial new pollutant toxicity level assessment model to perform model testing and obtain the corresponding model test accuracy;
[0039] If the model test accuracy is greater than the accuracy threshold, the optimal new pollutant toxicity level assessment model is output; otherwise, the optimization training continues.
[0040] Furthermore, the indicator dimension features of each historical sea area water quality detection key indicator data set are fused and labels are added to obtain the corresponding training data of several new pollutant toxicity level assessments containing historical toxicity level labels, including the following steps:
[0041] Use the random forest algorithm to obtain the feature contribution of each historical sea area water quality detection indicator data feature in the historical sea area water quality detection key indicator data set;
[0042] According to the feature contribution of the historical sea area water quality detection index data features, the corresponding feature selection standard value is obtained;
[0043] The indicator dimension feature fusion is performed on several historical sea area water quality detection indicator data features whose feature selection standard values are greater than the feature selection standard value threshold, and the new pollutant toxicity level assessment training data corresponding to the current historical sea area water quality detection key indicator data is obtained;
[0044] Setting corresponding historical toxicity level labels for the new pollutant toxicity level assessment training data corresponding to the current historical sea area water quality detection key indicator data, and obtaining new pollutant toxicity level assessment training data containing historical toxicity level labels;
[0045] The historical sea water quality detection key indicator dataset is traversed to obtain the corresponding toxicity level assessment training data of several new pollutants containing historical toxicity level labels.
[0046] Furthermore, the initial network parameters of the Elman network are optimized using the IWOA optimization algorithm to obtain the optimal initial network parameters of the Elman network, including the following steps:
[0047] The initial network parameters of the Elman network are used as the positions of the IWOA individuals in the IWOA optimization algorithm;
[0048] Initialize the algorithm parameters of the IWOA optimization algorithm and use the chaotic mapping sequence to initialize the IWOA population;
[0049] Calculate the fitness values of the IWOA individuals in the IWOA population, and retain the best IWOA individuals according to the fitness values of the IWOA individuals;
[0050] Randomly generate update parameters, and according to the update parameters, execute corresponding prey encirclement behavior, prey search behavior and bubble net attack behavior, update the position of the IWOA individual in the IWOA population, and obtain an updated IWOA population containing a number of updated IWOA individuals;
[0051] According to the updated IWOA population, dynamic reverse learning is performed to obtain each updated IWOA individual in the updated IWOA population, that is, the reverse solution corresponding to the forward solution;
[0052] Calculate the fitness values of the forward solution and the reverse solution, and retain the best IWOA individual in the updated IWOA population according to the fitness values of the forward solution and the reverse solution;
[0053] If the number of iterations is greater than the iteration threshold, or the optimal fitness value corresponding to the optimal IWOA individual is less than the fitness threshold, the position of the global optimal solution corresponding to the optimal IWOA individual is output to obtain the optimal initial network parameters of the Elman network. Otherwise, the IWOA population continues to be updated.
[0054] Furthermore, real-time sea area water quality detection data of the current sea area is collected, and the real-time sea area water quality detection data is retrieved and matched with a number of cluster centers to obtain the corresponding real-time new pollutant type, including the following steps:
[0055] Collecting real-time sea area water quality detection data of the current sea area, and normalizing the real-time sea area water quality detection data to obtain normalized real-time sea area water quality detection data;
[0056] Obtain the Euclidean distance between the normalized real-time marine water quality detection data and several cluster centers;
[0057] According to the Euclidean distance between the normalized real-time marine water quality detection data, several cluster centers are sorted in descending order;
[0058] The historical new pollutant type label corresponding to the first-ranked cluster center is used as the real-time new pollutant type corresponding to the real-time marine water quality detection data after normalization.
[0059] Further, based on the real-time key indicator data of marine water quality detection, the new pollutant toxicity level assessment model is used to assess the toxicity level of the new pollutant, and the toxicity level assessment result of the new pollutant is obtained, including the following steps:
[0060] Add the real-time new pollutant type label corresponding to the real-time new pollutant type to the real-time key indicator data of sea water quality detection, and obtain the real-time key indicator data of sea water quality detection after the label is added;
[0061] After adding the label, the key indicator data of real-time marine water quality detection are integrated with the indicator dimension features to obtain the corresponding new pollutant toxicity level assessment prediction data;
[0062] Input the new pollutant toxicity level assessment prediction data into the new pollutant toxicity level assessment model, conduct the new pollutant toxicity level assessment, and obtain the corresponding real-time toxicity level label;
[0063] The real-time new pollutant type and the real-time toxicity level label are output as the new pollutant toxicity level assessment results.
[0064] A new pollutant toxicity level assessment system based on big data analysis is used to implement a new pollutant toxicity level assessment method, the system includes a toxicity index setting unit, a big data processing unit, a historical data extraction unit, an assessment model building unit, a new pollutant matching unit, a real-time data extraction unit and a toxicity level assessment unit connected in sequence;
[0065] A toxicity index setting unit, used to set new pollutant types, new pollutant water quality detection indicators and corresponding pollution toxicity water quality detection indicators according to new pollutant definition rules, water quality detection rules and pollution toxicity assessment rules;
[0066] A big data processing unit, used for clustering the big data of historical sea water quality detection to obtain a number of cluster centers corresponding to the new pollutant type and a number of cluster clusters of historical sea water quality detection data corresponding to the cluster centers;
[0067] A historical data extraction unit is used to extract index data of the corresponding historical sea water quality detection clusters according to the new pollutant water quality detection index and the pollution toxicity water quality detection index of the new pollutant type, and obtain several historical sea water quality detection key index data sets corresponding to the new pollutant type;
[0068] The evaluation model building unit is used to optimize and train the key indicator data set of historical marine water quality detection using deep learning algorithms to build a new pollutant toxicity level evaluation model;
[0069] A new pollutant matching unit is used to collect real-time sea water quality detection data of the current sea area, and search and match the real-time sea water quality detection data with several cluster centers to obtain the corresponding real-time new pollutant type;
[0070] A real-time data extraction unit is used to collect real-time sea water quality detection data of the current sea area, and search and match the real-time sea water quality detection data with a number of cluster centers to obtain the corresponding real-time new pollutant types;
[0071] The toxicity level assessment unit is used to assess the toxicity level of new pollutants based on real-time key indicator data of marine water quality detection and use a new pollutant toxicity level assessment model to obtain new pollutant toxicity level assessment results.
[0072] The beneficial effects of the present invention are:
[0073] The present invention provides a method and system for evaluating the toxicity level of new pollutants based on big data analysis. By setting new pollutant types, new pollutant water quality detection indicators and corresponding pollution toxicity water quality detection indicators according to new pollutant definition rules, water quality detection rules and pollution toxicity assessment rules, the direct connection between the toxicity of new pollutants and water quality indicators is defined. Through quantitative data analysis, the process of evaluating the toxicity level of new pollutants is simplified, the human cost investment is reduced, the workload is alleviated, and through a deep learning algorithm, a new pollutant toxicity level evaluation model is constructed, which can explore the deep connection between water quality detection data and the toxicity level of new pollutants, improve the accuracy of the toxicity level evaluation of new pollutants, and improve the practicability in practical fields.
[0074] Other beneficial effects of the present invention will be further described in the specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0075] Figure 1 It is a flowchart of the new pollutant toxicity level assessment method based on big data analysis in the present invention.
[0076] Figure 2 It is a structural block diagram of the new pollutant toxicity level assessment system based on big data analysis in the present invention. DETAILED DESCRIPTION
[0077] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments.
[0078] Embodiment 1:
[0079] like Figure 1 As shown, this embodiment provides a new pollutant toxicity level assessment method based on big data analysis, including the following steps:
[0080] S1: According to the new pollutant definition rules, water quality detection rules and pollution toxicity assessment rules, set the new pollutant type, new pollutant water quality detection indicators and corresponding pollution toxicity water quality detection indicators, including the following steps:
[0081] S1-1: According to the new pollutant definition rules, set several new pollutant types;
[0082] S1-2: According to the water quality detection rules, set a number of new pollutant water quality detection indicators corresponding to each new pollutant type;
[0083] S1-3: According to the pollution toxicity assessment rules, set a number of pollution toxicity water quality detection indicators corresponding to each new pollutant type;
[0084] S2: Use the fuzzy C-mean (FCM) clustering algorithm to cluster the historical sea water quality detection big data to obtain several cluster centers corresponding to the new pollutant types and several historical sea water quality detection data cluster clusters corresponding to the cluster centers, including the following steps:
[0085] S2-1: Perform data preprocessing on the historical sea area water quality detection big data to obtain the preprocessed historical sea area water quality detection data set; the historical sea area water quality detection big data includes a number of historical sea area water quality detection data, and each historical sea area water quality detection data includes a number of historical sea area water quality detection indicator data;
[0086] Data preprocessing includes repeated data screening, erroneous data screening and normalization processing, which ensures the standardization of historical marine water quality testing big data, eliminates data magnitude differences, and improves model prediction accuracy and model building speed;
[0087] S2-2: Generate a number of corresponding initial cluster centers according to the number of new pollutant types;
[0088] S2-3: Obtain the initial membership degree of each preprocessed historical sea area water quality detection data and several initial cluster centers in the preprocessed historical sea area water quality detection data set;
[0089] S2-4: Update the objective function of the FCM clustering algorithm according to the initial cluster center and the initial membership degree;
[0090]
[0091] Where F(U,Z) is the objective function of the FCM clustering algorithm; U = (u ij ) n×c is the fuzzy membership matrix; Z = {z1,z2,...,z j ,...,z c} is the cluster center set; uij is the membership degree of the historical sea water quality detection data after the i-th preprocessing belonging to the j-th cluster center, which is the initial membership degree at the first update; x i is the historical sea area water quality detection data after the i-th preprocessing; m is the fuzzy factor; i is the data indicator; n is the total number of data; j is the cluster center indicator; c is the total number of cluster centers; z j is the jth cluster center, which is the initial cluster center during the first update;
[0092] S2-5: Obtain the merge function value and change value of the Lagrange multiplier method. If the merge function value is greater than the merge function value threshold, or the change value is less than the change value threshold, then output the final cluster center. Otherwise, update the cluster center to obtain an updated cluster center, and proceed to the next step.
[0093]
[0094] ΔJ t =J t -J t-1
[0095] In the formula, J t , J t-1 is the combined function value of the Lagrange multiplier method at the t and t-1 iterations; ΔJ t is the corresponding change value; i is the i-th characteristic parameter; t is the indicator of the number of iterations; uij is the degree of membership of the historical sea water quality detection data after the i-th preprocessing to the j-th cluster center; m is the fuzzy factor; i is the data indicator; n is the total number of data; j is the cluster center indicator; c is the total number of cluster centers; z j is the jth cluster center; is the Euclidean distance from the i-th preprocessed historical sea area water quality detection data to the j-th cluster center; i is the historical sea area water quality detection data after the i-th preprocessing;
[0096] S2-6: Obtain the updated membership of each pre-processed historical sea area water quality detection data and several updated cluster centers in the pre-processed historical sea area water quality detection data set, and re-update the merging function value;
[0097] The formula for updating the cluster center is:
[0098]
[0099] In the formula, z' j is the cluster center updated for the jth time; m is the fuzzy factor; i is the data indicator; n is the total number of data; j is the cluster center indicator; x i is the historical sea area water quality detection data after the i-th preprocessing; u ij is the degree of membership of the historical sea area water quality detection data after the i-th preprocessing to the j-th cluster center;
[0100] The formula for updating membership is:
[0101]
[0102] In the formula, x i is the historical sea area water quality detection data after the i-th preprocessing; i is the data indicator; j and k are the cluster center indicators; c is the total number of cluster centers; d ij ,d ik is the distance from the i-th preprocessed historical sea area water quality detection data to the j-th and k-th cluster centers; u' ij is the updated membership degree of the historical sea water quality detection data after the i-th preprocessing belonging to the j-th cluster center; d ij ,d ik is the Euclidean distance from the i-th preprocessed historical sea area water quality detection data to the j-th and k-th initial cluster centers;
[0103] S2-7: According to the final membership degree between the pre-processed historical sea water quality detection data and the final cluster center, the pre-processed historical sea water quality detection data is divided into the corresponding historical sea water quality detection data cluster clusters, and the historical new pollutant type label corresponding to the final cluster center is spread to the corresponding historical sea water quality detection data cluster cluster;
[0104] S3: According to the new pollutant water quality detection indicators and pollution toxicity water quality detection indicators of the new pollutant type, the corresponding historical sea area water quality detection clusters are extracted to obtain several historical sea area water quality detection key indicator data sets corresponding to the new pollutant type, including the following steps:
[0105] S3-1: According to the new pollutant types corresponding to the clusters of the current historical sea water quality detection clusters, match the corresponding new pollutant water quality detection indicators and pollution toxicity water quality detection indicators;
[0106] S3-2: According to the matched water quality detection indicators of several new pollutants, extract the data of several historical sea area water quality detection indicators corresponding to the current historical sea area water quality detection data in the current historical sea area water quality detection cluster, and obtain the corresponding data of several historical new pollutant water quality detection indicators;
[0107] S3-3: According to the matched several pollution toxicity water quality detection indicators, several historical sea area water quality detection indicator data corresponding to the current historical sea area water quality detection data in the current historical sea area water quality detection cluster are extracted to obtain the corresponding several historical pollution toxicity water quality detection indicator data;
[0108] S3-4: Integrate the new pollutant type labels corresponding to the current historical sea area water quality detection data, several historical new pollutant water quality detection index data, and several historical pollution toxicity water quality detection index data to obtain the corresponding historical sea area water quality detection key index data;
[0109] S3-5: Traverse the current historical sea water quality detection clusters to obtain the key indicator data set of historical sea water quality detection corresponding to the current new pollutant type;
[0110] S3-6: Traverse all historical sea water quality detection clusters to obtain several historical sea water quality detection key indicator data sets corresponding to all new pollutant types;
[0111] S4: Based on the historical data set of key indicators of marine water quality detection, the Elman-Improved Whale Optimization Algorithm (IWOA) algorithm is used to perform optimization training and build a new pollutant toxicity level assessment model, including the following steps:
[0112] S4-1: Perform indicator dimension feature fusion and label addition on each historical sea area water quality detection key indicator dataset to obtain corresponding training data on toxicity level assessment of several new pollutants containing historical toxicity level labels, including the following steps:
[0113] S4-1-1: Use the random forest algorithm to obtain the feature contribution of each historical sea area water quality detection indicator data feature in the historical sea area water quality detection key indicator data set;
[0114]
[0115] In the formula, is the characteristic contribution of the characteristic of the j'th historical sea area water quality detection index data; is the feature contribution of the j'th historical sea area water quality detection index data feature in the i'th tree of the random forest; j' is the indicator of the historical sea area water quality detection index data; i' is the indicator of the random forest decision tree;
[0116]
[0117] In the formula, is the change in the Gini index before and after the random forest node m' branches; m' is the indicator of the random forest decision tree node;
[0118]
[0119]
[0120] In the formula, GI m' GI l' GI r' is the Gini index of random forest node m', node l' and node r'; p mk is the proportion of category k' in random forest node m'; m', l', r' are all random forest decision tree node indicators; k' is the category indicator; K is the total number of categories;
[0121] S4-1-2: According to the characteristic contribution of the historical sea area water quality detection index data characteristics, the corresponding characteristic selection standard value is obtained. The formula is:
[0122]
[0123] In the formula, CFC j' Select the standard value for the characteristic of the j'th historical sea area water quality detection index data characteristics; VIM j” is the normalized characteristic contribution of the characteristic of the j’th historical sea area water quality detection index data; j’ is the indicator of the historical sea area water quality detection index data;
[0124] The formula for the feature contribution after normalization is:
[0125]
[0126] Where VIM j”is the normalized feature contribution of the jth historical sea area water quality detection index data feature; J is the total number of features;
[0127] S4-1-3: Perform indicator dimension feature fusion on several historical sea water quality detection indicator data features whose feature selection standard values are greater than the feature selection standard value threshold, and obtain new pollutant toxicity level assessment training data corresponding to the current historical sea water quality detection key indicator data;
[0128] S4-1-4: Set corresponding historical toxicity level labels for the new pollutant toxicity level assessment training data corresponding to the current historical sea area water quality detection key indicator data, and obtain new pollutant toxicity level assessment training data containing historical toxicity level labels;
[0129] S4-1-5: Traverse the historical sea water quality detection key indicator data set to obtain the corresponding toxicity level assessment training data of several new pollutants containing historical toxicity level labels;
[0130] S4-2: Divide the training data of toxicity level assessment of several new pollutants into corresponding model training sample sets and model test sample sets in a ratio of 7:3;
[0131] S4-3: Using the IWOA optimization algorithm to optimize the initial network parameters of the Elman network to obtain the optimal initial network parameters of the Elman network, including the following steps:
[0132] S4-3-1: The initial network parameters of the Elman network are used as the positions of the IWOA individuals in the IWOA optimization algorithm;
[0133] S4-3-2: Initialize the algorithm parameters of the IWOA optimization algorithm and use the chaotic mapping sequence to initialize the IWOA population;
[0134] The chaotic mapping sequence is a Tent-Logistic-Cosine chaotic mapping sequence, and the chaotic mapping sequence formula is:
[0135]
[0136] In the formula, x i”+1 The initial position of the IWOA individuals generated by the Tent-Logistic-Cosine chaotic mapping sequence; x i” is the randomly generated initial position of the IWOA individual; r is the preset parameter; i” is the IWOA individual indicator;
[0137] S4-3-3: Calculate the fitness values of the IWOA individuals in the IWOA population, and retain the best IWOA individuals according to the fitness values of the IWOA individuals;
[0138] The formula for the fitness value is:
[0139]
[0140] Where fit is the fitness function; E is the network output error function; y n’ is the true output value of the network node of the n'th Elman network; is the ideal output value of the network node of the n'th Elman network; n' is the network node indicator; N is the total number of network nodes of the Elman network;
[0141] S4-3-4: randomly generate update parameters, and according to the update parameters, execute corresponding prey encirclement behavior, prey search behavior and bubble net attack behavior, update the position of the IWOA individual in the IWOA population, and obtain an updated IWOA population containing a number of updated IWOA individuals;
[0142] Specifically, p is randomly generated. If p<0.5 and |A|<1, the prey encirclement behavior is executed and the IWOA population position is updated. If p<0.5 and |A|≥1, the prey search behavior is executed and the IWOA population position is updated. If p≥0.5, the bubble net attack behavior is executed and the IWOA population position is updated. Among them, p is the update parameter and A is the step coefficient.
[0143] S4-3-5: Perform dynamic reverse learning based on the updated IWOA population to obtain each updated IWOA individual in the updated IWOA population, that is, the reverse solution corresponding to the forward solution;
[0144] The formula for dynamic reverse learning is:
[0145] X' i” (t) = k (a (t) + b (t)) - X i” (t)
[0146] Where X' i” (t), X i” (t) are the reverse solution position and forward solution position of the “i” IWOA individual; a(t) and b(t) are the upper and lower bounds of the current IWOA population in the search dimension; k is the decreasing inertia factor, k = 0.9-0.5t / t max ; t, t max are the current number of iterations and the maximum number of iterations respectively; reducing search blind spots and more effectively avoiding premature algorithm maturation and falling into local optimal values;
[0147] S4-3-6: Calculate the fitness values of the forward solution and the reverse solution, and retain the best IWOA individual in the updated IWOA population according to the fitness values of the forward solution and the reverse solution;
[0148] S4-3-7: If the number of iterations is greater than the iteration number threshold, or the optimal fitness value corresponding to the optimal IWOA individual is less than the fitness threshold, then the position of the global optimal solution corresponding to the optimal IWOA individual is output to obtain the optimal initial network parameters of the Elman network, otherwise, continue to update the IWOA population;
[0149] S4-4: According to the optimal initial network parameters of the Elman network, the corresponding network structure is set, and the model training sample set is input to perform optimization training to construct the initial new pollutant toxicity level assessment model;
[0150] S4-5: input the model test sample set into the initial new pollutant toxicity level assessment model, perform model testing, and obtain the corresponding model test accuracy;
[0151] S4-6: If the model test accuracy is greater than the accuracy threshold, the optimal new pollutant toxicity level assessment model is output, otherwise, the optimization training continues;
[0152] S5: Collect the real-time sea water quality detection data of the current sea area, and search and match the real-time sea water quality detection data with several cluster centers to obtain the corresponding real-time new pollutant type, including the following steps:
[0153] S5-1: collect the real-time sea area water quality detection data of the current sea area, and perform normalization processing on the real-time sea area water quality detection data to obtain the normalized real-time sea area water quality detection data;
[0154] The normalization formula is:
[0155]
[0156] Where, q' is the real-time sea area water quality detection data after normalization; q is the real-time sea area water quality detection data; q max is the maximum value in the real-time marine water quality detection data; q min It is the minimum value in the real-time marine water quality detection data;
[0157] S5-2: Obtain the Euclidean distance between the normalized real-time marine water quality detection data and several cluster centers;
[0158] S5-3: sorting the cluster centers in descending order according to the Euclidean distance between them and the normalized real-time marine water quality detection data;
[0159] S5-4: The historical new pollutant type label corresponding to the first ranked cluster center is used as the real-time new pollutant type corresponding to the real-time marine water quality detection data after normalization;
[0160] S6: extracting index data from the real-time marine water quality detection data according to the new pollutant water quality detection index and the pollution toxicity water quality detection index corresponding to the real-time new pollutant type, and obtaining the corresponding real-time marine water quality detection key index data;
[0161] S7: Based on the real-time key indicator data of marine water quality detection, the toxicity level of new pollutants is evaluated using the new pollutant toxicity level evaluation model to obtain the new pollutant toxicity level evaluation results, including the following steps:
[0162] S7-1: adding the real-time new pollutant type label corresponding to the real-time new pollutant type to the real-time key indicator data of sea water quality detection, and obtaining the real-time key indicator data of sea water quality detection after the label is added;
[0163] S7-2: After adding the label, the key indicator data of real-time marine water quality detection are integrated with the indicator dimension features to obtain the corresponding new pollutant toxicity level assessment prediction data;
[0164] S7-3: Input the new pollutant toxicity level assessment prediction data into the new pollutant toxicity level assessment model, conduct new pollutant toxicity level assessment, and obtain the corresponding real-time toxicity level label;
[0165] S7-4: Output the real-time new pollutant type and the real-time toxicity level label as the new pollutant toxicity level assessment result.
[0166] Embodiment 2:
[0167] like Figure 2 As shown, this embodiment provides a new pollutant toxicity level assessment system based on big data analysis, which is used to implement a new pollutant toxicity level assessment method. The system includes a toxicity index setting unit, a big data processing unit, a historical data extraction unit, an assessment model building unit, a new pollutant matching unit, a real-time data extraction unit, and a toxicity level assessment unit connected in sequence;
[0168] A toxicity index setting unit, used to set new pollutant types, new pollutant water quality detection indicators and corresponding pollution toxicity water quality detection indicators according to new pollutant definition rules, water quality detection rules and pollution toxicity assessment rules;
[0169] A big data processing unit, used for clustering the big data of historical sea water quality detection to obtain a number of cluster centers corresponding to the new pollutant type and a number of cluster clusters of historical sea water quality detection data corresponding to the cluster centers;
[0170] A historical data extraction unit is used to extract index data of the corresponding historical sea water quality detection clusters according to the new pollutant water quality detection index and the pollution toxicity water quality detection index of the new pollutant type, and obtain several historical sea water quality detection key index data sets corresponding to the new pollutant type;
[0171] The evaluation model building unit is used to optimize and train the key indicator data set of historical marine water quality detection using deep learning algorithms to build a new pollutant toxicity level evaluation model;
[0172] A new pollutant matching unit is used to collect real-time sea water quality detection data of the current sea area, and search and match the real-time sea water quality detection data with several cluster centers to obtain the corresponding real-time new pollutant type;
[0173] A real-time data extraction unit is used to collect real-time sea water quality detection data of the current sea area, and search and match the real-time sea water quality detection data with a number of cluster centers to obtain the corresponding real-time new pollutant types;
[0174] The toxicity level assessment unit is used to assess the toxicity level of new pollutants based on real-time key indicator data of marine water quality detection and use a new pollutant toxicity level assessment model to obtain new pollutant toxicity level assessment results.
[0175] The present invention provides a method and system for evaluating the toxicity level of new pollutants based on big data analysis. By setting new pollutant types, new pollutant water quality detection indicators and corresponding pollution toxicity water quality detection indicators according to new pollutant definition rules, water quality detection rules and pollution toxicity assessment rules, the direct connection between the toxicity of new pollutants and water quality indicators is defined. Through quantitative data analysis, the process of evaluating the toxicity level of new pollutants is simplified, the human cost investment is reduced, the workload is alleviated, and through a deep learning algorithm, a new pollutant toxicity level evaluation model is constructed, which can explore the deep connection between water quality detection data and the toxicity level of new pollutants, improve the accuracy of the toxicity level evaluation of new pollutants, and improve the practicability in practical fields.
[0176] The present invention is not limited to the above optional implementations, and anyone can derive other various forms of products under the enlightenment of the present invention. The above specific implementations should not be understood as limiting the scope of protection of the present invention. The scope of protection of the present invention should be based on the definition in the claims, and the description can be used to interpret the claims.
Claims
1. A new pollutant toxicity level assessment method based on big data analysis, Features: The steps include: According to the new pollutant definition rules, water quality detection rules and pollution toxicity assessment rules, set new pollutant types, new pollutant water quality detection indicators and corresponding pollution toxicity water quality detection indicators; Perform clustering processing on the historical sea area water quality detection big data to obtain several cluster centers corresponding to the new pollutant types, and several historical sea area water quality detection data cluster clusters corresponding to the cluster centers; According to the new pollutant water quality detection indicators and pollution toxicity water quality detection indicators of the new pollutant types, the indicator data of the corresponding historical sea water quality detection clusters are extracted to obtain several historical sea water quality detection key indicator data sets corresponding to the new pollutant types; Based on the historical data set of key indicators of marine water quality testing, a deep learning algorithm is used to optimize training and build a new pollutant toxicity level assessment model; Collect the real-time sea water quality detection data of the current sea area, and search and match the real-time sea water quality detection data with several cluster centers to obtain the corresponding real-time new pollutant types; According to the new pollutant water quality detection indicators and pollution toxicity water quality detection indicators corresponding to the real-time new pollutant types, the real-time marine water quality detection data is extracted to obtain the corresponding real-time marine water quality detection key indicator data; According to the real-time key indicator data of marine water quality detection, the toxicity level of new pollutants is assessed using the new pollutant toxicity level assessment model to obtain the new pollutant toxicity level assessment results.
2. According to the method for evaluating the toxicity level of new pollutants based on big data analysis according to claim 1, Features: According to the new pollutant definition rules, water quality detection rules and pollution toxicity assessment rules, new pollutant types, new pollutant water quality detection indicators and corresponding pollution toxicity water quality detection indicators are constructed, including the following steps: According to the new pollutant definition rules, several new pollutant types are set; According to the water quality testing rules, set a number of new pollutant water quality testing indicators corresponding to each new pollutant type; According to the pollution toxicity assessment rules, a number of pollution toxicity water quality detection indicators corresponding to each new pollutant type are set.
3. According to the method for evaluating the toxicity level of new pollutants based on big data analysis according to claim 1, Features: The historical sea area water quality detection big data is clustered. Specifically, the FCM clustering algorithm is used to cluster the historical sea area water quality detection big data to obtain several cluster centers corresponding to the new pollutant types and several historical sea area water quality detection data cluster clusters corresponding to the cluster centers, including the following steps: Performing data preprocessing on the historical sea area water quality detection big data to obtain a preprocessed historical sea area water quality detection data set; the historical sea area water quality detection big data includes a number of historical sea area water quality detection data, and each historical sea area water quality detection data includes a number of historical sea area water quality detection indicator data; According to the number of new pollutant types, generate a number of corresponding initial cluster centers; Obtaining the initial membership degree of each preprocessed historical sea area water quality detection data and several initial clustering centers in the preprocessed historical sea area water quality detection data set; According to the initial cluster center and initial membership degree, the objective function of the FCM clustering algorithm is updated; Obtain the merge function value and change value of the Lagrange multiplier method. If the merge function value is greater than the merge function value threshold, or the change value is less than the change value threshold, output the final cluster center. Otherwise, update the cluster center, obtain the updated cluster center, and proceed to the next step. Obtain the updated membership of each pre-processed historical sea area water quality detection data and several updated cluster centers in the pre-processed historical sea area water quality detection data set, and re-update the merging function value; According to the final membership degree between the preprocessed historical sea water quality detection data and the final cluster center, the preprocessed historical sea water quality detection data are divided into the corresponding historical sea water quality detection data clustering clusters, and the historical new pollutant type label corresponding to the final cluster center is spread to the corresponding historical sea water quality detection data clustering cluster.
4. According to the method for evaluating the toxicity level of new pollutants based on big data analysis according to claim 3, Features: According to the new pollutant water quality detection indicators and pollution toxicity water quality detection indicators of the new pollutant type, the indicator data of the corresponding historical sea area water quality detection clusters are extracted to obtain several historical sea area water quality detection key indicator data sets corresponding to the new pollutant type, including the following steps: According to the new pollutant types corresponding to the clusters of the current historical sea water quality detection clusters, the corresponding new pollutant water quality detection indicators and pollution toxicity water quality detection indicators are matched; According to the matched water quality detection indicators of several new pollutants, several historical sea area water quality detection indicator data corresponding to the current historical sea area water quality detection data in the current historical sea area water quality detection cluster are extracted to obtain the corresponding several historical new pollutant water quality detection indicator data; According to the matched several pollution toxicity water quality detection indicators, several historical sea area water quality detection indicator data corresponding to the current historical sea area water quality detection data in the current historical sea area water quality detection cluster are extracted to obtain the corresponding several historical pollution toxicity water quality detection indicator data; Integrate the new pollutant type labels corresponding to the current historical sea area water quality detection data, several historical new pollutant water quality detection index data, and several historical pollution toxicity water quality detection index data to obtain the corresponding historical sea area water quality detection key index data; Traverse the current historical sea water quality detection clusters to obtain the key indicator data set of historical sea water quality detection corresponding to the current new pollutant type; Traverse all historical sea water quality detection clusters to obtain several historical sea water quality detection key indicator data sets corresponding to all new pollutant types.
5. According to the method for evaluating the toxicity level of new pollutants based on big data analysis according to claim 1, Features: Use deep learning algorithms to build a new pollutant toxicity level assessment model. Specifically, based on the historical sea area water quality detection key indicator data set, use the Elman-IWOA algorithm to perform optimization training and build a new pollutant toxicity level assessment model, including the following steps: The indicator dimension features of each historical sea area water quality detection key indicator data set are fused and labels are added to obtain the corresponding training data of several new pollutant toxicity level assessments containing historical toxicity level labels; The training data of toxicity level assessment of several new pollutants are divided into corresponding model training sample sets and model test sample sets in a ratio of 7:3; The IWOA optimization algorithm is used to optimize the initial network parameters of the Elman network to obtain the optimal initial network parameters of the Elman network; According to the optimal initial network parameters of the Elman network, the corresponding network structure is set, and the model training sample set is input for optimization training to construct the initial new pollutant toxicity level assessment model; The model test sample set is input into the initial new pollutant toxicity level assessment model to perform model testing and obtain the corresponding model test accuracy; If the model test accuracy is greater than the accuracy threshold, the optimal new pollutant toxicity level assessment model is output; otherwise, the optimization training continues.
6. According to claim 5, a new pollutant toxicity level assessment method based on big data analysis, Features: The indicator dimension features of each historical sea area water quality detection key indicator data set are fused and labels are added to obtain the corresponding training data of several new pollutant toxicity level assessments containing historical toxicity level labels, including the following steps: Use the random forest algorithm to obtain the feature contribution of each historical sea area water quality detection indicator data feature in the historical sea area water quality detection key indicator data set; According to the feature contribution of the historical sea area water quality detection index data features, the corresponding feature selection standard value is obtained; The indicator dimension feature fusion is performed on several historical sea area water quality detection indicator data features whose feature selection standard values are greater than the feature selection standard value threshold, and the new pollutant toxicity level assessment training data corresponding to the current historical sea area water quality detection key indicator data is obtained; Setting corresponding historical toxicity level labels for the new pollutant toxicity level assessment training data corresponding to the current historical sea area water quality detection key indicator data, and obtaining new pollutant toxicity level assessment training data containing historical toxicity level labels; The historical sea water quality detection key indicator dataset is traversed to obtain the corresponding toxicity level assessment training data of several new pollutants containing historical toxicity level labels.
7. According to claim 5, a new pollutant toxicity level assessment method based on big data analysis, Features: The IWOA optimization algorithm is used to optimize the initial network parameters of the Elman network to obtain the optimal initial network parameters of the Elman network, including the following steps: The initial network parameters of the Elman network are used as the positions of the IWOA individuals in the IWOA optimization algorithm; Initialize the algorithm parameters of the IWOA optimization algorithm and use the chaotic mapping sequence to initialize the IWOA population; Calculate the fitness values of the IWOA individuals in the IWOA population, and retain the best IWOA individuals according to the fitness values of the IWOA individuals; Randomly generate update parameters, and according to the update parameters, execute corresponding prey encirclement behavior, prey search behavior and bubble net attack behavior, update the position of the IWOA individual in the IWOA population, and obtain an updated IWOA population containing a number of updated IWOA individuals; According to the updated IWOA population, dynamic reverse learning is performed to obtain each updated IWOA individual in the updated IWOA population, that is, the reverse solution corresponding to the forward solution; Calculate the fitness values of the forward solution and the reverse solution, and retain the best IWOA individual in the updated IWOA population according to the fitness values of the forward solution and the reverse solution; If the number of iterations is greater than the iteration threshold, or the optimal fitness value corresponding to the optimal IWOA individual is less than the fitness threshold, the position of the global optimal solution corresponding to the optimal IWOA individual is output to obtain the optimal initial network parameters of the Elman network. Otherwise, the IWOA population continues to be updated.
8. According to claim 3, a new pollutant toxicity level assessment method based on big data analysis, Features: Collect the real-time sea water quality detection data of the current sea area, and search and match the real-time sea water quality detection data with several cluster centers to obtain the corresponding real-time new pollutant types, including the following steps: Collecting real-time sea area water quality detection data of the current sea area, and normalizing the real-time sea area water quality detection data to obtain normalized real-time sea area water quality detection data; Obtain the Euclidean distance between the normalized real-time marine water quality detection data and several cluster centers; According to the Euclidean distance between the normalized real-time marine water quality detection data, several cluster centers are sorted in descending order; The historical new pollutant type label corresponding to the first-ranked cluster center is used as the real-time new pollutant type corresponding to the real-time marine water quality detection data after normalization.
9. According to claim 6, a new pollutant toxicity level assessment method based on big data analysis, Features: According to the key indicator data of real-time marine water quality detection, the toxicity level of new pollutants is evaluated using the new pollutant toxicity level evaluation model to obtain the new pollutant toxicity level evaluation results, including the following steps: Add the real-time new pollutant type label corresponding to the real-time new pollutant type to the real-time key indicator data of sea water quality detection, and obtain the real-time key indicator data of sea water quality detection after the label is added; After adding the label, the key indicator data of real-time marine water quality detection are integrated with the indicator dimension features to obtain the corresponding new pollutant toxicity level assessment prediction data; Input the new pollutant toxicity level assessment prediction data into the new pollutant toxicity level assessment model, conduct the new pollutant toxicity level assessment, and obtain the corresponding real-time toxicity level label; The real-time new pollutant type and the real-time toxicity level label are output as the new pollutant toxicity level assessment results.
10. A new pollutant toxicity level assessment system based on big data analysis, used to implement the new pollutant toxicity level assessment method as claimed in any one of claims 1 to 9, Features: The system comprises a toxicity index setting unit, a big data processing unit, a historical data extraction unit, an evaluation model building unit, a new pollutant matching unit, a real-time data extraction unit and a toxicity level evaluation unit which are connected in sequence; A toxicity index setting unit, used to set new pollutant types, new pollutant water quality detection indicators and corresponding pollution toxicity water quality detection indicators according to new pollutant definition rules, water quality detection rules and pollution toxicity assessment rules; A big data processing unit, used for clustering the big data of historical sea water quality detection to obtain a number of cluster centers corresponding to the new pollutant type and a number of cluster clusters of historical sea water quality detection data corresponding to the cluster centers; A historical data extraction unit is used to extract index data of the corresponding historical sea water quality detection clusters according to the new pollutant water quality detection index and the pollution toxicity water quality detection index of the new pollutant type, and obtain several historical sea water quality detection key index data sets corresponding to the new pollutant type; The evaluation model building unit is used to optimize and train the key indicator data set of historical marine water quality detection using deep learning algorithms to build a new pollutant toxicity level evaluation model; A new pollutant matching unit is used to collect real-time sea water quality detection data of the current sea area, and search and match the real-time sea water quality detection data with several cluster centers to obtain the corresponding real-time new pollutant type; A real-time data extraction unit is used to collect real-time sea water quality detection data of the current sea area, and search and match the real-time sea water quality detection data with a number of cluster centers to obtain the corresponding real-time new pollutant types; The toxicity level assessment unit is used to assess the toxicity level of new pollutants based on real-time key indicator data of marine water quality detection and use a new pollutant toxicity level assessment model to obtain new pollutant toxicity level assessment results.
