Water quality anomaly detection feature selection method and system based on improved pigeon inspired optimization algorithm

Abstract

The invention relates to a water quality anomaly detection feature selection method and system based on an improved pigeon inspired optimization algorithm, and belongs to the field of machine learning. Executing an improved pigeon flock algorithm, and evaluating the pigeon flock; endowing the position and speed of the pigeon with the lowest fitness to the globally optimal pigeon; the speed of the pigeons is updated, the speed of the pigeons is converted through a Sigmoid function, and the positions of the pigeons are updated according to the output value of the Sigmoid function; when an iteration stopping condition is met, returning to a global optimal pigeon; otherwise, mutating the pigeon flock, and updating the position and the speed through the literanism; ranking the varied pigeons according to fitness values, calculating an ideal destination, and updating the positions of the pigeons; according to the method, the self-adaptive iterative map and the compass factor are set, and meanwhile, an Litalian mechanism is introduced, so that information interaction and cooperation can be carried out among pigeon individuals, the common evolution of a group is promoted, and the search capability of the group is further optimized.

Description

Technical Field

[0001] The invention relates to a method and system for selecting features for water quality anomaly detection based on an improved pigeon flock optimization algorithm, and belongs to the field of machine learning. Background Art

[0002] Water quality testing in water treatment plants is a key link in ensuring the safety of drinking water. It aims to comprehensively monitor and analyze various indicators of source water, treated water and factory water through scientific methods. Conventional testing indicators include turbidity, pH value, residual chlorine, total hardness, total nitrogen, total phosphorus, etc. These parameters directly affect the quality and treatment effect of water. Through water quality testing, water pollution sources and potential problems in the treatment process can be discovered in time to ensure the effectiveness of the water treatment process and that the water quality meets national drinking water standards. In addition, the application of modern water quality testing technologies, such as online monitoring equipment and automated control systems, has further improved the real-time and accuracy of water quality testing, provided a scientific basis for water plants, optimized water treatment processes, and ultimately ensured the health and safety of the public.

[0003] With the progress and development of the times, the field of MLOps (Machine Learning Operations) came into being, aiming to simplify the entire ML life cycle. The concept of MLOps comes from the DevOps (Development Operations) practice in software development, and has gradually evolved into a set of systems and processes specifically used to manage and operate machine learning projects in the field of machine learning. However, in the era of information explosion, data is showing an unprecedented growth trend. MLOps faces the challenge of processing massive and diversified data. These data not only come from a wide range of sources, but also in various forms, including structured data and semi-structured data. At the same time, after analyzing multi-source heterogeneous data, it is found that although there are many data features, there are also many redundant and invalid features. Eliminating redundant features and achieving dimensionality reduction for multi-source heterogeneous data can effectively reduce the time cost of MLOps, speed up the flow of MLOps data, and improve the accuracy of machine learning.

[0004] Feature selection is one of the key steps to achieve efficient MLOps. Feature selection aims to select the most relevant and informative feature subsets from the original data set to reduce the complexity of the model, improve the generalization ability of the model, and reduce the risk of overfitting. Feature selection methods can be divided into the following categories: 1. Filtering method: It is performed before feature selection, regardless of the performance of the model, and the importance of features is evaluated based on statistical tests. 2. Wrapping method: The feature selection process is regarded as a search problem, and the performance of the model is evaluated through different feature subsets. The performance of the model is used as a guide to select features. 3. Embedding method: Feature selection is performed during the model training process, and the model selection process and feature selection process are integrated. 4. Hybrid method: It combines the characteristics of filtering method and wrapping method. The filtering method is first used to reduce the feature space, and then the wrapping method is used for more detailed feature selection.

[0005] Swarm Intelligence (SI) is a class of heuristic optimization algorithms that simulate the behavior of biological groups in nature. They solve complex optimization problems by simulating the collective behavior of simple individuals. The core of swarm intelligence algorithms is to simulate the self-organizing behavior of biological groups. These algorithms usually do not require centralized control, but rely on local interactions between simple individuals to solve complex tasks. These algorithms combine prior information and a posteriori information. Altruism means showing selfless concern for the happiness of others. Humans and some animals sometimes show altruistic behavior to their family or friends, which gives other members a chance to survive or improve their ability to survive.

[0006] Pigeon-Inspired Optimization (PIO) is a newly developed bionic swarm intelligence algorithm. Pigeons have the ability to fly long distances to find food, which is due to their special homing behavior. When pigeons are far away from their nests, they reach their nests with the help of the geomagnetic field and landmark buildings. During the flight, pigeons will use different cruising tools according to different situations. First, they use the geomagnetic field to identify a general direction, and then use the terrain to correct the current direction until they reach the precise destination. Therefore, the pigeon homing in the PIO algorithm consists of two basic parts: the compass operator and the landmark operator. When pigeons are far away from their destination, they use the geomagnetic field to identify the direction. When they are close to the destination, they use local landmarks for navigation. In PIO, the map and pointer operator models are based on the geomagnetic field and the sun, while the landmark operator model is based on landmarks.

[0007] The traditional pigeon swarm algorithm tends to linger around a suboptimal solution, which means that it is more likely to fall into a local dilemma in a specific area of ​​the search space. At the same time, the traditional pigeon swarm algorithm faces the problem of low convergence accuracy, which means that it may not be able to reach a very accurate feature optimal solution. The problem of low accuracy can be effectively solved by introducing a binary process, but the binary process limits the effectiveness of the landmark operator. The working principle of the landmark operator is to calculate the location of the ideal pigeon destination as the average value of high-fitness pigeons, so the probability of generating a new and better solution from the average value of the binary vector is reduced, affecting the convergence rate and prolonging the time to reach the feature optimal solution. Therefore, in response to the above challenges, it is necessary to improve the traditional pigeon swarm algorithm, set an adaptive iteration factor, make full use of the map and compass factors; optimize the pigeon mutation process to increase the probability of generating a better solution, thereby improving the algorithm convergence rate and obtaining the optimal feature set. Summary of the invention

[0008] In view of the deficiencies in the prior art, the present invention provides a method and system for feature selection of water quality anomaly detection based on an improved pigeon flock optimization algorithm, which introduces an adaptive iterative map and a compass factor. The adaptive iterative map and the compass factor enable the pigeon flock algorithm to ensure sufficient solution space coverage through more diverse regional explorations in the early stage of the search. As the iteration proceeds, the particles gradually converge to the optimal solution, thereby realizing the development and optimization of the solution space. At the same time, an altruistic mutation mechanism is introduced to enable information exchange and collaboration between individual pigeons, thereby promoting the co-evolution of the group. Finally, in the iterative process, by identifying individuals with lower fitness and performing mutation operations on them, its search capability is further optimized.

[0009] The present invention first preprocesses the water treatment plant data so that it can be used as a data source by the method. The method defines a fitness function for evaluating the quality of a feature subset, and updates the position and speed of a simulated pigeon group in the search space to achieve global search and optimization of the solution space. An adaptive iterative map and a compass factor are set. The possibility of the population wandering around a suboptimal solution is reduced by the adaptive iterative map and the compass factor, and the problem of it falling into a local dilemma in a specific area of ​​the search space is reduced. The introduction of the adaptive iterative map and the compass factor can also adjust the balance between exploration and optimization when searching for the global optimal solution. At the same time, the altruism mechanism is introduced to enable information interaction and collaboration between individual pigeons, and promote the co-evolution of the group. In the iterative process, by identifying individuals with lower fitness and performing mutation operations on them, their search ability is further optimized. This mechanism not only improves the fitness mean of the group as a whole, but also enhances the possibility of finding the global optimal feature subset. In addition, the method significantly improves the convergence efficiency of the algorithm, reduces the calculation time required to reach the global optimal solution, and has good theoretical value and practical application potential.

[0010] The technical solution of the present invention is as follows: The first aspect of the present invention provides a method for selecting features for water quality anomaly detection based on an improved pigeon flock optimization algorithm, comprising: Step 1: Preprocess the water treatment plant data; Step 2: Evaluate the current iteration condition, which is the maximum number of iterations. If the condition is met, execute step 3, otherwise execute step 6. Step 3: Execute the improved pigeon flock algorithm to evaluate the fitness of the pigeon flock; Step 4: Assign the position and speed of the pigeon with the lowest fitness to the global optimal pigeon; execute steps 5 and 6; Step 5: Update the pigeon's speed value, convert the pigeon's speed using the Sigmoid function, and then update the pigeon's position according to the output value of the Sigmoid function; return to step 2; Step 6: When the iteration stop condition is reached, return the global optimal pigeon; otherwise, mutate the pigeon flock, let the pigeon flock converge to its optimal solution, and then use it to gain greater advantages; update the position and speed through altruism; rank the mutated pigeons according to the fitness value, calculate the ideal destination, and update the position of the pigeon flock; return to step 3.

[0011] Preferably, according to the present invention, preprocessing of water treatment plant data includes: Water treatment plant data include: 01 equipment power water pressure status, 02 equipment power water pressure status, 03 equipment power water pressure status (01, 02, 03 represent the pressure of the water supply system of the monitoring equipment in different areas, respectively, to ensure that the water pressure is within the normal range, to avoid the impact of excessive or low water pressure on equipment operation), compound sodium bisulfate dosing status (monitoring the addition of sodium bisulfate, used to adjust the pH of water), compound sodium bisulfate shortage alarm status (when the agent is insufficient, an alarm is issued to remind timely replenishment), compound chlorate dosing status (monitoring the addition of chlorate, used for disinfection or oxidation treatment), compound chlorate shortage alarm status (when the agent is insufficient), generator operation status (monitoring the operation of dosing equipment or disinfection equipment), compound dosing dosage (the total amount of mixed dosing of multiple agents), manual dosing dosage (manually operated dosing dosage), flow dosing dosage (dosing dosage automatically adjusted according to the inlet water flow); Data conversion: convert all symbolic data in the water treatment plant data into numerical data, and convert the input values ​​of the data column into binary form, where "0" represents normal records and "1" represents abnormal records, so as to unify the feature expression form, such as the switch composite chlorate dosing state, "0" represents normal, and "1" represents abnormal; Deduplication: Delete duplicate records in the water treatment plant data to avoid the classifier being biased towards frequently occurring records during training, improve its learning ability for uncommon records, and ensure the classifier's adaptability to diverse data; Data normalization: Normalize the water treatment plant data. Use the maximum value of the data in the data set minus the minimum value as the denominator, and the current data value minus the minimum value as the numerator. Use fractional operations to get the normalized result, eliminate the deviation of features with large values ​​in the data set, convert or scale each feature value into normalized data within a certain range of proportions, achieve data balance, and ensure the accuracy of classifier training; Preferably, according to the present invention, an improved pigeon flock algorithm is executed to evaluate the fitness of the pigeon flock; comprising: The preprocessed water treatment plant data is used as a data set, and the data set is divided into a training set and a test set; Select the classifier as a model based on the decision tree algorithm; the decision tree classification algorithm is a supervised learning method based on a tree structure, which is used to divide data into different categories. It is the application of the decision tree algorithm in classification tasks. By learning the relationship between input features and category labels, a series of decision rules are generated to achieve the classification of new samples, data classification and prediction; The model based on the decision tree algorithm is trained using the filtered feature subset; the improved pigeon flock algorithm is used to select features of the data set, and the data set after feature selection is the feature subset; each data in the data set is judged to determine whether it is abnormal water quality; Use the test set to validate the model based on the decision tree algorithm; The validated decision tree algorithm-based model is used to evaluate each pigeon in the flock; each pigeon represents a potential feature subset, and its fitness is determined by the fitness function (Formula (1)); the fitness function is as follows: (1) Among them, Fitness represents the fitness value, W 1 The weight parameter, W, represents the ratio of the number of selected features to the number of features in the dataset. 2 Represents the weight parameter of the false positive rate, W 3 represents the weight parameter of the inverse of the accuracy, and w 1 + w 2 + w 3 =1, TPR and FPR are equally important to the Fitness value, so the weight value is set to w 1 = 0.1, w 2 = w 3= 0.45, SF represents the number of selected features, NF represents the number of features in the data set, FPR represents the false positive rate, and TPR represents the correct rate.

[0012] Preferably, according to the present invention, the position and speed of the pigeon with the lowest fitness are assigned to the global optimal pigeon; comprising: Through the global optimal search function, the pigeon with the lowest fitness is searched according to the number of pigeon populations, and the pigeon with the lowest fitness value (Xp) is located. Once the pigeon with the lowest fitness value (Xp) is identified, the position and speed of the pigeon with the lowest fitness value are assigned to the global optimal pigeon (Xg). The global optimal pigeon is the pigeon closest to the destination in the pigeon group, that is, the pigeon with the lowest fitness value; The remaining pigeons adjust their search strategies, that is, they adjust the adaptive iterative map and compass factor search destination to the landmark factor search destination to follow the global optimal pigeon; Preferably, according to the present invention, the speed value of the pigeon is updated, and the pigeon speed is converted using a Sigmoid function, and then the pigeon position is updated according to the output value of the Sigmoid function; including: Update Rt through formula (2), and update the pigeon's speed value through formula (3). The formula is as follows:

[0013]

[0014]

[0015]

[0016]

[0017] (2) (3) Among them, dP represents the rate of change of the pigeon group, P represents the number of pigeons, and P e It is represented as the final state of the pigeon flock, P i Represents the current iteration state of the pigeon group, t e It is expressed as the total number of iterations of the pigeon group, t i It is represented by the number of iterations of the current pigeon group, c is the introduced constant, Rt is the adaptive iteration map and compass factor, rand is a uniform random number in the range of [0,1], X g represents the position of the global optimal pigeon individual, X i (t) represents the current position of the pigeon after iteration t, V i (t) represents the current speed of the pigeon after iteration t; V i(t+1) represents the current speed of the pigeon at iteration t+1; The Sigmoid function (Equation (4)) is used to map the speed value of each pigeon to a new value range. The position of each pigeon will be updated according to the output value of the Sigmoid function and the judgment of the random uniform number generated in the interval [0,1] (Equation (5)), as shown below: (4) (5) Among them, V i (t) is the speed of the pigeon at the tth iteration, r is a uniform random number, X(t) (i,P) [i] is the updated pigeon position.

[0018] Preferably, according to the present invention, a pigeon flock is mutated, the pigeon flock is allowed to converge to its optimal solution, and then it is used to gain greater advantage; the position and speed are updated through altruism; the mutated pigeons are ranked according to the fitness value, the ideal destination is calculated, and the position of the pigeon flock is updated; including: First, evaluate the fitness value of each pigeon in the previous iteration, that is, compare the fitness values, and sort all the obtained fitness values ​​from small to large accordingly. If the fitness value is low (the lower the fitness value, the better), retain the elite pigeons (such as top k%). Elite pigeons refer to the pigeons in the top k% of fitness values ​​after completing the fitness sorting. Let them converge to the optimal solution (converging to the optimal solution means not performing any processing and using the original speed and position to fly toward the destination); for the remaining pigeons, allow the pigeons ranked in the first half, that is, the better pigeons, and the pigeons in the second half, that is, the weaker pigeons, to show altruistic behavior in pairs, that is, assign the positions and speeds of the pigeons ranked in the first half (better) to the pigeons in the second half (weaker); In order to perform this task selectively, a random value is set. If the random value is greater than P and less than P + γ, the probability of a specific pigeon being selected is within the range (p, p + γ). The specific pigeon refers to the pigeon that will be mutated; where γ is the interval value of the mutation to be performed, and p is the starting value; Once the whole process is finished, the position and speed of the weaker pigeon are assigned by the position and speed of the better pigeon, and the speed and position of the better pigeon are assigned according to equations (4) and (5), these weaker pigeons reset the pigeon flock through altruism, that is, the better pigeon assigns the speed and position to the weaker pigeon, and the speed of the better pigeon is mutated by equation (6), and the position is mutated by equation (7) to obtain a possible better fitness; Among them, the speed and position update mechanism of the weaker pigeons, i.e., the part of pigeons at the bottom of the fitness value, is based on the state of the corresponding better pigeons, i.e., the part of pigeons with the top fitness value selected after removing the elite pigeons. Finally, through the implementation of the above mechanism, the entire altruism variation process is completed, as shown in the following formula: (6) (7) Among them, v good_idx represents the speed, good_idx represents the index value of the better pigeon after selecting the better ratio, bad_idx represents the index value of the weaker pigeon, X good_idx Indicates location; Then, the mutated pigeons are ranked according to their fitness values. In each generation, the number of pigeons is updated by formula (8), where only half of the number of pigeons are considered to calculate the ideal position of the central pigeon, that is, the position of the desired destination, while all other pigeons adjust their destinations according to the ideal destination position. The specific meaning of the desired destination is that after screening, the accuracy of the decision tree classification model can be maximized; as shown in the following formula: (8) Among them, N p (t+1) is the number of pigeons in the current iteration; The position of the desired destination is calculated by equation (9), and all other pigeons update their positions to the desired destination through equation (10), as shown below: (9) Among them, N p is the number of pigeons, X c (t+1) is the position of the central pigeon, i.e. the pigeon at the center of the flock (the desired destination), X i (t+1) is the current position of all pigeons, as shown in the following formula: (10) Here, rand is a uniform random number in the range [0,1].

[0019] Once all pigeons have reached their desired destination, feature selection for water quality anomaly detection is completed.

[0020] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a method for selecting features for water quality anomaly detection based on an improved pigeon flock optimization algorithm are implemented.

[0021] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a water quality anomaly detection feature selection method based on an improved pigeon flock optimization algorithm.

[0022] The second aspect of the present invention provides a water quality anomaly detection feature selection system based on an improved pigeon flock optimization algorithm, comprising: The preprocessing module is configured to: preprocess the water treatment plant data; The iteration condition evaluation module is configured to: evaluate the current iteration condition, which is the maximum number of iterations. If the condition is met, the pigeon flock evaluation module is executed, otherwise, the pigeon flock position update module is executed; The pigeon flock evaluation module is configured to: execute the improved pigeon flock algorithm to evaluate the fitness of the pigeon flock; The pigeon update module is configured to: assign the position and speed of the pigeon with the lowest fitness to the global optimal pigeon; execute the speed position update module and the pigeon group position update module; The speed and position updating module is configured to: update the speed value of the pigeon, convert the pigeon speed using the Sigmoid function, and then update the pigeon position according to the output value of the Sigmoid function; return to the iterative condition evaluation module; The pigeon flock position update module is configured as follows: when the iteration stop condition is reached, the global optimal pigeon is returned; otherwise, the pigeon flock is mutated and the position and speed are updated through altruism; the mutated pigeons are ranked according to the fitness value, the ideal destination is calculated, and the pigeon flock position is updated; and the module returns to the pigeon flock evaluation module.

[0023] The beneficial effects of the present invention are: 1. By setting the adaptive iteration map and compass factor, the diversity exploration of the solution space in the initial iteration stage is enhanced, effectively preventing the population from falling into a local suboptimal solution. In the subsequent iterations, the process of finding the optimal solution is strengthened, and the population converges quickly to the optimal solution. At the same time, the exploration of the feature space and the development of the optimal solution are balanced, which significantly improves the algorithm performance. This makes the improved pigeon swarm algorithm more stable on complex problems.

[0024] 2. By combining altruism and the Sigmoid pigeon algorithm, the problem of low convergence accuracy and inability to obtain accurate feature optimal solutions of traditional pigeon algorithms can be solved. The generation of Sigmoid pigeons can improve the convergence accuracy of pigeons. At the same time, adding altruism to the pigeons to perform mutation behaviors that are beneficial to the pigeons can effectively increase the probability of the Sigmoid pigeons generating the optimal solution, increase the convergence rate of the pigeons, and reduce the time to reach the optimal solution of the features.

[0025] 3. The feature selection method for water quality anomaly detection based on the improved pigeon swarm optimization algorithm can effectively reduce the number of features in the water treatment plant data set to reduce the complexity of model training, directly reduce the computing resources and time required for model training, so that the model can be iterated and optimized faster, improve the accuracy of water quality detection, and accelerate the flow of water treatment plant data in the MLOps data pipeline. At the same time, by eliminating irrelevant or redundant features, feature selection helps prevent model overfitting and improves the model's generalization ability on data, so that MLOps can have higher stability and reliability in water quality detection in water treatment plants. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a data preprocessing flow chart of the present invention; Figure 2 It is a flow chart of the feature selection method for water quality anomaly detection based on the improved pigeon flock optimization algorithm; Figure 3 This is a schematic diagram of the MLOps process for feature selection based on the improved pigeon swarm algorithm. DETAILED DESCRIPTION

[0027] The present invention will be further described below by way of embodiments in conjunction with the accompanying drawings, but is not limited thereto.

[0028] Example 1 A feature selection method for water quality anomaly detection based on improved pigeon flock optimization algorithm, such as Figure 2 As shown, including: Step 1: Preprocess the water treatment plant data; Step 2: Evaluate the current iteration condition, which is the maximum number of iterations. If the condition is met, execute step 3, otherwise execute step 6. Step 3: Execute the improved pigeon flock algorithm to evaluate the fitness of the pigeon flock; Step 4: Assign the position and speed of the pigeon with the lowest fitness to the global optimal pigeon; execute steps 5 and 6; Step 5: Update the pigeon's speed value, convert the pigeon's speed using the Sigmoid function, and then update the pigeon's position according to the output value of the Sigmoid function; return to step 2; Step 6: When the iteration stop condition is reached, return the global optimal pigeon; otherwise, mutate the pigeon flock, let the pigeon flock converge to its optimal solution, and then use it to gain greater advantages; update the position and speed through altruism; rank the mutated pigeons according to the fitness value, calculate the ideal destination, and update the position of the pigeon flock; return to step 3.

[0029] Example 2 The difference between the method for water quality anomaly detection feature selection based on the improved pigeon flock optimization algorithm described in Example 1 is that: The water treatment process in a water treatment plant usually includes the following main stages: Pretreatment: Raw water collection: Collect raw water from water sources (such as rivers, lakes, groundwater); Preliminary filtration: Remove large particles of impurities through a grid or screen; Adjust water quality: Add sodium bisulfate and other agents for adjustment according to the pH value of the raw water.

[0030] Coagulation and sedimentation: Adding coagulants: Adding coagulants (such as aluminum sulfate, polyaluminum chloride) makes the suspended matter and colloidal particles in the water condense into larger flocs; Mixing reaction: Through stirring or water mixing, the coagulant and water are fully in contact; Sedimentation and separation: The flocs settle in the sedimentation tank and the clear water rises.

[0031] Sand filtration: remove fine particles from the water through sand filters; Activated carbon filtration: remove organic matter, odor and color from the water; Backwashing: regularly backwash the filter to restore the filtration capacity.

[0032] Disinfection: Add disinfectants such as chlorate, chlorine dioxide, etc. to kill bacteria and viruses in water; contact time control: ensure that the disinfectant is in full contact with water to achieve the disinfection effect.

[0033] Reverse osmosis / nanofiltration: remove dissolved solids from water; effluent water quality testing: test the treated water to ensure that it meets national standards.

[0034] Water delivery: delivering qualified water to the water supply network; through the above data monitoring and processing process, the water treatment plant can efficiently convert raw water into drinking water that meets the standards to ensure the safety of residents' water use.

[0035] Preprocess the water treatment plant data, including: Water treatment plant data include: 01 equipment power water pressure status, 02 equipment power water pressure status, 03 equipment power water pressure status (01, 02, 03 represent the pressure of the water supply system of the monitoring equipment in different areas, respectively, to ensure that the water pressure is within the normal range, to avoid the impact of excessive or low water pressure on equipment operation), compound sodium bisulfate dosing status (monitoring the addition of sodium bisulfate, used to adjust the pH of water), compound sodium bisulfate shortage alarm status (when the agent is insufficient, an alarm is issued to remind timely replenishment), compound chlorate dosing status (monitoring the addition of chlorate, used for disinfection or oxidation treatment), compound chlorate shortage alarm status (when the agent is insufficient), generator operation status (monitoring the operation of dosing equipment or disinfection equipment), compound dosing dosage (the total amount of mixed dosing of multiple agents), manual dosing dosage (manually operated dosing dosage), flow dosing dosage (dosing dosage automatically adjusted according to the inlet water flow); Data conversion: convert all symbolic data in the water treatment plant data into numerical data, and convert the input values ​​of the data column into binary form, where "0" represents normal records and "1" represents abnormal records, so as to unify the feature expression form, such as switching on and off composite chlorate dosing, "0" represents normal use, and "1" represents abnormal use; Deduplication: Delete duplicate records in the water treatment plant data to avoid the classifier being biased towards frequently occurring records during training, improve its learning ability for uncommon records, and ensure the classifier's adaptability to diverse data; Data normalization: Normalize the water treatment plant data. Use the maximum value of the data in the data set minus the minimum value as the denominator, and the current data value minus the minimum value as the numerator. Use fractional operations to get the normalized result, eliminate the deviation of features with large values ​​in the data set, convert or scale each feature value into normalized data within a certain range of proportions, achieve data balance, and ensure the accuracy of classifier training; Set the parameters of the improved pigeon swarm algorithm, the number of pigeons (Np) is 70, the maximum number of iterations (Nc) is 150, initialize the pigeon swarm, use vector X as the current pigeon position, vector V as the current pigeon speed, the spatial dimension is the number of features of the selected data set 41, set the upper and lower bounds of the pigeon swarm speed (L, U) = (0, 1), set the random calculation value (R) to 0.09, and initialize the first generation of pigeon swarms; Determine whether the current number of iterations is greater than or equal to 1. If the number of iterations is greater than 1 (Nc>= 1), reduce the number of iterations by 1 and execute the next step to evaluate the pigeon group; if the number of iterations is less than one (Nc<1), jump to step 6.

[0036] Execute the improved pigeon flock algorithm to evaluate the fitness of the pigeon flock; including: The pre-processed water purification plant data is used as a data set, and the data set is divided into a training set and a test set; 60% of the data is randomly selected from the data set as a training set, and the remaining 40% of the data is used as a test set to ensure a balanced distribution of training and test samples; the classifier selects a model based on a decision tree algorithm for data classification and prediction. The decision tree algorithm is suitable for the pre-processed data features in this embodiment because of its clear structure and efficient classification ability; Select the classifier as a model based on the decision tree algorithm; the decision tree classification algorithm is a supervised learning method based on a tree structure, which is used to divide data into different categories. It is the application of the decision tree algorithm in the classification task. By learning the relationship between input features and category labels, a series of decision rules are generated to achieve the classification of new samples; perform data classification and prediction; The model based on the decision tree algorithm is trained using the filtered feature subset; the improved pigeon flock algorithm is used to select features of the data set, and the data set after feature selection is the feature subset; each data in the data set is judged to determine whether it is abnormal water quality; Use the test set to validate the model based on the decision tree algorithm; The validated decision tree algorithm-based model is used to evaluate each pigeon in the flock; each pigeon represents a potential feature subset, and its fitness is determined by the fitness function (Formula (1)); the fitness function is as follows: (1) Among them, Fitness represents the fitness value, W 1 The weight parameter, W, represents the ratio of the number of selected features to the number of features in the dataset. 2 Represents the weight parameter of the false positive rate, W 3 represents the weight parameter of the inverse of the accuracy, and w 1 + w 2 + w 3 =1, TPR and FPR are equally important to the Fitness value, so the weight value is set to w 1 = 0.1, w 2 = w 3 = 0.45, SF represents the number of selected features, NF represents the number of features in the data set, FPR represents the false positive rate, and TPR represents the correct rate.

[0037] Assign the position and speed of the pigeon with the lowest fitness to the global optimal pigeon; including: Through the global optimal search function, the pigeon with the lowest fitness is searched according to the number of pigeon populations, and the pigeon with the lowest fitness value (Xp) is located. Once the pigeon with the lowest fitness value (Xp) is identified, the position and speed of the pigeon with the lowest fitness value are assigned to the global optimal pigeon (Xg). The global optimal pigeon is the pigeon closest to the destination in the pigeon group, that is, the pigeon with the lowest fitness value; The remaining pigeons adjust their search strategies, that is, they adjust the adaptive iterative map and compass factor search destination to the landmark factor search destination to follow the global optimal pigeon; Update the pigeon's speed value, convert the pigeon's speed using the Sigmoid function, and then update the pigeon's position according to the output value of the Sigmoid function; including: Update Rt through formula (2), and update the pigeon's speed value through formula (3). The formula is as follows:

[0038]

[0039]

[0040]

[0041]

[0042] (2) (3) Among them, dP represents the rate of change of the pigeon group, P represents the number of pigeons, and P e It is represented as the final state of the pigeon flock, P i Represents the current iteration state of the pigeon group, t e It is expressed as the total number of iterations of the pigeon group, t i It is represented by the number of iterations of the current pigeon group, c is the introduced constant, Rt is the adaptive iteration map and compass factor, rand is a uniform random number in the range of [0,1], X g represents the position of the global optimal pigeon individual, X i (t) represents the current position of the pigeon after iteration t, V i (t) represents the current speed of the pigeon after iteration t; V i (t+1) represents the current speed of the pigeon at iteration t+1; The Sigmoid function (Equation (4)) is used to map the speed value of each pigeon to a new value range. The position of each pigeon will be updated according to the output value of the Sigmoid function and the judgment of the random uniform number generated in the interval [0,1] (Equation (5)), as shown below: (4) (5) Among them, V i (t) is the speed of the pigeon at the tth iteration, r is a uniform random number, X(t) (i,P) [i] is the updated pigeon position.

[0043] Mutate the pigeon flock, let it converge to its optimal solution, and then use it to gain greater advantage; update the position and speed through altruism; rank the mutated pigeons according to the fitness value, calculate the ideal destination, and update the position of the pigeon flock; including: First, evaluate the fitness value of each pigeon in the previous iteration, that is, compare the fitness values, and sort all the obtained fitness values ​​from small to large accordingly. If the fitness value is low (the lower the fitness value, the better), retain the elite pigeons (such as top k%). Elite pigeons refer to the pigeons in the top k% of fitness values ​​after completing the fitness sorting. Let them converge to the optimal solution (converging to the optimal solution means not performing any processing and using the original speed and position to fly toward the destination); for the remaining pigeons, allow the pigeons ranked in the first half, that is, the better pigeons, and the pigeons in the second half, that is, the weaker pigeons, to show altruistic behavior in pairs, that is, assign the positions and speeds of the pigeons ranked in the first half (better) to the pigeons in the second half (weaker); In order to perform this task selectively, a random value is set. If the random value is greater than P and less than P + γ, the probability of a specific pigeon being selected is within the range (p, p + γ). The specific pigeon refers to the pigeon that will be mutated; where γ is the interval value of the mutation to be performed, and p is the starting value; Once the whole process is finished, the position and speed of the weaker pigeon are assigned by the position and speed of the better pigeon, and the speed and position of the destination better pigeon are assigned according to equations (4) and (5), these weaker pigeons reset the pigeon flock through altruism, that is, the better pigeon assigns the speed and position to the weaker pigeon, and the speed of the better pigeon is mutated by equation (6), and the position is mutated by equation (7) to obtain a possible better fitness; Among them, the speed and position update mechanism of the weaker pigeons, i.e., the pigeons at the bottom of the fitness value, is based on the state of the corresponding better pigeons, i.e., the pigeons with the top fitness values ​​selected after removing the elite pigeons. Finally, through the implementation of the above mechanism, the entire altruism variation process is completed; Adding altruism variation to the iterative process of the Sigmoid pigeon population includes: Input: N represents the number of features, k represents the intercepted ratio, i is the loop operation record value, z is the optimal ratio to be selected for alturism_rank, alturism_rank is an array tuple sorted in descending order according to fitness, good_idx is the index value of the better feature after z selects the better ratio in alturism_rank, bad_idx is the index value of the weaker feature selected by alturism_rank, v is the speed, x is the position, where [N=41,k=0.3,z=0.4]; Specific operation: pass the speed and position of the good_idx-th pigeon in the pigeon group, as well as the speed and position of the bad_idx-th pigeon in the pigeon group, into the altruism mechanism; generate a random number 1 for mutation (control the bad_idx pigeon for mutation), and assign the speed value and position of the good_idx-th pigeon to the bad_idx-th pigeon; generate a random number 2 for mutation (control the good_idx pigeon for mutation) and mutate the speed of the good_idx-th pigeon through formula (6), and mutate the position of the good_idx-th pigeon through formula (7). After the execution is completed, return the transformed position and speed of the good_idx pigeon and the position and speed of the bad_idx pigeon; after completing the entire k*N loop mutation process, exit the loop and complete the mutation operation, as shown in the following formula: (6) (7) Among them, v good_idx represents the speed, good_idx represents the index value of the better pigeon after selecting the better ratio, bad_idx represents the index value of the weaker pigeon, X good_idx Indicates location; Then, the mutated pigeons are ranked according to their fitness values. In each generation, the number of pigeons is updated by formula (8), where only half of the number of pigeons are considered to calculate the ideal position of the central pigeon, that is, the position of the desired destination, while all other pigeons adjust their destinations according to the ideal destination position. The specific meaning of the desired destination is that after screening, the accuracy of the decision tree classification model can be maximized; as shown in the following formula: (8) Among them, N p (t+1) is the number of pigeons in the current iteration; The position of the desired destination is calculated by equation (9), and all other pigeons update their positions to the desired destination through equation (10), as shown below: (9) Among them, N p is the number of pigeons, X c (t+1) is the position of the central pigeon, i.e. the pigeon at the center of the flock (the desired destination), X i (t+1) is the current position of all pigeons, as shown in the following formula: (10) Here, rand is a uniform random number in the range [0,1].

[0044] Once all pigeons have reached their desired destination, feature selection for water quality anomaly detection is completed.

[0045] Visual display, use the formula to calculate TPR, use the formula to calculate FPR, use the formula to calculate acc, use the formula to calculate F-score, intercept the selected features, and after completing the above calculations, intercept the selected features and their corresponding performance index results and output them to a text document for users to review and analyze. This process not only provides an intuitive display of the effect of feature selection, but also enables users to deeply understand the specific impact of feature selection on model performance through detailed numerical results.

[0046] Example 3 According to the feature selection method for water quality anomaly detection based on the improved pigeon flock optimization algorithm described in Example 1, the difference is that the feature selection method based on the improved pigeon flock algorithm is added to the data flow in MLOps, such as Figure 3 As shown, including: The data of the water treatment plant is transmitted to the data lake (hudi) through the Kafka stream processing system, where Apache Hudi (hudi) is an open source data lake storage framework that supports the storage of various types of data, efficient data update, deletion and incremental processing, and provides transactional ACID operations and time travel functions. In the present invention, it is used to store raw data and primary feature data. As a centralized storage warehouse, the data lake carries various types of water quality detection data of the water treatment plant and provides support for subsequent MLOps processing and analysis. Data stream processing engine technology (such as Flink) efficiently preprocesses the original data under the premise of ensuring the integrity of heterogeneous data storage. The data preprocessing step includes processing methods such as deleting missing values ​​and filling missing values ​​with mean values ​​to ensure the integrity and consistency of the data. The preprocessed data is transmitted to the feature storage system (hive), where Apache Hive (hive) is a data warehouse tool based on Hadoop for data storage, query and analysis of large-scale data sets. In the present invention, it is used to store processed data, and the data stored in the feature repository is the feature data. Next, feature selection is performed based on the improved pigeon swarm algorithm to extract the best features, and these features are passed to feature extraction tools (such as OpenMLDB). The feature extraction tool inputs the extracted features into the machine learning model to complete the data flow and analysis tasks in the MLOps data pipeline. This process significantly accelerates the MLOps process through efficient data flow and feature processing methods, thereby improving the real-time and accuracy of water quality anomaly detection and analysis in water treatment plants.

[0047] Obviously, the examples listed in the specific implementation are only some examples of the present invention, not all examples. Based on the examples of the present invention, all other examples obtained by technicians without creative work should fall within the protection scope of the present invention.

[0048] Example 4 A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the water quality anomaly detection feature selection method based on the improved pigeon flock optimization algorithm described in any one of Examples 1-3 are implemented.

[0049] Example 5 A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the water quality anomaly detection feature selection method based on an improved pigeon flock optimization algorithm described in any one of Examples 1-3 are implemented.

[0050] Example 6 A water quality anomaly detection feature selection system based on an improved pigeon flock optimization algorithm, comprising: The preprocessing module is configured to: preprocess the water treatment plant data; The iteration condition evaluation module is configured to: evaluate the current iteration condition, which is the maximum number of iterations. If the condition is met, the pigeon flock evaluation module is executed, otherwise, the pigeon flock position update module is executed; The pigeon flock evaluation module is configured to: execute the improved pigeon flock algorithm to evaluate the fitness of the pigeon flock; The pigeon update module is configured to: assign the position and speed of the pigeon with the lowest fitness to the global optimal pigeon; execute the speed position update module and the pigeon group position update module; The speed and position updating module is configured to: update the speed value of the pigeon, convert the pigeon speed using the Sigmoid function, and then update the pigeon position according to the output value of the Sigmoid function; return to the iterative condition evaluation module; The pigeon flock position update module is configured as follows: when the iteration stop condition is reached, the global optimal pigeon is returned; otherwise, the pigeon flock is mutated and the position and speed are updated through altruism; the mutated pigeons are ranked according to the fitness value, the ideal destination is calculated, and the pigeon flock position is updated; and the module returns to the pigeon flock evaluation module.

Claims

1. A feature selection method for water quality anomaly detection based on improved pigeon flock optimization algorithm, characterized in that: include: Step 1: Preprocess the water treatment plant data; Step 2: Evaluate the current iteration condition, which is the maximum number of iterations. If the condition is met, execute step 3, otherwise execute step 6. Step 3: Execute the improved pigeon flock algorithm to evaluate the fitness of the pigeon flock; Step 4: Assign the position and speed of the pigeon with the lowest fitness to the global optimal pigeon; execute steps 5 and 6; Step 5: Update the pigeon's speed value, convert the pigeon's speed using the Sigmoid function, and then update the pigeon's position according to the output value of the Sigmoid function; Return to step 2; Step 6: When the iteration stop condition is reached, return the global optimal pigeon; otherwise, mutate the pigeon flock; update the position and speed through altruism; rank the mutated pigeons according to the fitness value, calculate the ideal destination, and update the position of the pigeon flock; Return to step 3.

2. According to claim 1, a method for selecting features for water quality anomaly detection based on an improved pigeon flock optimization algorithm is characterized in that: Preprocess the water treatment plant data, including: Data conversion: convert all symbolic data in the water treatment plant data into numerical data, and convert the input values ​​of the data column into binary form, where "0" represents a normal record and "1" represents an abnormal record; Deduplication: Delete duplicate records in water treatment plant data; Data normalization: Normalize the water treatment plant data.

3. According to claim 1, a method for selecting features for water quality anomaly detection based on improved pigeon flock optimization algorithm is characterized in that: Execute the improved pigeon flock algorithm to evaluate the fitness of the pigeon flock; including: The pre-processed water treatment plant data is used as a data set, and the data set is divided into a training set and a test set; the classifier is selected as a model based on the decision tree algorithm; Use the filtered feature subset to train the model based on the decision tree algorithm; use the test set to verify the model based on the decision tree algorithm; Each pigeon in the flock is evaluated using the validated decision tree algorithm-based model; its fitness is determined by the fitness function; the fitness function is as follows: （1） Among them, Fitness represents the fitness value, W1 represents the weight parameter of the ratio of the number of selected features to the number of features in the data set, W2 represents the weight parameter of the false positive rate, W3 represents the weight parameter of the inverse of the accuracy rate, and w1 + w2 + w3 = 1, the weight values ​​are set to w1 = 0.1, w2= w3 = 0.45, SF represents the number of selected features, NF represents the number of features in the data set, FPR represents the false positive rate, and TPR represents the accuracy rate.

4. According to claim 3, a method for selecting features for water quality anomaly detection based on improved pigeon flock optimization algorithm is characterized in that: Assign the position and speed of the pigeon with the lowest fitness to the global optimal pigeon; including: The global optimal search function is used to search for the pigeon with the lowest fitness according to the number of pigeon populations, and the pigeon with the lowest fitness value is located. The position and speed of the pigeon with the lowest fitness value are assigned to the global optimal pigeon. The global optimal pigeon is the pigeon closest to the destination in the pigeon group, that is, the pigeon with the lowest fitness value. The remaining pigeons adjusted their search strategy to use landmark factors to search for the destination.

5. According to claim 4, a method for selecting features for water quality anomaly detection based on improved pigeon flock optimization algorithm is characterized in that: Update the pigeon's speed value, convert the pigeon's speed using the Sigmoid function, and then update the pigeon's position according to the output value of the Sigmoid function; including: Update Rt through formula (2), and update the pigeon's speed value through formula (3). The formula is as follows: （2） （3） Among them, P represents the pigeon group, Rt is the adaptive iterative map and compass factor, rand is a uniform random number in the range of [0,1], and X g represents the position of the global optimal pigeon individual, X i (t) represents the current position of the pigeon after iteration t, V i (t) represents the current speed of the pigeon after iteration t; V i (t+1) represents the current speed of the pigeon at iteration t+1; The Sigmoid function is used to map the speed value of each pigeon to a new value range. The position of each pigeon will be updated according to the output value of the Sigmoid function and the judgment of the random uniform number generated in the [0,1] interval, as shown below: （4） （5） Among them, V i (t) is the speed of the pigeon at the tth iteration, r is a uniform random number, X(t) (i,P) [i] is the updated pigeon position.

6. A method for selecting features for water quality anomaly detection based on improved pigeon flock optimization algorithm according to claim 5, characterized in that: Mutate the pigeon flock and update its position and speed through altruism; Rank the mutated pigeons according to their fitness values, calculate the ideal destination, and update the location of the pigeon flock; including: First, the fitness value of each pigeon in the previous iteration is evaluated, that is, the size of the fitness value is compared, and all the obtained fitness values ​​are sorted from small to large accordingly, and the elite pigeons are retained. The elite pigeons refer to the pigeons with the top k% fitness values ​​after the fitness sorting is completed, and they are allowed to converge to the optimal solution; for the remaining pigeons, the pigeons ranked in the first half, that is, the better pigeons, are allowed to show altruistic behavior in pairs with the pigeons in the second half, that is, the positions and speeds of the pigeons ranked in the first half are assigned to the pigeons in the second half; Set a random value. If the random value is greater than P and less than P + γ, the probability of a specific pigeon being selected is in the range (p, p +γ). The specific pigeon is the pigeon that will be mutated. γ is the interval value of the mutation to be performed, and p is the starting value. The speed of the better pigeon is varied by equation (6), and the position is varied by equation (7); as shown in the following equations: （6） （7） Among them, v good_idx represents the speed, good_idx represents the index value of the better pigeon after selecting the better ratio, bad_idx represents the index value of the weaker pigeon, X good_idx Indicates location; Then, the mutated pigeons are ranked according to their fitness values. In each generation, the number of pigeons is updated by equation (8), as shown below: （8） Among them, N p (t+1) is the number of pigeons in the current iteration; The position of the desired destination is calculated by equation (9), and all other pigeons update their positions to the desired destination through equation (10), as shown below: （9） Where N p is the number of pigeons, X c (t+1) is the position of the central pigeon, i.e. the pigeon at the center of the pigeon group, X i (t+1) is the current position of all pigeons, as shown in the following formula: （10） Here, rand is a uniform random number in the range [0,1].

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the water quality anomaly detection feature selection method based on the improved pigeon flock optimization algorithm described in any one of claims 1-6 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the water quality anomaly detection feature selection method based on the improved pigeon flock optimization algorithm described in any one of claims 1-6 are implemented.

9. A water quality anomaly detection feature selection system based on improved pigeon flock optimization algorithm, characterized in that: include: The preprocessing module is configured to: preprocess the water treatment plant data; The iteration condition evaluation module is configured to: evaluate the current iteration condition, which is the maximum number of iterations. If the condition is met, the pigeon flock evaluation module is executed, otherwise, the pigeon flock position update module is executed; The pigeon flock evaluation module is configured to: execute the improved pigeon flock algorithm to evaluate the fitness of the pigeon flock; The pigeon update module is configured to: assign the position and speed of the pigeon with the lowest fitness to the global optimal pigeon; execute the speed position update module and the pigeon group position update module; The speed and position updating module is configured to: update the speed value of the pigeon, convert the pigeon speed using the Sigmoid function, and then update the pigeon position according to the output value of the Sigmoid function; return to the iterative condition evaluation module; The pigeon group position update module is configured to: return the global optimal pigeon when the iteration stop condition is reached; Otherwise, the pigeon flock will be mutated, and the position and speed will be updated through altruism; the mutated pigeons will be ranked according to their fitness values, the ideal destination will be calculated, and the position of the pigeon flock will be updated; and the process will return to the pigeon flock evaluation module.

Images