A multi-source data anomaly detection and bad data correction method for pig farms

Abstract

The application discloses a pig farm multi-source data anomaly detection and bad data correction method, comprising: acquiring heterogeneous data; classifying pig breeding data sources and analyzing data characteristics, preprocessing the collected heterogeneous data; based on the pig farm data anomaly detection algorithm, improving the evaluation index and the specific anomaly detection algorithm weight calculation method, integrating and optimizing the multi-source heterogeneous data anomaly detection method; using collaborative filtering recommendation algorithm, reducing the input dependence of error data repair in an unsupervised manner, starting from horizontal correlation, finding a similar historical monitoring point data set at the current time, calculating the range recommendation degree, using genetic algorithm to optimize the recommended data set, and finding the best correction value of the pig breeding bad data. Thus, the accuracy and reliability of the pig farm data are improved, and high-quality data support is provided for greenhouse gas emission early warning, environmental quality comprehensive evaluation and lean management strategy making of the pig farm.

Description

Technical Field

[0001] This application relates to the field of pig farming technology, specifically to a method for detecting anomalies and correcting bad data from multiple sources in pig farms. Background Technology

[0002] my country is the world's largest producer and consumer of pork, and the pig industry is a pillar industry of my country's livestock sector. Currently, Denmark, the Netherlands, and the United States have significantly higher feed utilization rates, breeding technologies, and production efficiency in pig farming than the Chinese industry average. While the scale and intensification of pig farming in my country are continuously improving, problems remain, including low levels of intelligent technology, outdated automation equipment, low management efficiency, environmental pollution, low energy efficiency, and a lack of big data platform support.

[0003] In recent years, the academic community has achieved considerable results in big data analysis and personalized recommendation. However, in the field of livestock application, problems remain, including unstable data sources, low data quality, and low accuracy in information mining and decision-making. The accuracy and reliability of data are crucial for the management and decision-making of pig farms. Given the various complex factors and environmental variables present in pig farms, how to obtain complete, timely, and reliable comprehensive data on the breeding process, and how to effectively analyze, process, and utilize this information to achieve lean management of pig farms, has become a critical issue that urgently needs to be addressed. Summary of the Invention

[0004] In order to solve the above-mentioned technical problems, this application proposes the following technical solution:

[0005] In a first aspect, embodiments of this application provide a method for detecting anomalies and correcting problematic data in multi-source data from pig farms, including:

[0006] Acquire heterogeneous data on monitoring, collection, transmission, energy consumption, and management of large-scale pig farms;

[0007] We conducted data source classification and data feature analysis for pig farming, and preprocessed the collected heterogeneous data, including outlier removal, data normalization, and redundancy removal.

[0008] Based on the anomaly detection algorithm for pig farm data, we improve the evaluation index and the weight calculation method for specific anomaly detection algorithms, and integrate and optimize the anomaly detection method for multi-source heterogeneous data.

[0009] A collaborative filtering recommendation algorithm is adopted to reduce the input dependence of error data repair in an unsupervised manner. Starting from the horizontal correlation, a set of historical monitoring point data similar to the current moment is found, the range recommendation degree is calculated, and a genetic algorithm is used to optimize the recommended data set to find the best correction value for poor data in pig farming.

[0010] In one possible implementation, the acquisition of heterogeneous data related to the monitoring, collection, transmission, energy consumption, and management of large-scale pig farms includes:

[0011] Based on the environmental information monitoring needs of pig farms, sensor / data acquisition nodes are deployed and various sensors are installed, including temperature sensors, relative humidity sensors, and ammonia / hydrogen sulfide / sulfur dioxide / carbon dioxide sensors.

[0012] The system collects various data indicators in the pigsty in real time using deployed sensors, and transmits the data through a 5G transmission module.

[0013] Obtain outdoor climate data, including outdoor temperature, relative humidity, wind speed, and air quality data;

[0014] Based on the pig farm production management system, data on procurement management, cost accounting, pig production, body size characteristics, and physiological health monitoring are obtained;

[0015] Based on the integrated energy consumption monitoring and management system for pig farms, data on the combined energy consumption of biogas, heat, and electricity is obtained.

[0016] In one possible implementation, the classification of pig farming data sources and analysis of data characteristics include:

[0017] Data on pig farming: including the number of pigs, breeds, body size and physical characteristics, physiological health, feed consumption, and vaccine usage;

[0018] Environmental monitoring data includes temperature, humidity, climate conditions, ammonia concentration, hydrogen sulfide concentration, sulfur dioxide concentration, and carbon dioxide concentration.

[0019] Pigsty management data includes pigsty layout, equipment usage, and hygiene inspection records;

[0020] Disease surveillance data: including pig health status and epidemic monitoring;

[0021] Feeding program data: including feed formulation and feeding schedule;

[0022] Cost-benefit data: including feed costs, labor costs, and farming profits;

[0023] Pesticide usage data: including the types and amounts of pesticides used to control pests and diseases;

[0024] Energy consumption data: including power consumption for lighting equipment, ventilation and air conditioning, feed processing and handling equipment, cleaning and disinfection equipment, water pumps and irrigation systems.

[0025] In one possible implementation, the preprocessing of the collected heterogeneous data includes outlier removal, data normalization, and redundancy removal, comprising:

[0026] The raw data was cleaned using the k-means clustering algorithm.

[0027] The cleaned data is normalized: by standardizing the pig farming data, the data is transformed and normalized to the same range.

[0028] The G-Relief F algorithm was used to select and remove redundancy from data from pig farms, and the feature weights were calculated.

[0029] In one possible implementation, the simple cleaning of the collected raw data using the k-means clustering algorithm includes:

[0030] The dataset is divided into groups based on the similarity of the data. Objects in the same cluster are similar to each other, while objects in different clusters are different.

[0031] Using cluster C i The centroid represents the cluster, cluster C. i The quality is measured by intra-cluster variation, defined as: dist(p,c i ) represents the object p∈C i With this cluster representing C i difference;

[0032] Assume the nearest center to object p is c. p c p Assigned to c p Average distance between objects l cp Define ratio Outlier samples are identified and removed based on the ratio R.

[0033] In one possible implementation, the use of the G-Relief F algorithm for data feature selection and redundancy removal in pig farms includes:

[0034]

[0035] Features A and B are each represented by an n-dimensional array, where A = {a1, a2, ..., a...} n B = {b1, b2, ..., b} n Each sample is a point in an n-dimensional space;

[0036]

[0037] The difference between samples R1 and R2 on feature A is represented by diff(A,R1,R2).

[0038] In one possible implementation, the calculation of the feature weights includes:

[0039] The formula for calculating feature weights is:

[0040]

[0041]

[0042] diff(A,R,H j ) represents samples R and H j Regarding the differences in feature A; H j With M j represents the nearest sample of the same class and the nearest sample of a different class in the training set, respectively; m represents the number of samplings; K represents the number of nearest neighbor samples; P(Class(R)) is the proportion of a randomly selected sample of a certain class; P(C) is the proportion of the number of samples of class C to the total number of samples;

[0043]

[0044] The main characteristic types are m * Each evaluation object contains n evaluation indicators; X is selected from the feature matrix. Tm *The remaining features serve as "comparison sequences" and are used as "reference sequences".

[0045]

[0046] 0<ξ oi (k)≤1, P∈[0-1]

[0047]

[0048] (i = 1, 2, ..., m) * (k = 1, 2, ..., n)

[0049] X ij =Inter(X) i →X j ),X i ,X j ∈A

[0050] Compare sequence X at time k Ti With reference sequence X To The relative difference is used as X Ti For X To The correlation coefficient at time k is ξ. oi (k), Δ 0i (min)=min (i) min (k) |X To (k)-XTi (k)| is called the two-level minimum difference, i.e., X To With all X Ti Select the smallest difference from the smallest absolute differences; Δ 0i (max)=max (i) max (k) |X To (k)-X Ti (k)| is called the maximum difference between the two levels, i.e., X To With all X Ti Select the largest difference from the largest absolute differences; |X To (k)-X Ti (k)|=Δ 0i (k) is called the absolute difference between the reference sequence and the comparison sequence at time k; P is the resolution coefficient; ω k The value of directly affects the degree of correlation. The larger the value, the stronger the correlation between the two features.

[0051] In one possible implementation, the method for improving evaluation indicators and calculating weights for specific anomaly detection algorithms based on pig farm data anomaly detection algorithms, and integrating and optimizing multi-source heterogeneous data anomaly detection methods, includes:

[0052] Select specific scenario datasets based on the actual business needs of pig farms;

[0053] Select a single anomaly detection algorithm and train it to obtain a weak detector in the ensemble learning algorithm;

[0054] Improve the evaluation indicators for data detection effectiveness based on the specific application scenarios of pig farms;

[0055] Improve the weight calculation method of a single detection algorithm in the ensemble learning process, and integrate a strong data anomaly detection model.

[0056] In one possible implementation, the improved method for calculating the weights of a single detection algorithm during ensemble learning, integrating a strong data anomaly detection model, includes:

[0057] By comparing the identification accuracy, execution time, prediction accuracy, and Kappa statistics of typical data anomaly detection algorithms, the weak detector G in the ensemble learning training of the detection algorithm is determined. p m is the number of specific identifiers, p = 1, ..., m;

[0058] The method for calculating the prediction error rate has been improved by comprehensively considering the identification accuracy, prediction accuracy, and kappa statistics. The entropy weight method is used to calculate the weight ω of the evaluation index. j :

[0059]

[0060] l represents the number of data samples, i = 1, ..., l; n represents the number of evaluation indicators, j = 1, ..., n, a ij s represents the evaluation index value in a real-world pig farm setting. ij The root is the proportion of the i-th sample index value under the j-th index, d j To calculate the information entropy value of the j-th item, the comprehensive score g of the weak detector's detection performance is calculated based on the index weights. p Then, the average error rate e is obtained. p ;

[0061]

[0062] The coefficient α of the single detection model is calculated based on the average error rate. p This leads to a linear combination of six specific weak detectors.

[0063] The final integrated data anomaly detection model G:

[0064] G=sign(α1*G1+α2*G2+···α p *G p )

[0065]

[0066] In one possible implementation, the faulty data is corrected according to the following steps:

[0067] We collect the current time-series data set Sw(t0) of the pig farming business scenario and iteratively analyze each point s. i Determine the set of neighboring nodes of the abnormal data, denoted as t′0 (t′0∈t0);

[0068] Based on t′0, a certain amount of historical data is read to form a time node set {t′0, t′}. -1 ,...,t′ k} as input to the collaborative filtering algorithm;

[0069] Find the set {t′0,t′} of t′0. -1 ,...,t′ k The correlation coefficient r of other vectors in} k The data is then sorted, and a certain threshold is set to obtain a similar time-series dataset Rec(t′).

[0070]

[0071] After obtaining the similar time-series data set, recommendation analysis was performed, and the upper and lower bound estimates for outlier data were:

[0072]

[0073] A genetic algorithm is used to optimize the recommended dataset in order to determine the best correction value for bad data.

[0074] In this embodiment, various sensors, monitoring equipment, and 5G wireless transmission devices are deployed in pig farms to acquire multi-source heterogeneous data on production, management, external climate, and internal environment of large-scale pig farms. Data source classification and data feature analysis are then performed. Research is conducted on data quality detection, defective data correction, and data quality improvement technologies in the context of big data in pig farming, thereby enhancing the accuracy and reliability of pig farm data and providing high-quality data support for greenhouse gas emission early warning, comprehensive environmental quality assessment, and lean management strategy formulation for pig farms. Attached Figure Description

[0075] Figure 1 A flowchart illustrating a method for detecting anomalies and correcting bad data in multi-source data from a pig farm, provided in an embodiment of this application;

[0076] Figure 2 This is a schematic diagram of environmental data monitoring and transmission in a pig farm, provided in an embodiment of this application.

[0077] Figure 3 A schematic diagram illustrating the data feature selection and redundancy removal process for pig farms provided in this application embodiment;

[0078] Figure 4 This is a schematic diagram of multi-source data anomaly detection in pig farms provided in an embodiment of this application;

[0079] Figure 5 A schematic diagram of the defective data correction method provided in the embodiments of this application. Detailed Implementation

[0080] The present solution will now be described in conjunction with the accompanying drawings and specific embodiments.

[0081] See Figure 1 The method for detecting anomalies and correcting bad data from multi-source data in pig farms in this embodiment includes:

[0082] S101 acquires heterogeneous data for monitoring, collecting, transmitting, energy consumption, and management in large-scale pig farms.

[0083] See Figure 2Based on the environmental information monitoring needs of pig farms, sensor / data collector nodes are deployed, and various sensors are installed, including temperature sensors, relative humidity sensors, and ammonia / hydrogen sulfide / sulfur dioxide / carbon dioxide sensors, to collect various data indicators in the pigsty in real time and transmit the data through a 5G transmission module; external climate data, including outdoor temperature, relative humidity, wind speed, and air quality, are acquired; based on the pig farm production management system, data on procurement management, cost accounting, pig production, body shape characteristics, and physiological health monitoring are acquired; and based on the pig farm comprehensive energy consumption monitoring and management system, data on biogas / heat / electricity comprehensive energy consumption are acquired.

[0084] S102, classify and analyze the data sources of pig farming, and preprocess the collected heterogeneous data, including outlier removal, data normalization and redundancy removal.

[0085] The data source and characteristics analysis of pig farms are summarized below:

[0086] 1) Data on breeding conditions: including the number of pigs, breeds, body size and physical characteristics, physiological health, feed consumption, and vaccine usage.

[0087] 2) Environmental monitoring data: including temperature, humidity, climate conditions, ammonia concentration, hydrogen sulfide concentration, sulfur dioxide concentration, carbon dioxide concentration, etc.

[0088] 3) Pigsty management data: including pigsty layout, equipment usage, hygiene inspection records, etc.

[0089] 4) Disease monitoring data: including pig health status, epidemic monitoring, etc.

[0090] 5) Feeding program data: including feed formulation, feeding plan, etc.

[0091] 6) Cost and benefit data: including feed costs, labor costs, and breeding profits.

[0092] 7) Pesticide usage data: including the types and amounts of pesticides used to control pests and diseases.

[0093] 8) Energy consumption data: including power consumption of lighting equipment, ventilation and air conditioning, feed processing and handling equipment, cleaning and disinfection equipment, water pumps and irrigation systems, etc.

[0094] 1) The collected raw data was cleaned using the k-means clustering algorithm:

[0095] K-means clustering is a simple and efficient clustering algorithm. It divides a dataset into groups based on data similarity; objects within the same cluster are similar, while objects in different clusters are dissimilar. Using cluster C...i The centroid (the mean of the objects assigned to this cluster) represents the cluster, cluster C. i The quality can be expressed as intra-cluster variation (cluster C) i All objects and centroid c i The sum of squares of the errors between them is a measure, defined as follows: dist(p,c i ) represents the object p∈C i With this cluster representing C i The difference. Assume the nearest center to object p is c. p c p Assigned to c p Average distance between objects l cp Define ratio Outlier samples are identified and removed based on the ratio R.

[0096] 2) Normalize the cleaned data: By standardizing the pig farming data, the data is transformed and normalized to the same range.

[0097] 3) The G-Relief F algorithm is used for data feature selection and redundancy removal in pig farms. Specific implementation steps are detailed below. Figure 3 .

[0098]

[0099] Features A and B are each represented by an n-dimensional array, where A = {a1, a2, ..., a...} n B = {b1, b2, ..., b} n Each sample is a point in an n-dimensional space.

[0100]

[0101] The difference between samples R1 and R2 on feature A is represented by diff(A,R1,R2).

[0102] 4) The formula for calculating feature weights is:

[0103]

[0104]

[0105] diff(A,R,H j ) represents samples R and H j Regarding the differences in feature A; H j With M jrepresents the nearest sample of the same class and the sample of a different class to sample R in the training set, respectively; m represents the number of samplings; K represents the number of nearest neighbor samples; P(Class(R)) is the proportion of a randomly selected sample class; P(C) is the proportion of the number of samples of class C to the total number of samples.

[0106]

[0107] The main characteristic types are m * Each evaluation object contains n evaluation indicators; the feature matrix is ​​selected... The remaining features serve as "comparison sequences" and are used as "reference sequences".

[0108]

[0109] 0<ξ oi (k)≤1, P∈[0-1] (6)

[0110]

[0111] (i = 1, 2, ..., m) * (k = 1, 2, ..., n) (7)

[0112] X ij =Inter(X) i →X j ),X i ,X j ∈A (8)

[0113] Compare sequence X at time k Ti With reference sequence X To The relative difference is used as X Ti For X To The correlation coefficient at time k is ξ. oi (k), Δ 0i (min)=min (i) min (k) |X To (k)-X Ti (k)| is called the two-level minimum difference, i.e., X To With all X Ti Select the smallest difference from the smallest absolute differences; Δ 0i (max)=max (i) max (k) |X To (k)-X Ti (k)| is called the maximum difference between the two levels, i.e., X To With all X Ti Select the largest difference from the largest absolute differences; |X To (k)-XTi (k)|=Δ 0i (k) is called the absolute difference between the reference sequence and the comparison sequence at time k; P is the resolution coefficient, which subjectively reflects the researcher's judgment on Δ. 0i The degree of importance attached to (max) objectively reflects the indirect influence of each factor in the system on the degree of correlation. ω k The value of directly affects the degree of correlation. The larger the value, the stronger the correlation between the two features.

[0114] S103, based on the anomaly detection algorithm for pig farm data, improves the evaluation index and the weight calculation method for specific anomaly detection algorithms, and integrates and optimizes the anomaly detection method for multi-source heterogeneous data.

[0115] See Figure 4 Based on the actual business needs of pig farms, specific scenario datasets are selected.

[0116] Most anomaly detection algorithms assume that the data distribution is constant, meaning that historical and future data follow the same distribution. However, in specific application scenarios, the data distribution may change over time and with state transitions. Therefore, this invention selects a dataset to verify the detection performance based on the actual application scenarios of pig farms (such as greenhouse gas emission early warning, environmental quality assessment, and pig farm energy consumption prediction).

[0117] A single anomaly detection algorithm is selected, and a weak detector is trained within the ensemble learning algorithm. Several typical anomaly detection algorithms selected in this invention are summarized below:

[0118] 1) KNN algorithm for outlier detection: This invention sets a distance score based on the distance of a sample data point to its k-th nearest neighbor. Let the sample set X = {x1, x2, ..., x...} n}, Y = {y1, y2, ..., y n The formula for calculating the nearest sample distance d(X,Y) is as follows:

[0119] 2) CBLOF algorithm for outlier detection: This invention divides the data into small clusters and large clusters, and then calculates the outlier score based on the size of the cluster to which the data sample point belongs and the distance to the nearest large cluster.

[0120] 3) IF Algorithm for Outlier Detection: The IF algorithm uses a binary search tree structure called an isolation tree (iTree) to isolate sample points and detect outliers. Outliers are closer to the root node of the iTree, while normal values ​​are farther away. In the implementation of the IF algorithm in this invention, a set of trees is used to partition the data, an outlier score is set to check the degree of isolation of sample points in the structure, and then the outlier score is used to identify outliers in the sample data.

[0121] 4) FB Algorithm for Outlier Detection: The functional bagging detection method installs many basic detectors on each subsample of the dataset. Each detector identifies different outliers, and the outlier values ​​corresponding to their outlier probabilities are assigned to all data records. Then, the outlier values ​​calculated by each outlier detector are merged to find outliers of higher quality. This invention uses local outlier factors for basic detection.

[0122] 5) HBOS algorithm for outlier detection: Assume the probability density of the i-th feature of sample p is P i (p), through the scoring model Calculate the scores for outliers.

[0123] Improve the evaluation indicators for data detection effectiveness based on the specific application scenarios of pig farms.

[0124] Based on traditional data anomaly detection and evaluation indicators, this invention selects evaluation indicators according to the actual application scenarios in the pig farming process. For example, in scenarios such as greenhouse gas emission early warning and pig farm energy consumption prediction, it considers supplementing prediction accuracy, prediction precision, and Kappa statistics as evaluation indicators for detection effectiveness; in scenarios such as pig farm environmental quality assessment, it considers supplementing quality assessment rationality as evaluation indicator for detection effectiveness.

[0125] Improve the weight calculation method of a single detection algorithm in the ensemble learning process, and integrate a strong data anomaly detection model.

[0126] Ensemble learning algorithms train different weak classifiers on the same training set and then combine these weak classifiers into a strong classifier. The complementary nature of the weak classifiers gives the combined strong classifier strong generalization ability. The core idea of ​​this algorithm is to increase the weights of samples with poor training performance and weak classifiers with strong learning ability in the training set, while decreasing the weights of samples with good training performance and weak classifiers with weak learning ability. This invention selects a single detection algorithm to train a weak detector, and then integrates a strong data anomaly detection model by improving the weight calculation method. The specific implementation steps are as follows:

[0127] 1) By comparing the identification accuracy, execution time, prediction accuracy, and Kappa statistics of typical data anomaly detection algorithms, the weak detector G in the training ensemble learning of six detection algorithms was determined. p m is the number of specific identifiers, and p = 1, ..., m.

[0128] 2) Improved prediction error rate calculation method. This invention comprehensively considers identification accuracy, prediction accuracy, and kappa statistics, and uses the entropy weight method to calculate the weights of the evaluation indicators, with the weight ω. j

[0129]

[0130] l represents the number of data samples, i = 1, ..., l; n represents the number of evaluation indicators, j = 1, ..., n, a ij s represents the evaluation index value in a real-world pig farm setting. ij The root is the proportion of the i-th sample index value under the j-th index, d j To calculate the information entropy value of the j-th item, the comprehensive score g of the weak detector's detection performance is calculated based on the indicator weights. p Then, the average error rate e is obtained. p .

[0131]

[0132] 3) Calculate the coefficient α of the single detection model based on the average error rate. p This leads to a linear combination of six specific weak detectors, which ultimately integrates a strong data anomaly detection model G.

[0133] G = sign(α1*G1 + α2*G2 + ... + α) p *G p )

[0134]

[0135] S104 employs a collaborative filtering recommendation algorithm to reduce the input dependency of error data repair in an unsupervised manner. Starting from horizontal correlation, it searches for a set of historical monitoring point data similar to the current moment, calculates the range recommendation degree, and uses a genetic algorithm to optimize the recommended data set to find the best correction value for poor data in pig farming.

[0136] See Figure 5 The embodiments of this application correct defective data according to the following steps:

[0137] 1) Collect the current time series data set Sw(t0) of the pig farming business scenario, and iteratively analyze each point s. i Determine the set of neighboring nodes of the abnormal data, denoted as t′0 (t′0∈t0).

[0138] 2) Based on t′0, read a certain amount of historical data to form a time node set {t′0, t′}. -1 ,...,t′ k} is used as input to the collaborative filtering algorithm.

[0139] 3) Find the set {t′0,t′} of t′0. -1 ,...,t′ k The correlation coefficient r of other vectors in} k The data are then sorted. A certain threshold is set to obtain a similar time-series dataset Rec(t′).

[0140]

[0141] After obtaining the similar time-series data set, recommendation analysis was performed, and the upper and lower bound estimates for outlier data were:

[0142]

[0143] A genetic algorithm is used to optimize the recommended dataset in order to determine the best correction value for bad data.

[0144] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0145] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for detecting anomalies and correcting bad data in multi-source data from pig farms, characterized in that, include: Acquire heterogeneous data on monitoring, collection, transmission, energy consumption, and management of large-scale pig farms; We conducted data source classification and data feature analysis for pig farming, and preprocessed the collected heterogeneous data, including outlier removal, data normalization, and redundancy removal. Based on anomaly detection algorithms for pig farm data, this paper improves the evaluation index and the weight calculation method for specific anomaly detection algorithms, and integrates and optimizes anomaly detection methods for multi-source heterogeneous data, including: Select specific scenario datasets based on the actual business needs of pig farms; Select a single anomaly detection algorithm and train it to obtain a weak detector in the ensemble learning algorithm; Improve the evaluation indicators for data detection effectiveness based on the specific application scenarios of pig farms; Improve the weight calculation method of a single detection algorithm in the ensemble learning process, and integrate a strong data anomaly detection model; The improved method for calculating the weights of a single detection algorithm during ensemble learning, and the ensemble model for strong detection of data anomalies, include: By comparing the identification accuracy, execution time, prediction accuracy, and Kappa statistics of typical data anomaly detection algorithms, weak detectors in the ensemble learning training of detection algorithms are identified. G p , For a specific number of identifiers, ; The method for calculating the prediction error rate has been improved by comprehensively considering the identification accuracy, prediction accuracy, and kappa statistics, and the entropy weight method is used to calculate the weights of the evaluation indicators. ω j : The number of data samples. ; For the number of evaluation indicators, , These are evaluation index values ​​used in actual pig farm scenarios. The root is the first The first item under the indicator The proportion of each sample indicator value To calculate the first The information entropy value is used to calculate the comprehensive score of the weak detector's detection performance based on the indicator weights. Then, the average error rate is obtained. ; Calculate the coefficients of a single detection model based on the average error rate. α p This leads to a linear combination of six specific weak detectors, ultimately integrating a strong data anomaly detection model. G : G=sign ( α 1 G 1 + α 2 G 2 +··· α p G p ) ； A collaborative filtering recommendation algorithm is adopted to reduce the input dependence of error data repair in an unsupervised manner. Starting from the horizontal correlation, a set of historical monitoring point data similar to the current moment is found, the range recommendation degree is calculated, and a genetic algorithm is used to optimize the recommended data set to find the best correction value for poor data in pig farming.

2. The method for detecting anomalies and correcting bad data in multi-source data of pig farms according to claim 1, characterized in that, The acquisition of heterogeneous data related to monitoring, collection, transmission, energy consumption, and management of large-scale pig farms includes: Based on the environmental information monitoring needs of pig farms, sensor / data acquisition nodes are deployed and various sensors are installed, including temperature sensors, relative humidity sensors, and ammonia / hydrogen sulfide / sulfur dioxide / carbon dioxide sensors. The system collects various data indicators in the pigsty in real time using deployed sensors, and transmits the data through a 5G transmission module. Obtain outdoor climate data, including outdoor temperature, relative humidity, wind speed, and air quality data; Based on the pig farm production management system, data on procurement management, cost accounting, pig production, body size characteristics, and physiological health monitoring are obtained; Based on the integrated energy consumption monitoring and management system for pig farms, data on the combined energy consumption of biogas, heat, and electricity is obtained.

3. The method for detecting anomalies and correcting bad data in multi-source data of pig farms according to claim 1, characterized in that, The classification and feature analysis of data sources for pig farming includes: Data on pig farming: including the number of pigs, breeds, body size and physical characteristics, physiological health, feed consumption, and vaccine usage; Environmental monitoring data includes temperature, humidity, climate conditions, ammonia concentration, hydrogen sulfide concentration, sulfur dioxide concentration, and carbon dioxide concentration. Pigsty management data includes pigsty layout, equipment usage, and hygiene inspection records; Disease surveillance data: including pig health status and epidemic monitoring; Feeding program data: including feed formulation and feeding schedule; Cost-benefit data: including feed costs, labor costs, and farming profits; Pesticide usage data: including the types and amounts of pesticides used to control pests and diseases; Energy consumption data: including power consumption for lighting equipment, ventilation and air conditioning, feed processing and handling equipment, cleaning and disinfection equipment, water pumps and irrigation systems.

4. The method for detecting anomalies and correcting bad data in multi-source data of pig farms according to claim 1, characterized in that, The preprocessing of the collected heterogeneous data includes outlier removal, data normalization, and redundancy removal, including: The raw data was cleaned using the k-means clustering algorithm. The cleaned data is normalized: by standardizing the pig farming data, the data is transformed and normalized to the same range. The G-Relief F algorithm was used to select and remove redundancy from data from pig farms, and the feature weights were calculated.

5. The method for detecting anomalies and correcting bad data in multi-source data of pig farms according to claim 4, characterized in that, The process of cleaning the collected raw data using the k-means clustering algorithm includes: The dataset is divided into groups based on the similarity of the data. Objects in the same cluster are similar to each other, while objects in different clusters are different. Use clusters The centroid represents the cluster, and the cluster The quality is measured by intra-cluster variation, defined as: , , Representation Object With the cluster representative difference; Assuming to object The nearest center is , With assignment Average distance between objects Define ratio According to the ratio Identify and remove outlier samples.

6. The method for detecting anomalies and correcting bad data in multi-source data of pig farms according to claim 4 or 5, characterized in that, The method of using the G-Relief F algorithm for data feature selection and redundancy removal in pig farms includes: feature A , B Use respectively n 3D array representation, Each sample is n A point in 3D space; sample and samples In features A The difference on express.

7. The method for detecting anomalies and correcting bad data in multi-source data of pig farms according to claim 6, characterized in that, The calculation of feature weights includes: The formula for calculating feature weights is: Representative sample R and Regarding features A Differences; and Representing the training set and samples respectively R The nearest samples of the same class and samples of different classes; m Represents the number of samples; K Represents the number of nearest neighbor samples; PClass(R) The proportion of a randomly selected sample category; It is the first C The proportion of class samples to the total number of samples; The main characteristic types are m Each evaluation object contains the following number of evaluation indicators: n Selecting from the feature matrix The remaining features serve as "comparison sequences" and are used as "reference sequences". No. Time Comparison Sequence With reference sequence The relative difference as right exist The correlation coefficient at time is , This is called the two-level minimum difference, i.e. With all Then select the smallest difference from the smallest absolute differences; This is called the maximum difference between the two levels, i.e. With all Select the largest difference from the largest absolute differences; The reference sequence and the comparison sequence are called the first two sequences. k The absolute difference in time; P The resolution coefficient; The value of directly affects the degree of correlation. The larger the value, the stronger the correlation between the two features.

8. The method for detecting anomalies and correcting bad data in multi-source data of pig farms according to claim 1, characterized in that, Correct the defective data according to the following steps: Statistics on the current time series of pig farming business scenario data set Analyze each point in a loop Determine the set of neighboring nodes of the abnormal data, denoted as . ( ); according to Read a certain amount of historical data to form a time node set. As input to the collaborative filtering algorithm; beg gather Correlation coefficients of other vectors The datasets are then sorted, and a certain threshold is set to obtain similar time-series datasets. ; After obtaining the similar time-series data set, recommendation analysis was performed, and the upper and lower bound estimates for outlier data were: A genetic algorithm is used to optimize the recommended dataset in order to determine the best correction value for bad data.

Images