A method and system for handling imbalanced data based on nearest neighbor constraints and consistency screening.

Abstract

This invention relates to the field of data processing technology, and in particular to a method and system for processing imbalanced data based on nearest neighbor constraints and consistency screening. The method includes acquiring industrial fault detection data; performing data preprocessing based on the acquired industrial fault detection data; constructing a minority class MNN skeleton based on DenMune clustering and performing two-layer noise removal; local adaptive oversampling based on MNN sparsity double-layer weights and MNN neighborhood constraint interpolation; dual-threshold screening based on source sub-cluster consistency and original sample dominance; and using a trained model to predict and classify fault samples. This effectively suppresses the introduction of new noise during oversampling, making the final training dataset more stable near the local structure and decision boundary, which is beneficial for improving the stability and robustness of the industrial fault detection model in imbalanced scenarios.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an imbalanced data processing method and system based on nearest neighbor constraints and consistency screening. Background Technology

[0002] With the continuous improvement of intelligent manufacturing and industrial automation, large-scale plants, key equipment, and complex processes in modern chemical, metallurgical, power, and petrochemical process industries increasingly rely on continuous and stable operation. Abnormal operating conditions or equipment failures can lead not only to fluctuations in product quality and increased energy consumption, but also to downtime, equipment damage, and even safety accidents. Therefore, fault detection and diagnosis technologies for industrial processes have become a crucial foundation for ensuring production safety and improving operational efficiency.

[0003] However, in actual industrial operations, enterprises typically aim for "fewer failures and less downtime," with systems operating under normal conditions for extended periods. In contrast, real failures, especially serious and hidden ones, occur less frequently, resulting in a significantly smaller number of failure samples compared to normal samples. Furthermore, there are substantial differences in sample size between different failure types, leading to a typical class imbalance problem at the data level. If a classification model is directly trained on such datasets, the model is often dominated by the more prevalent normal class or some common failure classes, lacking the ability to identify minority failure classes. This can easily lead to "misclassification as normal" or "only recognizing common failures," severely impacting the practicality and reliability of industrial fault detection.

[0004] Taking the classic Tennessee-Eastman (TE) chemical process dataset as an example, this dataset simulates a continuous chemical process consisting of multiple units such as reactors, condensers, separators, and compressors. It includes normal operating conditions and various preset fault conditions. The number of normal operating condition samples is sufficient, while the number of samples for each fault condition is significantly insufficient. Furthermore, different fault types often exhibit a complex pattern in the feature space: high-density clustering within small areas and sparse distribution over a large area overall. Similar phenomena can be observed in the operational datasets of coal mills and feeders in actual thermal power plants: normal samples constitute the vast majority, while some critical faults have only a very small number of recorded samples. Moreover, minority fault samples are often distributed at the boundaries or transition regions of the feature space, mixed with noise points. This imbalanced and structurally complex distribution pattern further weakens the ability of conventional classification models to distinguish minority fault states, making it difficult to meet the safety requirement of "better to overreport than underreport" in industrial scenarios.

[0005] Currently, solutions to class imbalance problems in industrial fault detection mainly fall into three categories: data-level methods, algorithm-level methods, and ensemble methods. Data-level methods directly adjust the distribution of training data through undersampling or oversampling; algorithm-level methods improve the model's focus on the minority class by modifying the loss function and introducing cost-sensitive mechanisms; and ensemble methods improve overall performance by constructing multiple classifiers and making ensemble decisions. Because data-level methods do not depend on a specific classifier, they are easily integrated with existing industrial fault detection models and have strong versatility and engineering transferability, thus being widely adopted in industrial imbalanced data processing scenarios.

[0006] Among these, data-level oversampling techniques, represented by SMOTE and its variants, are the most widely used. These methods typically use Euclidean distance to interpolate and synthesize samples among their nearest neighbors in the minority class, which alleviates the problem of sample imbalance to some extent. However, existing methods generally have two limitations: First, most of them only use "point-to-point" local nearest neighbor relationships for interpolation, failing to explicitly characterize the internal structure of the minority class and making it difficult to distinguish between dense, sparse, and noisy regions within a cluster. This results in synthesized samples that cannot fully reflect the true form of minority class faults in the feature space. Second, the synthesis process lacks constraints on the original structure. It neither sufficiently removes noisy samples before oversampling nor has a mechanism for "structural acceptance" of the synthesized samples in the original feature space. This easily introduces new noisy samples near the class boundaries, causing distortion of the decision boundary and limiting further improvement in the performance of industrial fault detection models. Summary of the Invention

[0007] To address the aforementioned problems, this invention provides a method and system for processing imbalanced data based on nearest neighbor constraints and consistency screening.

[0008] In a first aspect, the present invention provides an imbalanced data processing method based on nearest neighbor constraints and consistency screening, which adopts the following technical solution:

[0009] An imbalanced data processing method based on nearest neighbor constraints and consistency screening includes:

[0010] Acquire industrial fault detection data;

[0011] Data preprocessing is performed based on the acquired industrial fault detection data;

[0012] Minority MNN skeleton construction and two-layer noise removal based on DenMune clustering;

[0013] Local adaptive oversampling based on MNN sparse double-layer weights and MNN neighborhood constraint interpolation;

[0014] Dual threshold screening based on source sub-cluster consistency and original sample dominance;

[0015] Fault sample prediction and classification are performed using a trained model.

[0016] Secondly, an imbalanced data processing system based on nearest neighbor constraints and consistency screening includes:

[0017] The data acquisition module is configured to acquire industrial fault detection data;

[0018] The preprocessing module is configured to perform data preprocessing based on the acquired industrial fault detection data;

[0019] The removal module is configured to construct minority MNN skeletons and perform two-layer noise removal based on DenMune clustering.

[0020] The oversampling module is configured to perform local adaptive oversampling based on MNN sparse double-layer weights and MNN neighborhood constraint interpolation.

[0021] The filtering module is configured to perform dual-threshold filtering based on source subcluster consistency and original sample dominance.

[0022] The classification module is configured to use the trained model to predict and classify fault samples.

[0023] Thirdly, the present invention provides a computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device of the aforementioned unbalanced data processing method based on nearest neighbor constraints and consistency screening.

[0024] Fourthly, the present invention provides a terminal device, including a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, the instructions being adapted to be loaded and executed by the processor to provide the aforementioned imbalanced data processing method based on nearest neighbor constraints and consistency screening.

[0025] In summary, the present invention has the following beneficial technical effects:

[0026] 1. Compared with the background technologies represented by SMOTE, which fail to distinguish the intra-class structure of minority classes and synthesize samples according to a uniform strategy, easily generating a large number of redundant samples in high-density regions and insufficient coverage in sparse regions, this invention is based on the nearest neighbor (MNN) structure output by the DenMune algorithm. It constructs an MNN skeleton dominated by strong points within the minority class and automatically identifies and deletes two types of noise samples through "two-layer noise removal." Furthermore, it utilizes the local sparsity of the samples defined by the number of MNNs. and sub-cluster sparsity Construct sub-cluster level Compared with sample level The dual-layer adaptive sampling weights enable the oversampling intensity to be adaptively adjusted according to the local sparsity. New samples are restricted to local neighborhood interpolation generation within each DenMune sub-cluster supported by the MNN structure. Overall, this prioritizes enhancing the sample density of the minority class in sparse regions and improves the local coverage of the minority class in the feature space, providing a structural foundation for improving the separability and recognition accuracy of rare fault conditions in industrial processes.

[0027] 2. Compared with existing oversampling methods, which typically interpolate only among the nearest neighbors of a minority class, lack posterior structure verification of the synthesized samples, and are prone to introducing new noise and distorting decision boundaries near class boundaries, this invention reconstructs the enhanced minority class set in the original normalized feature space after completing local adaptive oversampling based on MNN. Nearest neighbor and mutual nearest neighbor relationships are used to calculate the consistency index of the source subclusters for each synthetic sample. and the dominant indicators of the original sample and through dual thresholds , A structural consistency screening process is implemented. Samples that cannot form stable MNN relationships with the original samples in the source subcluster or are primarily neighbors of the synthetic samples are automatically removed, retaining only high-quality synthetic samples with consistent structure for model training. This mechanism effectively suppresses the introduction of new noise during oversampling without relying on complex additional hyperparameters, making the final training dataset more stable near the local structure and decision boundary. This is beneficial for improving the stability and robustness of industrial fault detection models in imbalanced scenarios.

[0028] 3. Compared with 11 typical oversampling methods, including SMOTE, Borderline-SMOTE, ADASYN, MWMOTE, CBSO, DBSMOTE, K-means SMOTE, G-SOMO, IR-SMOTE, OREM, and SD-CSMOTE, this invention conducted comparative experiments on four sets of public datasets based on a decision tree (DT) classifier. Friedman ranking analysis results show that, under the conditions of the four selected imbalanced datasets and the DT classifier, the MNSCO method proposed in this invention achieves a high average ranking across all three metrics, demonstrating superior overall performance and showing an advantage over the 11 typical oversampling methods compared. Experimental results indicate that, under imbalanced data distribution conditions, this method can simultaneously improve minority class identification ability and overall discriminative performance, exhibiting good generalization and adaptability. It can serve as a sample balancing scheme in industrial fault detection tasks and can be extended to other imbalanced classification scenarios in industrial processes. Attached Figure Description

[0029] Figure 1This is a schematic diagram of an imbalanced data processing method based on nearest neighbor constraint and consistency screening according to Embodiment 1 of the present invention;

[0030] Figure 2 This is a schematic diagram of the architecture of an imbalanced data processing method based on nearest neighbor constraints and consistency screening according to Embodiment 1 of the present invention;

[0031] Figure 3 This is the original data distribution diagram of the wine dataset in Embodiment 1 of the present invention.

[0032] Figure 4 This is a data distribution diagram of the wine dataset after processing by the MNSCO method in Embodiment 1 of the present invention.

[0033] Figure 5 This is the Friedman ranking chart of the F1 index of each method based on the decision tree (DT) classifier in Embodiment 1 of the present invention.

[0034] Figure 6 This is the Friedman ranking chart of the G-mean index of each method based on the decision tree (DT) classifier in Embodiment 1 of the present invention.

[0035] Figure 7 This is the Friedman ranking chart of the AUC index of each method based on the decision tree (DT) classifier in Embodiment 1 of the present invention. Detailed Implementation

[0036] The present invention will be further described in detail below with reference to the accompanying drawings.

[0037] Example 1

[0038] Reference Figure 1 This embodiment of an imbalanced data processing method based on nearest neighbor constraints and consistency screening includes:

[0039] In the industrial fault detection scenario described in this invention, industrial fault detection data mainly refers to multivariable process and state quantities, such as temperature, pressure, flow rate, valve opening, rotational speed, and vibration amplitude, collected by distributed control systems (DCS), monitoring and data acquisition systems (SCADA), or online monitoring devices. These data are then processed through time window slicing and feature extraction to form structured numerical sample data. Specifically, the continuously acquired process monitoring time series is divided according to a preset time window, and statistical or engineering features such as mean, variance, extreme values, and trend coefficients are extracted within each time window. This transforms the original time series into a feature matrix for subsequent fault mode identification and classification.

[0040] This method targets industrial fault detection datasets and includes the following steps: preprocessing of the industrial fault detection dataset, construction of a minority class MNN skeleton based on DenMune and two-layer noise removal, local adaptive oversampling based on MNN sparsity two-layer weights and MNN neighborhood constraint interpolation, dual threshold screening based on source subcluster consistency and original sample dominance, and construction and application of the industrial fault detection model. The overall processing flow is as follows: Figure 1 As shown.

[0041] like Figure 2 As shown, the method of this invention takes an industrial fault detection dataset as input. During the model construction and training phase, the dataset is divided into a training set and a test set according to a preset ratio. Subsequently, the data preprocessing operations described in step S1 are performed on the training set and the test set respectively, including feature numerical conversion, missing value handling, and normalization, to obtain preprocessed training and test sets. In the preprocessed training set, samples are divided into majority class samples and minority class samples according to the class labels. The minority class samples in the training set are sent to the sample enhancement layer, which successively undergoes minority class MNN skeleton construction and double-layer noise removal based on DenMune, local adaptive oversampling based on MNN sparsity double-layer weights and MNN neighborhood constraint interpolation, and double threshold screening based on source subcluster consistency and original sample dominance to obtain enhanced high-quality minority class samples. The enhanced minority class samples are merged with the preprocessed majority class samples in the training set to form a balanced training set, which is used for training the industrial fault detection model and the classification model in the prediction layer. In industrial fault detection scenarios, the original label set typically includes normal operating conditions and various fault categories. Each fault category is usually significantly smaller in sample size than the normal operating conditions and can be considered as multiple minority fault categories. Augmentation is performed separately for each fault category; the augmentation scale for each category is determined by the difference in size between the augmented and reference sample sets. Finally, the augmented samples for all fault categories are merged with the normal operating condition samples to form a more balanced training dataset.

[0042] The test set samples undergo only data preprocessing and do not participate in sample augmentation layer operations. After training is complete, during the evaluation phase, the preprocessed test set samples are input into the trained model, which outputs the prediction results of fault categories or normal operating conditions for objective evaluation of model performance.

[0043] S1. Preprocessing of Industrial Fault Detection Datasets

[0044] This step aims to transform the sample data in the industrial fault detection dataset into a numerical feature representation suitable for subsequent distance measurement, cluster analysis, and sampling processing, which helps to ensure the stability of subsequent clustering and oversampling mechanisms based on the nearest neighbor structure.

[0045] The industrial fault detection dataset refers to a "sample-feature" structured numerical matrix formed by dividing multivariate process quantities and state quantities collected by an industrial process monitoring system into time windows and extracting features.

[0046] The preprocessing includes: feature numerical conversion, missing value handling, and normalization operations, in order to reduce the impact of features of different dimensions and types on the stability of subsequent algorithms.

[0047] S1.1. Feature Numerical Conversion: In industrial fault detection data, in addition to a large number of numerical features, there are usually also categorical features such as "operating condition indicators," "equipment operating modes," and "valve on / off status." Since these non-numerical features cannot be directly used in numerical calculations, subsequent operations such as clustering will fail if they are not converted. Therefore, they need to be numerically converted first.

[0048] This step employs one-hot encoding to convert all categorical feature values ​​in the dataset into numerical values, directly retaining features that were originally numerical. Specifically, for each categorical feature dimension, all its values ​​are first counted, and a binary indicator variable is constructed for each value. When a sample takes that value for that feature dimension, the corresponding indicator variable is set to 1; otherwise, it is set to 0, replacing the original categorical feature. After this processing, all categorical features are converted into numerical features, forming a unified numerical feature matrix together with the original numerical features, providing a unified numerical basis for subsequent distance metrics and cluster analysis.

[0049] S1.2. Missing Value Handling: After completing the feature numerical transformation and obtaining a unified numerical feature matrix, it is necessary to handle the missing values ​​in this matrix. In the actual industrial process data acquisition and feature extraction process, some variables may have missing values ​​due to data acquisition anomalies or other reasons. If distance calculations or cluster analyses are performed directly on the numerical matrix containing missing values, it will lead to measurement distortion and may even cause algorithm errors.

[0050] This step may include, but is not limited to, the following methods:

[0051] Features or samples whose missing proportions do not meet the preset completeness requirements are removed to avoid interference from extremely missing data on the overall distribution;

[0052] For data retained from the original numerical features, missing values ​​can be estimated using methods such as mean imputation and median imputation.

[0053] For data derived from original categorical features through one-hot encoding, missing values ​​can be handled by treating "missing" as an independent value of the feature.

[0054] In one specific embodiment of the present invention, for data with a low missing rate that is retained from the original numerical features, the mean of the feature is used for imputation; for data converted from the original categorical features through one-hot encoding, the missing value is treated as an independent value of the feature; through the above missing value processing, the feasibility and stability of subsequent distance measurement and matrix operation can be guaranteed without significantly changing the overall data distribution.

[0055] S1.3. Normalization: After completing the feature numerical transformation and missing value handling, the features in the resulting numerical feature matrix may still have different dimensions and value ranges. For example, the dimensions of temperature, pressure, flow rate, power, and vibration amplitude may differ significantly. If directly used for distance calculation, features with large numerical ranges will have an excessive influence on the results. To avoid this problem, this step can use Min-Max normalization to map each feature to a numerical range with a comparable scale.

[0056] In one specific embodiment of the present invention, the Min-Max normalization method is used to map the values ​​of each feature dimension to the [0,1] interval. For the data matrix... The dimensional feature, and its normalization calculation formula are as follows:

[0057]

[0058] in, Describes any sample at the th The original values ​​on the dimensional features, This represents the minimum value of this feature across all samples. This represents the maximum value of this feature across all samples. This represents the normalized feature value. When a certain feature dimension takes the same value across all samples (…), This dimension of the feature can be uniformly mapped to 0 to avoid the abnormal situation where the denominator is 0.

[0059] After processing each dimension of the numerical feature matrix according to formula (1), the normalized data matrix is ​​obtained. ,in The total number of samples, The characteristic dimension is denoted by Matrix. It can be used for subsequent distance measurement, cluster analysis and sampling processing.

[0060] In other embodiments, data preprocessing may further include a feature selection step to further remove redundant or irrelevant features based on the specific application scenario.

[0061] After the above preprocessing steps, the categorical features in the industrial fault detection dataset are converted into computable numerical features, missing values ​​are filled in appropriately, and numerical features of different dimensions are mapped to the same or similar numerical ranges, providing a unified and stable data foundation for subsequent DenMune-based clustering analysis and local adaptive oversampling.

[0062] S2. Minority MNN Backbone Construction and Two-Layer Noise Removal Based on DenMune

[0063] This step utilizes the Mutual Nearest Neighbors (MNN) structure obtained from the DenMune clustering algorithm on the minority class sample subset to construct an MNN skeleton composed of strong points, and obtains a clean set of minority class sub-clusters through a "two-layer noise removal" mechanism. Unlike existing methods that directly apply general density clustering to the overall data or the minority class, this step does not merely use DenMune as a clustering tool, but explicitly extracts MNN structural information for noise identification and sub-cluster structure construction, providing a dedicated structural information foundation for subsequent oversampling under MNN neighborhood constraints.

[0064] It should be noted that: steps S2 to S4 below are performed on a per-fault-category basis; the fault-category refers to a fault category (minority category) currently being processed in the training set, excluding the normal operating condition category.

[0065] This step uses the normalized feature matrix obtained in step S1 on the training set. Based on this, targeting the minority subset of fault categories. First of all Perform t-SNE dimensionality reduction mapping to a lower-dimensional space to obtain the dimensionality-reduced representation matrix. Then The DenMune algorithm is used for clustering to identify the distribution within minority classes of the fault categories and remove original noise. Let... ,in A vector representing a minority class sample. The total number of minority class samples for the fault category. The DenMune algorithm identifies dense regions based on MNN, where the neighborhood size is... These are the core parameters that need to be set. Among them, the neighborhood size... A positive integer, used to construct mutual... Neighborly relations. To avoid Too small a value can cause the MNN structure to break down, or Excessive size leads to blurred cluster boundaries; this invention will... The candidate range is defined as follows ,in ; and within the training set, candidates use Adaptive selection using cross-validation: for each candidate Perform steps S2 to S4 and calculate the cross-validation mean of the target evaluation index, then select the index that is optimal for that index. As the final value; when multiple When indicators are similar, prioritize the smaller one. To reduce the risk of excessive interconnection. In this embodiment, for ease of explanation, [the following is taken as an example]. As an example value, the algorithm is applicable to larger values. It exhibits good stability within the specified value range, automatically detects and removes noise points, and accurately identifies the target cluster. By performing DenMune clustering on the minority class samples of the fault category, it can divide them into several sub-clusters, providing a reasonable local synthesis range for subsequent oversampling; it can also effectively filter noise points, avoiding the negative impact of these noises on the quality and representativeness of the synthesized samples during the sample synthesis process.

[0066] Related studies have shown that t-SNE performs well in preserving the local neighborhood structure of high-dimensional data, which is beneficial for revealing the sub-cluster structure based on MNN in a low-dimensional space. In a preferred embodiment of the present invention, t-SNE is used to perform two-dimensional mapping on the minority class samples of the fault category in order to construct a stable MNN skeleton; in other embodiments, other dimensionality reduction methods such as PCA can also be used to obtain a low-dimensional representation, and then DenMune clustering can be performed on this low-dimensional space.

[0067] S2.1. Dimensionality Reduction Based on t-SNE

[0068] Because the original feature space has a high dimensionality, and clustering algorithms are better able to capture cluster structures of arbitrary shapes and densities in low-dimensional spaces, this step first reduces the dimensionality of the minority class sample subset corresponding to the fault category. For the minority class sample subset of the fault category obtained in step S1... ,in The original feature dimension is given. The t-SNE algorithm is used to map it to a lower-dimensional space, yielding the matrix. ,in The feature dimension after dimensionality reduction is taken as [value]. In a specific embodiment of the present invention, [value] is taken as [value]. t-SNE can effectively preserve the local structure in high-dimensional data, and is particularly suitable for discovering complex structures and cluster morphologies, providing an intuitive and structurally clear two-dimensional representation for subsequent distance-based clustering analysis.

[0069] It should be noted that the dimensionality reduction process of this invention is performed on the minority class sample subset corresponding to the fault category, rather than on the entire training set, thereby significantly reducing computational overhead. Furthermore, in high-dimensional, large-scale scenarios, the following engineering acceleration method can be adopted without affecting the principle of this invention: first, pre-reduce the minority class subset to a lower dimension before performing t-SNE.

[0070] S2.2. Definition of Nearest Neighbor Relationship

[0071] In dimensional reduction space In the context, for any two sample points and Its Euclidean distance is calculated according to equation (2):

[0072]

[0073] Based on Euclidean distance, for each sample The following three neighborhood sets are defined to characterize their local spatial relationships:

[0074] 1. Neighbor set : From distance samples Recent A set consisting of 100 sample points.

[0075] 2. Reverse Neighbor set By all Incorporate it into itself The set of sample points in the nearest neighbor set, i.e. .

[0076] 3. Mutual Neighbor set : refers to all samples mutual The nearest neighbor set, i.e., both sides include each other in their own set of samples. In the nearest neighbor. Mathematically defined as a sample. of Nearest neighbor set and reverse The intersection of nearest neighbor sets, i.e. . indicates those related to The set of sample points that are each other's nearest neighbors is used to characterize stable local interconnections.

[0077] S2.3. Sample pre-classification based on local density

[0078] To evaluate the local density of each sample, this invention first constructs the mutual density of each sample in the reduced-dimensional space. Neighbor set For any sample :

[0079] when If the sample does not form a nearest neighbor relationship with any other sample, it cannot be integrated into any locally high-density interconnection structure. Such samples are directly classified as Class I noise points, eliminated in the first stage of the algorithm, and will not be used in subsequent MNN skeleton construction and oversampling processes.

[0080] For satisfying To quantify the local density of the samples, a local density index is introduced. Its definition is:

[0081]

[0082] in, For the sample The reverse Nearest neighbor set, representing the set of neighbors. Incorporate into itself The sample set of the nearest neighbor set. In this invention, Neighbor set The size is fixed as ,therefore The above formula can be equivalently written as . The larger the value, the more neighboring points the point is considered as a neighbor, the higher its local density, and the more central the data distribution; conversely, the smaller the value, the more isolated the point is.

[0083] In removal After the first type of noise point, only the remaining satisfying The sample, according to The values ​​are pre-classified and divided into two categories:

[0084] Strengths: Satisfies The sample. This means the sample The number of its reverse nearest neighbors is not less than the number of its nearest neighbors, indicating that it is in a highly interconnected and stable region, and is an ideal candidate point for forming the MNN skeleton.

[0085] Weakness: Satisfaction The sample. These points have a certain number of reverse nearest neighbors, but are not recognized by all their positive neighbors, and their final category assignment is somewhat uncertain.

[0086] Therefore, from an overall perspective, the minority samples of the aforementioned fault categories can be considered to consist of three types of samples: first, the first type of noise points identified and eliminated in the aforementioned steps; second, the remaining samples based on... The third strength of the judgment is based on Weaknesses are identified. Subsequent MNN skeleton construction relies solely on strong points; weak samples are incorporated into their corresponding subclusters via a voting mechanism after the MNN skeleton is built.

[0087] S2.4. Constructing the MNN skeleton:

[0088] The MNN skeleton uses all strong points as skeleton nodes, and the connections between these strong points are determined by their mutual relationships. The nearest neighbor set is used to determine this. The specific steps are as follows: Traverse all strong points, and for any two strong points... and If its extended set and There exists an intersection, meaning the following conditions are met:

[0089] If the two are considered to be locally connected, they are grouped into the same MNN skeleton cluster. By iterating this process, all strong points that are directly or indirectly connected through this neighborhood relationship will be gradually classified into the same skeleton, eventually forming multiple independent MNN skeleton clusters.

[0090] S2.5. Weakness Merging Based on Hard Voting Mechanism for Type II Noise Identification

[0091] Weak samples in the pre-classification process need to be appropriately assigned to the established MNN backbone clusters, or classified as Class II noise. This process employs a hard voting mechanism to ensure robustness of the assignment:

[0092] For each weakness Examine itself and its interactions Groups formed by neighbors .

[0093] If this group intersects with a certain MNN skeleton cluster, then voting is initiated. Statistics are then compiled for this group. The number of strong points already assigned to each MNN backbone cluster. Weak points... Assign it to the cluster with the most strong nodes. This approach essentially allows the weak nodes' MNNs to jointly determine their affiliation, thus enabling them to reliably attach to the most relevant core clusters.

[0094] If this group If a weak sample and its MNN have no intersection with any of the constructed MNN backbone clusters, it indicates that the weak sample and its MNN have no reliable connection with any of the core clusters and are in a free state. Therefore, this type of weakness is marked as a second type of noise point and will be removed in subsequent steps.

[0095] After the above steps, the DenMune algorithm provides a low-dimensional representation of the minority class samples of the input fault category. Cluster partitioning and noise identification were completed. Both Class I and Class II noise points were considered noise samples that might interfere with the quality of subsequent synthesized samples and were removed. The final cluster set was obtained. Each cluster It contains all strong samples belonging to this cluster and weak samples merged in through a voting mechanism. (Set) The sub-cluster structure defined by the MNN relation within the minority class samples of the fault category after denoising was accurately characterized. Each sub-cluster was automatically generated by DenMune on the MNN skeleton and two types of noise points were removed. This provides a clear and clean data partitioning basis for the next stage of oversampling in the local neighborhood jointly defined by DenMune sub-clusters and MNN.

[0096] Unlike many cluster-based oversampling methods that mainly utilize cluster labels after density clustering and perform conventional SMOTE or random interpolation independently within each cluster, this invention not only obtains the denoised sub-cluster partitioning results on minority class samples, but also explicitly preserves the MNN relationship in the DenMune calculation process.

[0097] In subsequent steps, these denoised subclusters and their MNN structures are not only used to indicate the cluster to which a sample belongs, but are further used to construct a system based on... The sparsity index is determined, the sampling weights at the sub-cluster level and the sample level are determined, and the new samples are constrained to be generated along the neighborhood of the MNN, so that the structural information obtained in the clustering stage can be integrated into the entire oversampling process.

[0098] S3. Local adaptive oversampling based on MNN sparse double-layer weights and MNN neighborhood constraint interpolation

[0099] This step constructs a two-layer adaptive oversampling mechanism driven entirely by the MNN structure: at the sub-cluster level, the oversampling intensity is automatically allocated according to the MNN sparsity of the DenMune sub-cluster; at the sample level, seed samples are selected according to the local MNN sparsity; and linear interpolation is performed only in the local neighborhood jointly defined by the DenMune sub-cluster and its MNN structure, thereby prioritizing the enhancement of sparse regions and avoiding overfitting to high-density regions.

[0100] In step S2, DenMune clustering and the construction of nearest neighbor relationships are both performed using the dimensionality-reduced representation of the minority class samples of the fault category. The process is completed in the middle; simultaneously, the representation of the minority class samples of the fault category in the normalized feature space is denoted as... The two are linked one-to-one through sample indices, thus allowing for [interaction / coordination] in the feature space. While performing sample synthesis, it still utilizes the space of dimensionality reduction. The MNN structure and sub-cluster partitioning results obtained from the calculation.

[0101] The minority class sample set of the fault categories participating in DenMune processing in step S2 is Its dimensionality reduction representation is After noise removal, only samples belonging to each sub-cluster are retained, resulting in the denoised minority class sample set of the fault category, denoted as . Accordingly, we denote the representation of these samples in the reduced-dimensional space as... Its index set is denoted as ,satisfy:

[0102]

[0103] in, Subclusters The set of indexes for the samples in the middle. For index The vector representation of the sample in the reduced-dimensional space. Since the sample has already satisfied the condition in step S2... The samples were identified as Class I noise and removed. In the denoised set, each sample satisfies... .

[0104] Based on the DenMune sub-cluster and MNN structure described above, this step uses the local sparsity index defined by MNN to drive the oversampling intensity allocation. Under the constraints of the MNN structure, interpolation synthesis is performed only in the local neighborhood jointly defined by the DenMune sub-cluster and its MNN structure, so as to prioritize the enhancement of sparse regions and avoid overfitting in already highly dense regions.

[0105] S3.1. Subcluster Sparsity Measurement Based on MNN

[0106] To characterize the sparsity within each DenMune subcluster, this step is based on the mutual data constructed in step S2. Neighbor set First, a local sparsity index is defined for each sample, and then the overall sparsity index of the subcluster is obtained.

[0107] For subclusters Any sample point within Record their mutual Nearest neighbor size is In the denoised minority class sample set of the aforementioned fault categories, each sample satisfies... This step defines the local sparsity index of the sample as:

[0108]

[0109] When a sample is located in a locally high-density interconnection structure, the number of samples that are its nearest neighbors is relatively large. Larger Smaller; when the sample area is sparse and has few MNN relationships with other samples. Smaller Larger. Therefore, It can be used as a measure of the local sparsity of MNN-based structures.

[0110] At the sub-cluster level, for sub-clusters The local sparsity of all samples within the cluster is averaged to obtain the sparsity index of the subcluster:

[0111]

[0112] in, sub-clusters The number of samples. The larger the value, the fewer MNN relationships there are in the subcluster as a whole, and the sparser the overall distribution, indicating that it belongs to a sparse region that needs to be strengthened.

[0113] It can be seen that, and Completely determined by the number of each sample's K-nearest neighbors This approach avoids the problem of needing to pre-set density thresholds or distance bandwidths in traditional methods, allowing sparsity metrics to be naturally integrated with the MNN structure.

[0114] S3.2. Adaptive Sampling Weight Allocation Based on MNN Sparsity

[0115] For the denoised minority class sample set corresponding to the fault categories in the training set The total number of samples is The number of samples in the normal operating condition category in the training set is denoted as The number of fault categories in the training set (excluding normal operating conditions) is denoted as To establish a correlation between the enhancement scale of each fault category and the number of normal operating condition samples, and to avoid padding each fault category to the required level... This leads to an increase in sample size; the reference sample size for each fault category is pre-set to be... To mitigate the sample size difference between the fault category and the reference sample size, the total number of minority class samples required to be synthesized for the fault category is:

[0116]

[0117] At the sub-cluster level, based on sub-cluster sparsity Determine the sampling weight for each DenMune subcluster. The specific steps are as follows:

[0118] The total number of synthesized samples required to synthesize the minority classes of the aforementioned fault categories. The sparsity of subclusters is proportionally allocated. First, the relative sparsity weight of each subcluster is calculated according to the following formula:

[0119]

[0120] This yields a set satisfy and It can be directly used for subsequent allocation of synthetic sample numbers.

[0121] According to sparsity weight The total number of synthetic samples for the fault categories The subclusters are assigned to each subcluster, thus obtaining the subclusters. The corresponding number of synthetic samples:

[0122]

[0123] in, This indicates a rounding operation. In actual implementation, the number of synthesized samples in the last sub-cluster can be adjusted compensatorily to ensure... .

[0124] Within the sub-cluster, the local sparsity index of the samples is further utilized. As a weight for selecting synthetic seed samples. For a given sub-cluster Based on the local sparsity index of each sample within the sub-cluster Determine the internal weights of the samples selected as seed samples:

[0125]

[0126] The larger the sample size, the better. The sparser the local region, the higher the probability that it will be selected as a "seed sample" in subsequent sample synthesis, thereby achieving key compensation for sparse regions within the sub-cluster.

[0127] Thus, a set of conditions is formed at the sub-cluster level. The sparsity weights are used to calculate the total number of synthetic samples for the fault category. They are allocated to each sub-cluster according to the principle of "the sparser, the greater the weight". Within each sub-cluster, further allocation is based on... Constructing sample-level weights This achieves targeted compensation for sparse regions within the cluster, and the two together constitute the "DenMune sub-cluster level" ( ) + Sample level ( The two-layer adaptive sampling weight system is ").

[0128] S3.3. Local Sample Synthesis Strategy Based on MNN Structure Constraints

[0129] After calculating the sampling weights at the sub-cluster and sample levels, this step performs specific oversampling operations under the constraints of the MNN structure. Unlike traditional interpolation methods that randomly select paired samples based solely on Euclidean distance, this invention strictly restricts the generation of new samples to a local neighborhood within the DenMune sub-cluster supported by the MNN structure. This ensures that new samples only appear in structurally reliable regions, thereby avoiding disruption of the original local structure of the minority classes.

[0130] For each subcluster Repeat the following process until generated One sample:

[0131] (1) Seed sample selection

[0132] According to weight Within a subcluster, the set of indices Perform weighted random sampling and select a sample index. As a seed sample, its vector in the normalized feature space is The sparser the local area, the more... The larger, The larger the value, the higher the probability of it being selected as a seed sample, thus synthesizing more new samples in the sparse regions within the sub-cluster.

[0133] (2) Paired sample selection under MNN constraints

[0134] For clusters The set of indices for the samples is then defined within the sub-cluster:

[0135]

[0136] That is, to consider and They are nearest neighbors and belong to the same subcluster The samples are used as candidate pairing points. If If not empty, then randomly select an index from them. Its vector in the normalized feature space is denoted as In this way, new samples are always generated within the same DenMune sub-cluster and along the neighborhood supported by the MNN structure, avoiding unreasonable interpolation across sub-clusters or weakly connected regions.

[0137] When individual subclusters exist When a point is empty, you can fall back to using the empty point subcluster. Neighbor set As a candidate pairing set, any sample point is randomly selected from the candidate pairing set as a candidate pairing point to ensure the stability of the synthesis process.

[0138] (3) Sample synthesis:

[0139] Generate new sample vectors using linear interpolation. :

[0140]

[0141] in, It is a random number within the interval (0,1).

[0142] (4) Sample labeling and dataset update:

[0143] Synthetic sample As a new minority class sample added to the denoised minority class set, the set of minority class samples after the oversampling enhancement of the fault category can be denoted as follows:

[0144]

[0145] After completing the above synthesis process for all subclusters, an enhanced minority class sample set of the fault category can be obtained. .

[0146] By employing a local adaptive oversampling strategy based on MNN sparse double-layer weights and MNN neighborhood constraint interpolation, it can be ensured that new samples mainly fall within the reduced-dimensional space. The DenMune subclusters and their MNN structures are characterized and mapped to the normalized feature space. Within the corresponding local neighborhood, it can selectively fill sparse regions while avoiding crossing different subclusters or introducing significant noise. This improves the representativeness of the synthesized samples and the generalization ability of subsequent fault detection models.

[0147] S4. Dual-threshold screening based on source subcluster consistency and original sample dominance

[0148] For the minority class samples of the fault category, after the local adaptive oversampling in step S3, the enhanced minority class sample set of the fault category is obtained. .in, This refers to the original minority class sample set of the fault categories retained after DenMune noise removal in step S2. A minority class sample set is synthesized for the fault categories generated in step S3.

[0149] During the synthesis process, each synthesized sample Each inherits its seed sample during generation. The subcluster number. That is, the seed sample index. Then the new sample will be marked as originating from the sub-cluster. For ease of description, please note... This represents the index set of all minority class samples synthesized from the fault categories generated in step S3. ,and for The complete set of sample indexes.

[0150] This step is performed in the original normalized feature space. The MNN structure is reconstructed, and based on the dual constraints of "source subcluster consistency" and "original sample dominance", the synthetic samples of the fault category are screened for structural consistency, thereby eliminating synthetic samples that deviate from the original spatial MNN structure, may cause class boundary ambiguity, or do not conform to the distribution.

[0151] It should be noted that the DenMune clustering and the construction of the initial MNN structure in this invention are performed in a dimensionality reduction space. The process is completed in step S1, while the structural consistency screening of the synthesized samples is performed in conjunction with the data matrix in step S1. Both processes operate within the same original normalized feature space, but each serves a different function. On one hand, t-SNE maps high-dimensional minority class samples to a low-dimensional space. This is beneficial for DenMune to reveal arbitrarily shaped clusters and local MNN structures on complex manifolds; therefore, step S2 is chosen to be... Clustering and noise removal are performed in the spatial representation; on the other hand, the new samples synthesized in step S3 are directly generated by linear interpolation within the original normalized feature space, and the subsequent industrial fault detection model is also trained in the original normalized feature space. Based on the above reasons, in step S4, this invention uses the minority class sample set enhanced with the fault category within the original normalized feature space. Reconstruct the node set The nearest neighbor and MNN relationships are used to calculate the sub-cluster consistency index and the original sample dominance index. This allows for structural consistency screening of the synthesized samples for the fault categories, thereby achieving "in dimensionality reduction space..." The division of labor mechanism is to "perform robust modeling and denoising in the middle and complete the final structural acceptance in the original normalized feature space".

[0152] S4.1. MNN relation reconstruction on the enhanced minority class set

[0153] In the original normalized feature space, the minority class sample set after the fault category enhancement can be represented as: For any two samples , Its Euclidean distance is defined as:

[0154]

[0155] Based on the above distance metric, in The above is for each sample Rebuild Nearest neighbor and mutual nearest neighbor relationships. The specific definitions are as follows:

[0156] Nearest neighbor set: for each sample , record it in On Nearest neighbor set ,in .

[0157] Reverse Nearest neighbor set: corresponding reverse Nearest neighbor set .

[0158] mutual Neighbors: each other Nearest neighbor set is defined as That is, on the minority class set after oversampling, and mutual The set of nearest neighbor samples.

[0159] If for a certain synthetic sample have An empty set indicates that the sample After oversampling, if a sample fails to form a nearest neighbor relationship with any other sample in the minority class sample set and is in a clearly isolated state, such a synthetic sample will be directly regarded as an unqualified sample in the subsequent structural consistency screening.

[0160] In the specific implementation, to reduce the number of minority classes after enhancement Reconstruction Neighbors and mutual The computational cost of nearest neighbor relationships can be reduced to only apply to the original normalized feature space. Build an efficient nearest neighbor retrieval index and use batch queries to obtain the results of each sample at once. Neighbor set KD-Tree can be used for low- to medium-dimensional scenarios, while BallTree can be used for higher-dimensional scenarios. The query process can be parallelized, which can significantly reduce the reconstruction time in the structure acceptance stage when the number of synthesized samples is large.

[0161] S4.2. Subcluster Consistency and Original Sample Dominance Indicators Based on MNN

[0162] For each of the aforementioned fault categories, a minority class of synthetic samples The source subcluster number was recorded during the oversampling phase and is denoted as... ,Right now sub-clusters A seed sample within the sample was obtained through interpolation. At this point, it is still based on... Subclusters The index set of the original denoised minority class samples. It is still the set of indices of the original minority class samples after denoising the fault category.

[0163] In the enhanced MNN structure The consistency between the synthesized samples and the original MNN structure is measured from the following two aspects:

[0164] 1. Subcluster consistency index:

[0165]

[0166] This indicator reflects the synthetic sample What percentage of the nearest neighbors of a cluster come from its source sub-cluster? The original denoised sample. If Higher, indicating In the enhanced MNN structure, it remains embedded within the stable interconnect structure of the source subcluster and its neighborhood, without being significantly shifted to the vicinity of other subcluster boundaries.

[0167] 2. Dominant Indicators of the Original Sample

[0168]

[0169] This indicator reflects What percentage of the nearest neighbors are original denoised minority class samples of the fault category, rather than synthetic samples? Higher, indicating The samples are mainly "surrounded" by the original samples of the fault category, and there will be no situation where a group of synthetic samples act as each other's neighbors, creating a sample cluster that does not match the true distribution in the originally sparse area.

[0170] when When the set is empty, it indicates that the synthesized sample cannot be integrated into the oversampled MNN structure, and in this invention, it is agreed that no further calculation is required. and The synthesized sample was directly regarded as a structurally abnormal sample and was removed in subsequent screening.

[0171] S4.3. Structural Consistency Screening Based on Dual Thresholds and Construction of the Final Minority Class Set

[0172] To enable automatic screening of the synthesized samples for the aforementioned fault categories, this step pre-sets two structural consistency threshold parameters:

[0173] Subcluster Consistency Threshold ;

[0174] Original sample dominance threshold .

[0175] In the specific implementation, and The threshold values ​​can be adjusted according to the characteristics of the dataset and the requirements of the fault detection task. This invention does not limit the specific values, only requiring that both values ​​be within the (0,1) interval. In a preferred embodiment of this invention, to balance strict screening with effective sample retention, a dual threshold is used. and The default value is set to 0.5, but different industrial scenarios have different requirements for the "reliability of synthetic samples" (e.g., high-risk fault detection requires more stringent screening). Therefore, a grid search can be used to adaptively select the appropriate value. and The optimal value is determined to balance the risk of false negatives and the sample retention rate. The performance of different thresholds is evaluated on the training set using the F1 metric, and the threshold that best suits the real-world scenario is selected.

[0176] For each synthetic sample index The following determination rules shall be applied:

[0177] like Then directly take the sample Samples marked as structurally inconsistent are removed.

[0178] like Then calculate its subcluster consistency index. Dominant indicators of the original sample Only if both conditions are met At that time, retain the sample Otherwise, it is considered a synthetic noise sample that is inconsistent with the original spatial MNN structure and is discarded.

[0179] Let the set of synthetic sample indices of the fault categories retained after consistency screening using the above dual-threshold structure be denoted as . Then the final minority class sample set of the fault category can be represented as:

[0180]

[0181] in, The original minority class sample set of the fault categories retained after denoising in DenMune. A high-quality minority class sample set is synthesized for the fault categories retained after structural consistency screening.

[0182] Therefore, this step further filters out unreliable synthetic samples without excessively altering the original minority class structure and DenMune sub-cluster partitioning. This ensures that the final retained synthetic samples can both enhance the coverage of sparse regions and maintain consistency with the original spatial MNN structure, thereby improving the credibility and discriminative ability of the training data for the industrial fault detection model.

[0183] S5. Construction and Application of Industrial Fault Detection Models

[0184] S5.1. Construction and Training of Industrial Fault Detection Model

[0185] In the model building and training phases, as described above Figure 2 As shown, the industrial fault detection dataset is divided into a training set and a test set according to a preset ratio. For each round of training, the present invention performs the following steps on the training set in sequence: industrial fault detection dataset preprocessing, DenMune-based minority class MNN skeleton construction and double-layer noise removal, local adaptive oversampling based on MNN sparsity double-layer weights and MNN neighborhood constraint interpolation, and double threshold screening based on source subcluster consistency and original sample dominance, to obtain a balanced and structurally stable training dataset for model learning; while the corresponding test set only performs feature encoding, missing value processing and normalization in step S1, and the feature subset extraction operation corresponding to the feature selection in the preprocessing stage, and does not perform the oversampling and structural screening operations in steps S2 to S4, thereby maintaining the objectivity of the test data distribution.

[0186] Based on the above, an industrial fault detection model is constructed using the balanced training set obtained through the data processing flow of this invention as the model input and the corresponding fault category label as the output. The model directly distinguishes between normal operating conditions and fault categories. The industrial fault detection model can be any of the supervised learning classification models such as decision tree (DT), random forest (RF), or gradient boosting (GB). The specific model type and hyperparameter settings can be selected and adjusted according to the actual needs of the industrial process.

[0187] In a preferred embodiment of the present invention, a decision tree model is selected, and the balanced training set obtained through steps S1 to S4 is used as input to train the model parameters, thereby improving its comprehensive performance on imbalanced classification metrics such as F1, G-mean, and AUC. Through the above training process, an industrial fault detection model fitted to the balanced training dataset can be obtained. Since the training data has undergone local adaptive oversampling based on DenMune minority class MNN skeleton construction and double-layer noise removal, MNN sparsity double-layer weight and MNN neighborhood constraint interpolation, and dual threshold screening based on source subcluster consistency and original sample dominance, the model can more fully learn the discriminative features of minority class fault samples, reduce the interference of structurally inconsistent synthetic samples on the decision boundary, and thus improve the ability to identify rare fault conditions.

[0188] S5.2. Using the trained model for fault sample prediction and classification

[0189] In practical use, for new industrial fault detection samples, feature encoding, missing value processing and normalization are first performed according to the process specified in step S1. When feature selection is used in the preprocessing stage, the same feature subset as in the model training stage is extracted to ensure that its feature representation is consistent with that in the model training stage.

[0190] The sample processed by S1 is input into a trained industrial fault detection model, which outputs a prediction of whether the sample belongs to each candidate fault category or a normal operating condition. Combined with existing alarm mechanisms, the model output can be used as an auxiliary decision-making basis to indicate possible abnormal operating conditions or fault types, thereby providing reference information for operators and improving the safety and reliability of industrial process operation.

[0191] Experimental verification:

[0192] Although this invention primarily addresses imbalanced fault detection scenarios in industrial applications, its core principles are equally applicable to general imbalanced classification tasks. To facilitate a fair comparison with existing sampling methods, this embodiment selects four publicly available imbalanced datasets for empirical research and evaluates them uniformly as binary imbalanced classification tasks. Detailed information about the datasets is shown in Table 1.

[0193] To visually demonstrate the improvement effect of the MNSCO method on the distribution of minority class samples, the wine dataset is used as an example. Figure 3 The image shows the original data distribution of the wine dataset in Embodiment 1 of this invention. The minority class samples are significantly insufficient, and the data exhibits a relatively dense distribution in local areas of the feature space, while the overall space shows a sparse and uneven distribution. After processing using the MNSCO method of this invention, as shown... Figure 4 As shown ( Figure 4This is a data distribution diagram of the wine dataset after processing by the MNSCO method in Embodiment 1 of the present invention. The sample density of the minority class in the originally sparse region is enhanced in a targeted manner, the noise samples are effectively cleaned up, the relationship between the number of classes is more balanced, and the overall distribution is more stable.

[0194] Regarding the classification model, this embodiment constructs a decision tree (DT)-based classifier and compares the performance of the method of this invention with existing oversampling methods on general imbalanced classification tasks. All experiments use 5-fold cross-validation. In each round of 5-fold cross-validation, the dataset is first divided into training and test sets.

[0195] For the training set, the following steps are performed sequentially: industrial fault detection dataset preprocessing, DenMune-based minority class MNN skeleton construction and double-layer noise removal, local adaptive oversampling based on MNN sparsity double-layer weights and MNN neighborhood constraint interpolation, and dual threshold screening based on source sub-cluster consistency and original sample dominance. This results in a training dataset with a more balanced number of samples and stable local structure, which is used for learning classification model parameters.

[0196] For the corresponding test set, only feature encoding, missing value handling, and normalization in the industrial fault detection dataset preprocessing stage are performed. The test set does not perform steps S2 to S4, such as DenMune-based minority class MNN skeleton construction and double-layer noise removal, local adaptive oversampling based on MNN sparsity double-layer weights and MNN neighborhood constraint interpolation, and double threshold screening based on source sub-cluster consistency and original sample dominance. This maintains the original sample quantity and class distribution of the test data and is used to objectively evaluate the generalization performance of the model.

[0197] Five evaluation metrics were obtained for each dataset under 5-fold cross-validation, and the final result was the average of the 5-fold cross-validations as the experimental performance on that dataset.

[0198] Table 1 Information on Imbalanced Datasets

[0199] Dataset Sample size Feature number Imbalance ratio wine 178 13 1.5 glass-0-6_vs_5 108 9 11 dermatology-6 358 34 16.9 glass5 214 9 22.78

[0200] This embodiment uses F1, G-mean, and AUC as performance evaluation metrics. The F1 score measures the classifier's ability to identify minority class samples; a higher F1 score indicates stronger minority class performance. The G-mean score comprehensively evaluates the accuracy of both majority and minority class scores; it is essentially the geometric mean of recall and specificity, and a higher G-mean score indicates a more balanced classification of the two classes. The AUC score, representing the area under the ROC curve, indicates better overall classifier performance.

[0201] Table 2 Confusion Matrix

[0202] Predicted Value\Real value Positive Negative Positive TP FP Negative FN TN

[0203] Table 2 shows a typical confusion matrix, where TP represents the number of correctly classified positive samples, FN represents the number of misclassified positive samples, FP represents the number of misclassified negative samples, and TN represents the number of correctly classified negative samples. The formulas for calculating F1 and G-means are shown below. Furthermore, AUC is measured as the area under the ROC curve.

[0204]

[0205]

[0206]

[0207]

[0208]

[0209] Regarding the sampling method, this embodiment compares the MNSCO method proposed in this invention with 11 typical oversampling methods, namely SMOTE, Borderline-SMOTE, ADASYN, MWMOTE, CBSO, DBSMOTE, K-means SMOTE, G-SOMO, IR-SMOTE, OREM, and SD-CSMOTE (referred to as SMO, BS, ADA, MWM, CBS, DBS, KMS, GSO, IRS, ORE, and SDC in the figure, and the MNSCO method of this invention is referred to as MNSCO in the figure).

[0210] This embodiment uses Friedman ranking to analyze the overall performance of each sampling method on four selected imbalanced datasets. Specifically, on each dataset, the methods are ranked from best to worst according to the results of F1, G-mean, and AUC. The ranking results of each method on all datasets are then averaged to obtain the corresponding Friedman average ranking. The lower the ranking, the better the overall performance. Figure 5 , Figure 6 , Figure 7 The Friedman ranking graph based on the decision tree (DT) classifier is given, where Figure 5 , Figure 6 and Figure 7 The rankings of each sampling method under the F1, G-mean, and AUC metrics respectively are presented, and the overall performance differences of different sampling methods under the DT classifier are compared.

[0211] Depend on Figure 5 , Figure 6 , Figure 7As can be seen, under the DT classifier, the MNSCO method consistently ranks highly in Friedman's score for F1, G-mean, and AUC, demonstrating superior performance in most cases. The results show that on the four selected imbalanced datasets, the method of this invention exhibits stable and good performance in minority class identification, class balance, and overall discriminative ability.

[0212] The MNSCO method proposed in this invention can achieve excellent performance on three metrics: F1, G-mean, and AUC, under the selected four imbalanced datasets and DT classifier, further verifying the effectiveness of this sampling method in general imbalanced classification tasks.

[0213] A computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device of the aforementioned imbalanced data processing method based on nearest neighbor constraints and consistency screening.

[0214] A terminal device includes a processor and a computer-readable storage medium, the processor being configured to implement various instructions; the computer-readable storage medium being configured to store multiple instructions adapted for loading and execution by the processor of the aforementioned imbalanced data processing method based on nearest neighbor constraints and consistency screening.

[0215] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for processing imbalanced data based on nearest neighbor constraints and consistency screening, characterized in that, include: Acquire industrial fault detection data; Data preprocessing is performed based on the acquired industrial fault detection data; Minority MNN skeleton construction and two-layer noise removal based on DenMune clustering; Local adaptive oversampling based on MNN sparse double-layer weights and MNN neighborhood constraint interpolation; Dual threshold screening based on source sub-cluster consistency and original sample dominance; Fault sample prediction and classification are performed using a trained model.

2. The imbalanced data processing method based on nearest neighbor constraint and consistency screening according to claim 1, characterized in that, The data preprocessing based on the acquired industrial fault detection data includes: using one-hot encoding to numerically transform all categorical feature values ​​in the dataset; retaining features that were originally numerical, and then performing feature numerical transformation, where for each categorical feature, all its values ​​are counted, and a binary indicator variable is constructed for each value; when a sample takes that value for that feature dimension, the corresponding indicator variable is set to 1, otherwise it is set to 0; the original categorical features are replaced with this 0-1 indicator variable set, and a unified numerical feature matrix is ​​formed by combining all the categorical features converted to numerical features with the original numerical features; finally, missing value processing is performed, where, after obtaining the unified... After generating the first numerical feature matrix, missing values ​​in the matrix are processed, including removing missing features and samples to avoid interference from extremely missing data on the overall distribution. For data retained from the original numerical features, missing values ​​are estimated using mean imputation and median imputation. For data derived from the original categorical features through one-hot encoding, missing values ​​are treated as independent values ​​of the features. After completing the feature numerical transformation and missing value processing, the Min-Max normalization method is used to map the values ​​of each feature dimension to the [0,1] interval. The dimensional feature, whose normalization calculation formula is: in, Describes any sample at the th The original values ​​on the dimensional features, This represents the minimum value of this feature across all samples. This represents the maximum value of this feature across all samples. This represents the normalized eigenvalues.

3. The imbalanced data processing method based on nearest neighbor constraint and consistency screening according to claim 2, characterized in that, The minority class MNN skeleton construction and two-layer noise removal based on DenMune clustering include dimensionality reduction of the minority class sample subset corresponding to the fault category. ,in The original feature dimension is used to map it to a lower-dimensional space using the t-SNE algorithm, resulting in a matrix. ,in The feature dimension is determined by the dimension reduction; then, the nearest neighbor relationship is defined in the dimension reduction space. In the context, for any two sample points and The Euclidean distance is calculated as follows: Based on Euclidean distance, for each sample A neighborhood set is defined to characterize the local spatial relationships; finally, sample pre-classification based on local density is performed, where, to evaluate the local density of each sample, a mutual relation between each sample is first constructed in the reduced-dimensional space. Neighbor set For any sample :when When a sample is identified as a Class I noise point, it is removed during MNN skeleton construction and oversampling. For samples that meet the following criteria... To quantify the local density of the samples, a local density index is introduced. Defined as: ,in, For the sample The reverse Nearest neighbor set, representing the set of neighbors. Incorporate into itself The nearest neighbor set of samples; in the process of eliminating After the first type of noise point, only the remaining satisfying The sample, according to The values ​​are pre-classified into strengths and weaknesses.

4. The imbalanced data processing method based on nearest neighbor constraint and consistency screening according to claim 3, characterized in that, The minority class MNN skeleton construction and two-layer noise removal based on DenMune clustering also includes constructing an MNN skeleton, where all strong points are used as skeleton nodes, and the connection relationship is determined by the mutual relationships between the strong points. The nearest neighbor set is determined by the following steps: traverse all strong points, and for any two strong points... and If the set is expanded and There exists an intersection, that is, the following conditions are met: If two points are considered to be locally connected, they are grouped into the same MNN skeleton cluster. Through iterative traversal, all strong points connected by neighborhood relationships will be gradually classified into the same skeleton, eventually forming multiple independent MNN skeleton clusters. Then, a weakness merging based on a hard voting mechanism is performed for second-type noise identification, which is used to reasonably allocate the pre-classified weakness samples to the established MNN skeleton clusters. For each weakness... Examine oneself and its interrelationships Groups formed by neighbors If a group intersects with a certain MNN skeleton cluster, a voting process is initiated, and the group is counted. The number of strong points already assigned to each MNN skeleton cluster, and the number of weak points. Assign to the cluster with the most strong nodes, and let the weaker MNNs jointly determine the affiliation; if the group If a weak sample and its MNN have no intersection with any of the constructed MNN backbone clusters, it indicates that the weak sample and its MNN have no reliable connection with any of the core clusters. These weak samples are then marked as second-type noise points and removed, resulting in the final cluster set. , where each cluster It includes all strong samples belonging to the cluster and weak samples merged in through a voting mechanism.

5. The imbalanced data processing method based on nearest neighbor constraint and consistency screening according to claim 4, characterized in that, The local adaptive oversampling based on MNN sparse double-layer weights and MNN neighborhood constraint interpolation includes based on the constructed mutual Neighbor set First, a local sparsity index is defined for each sample, and then the overall sparsity index of the sub-cluster is obtained. Specifically, for the sub-cluster... Any sample point within Record their mutual Nearest neighbor size is In the denoised minority class sample set, each sample satisfies The local sparsity index of the samples is defined as: ,use As a measure of local sparsity based on MNN structure; At the sub-cluster level, for sub-clusters The local sparsity of all samples within the cluster is averaged to obtain the sparsity index of the subcluster: ,in, sub-clusters The number of samples, The larger the value, the fewer MNN relationships there are in the subcluster as a whole. and Completely determined by the interaction of samples Number of neighbors The decision avoids the problem of needing to pre-set density thresholds and distance bandwidth in traditional methods, allowing sparsity metrics to be naturally integrated with the MNN structure.

6. The imbalanced data processing method based on nearest neighbor constraint and consistency screening according to claim 5, characterized in that, The local adaptive oversampling based on MNN sparse double-layer weights and MNN neighborhood constraint interpolation also includes adaptive sampling weight allocation based on MNN sparsity, wherein, for the minority class sample set after denoising fault categories in the training set... The total number of samples is The number of samples in the normal operating condition category in the training set is denoted as The number of fault categories in the training set is denoted as The reference sample size for the preset fault category is 1. To mitigate the sample size difference between the fault category and the reference sample size, the total number of synthesized samples required for the fault category is: At the sub-cluster level, based on sub-cluster sparsity The sampling weight for each DenMune subcluster is determined by the following operation: the total number of synthesized samples required to synthesize the fault categories. The sparsity of each sub-cluster is proportionally allocated based on its sparsity, and the relative sparsity weight of each sub-cluster is calculated, expressed as follows: A set of satisfy and Then according to sparsity weights The total number of synthesized samples required for each fault category The subclusters are assigned to each subcluster, thus obtaining the subclusters. The corresponding number of synthetic samples: ,in, This indicates a rounding operation, within a sub-cluster, utilizing the local sparsity index of the samples. As a weight for selecting synthetic seed samples, for a given sub-cluster According to sub-clusters Local sparsity index of each sample Determine the internal weights of the samples selected as seed samples: , The larger the sample size, the better. The sparser the local region, the higher the probability that it will be selected as a seed sample in subsequent sample synthesis. This allows for targeted compensation of sparse regions within the sub-cluster, ultimately forming a set of samples at the sub-cluster level that satisfy... The sparsity weights are used to determine the total number of synthesized samples required for the fault category. The data is distributed to each sub-cluster according to the principle that the sparser the data, the greater its weight. Within each sub-cluster, further distribution is based on... Constructing sample-level weights This enables targeted compensation for sparse regions within the cluster, and together the two constitute the DenMune sub-cluster level. +Sample level A two-layer adaptive sampling weighting system.

7. The imbalanced data processing method based on nearest neighbor constraint and consistency screening according to claim 6, characterized in that, The local adaptive oversampling based on MNN sparse double-layer weights and MNN neighborhood constraint interpolation further includes performing specific oversampling operations under MNN structural constraints after calculating the sampling weights at the sub-cluster and sample levels. By restricting the generation of new samples to the local neighborhood supported by the MNN structure within the DenMune sub-cluster, new samples only appear in structurally reliable regions, avoiding disruption of the original local structure of the minority classes. Specifically, for each sub-cluster... Repeat the sampling and synthesis operation until the result is generated. Each sample includes: seed sample selection, based on weights. Within a subcluster, the index set Perform weighted random sampling and select a sample index. As a seed sample, its vector in the normalized feature space is Paired sample selection under MNN constraints For clusters The set of indices for the samples is then defined within the sub-cluster: That is, to consider and They are nearest neighbors and belong to the same subcluster If the samples are used as candidate pairing points, If not empty, then randomly select an index from them. Its vector in the normalized feature space is denoted as This ensures that new samples are always generated within the same DenMune sub-cluster and along the neighborhood supported by the MNN structure, avoiding unreasonable interpolation across sub-cluster regions; when individual sub-clusters contain... When there are empty points, use the empty point sub-cluster. Neighbor set As a candidate pairing set, any sample point is randomly selected from the candidate pairing set as a candidate pairing point to ensure the stability of the synthesis process, which is expressed as: (1) Sample synthesis: a new sample vector is generated by linear interpolation. : ,in, (2) Sample label and dataset update: The synthetic sample is a random number in the interval (0,1); As new minority class samples added to the denoised minority class set of the fault category, the set of minority class samples after the oversampling enhancement of the fault category is denoted by the following notation: After completing the sampling and synthesis process for all subclusters of the fault category, the enhanced minority class sample set of the fault category is obtained. .

8. The imbalanced data processing method based on nearest neighbor constraint and consistency screening according to claim 7, characterized in that, The dual-threshold screening based on source sub-cluster consistency and original sample dominance includes MNN relation reconstruction after enhancing the minority class set for fault categories, wherein the enhanced minority class sample set for fault categories is represented in the original normalized feature space as follows: For any two samples , The Euclidean distance is defined as: ,exist The above is for each sample Rebuild The nearest neighbor and mutual nearest neighbor relationships are defined as follows: Nearest neighbor set: for each sample , record it in On Nearest neighbor set ,in ; reverse Nearest neighbor set: corresponding reverse Nearest neighbor set ;mutual Neighbors: each other Nearest neighbor set is defined as That is, on the minority class set after oversampling of the fault categories, and... mutual The nearest neighbor sample set; for each minority class synthetic sample of the fault category. The source subcluster number was recorded during the oversampling phase and is denoted as... ,Right now Sub-clusters A seed sample within the MNN structure is obtained through interpolation; in the enhanced MNN structure The consistency between the synthesized samples and the original MNN structure is measured using both the subcluster consistency index and the original sample dominance index: The subcluster consistency index is expressed as: ,like Higher, indicating The enhanced MNN structure remains embedded within the stable interconnection structure of the source subclusters and their neighborhood; the original sample dominance index is expressed as: ,like Higher, indicating Primarily surrounded by the original sample; when When the set is empty, it means that the synthesized sample cannot be integrated into the oversampled MNN structure and will be removed in subsequent screening.

9. The imbalanced data processing method based on nearest neighbor constraint and consistency screening according to claim 8, characterized in that, The dual-threshold screening based on source sub-cluster consistency and original sample dominance also includes structural consistency screening based on dual thresholds and final minority class set construction. To achieve automatic screening of synthetic samples, two structural consistency threshold parameters are pre-set: sub-cluster consistency threshold... Original sample dominance threshold For each synthetic sample index The following determination rules shall be applied: like Then directly take the sample Samples marked as structurally inconsistent are removed. like Then calculate its subcluster consistency index. Dominant indicators of the original sample Only if both conditions are met At that time, retain the sample ; Otherwise, it is considered a synthetic noise sample inconsistent with the original spatial MNN structure and is discarded; let the set of synthetic sample indices retained after the double-threshold structural consistency screening be denoted as . Then the final minority class sample set of the fault category is represented as: ， in, The original minority class sample set retained after denoising for the fault category DenMune. This is a set of high-quality synthetic minority class samples retained after structural consistency screening.

10. An imbalanced data processing system based on nearest neighbor constraints and consistency screening, characterized in that, include: The data acquisition module is configured to acquire industrial fault detection data; The preprocessing module is configured to perform data preprocessing based on the acquired industrial fault detection data; The removal module is configured to construct the minority MNN skeleton based on DenMune clustering and perform two-layer noise removal. The oversampling module is configured to perform local adaptive oversampling based on MNN sparse double-layer weights and MNN neighborhood constraint interpolation. The filtering module is configured to perform dual-threshold filtering based on source subcluster consistency and original sample dominance. The classification module is configured to use the trained model to predict and classify fault samples.

Images