Data recognition method and system based on supervised learning

Abstract

The present invention discloses a data recognition method and system based on supervised learning, which specifically relates to the technical field of data recognition, and is used to solve the deficiencies of existing supervised learning methods in low-frequency feature capture and feature association modeling. The method collects training data from multiple data sources and filters samples that do not meet the quality threshold; measures information gain of features through a dimensionality reduction algorithm, and screens feature sets with high information gain; generates a sparse feature combination set using an association rule algorithm, extracts key low-frequency features and their interactive relationships; inputs the low-frequency feature set into a gradient boosting machine for supervised training, and strengthens the influence of low-frequency features through an adaptive weighting strategy; monitors the gradient update trend and Shapley value distribution of low-frequency features during the training process, and adjusts the weighting strategy for abnormal features to correct the model weight distribution, thereby improving the generalization ability and prediction accuracy of the model, and being suitable for feature recognition and analysis of complex data scenarios.

Description

Technical Field

[0001] The present invention relates to the field of data recognition technology, and more specifically, to a data recognition method and system based on supervised learning. Background Art

[0002] Supervised learning methods have shown important value in processing high-dimensional feature data and modeling complex feature relationships. In large-scale data processing, supervised learning methods usually rely on feature extraction and selection techniques. By analyzing and optimizing the features of training data, models are constructed to improve prediction performance. In practical applications, the analysis and utilization of feature combinations are of great significance for mining potential associations in data. These features often have the characteristics of uneven distribution and complex correlations. Therefore, effective algorithms and strategies are needed to process them in order to improve the accuracy and stability of the model.

[0003] In the existing technology, supervised learning methods have difficulty in effectively capturing the correlation between key features when processing complex feature combinations, which will lead to limited model generalization ability and incorrect or missed recognition. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, embodiments of the present invention provide a data recognition method and system based on supervised learning to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] The data recognition method based on supervised learning includes the following steps:

[0007] S1: Collect training data from multiple data sources and perform filtering operations on the training data to remove samples that do not meet the quality threshold;

[0008] S2: Perform feature analysis on the filtered training data, and use the dimension reduction algorithm to perform information gain measurement on each feature to generate a feature set with high information gain;

[0009] S3: Generate a sparse feature combination set in the feature space of the feature set with high information gain using the association rule algorithm, and retain the key low-frequency features and their interactions by setting the frequency threshold to obtain a low-frequency feature set;

[0010] S4: Input the low-frequency feature set into the gradient boosting machine for preliminary supervised training, and set the initial weights through an adaptive weighting strategy to strengthen the influence of low-frequency features in training;

[0011] S5: During supervised training, based on the gradient update trend of low-frequency features and the feature contribution distribution of Shapley values, monitor whether the weight changes of low-frequency features and their impact on the gradient boosting machine are abnormal. If abnormal, adjust the adaptive weighting strategy to correct the impact of low-frequency features.

[0012] In a preferred embodiment, training data is collected from multiple data sources, and filtering operations are performed on the training data to remove samples that do not meet the quality threshold, specifically including:

[0013] Obtain the original data set containing the target features, which comes from different data acquisition devices and platforms;

[0014] Perform data cleaning operations on the original data set to remove data entries with missing values, data that do not meet format requirements, and data that contain obvious errors;

[0015] Perform quality assessment on the original data set after data cleaning, screen the data samples based on the preset quality assessment criteria, and remove data samples that do not meet the quality assessment criteria to generate a training data set that meets the requirements.

[0016] In a preferred embodiment, feature analysis is performed on the filtered training data, and a dimension reduction algorithm is used to perform information gain measurement on each feature to generate a feature set with high information gain, specifically including:

[0017] Construct a feature space based on the value range and distribution of each feature in the training data. The feature space contains multiple dimensions for describing the features of the training data.

[0018] The dimensionality reduction algorithm is used to screen and optimize the dimensions of the feature space. The dimensionality reduction algorithm includes an analysis method based on information gain measurement.

[0019] The information gain value is calculated for each dimension of the feature space. The information gain value is based on the entropy difference between the distribution of the target label and the conditional distribution of the feature value.

[0020] According to a preset threshold, features with information gain values ​​higher than the threshold are retained to form a feature set with high information gain.

[0021] In a preferred embodiment, an association rule algorithm is used to generate a sparse feature combination set in the feature space of a feature set with high information gain, and a frequency threshold is set to retain key low-frequency features and their interactions to obtain a low-frequency feature set, specifically including:

[0022] Extract each feature in the feature set with high information gain, and construct a feature space to store all possible feature combinations of the features;

[0023] Use association rule algorithm to count the frequency of all feature combinations in feature space and calculate the support of each feature combination. The support is defined as the proportion of feature combinations that appear simultaneously in the training data.

[0024] Generate a sparse feature combination set based on the support calculation result, and the sparse feature combination set includes all feature combinations that meet the set support range;

[0025] Extract feature combinations whose support is lower than a preset frequency threshold from the sparse feature combination set, retain feature combinations containing low-frequency features, and analyze the interaction relationship between low-frequency features;

[0026] The retained low-frequency features and their interaction relationships are formed into a low-frequency feature set, and the low-frequency feature set only includes low-frequency features and their interaction relationships whose support is lower than a preset frequency threshold.

[0027] In a preferred embodiment, the low-frequency feature set is input into the gradient boosting machine for preliminary supervised training, and the initial weights are set by an adaptive weighting strategy to strengthen the influence of the low-frequency features in the training, specifically including:

[0028] Format the low-frequency features in the low-frequency feature set into an input form suitable for the gradient boosting machine, including normalization of numerical features and one-hot encoding of categorical features;

[0029] Initialize the training parameters of the gradient boosting machine, including learning rate, maximum tree depth, subsampling ratio, and minimum number of samples for splitting nodes;

[0030] In the model initialization stage, according to the weight allocation rules of low-frequency features, the initial weights are allocated to low-frequency features through an adaptive weighting strategy. The weight allocation rules include: calculating the initial weight of the low-frequency features according to their support, and the weight is proportional to the inverse of the support value; if there is a significant interaction between low-frequency features, their weights are superimposed and corrected based on the strength of the interaction;

[0031] The gradient boosting machine is supervised and trained using training data containing a set of low-frequency features. The model parameters, including feature splitting points, leaf node weights, and loss function values, are updated in each iteration.

[0032] In a preferred embodiment, during the supervised training process, based on the gradient update trend of the low-frequency features and the feature contribution distribution of the Shapley value, the weight change of the low-frequency features and the influence on the gradient boosting machine are monitored to see if they are abnormal, and when abnormal, the adaptive weighting strategy is adjusted to correct the influence of the low-frequency features, specifically including:

[0033] In each round of supervised training, the gradient update value of the low-frequency feature is calculated, and an update trend sequence reflecting the gradient change is generated to identify whether there is a sudden increase or decrease in the gradient of the low-frequency feature;

[0034] Calculate the Shapley value of low-frequency features, generate a Shapley value distribution sequence based on all supervised training rounds, and detect whether the Shapley value distribution sequence of low-frequency features has abnormal fluctuations;

[0035] The low-frequency features with abnormal fluctuations or sudden increases or decreases in the Shapley value distribution sequence are marked as abnormal low-frequency features;

[0036] The adaptive weighting strategy is adjusted for the detected abnormal low-frequency features, and their influence on the gradient boosting machine is corrected by modifying the weight value.

[0037] In a preferred embodiment, during the supervised training process, the Shapley value of the low-frequency feature is calculated, and a Shapley value distribution sequence is generated based on all supervised training rounds to detect whether the Shapley value distribution sequence of the low-frequency feature has abnormal fluctuations, specifically:

[0038] After each round of training, the Shapley value of the low-frequency feature is calculated using the following formula: ;in, The Shapley value representing the low-frequency feature, represents a set of all low-frequency features, Indicates the current low-frequency characteristics, represents a subset of low-frequency features, represents the size of the subset of low-frequency features, represents the size of the set containing all low-frequency features, Indicates the contribution of the subset of low-frequency features to the model prediction results, It represents the contribution of the subset of low-frequency features plus the low-frequency features to the model prediction results;

[0039] Detect whether the Shapley value distribution has abnormal fluctuations. The determination formula for abnormal fluctuations is: ;in, Indicates the Shapley value of the current training round, represents the historical average Shapley value, Indicates the fluctuation threshold.

[0040] On the other hand, the present invention provides a data recognition system based on supervised learning, including a data acquisition and filtering module, a feature gain screening module, a sparse feature generation module, a low-frequency feature weighting module, and a feature weight adjustment module;

[0041] Data collection and filtering module: collects training data from multiple data sources and performs filtering operations on the training data to remove samples that do not meet the quality threshold;

[0042] Feature gain screening module: performs feature analysis on the filtered training data, and uses the dimension reduction algorithm to perform information gain measurement on each feature to generate a feature set with high information gain;

[0043] Sparse feature generation module: uses association rule algorithm to generate sparse feature combination sets in the feature space of feature sets with high information gain, and retains key low-frequency features and their interactions by setting frequency thresholds to obtain low-frequency feature sets;

[0044] Low-frequency feature weighting module: The low-frequency feature set is input into the gradient boosting machine for preliminary supervised training, and the initial weights are set through an adaptive weighting strategy to strengthen the influence of low-frequency features in training;

[0045] Feature weight adjustment module: During supervised training, based on the gradient update trend of low-frequency features and the feature contribution distribution of Shapley values, monitor the weight changes of low-frequency features and whether their impact on the gradient boosting machine is abnormal. If abnormal, adjust the adaptive weighting strategy to correct the impact of low-frequency features.

[0046] The technical effects and advantages of the data recognition method and system based on supervised learning of the present invention are as follows:

[0047] 1. By screening efficient feature sets through information gain measurement and combining with association rule algorithm to generate sparse feature combination sets, the key low-frequency features and their interactive relationships are retained while effectively compressing the feature space, significantly improving the pertinence of feature selection; the low-frequency feature set is extracted using the set frequency threshold, ensuring that even in the case of insufficient sample size, the important contribution of low-frequency features to data recognition can be mined, thereby reducing the risk of feature loss.

[0048] 2. By constructing a supervised learning model based on a gradient boosting machine and combining it with an adaptive weighting strategy, the weights of low-frequency features are dynamically adjusted to achieve a reasonable distribution of feature contributions in the model. During the training process, the gradient update trend and the feature contribution distribution of the Shapley value are used to monitor the weight changes of low-frequency features, and the weights of features that may be abnormal are corrected in time to avoid the imbalance of feature influence. This method effectively improves the generalization ability and prediction accuracy of the model, solves the problem of difficulty in capturing the key correlation of low-frequency features in existing technologies, and has important practical application value and innovative significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of a data identification method based on supervised learning according to the present invention;

[0050] Figure 2 It is a structural diagram of the data recognition system based on supervised learning of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Embodiment 1: Figure 1 The data recognition method based on supervised learning of the present invention is provided, which comprises the following steps:

[0053] S1: Collect training data from multiple data sources and perform filtering operations on the training data to remove samples that do not meet the quality threshold.

[0054] S2: Perform feature analysis on the filtered training data, and use the dimensionality reduction algorithm to perform information gain measurement on each feature to generate a feature set with high information gain.

[0055] S3: Use the association rule algorithm to generate a sparse feature combination set in the feature space of the feature set with high information gain, and retain the key low-frequency features and their interactions by setting the frequency threshold to obtain a low-frequency feature set.

[0056] S4: The low-frequency feature set is input into the gradient boosting machine for preliminary supervised training, and the initial weights are set through an adaptive weighting strategy to enhance the influence of low-frequency features in training.

[0057] S5: During supervised training, based on the gradient update trend of low-frequency features and the feature contribution distribution of Shapley values, monitor whether the weight changes of low-frequency features and their impact on the gradient boosting machine are abnormal. If abnormal, adjust the adaptive weighting strategy to correct the impact of low-frequency features.

[0058] Collect training data from multiple data sources and perform filtering operations on the training data to remove samples that do not meet the quality threshold, including:

[0059] Get the original data set containing the target features. The original data set comes from different data acquisition devices and platforms:

[0060] Obtain raw data from different types of data acquisition devices (such as sensors, cameras, and logging systems) to ensure that the raw data covers different scenarios or conditions of the target features; extract raw data sets from multiple data platforms (such as databases, cloud storage, or distributed file systems) and use interfaces or query statements to read batch data; perform consistency checks on data from different sources to ensure that the raw data's field format, timestamp, device identification, and other information are consistent, providing a unified data foundation for subsequent data processing.

[0061] Perform data cleaning on the original data set to remove data entries with missing values, data that do not meet format requirements, and data with obvious errors:

[0062] Detect missing values ​​in the data and mark missing value samples based on set rules (such as empty fields, abnormal values, etc.); adopt a elimination strategy for detected missing value samples to directly remove incomplete data entries; perform checks on the data format to verify whether the field type meets the requirements (such as whether the numeric field is a legal number), and remove data that does not meet the format requirements; detect obvious erroneous entries in the data, such as numeric fields that exceed the reasonable range (such as negative temperature values) or invalid time fields (such as empty timestamps), and delete obvious erroneous entries.

[0063] Perform quality assessment on the original data set after data cleaning, screen the data samples based on the preset quality assessment criteria, and remove the data samples that do not meet the quality assessment criteria to generate a training data set that meets the requirements:

[0064] Establish quality assessment standards, including but not limited to data collection time range, sample integrity, numerical range, and target feature coverage; check each cleaned data sample to see if it meets the quality assessment standards, and score or mark each data sample; eliminate data samples that do not meet the quality assessment standards, such as samples with scores below the threshold or that do not cover the target features; include data samples that pass the quality assessment into the training data set, and ensure the consistency and completeness of the data set.

[0065] For example, in an industrial quality inspection scenario, raw data sets containing product defect images are collected from multiple sources; these data include high-definition images collected from three different types of cameras, defect record logs generated by automatic inspection equipment, and historical defect data in the cloud storage platform; for the collected data, field consistency verification is first performed, such as ensuring that all image data has a uniform resolution (1920×1080 pixels) and verifying the timestamp format of the log data (YYYY-MM-DD HH:MM:SS); in the data cleaning stage, log records with missing fields (such as missing defect location or time information) are eliminated, and image files with incorrect formats (such as inconsistent file extensions or abnormal file sizes) are deleted; then the cleaned data set is evaluated for quality, with the standard set as "complete defect marking, image resolution meets requirements, and log timestamp covers the last 30 days"; finally, only high-quality data samples that meet the above standards are retained for subsequent model training.

[0066] Perform feature analysis on the filtered training data, and use the dimension reduction algorithm to perform information gain measurement on each feature to generate a feature set with high information gain, including:

[0067] The feature space is constructed based on the value range and distribution of each feature in the training data. The feature space contains multiple dimensions used to describe the characteristics of the training data:

[0068] Extract all features from the training data and treat each feature as a dimension. For example, the features extracted in sales forecasting include sales volume, price, promotion frequency, etc.; count the value range of each feature, record the minimum, maximum and mean value for numerical features, and record the number of categories and distribution for categorical features; perform a preliminary check on the extracted features to ensure that the format of the features is consistent, such as unifying the units or normalizing the numerical features and unifying the encoding of the categorical features; construct a feature space based on all extracted features. The feature space contains multiple dimensions that describe the training data, and each dimension corresponds to a feature.

[0069] The dimension reduction algorithm is used to screen and optimize the dimensions of the feature space. The dimension reduction algorithm includes an analysis method based on information gain measurement:

[0070] Select a dimensionality reduction algorithm based on information gain measurement to analyze the importance of features to the target label; process each dimension of the feature space one by one, and calculate the conditional distribution of each feature. The conditional distribution is defined as the distribution of each feature value under a given target label; by calculating the conditional distribution, determine the correlation between the feature and the target label, laying the foundation for the subsequent calculation of the information gain value; in the screening process, eliminate features that cannot calculate the conditional distribution (such as features with too high missing rate or no distribution change) to optimize the feature space.

[0071] The information gain value is calculated for each dimension of the feature space, which is based on the entropy difference between the distribution of the target label and the conditional distribution of the feature value:

[0072] The information gain value is calculated for each feature. The information gain value is based on the entropy difference between the distribution of the target label and the conditional distribution of the feature value. The specific formula is as follows: ;in, Represents the information gain value of the feature; Indicates the target label to be calculated The entropy of represents the overall uncertainty of the target label without considering any features; Indicates the target label The conditional entropy is a measure of the known features In the case of target label the degree of information confusion.

[0073] ;in, Represents the target label, which is a field in the training data and is used for supervised learning model prediction; Indicates the target label Total number of categories in ; Indicates the category in the target label The probability of the category Frequency of occurrence in the training data; Indicates the target label The index of the category in the range 1 to ; Indicates the target label No. categories.

[0074] ;in, Represents a feature, which is a field in the training data and serves as the model input; Representation characteristics The total number of values ​​of ; Representation characteristics Medium value The probability of Frequency of occurrence in the training data; Representation characteristics No. A value; Representation characteristics The index of the value in the range 1 to ; Indicated in the feature The value is When the target label for The conditional probability of .

[0075] The information gain value of each feature in the feature space is calculated one by one, and the results are stored as a corresponding list of features and their information gain values.

[0076] According to the preset threshold, features with information gain values ​​higher than the threshold are retained to form a feature set with high information gain:

[0077] Set a threshold for the information gain value, such as selecting a fixed value (such as 0.1) or selecting the optimal threshold through cross-validation; filter out features with information gain values ​​greater than the threshold from the information gain value list and include them in the feature set with high information gain; perform redundancy removal on the filtered feature set, such as using the Pearson correlation coefficient to calculate the correlation between features. If the correlation is higher than the set threshold, retain the features that contribute more to the target label; the final feature set with high information gain contains the most important features for predicting the target label, which is used for subsequent data processing and model training.

[0078] The association rule algorithm is used to generate a sparse feature combination set in the feature space of the feature set with high information gain, and the key low-frequency features and their interactions are retained by setting the frequency threshold to obtain a low-frequency feature set, which specifically includes:

[0079] Extract each feature in the feature set with high information gain and construct a feature space to store all possible feature combinations of the features:

[0080] Extract all features from the feature set with high information gain generated in the previous step. For example, in user behavior analysis, the extracted features may include "monthly consumption amount", "login frequency", "active days", etc.

[0081] Construct a feature space where each feature represents a dimension and the feature space contains all possible feature combinations in the feature set. For example, if the feature set contains three features T1, T2, and T3, the combinations contained in the feature space include a single feature (such as T1), a double feature combination (such as T1, T2), and a multi-feature combination (such as T1, T2, T3).

[0082] Assign a unique identifier to each feature and store its properties (such as value range, number of categories, etc.) to ensure consistent identification and processing of features in subsequent steps.

[0083] Use the association rule algorithm to count the frequency of all feature combinations in the feature space and calculate the support of each feature combination. The support is defined as the proportion of feature combinations that appear simultaneously in the training data:

[0084] The calculation formula for defining support is as follows: ;in, represents the support of feature combination, represents the number of times the feature combination appears in the training data, Indicates the total number of training data.

[0085] Perform frequency statistics on each feature combination in the feature space and record the number of times each feature combination appears in the training data. For example, in the user behavior data, the feature combination "high consumption + high frequency login" appears 50 times, and the total number of training data is 1000, so the support is 50 / 1000=0.05.

[0086] Each feature combination and its support are stored as a key-value pair structure to form a support statistics table for subsequent screening.

[0087] Generate a sparse feature combination set based on the support calculation results. The sparse feature combination set contains all feature combinations that meet the set support range:

[0088] Set the range of support (for example, [Smin, Smax]), filter feature combinations with support within the range, and generate a sparse feature combination set.

[0089] The sparse feature combination set contains all feature combinations that meet the set support range. For example, if the support range is set to [0.01, 0.2], the feature combinations "low consumption + high complaint rate" (support is 0.15) and "high consumption + high frequency login" (support is 0.05) within this range are selected.

[0090] The support of each combination in the sparse feature combination set is recorded to support the subsequent screening of low-frequency features.

[0091] Extract feature combinations whose support is lower than the preset frequency threshold from the sparse feature combination set, retain feature combinations containing low-frequency features, and analyze the interaction between low-frequency features:

[0092] Set the frequency threshold Slow to extract feature combinations with support lower than the threshold. For example, if the frequency threshold is 0.05, the feature combination "low consumption + low-frequency login" (support is 0.03) with support lower than 0.05 is selected.

[0093] The extracted low-frequency feature combinations are retained, and their component features and support are recorded to form a low-frequency feature screening table.

[0094] Perform interactive relationship analysis on low-frequency feature combinations, including:

[0095] Count the number of co-occurrences between low-frequency features;

[0096] Calculate the correlation between low-frequency features, for example using the mutual information formula: ;in, Represents low frequency features and low frequency characteristics The mutual information between Represents low frequency features The eigenvalue of and low frequency characteristics The eigenvalue of The probability of simultaneous occurrence, Represents low frequency features The eigenvalue of The marginal probability of Represents low frequency features The eigenvalue of The marginal probability of .

[0097] Based on the analysis results, low-frequency feature combinations with significant interactions are retained to construct low-frequency feature sets.

[0098] The retained low-frequency features and their interactions are formed into a low-frequency feature set, which only contains low-frequency features and their interactions whose support is lower than the preset frequency threshold:

[0099] De-redundancy processing is performed on low-frequency feature combinations. For example, correlation analysis methods (such as the Pearson correlation coefficient) are used to eliminate redundant combinations, and only feature combinations that are valuable for subsequent model training are retained.

[0100] A low-frequency feature set is formed, which includes: low-frequency features whose support is lower than a preset frequency threshold; and the interaction relationship between these low-frequency features (such as mutual information value or common occurrence frequency).

[0101] The low-frequency feature set is stored as the model input data format for subsequent model training and optimization.

[0102] The low-frequency feature set is input into the gradient boosting machine for preliminary supervised training, and the initial weights are set through an adaptive weighting strategy to strengthen the influence of low-frequency features in training, including:

[0103] Format the low-frequency features in the low-frequency feature set into an input form suitable for the gradient boosting machine, including normalization of numerical features and unique hot encoding of categorical features:

[0104] Extract all numerical features from the low-frequency feature set and record the minimum, maximum, and mean of each numerical feature.

[0105] Use the min-max normalization formula to normalize numerical features.

[0106] Extract the categorical features from the low-frequency feature set and record all possible values ​​of each categorical feature.

[0107] Perform one-hot encoding on categorical features to convert each category into a binary feature. For example, in "User Type", "Ordinary User" is encoded as [1, 0, 0], "Member User" is encoded as [0, 1, 0], and "Senior Member" is encoded as [0, 0,1].

[0108] The processed numerical features and categorical features are combined to form a feature matrix that adapts to the input requirements of the gradient boosting machine. Each row of the feature matrix represents a data sample and each column represents a feature.

[0109] Initialize the training parameters of the gradient boosting machine, including the learning rate, maximum tree depth, subsampling ratio, and the minimum number of samples for splitting nodes:

[0110] Set the learning rate of the gradient boosting machine. The learning rate determines the step size of each parameter update of the model. For example, if the learning rate is set to 0.1, it means that the update amplitude of the model in each iteration is 10% of the original gradient.

[0111] Set the maximum tree depth of the gradient boosting machine. The tree depth determines the complexity of a single decision tree. For example, if the maximum tree depth is set to 6, the maximum number of split layers for each tree is 6.

[0112] Set the subsampling ratio, which determines the proportion of data samples used when training each tree. For example, if the subsampling ratio is set to 0.8, 80% of the data samples will be used for each training.

[0113] Set the minimum number of samples for splitting nodes. The minimum number of samples determines the lower limit of the number of data samples in each leaf node. For example, if the minimum number of samples for splitting nodes is set to 10, it means that each leaf node must contain at least 10 data samples.

[0114] Complete the initialization of training parameters and store the parameters as the initial configuration of the model.

[0115] In the model initialization stage, according to the weight allocation rule of low-frequency features, the initial weights are allocated to low-frequency features through an adaptive weighting strategy. The weight allocation rule includes: calculating the initial weight of the low-frequency feature according to its support, and the weight is proportional to the inverse of the support value; if there is a significant interaction between low-frequency features, the weight is superimposed and corrected based on the strength of the interaction:

[0116] Among them, the initial weight of the low-frequency feature can be expressed as the inverse of the support of the low-frequency feature.

[0117] The weight of the corrected low-frequency feature can be expressed as the sum of the initial weight of the low-frequency feature and the product of the mutual information value between the low-frequency features and the correction coefficient, where the correction coefficient is used to control the influence of the interaction relationship on the weight.

[0118] The calculated weights of the low-frequency features are assigned to each low-frequency feature, and the weight values ​​are passed to the gradient boosting machine as the importance indicators of the input features.

[0119] Use training data containing a set of low-frequency features to supervise the gradient boosting machine training, and update the model parameters in each iteration, including feature split points, leaf node weights, and loss function values:

[0120] The formatted low-frequency feature set is used as the input of training data, and the target label is used as the output, which is input into the gradient boosting machine for supervised training.

[0121] During the training process, the gradient boosting machine updates the model parameters according to the following steps: use the input data to train the decision tree for each iteration, determine the feature splitting point, and the selection of the splitting point is based on the contribution of the feature to the target label; calculate the weight of each leaf node and calculate the node value based on the distribution of samples in the leaf node; update the loss function value to optimize the overall performance of the model.

[0122] In each iteration, the weights of the features are updated according to the errors in the training data to ensure that low-frequency features continue to be important in subsequent training.

[0123] During supervised training, based on the gradient update trend of low-frequency features and the feature contribution distribution of Shapley values, monitor the weight changes of low-frequency features and whether their impact on the gradient boosting machine is abnormal. If abnormal, adjust the adaptive weighting strategy to correct the impact of low-frequency features. Specifically,

[0124] In each round of supervised training, the gradient update value of the low-frequency feature is calculated, and an update trend sequence reflecting the gradient change is generated to identify whether there is a sudden increase or decrease in the gradient of the low-frequency feature:

[0125] In each round of supervised training, the gradient update value of each low-frequency feature is calculated. The gradient update value indicates the contribution of the feature to the reduction of the current loss function. The gradient update value is obtained by calculating the partial derivative of the current model loss function with respect to the feature weight, which indicates the contribution of the current low-frequency feature to the optimization of the loss function. Specifically, in each round of supervised training, the model will calculate the adjustment direction and amplitude of the feature weight based on the loss function. The gradient update value is the influence of the low-frequency feature on the overall optimization in this round of training. The size and direction of the gradient update value reflect the direct effect of the feature on the improvement of model performance, and provide a basis for subsequent feature weight adjustments.

[0126] Record the gradient update value in each round of training and generate an update trend sequence that reflects its changes. For example, if the gradient update values ​​in the first, second, and third rounds are 0.02, 0.03, and 0.025, respectively, then the update trend sequence is [0.02, 0.03, 0.025].

[0127] Analyze the update trend sequence to detect whether there is a sudden increase or a sudden decrease. A sudden increase is defined as the gradient growth exceeding the threshold, and a sudden decrease is defined as the gradient decrease exceeding the threshold. The specific formula is as follows:

[0128] or ;in, Represents the gradient change between two consecutive rounds of training; Indicates the preset burst threshold.

[0129] The preset sudden drop threshold can be set to the negative preset sudden increase threshold, and the specific setting needs to be adjusted according to the specific application scenario. Generally speaking, the absolute values ​​of the sudden increase threshold and the sudden drop threshold should be equal to maintain consistency in the positive and negative directions of the gradient change. For example, if the sudden increase threshold is set to 0.01, the sudden drop threshold is -0.01. This setting ensures that the model is symmetrical when detecting sudden changes, facilitates unified analysis of gradient anomalies, and avoids inconsistent judgment criteria for positive and negative changes that may lead to result deviations.

[0130] Calculate the Shapley value of low-frequency features, generate a Shapley value distribution sequence based on all supervised training rounds, and detect whether the Shapley value distribution sequence of low-frequency features has abnormal fluctuations:

[0131] After each round of training, the Shapley value of the low-frequency feature is calculated to represent the marginal contribution of the feature to the model prediction. The calculation formula is as follows: ;in, Represents the Shapley value of low-frequency features, which is used to measure the marginal contribution of low-frequency features to model prediction; Represents a set of all low-frequency features. For example, a feature set may include multiple numerical or categorical features. Represents the current low-frequency feature, that is, the feature for which the Shapley value needs to be calculated; Represents a subset of low-frequency features, the subset is from Select from, indicating that except A part of the low-frequency feature combination other than The size of the subset representing the low-frequency features, that is, the number of features contained in the subset; Represents the size of the set containing all low-frequency features, that is, the total number of all features; Indicates the contribution of the subset of low-frequency features to the model prediction results; It represents the contribution of the subset of low-frequency features plus the low-frequency features to the model prediction results.

[0132] The Shapley values ​​in each round of training are recorded as a distribution sequence. For example, if the Shapley values ​​of a low-frequency feature in the first, second, and third rounds are 0.1, 0.15, and 0.12, respectively, then the distribution sequence is [0.1, 0.15, 0.12].

[0133] Detect whether the Shapley value distribution has abnormal fluctuations. The determination formula for abnormal fluctuations is: ;in, Indicates the Shapley value of the current training round; represents the historical average Shapley value; Indicates the fluctuation threshold.

[0134] The fluctuation threshold is usually set according to the standard deviation of historical data or the tolerance range of a specific application, for example, it is set to the mean of the characteristic Shapley value plus or minus two times the standard deviation to adapt to the data fluctuation characteristics.

[0135] The low-frequency features with abnormal fluctuations or sudden increases or decreases in the Shapley value distribution sequence are marked as abnormal low-frequency features:

[0136] If it is detected that the gradient update trend of the low-frequency feature suddenly increases or decreases, or the Shapley value distribution fluctuates abnormally, the low-frequency feature is marked as an abnormal low-frequency feature.

[0137] Record relevant information about abnormal characteristics, including abnormal type, change range and detection round.

[0138] Adjust the adaptive weighting strategy for the detected abnormal low-frequency features and modify their impact on the gradient boosting machine by modifying the weight value:

[0139] For abnormal low-frequency features, adjust their weight distribution. Specifically, for low-frequency features with sudden gradient increases, increase the weight decay coefficient to reduce the influence of the feature in the model, thereby avoiding over-reliance on a single feature; for low-frequency features with sudden gradient drops or abnormal fluctuations in Shapley values, reduce the weight decay coefficient to enhance their influence on the model to ensure that their importance is not weakened. The weight adjustment is based on the change amplitude and fluctuation degree of the feature. The adjusted weight will take effect in the next round of training, thereby dynamically optimizing the importance distribution of features during the overall training process and maintaining the stability and performance of the model.

[0140] Example 2: The difference between Example 2 of the present invention and Example 1 is that this example introduces a data recognition system based on supervised learning.

[0141] Figure 2 A structural schematic diagram of a data recognition system based on supervised learning of the present invention is given. The data recognition system based on supervised learning includes a data acquisition and filtering module, a feature gain screening module, a sparse feature generation module, a low-frequency feature weighting module and a feature weight adjustment module.

[0142] Data collection and filtering module: collects training data from multiple data sources and performs filtering operations on the training data to remove samples that do not meet the quality threshold.

[0143] Feature gain screening module: performs feature analysis on the filtered training data, and uses a dimensionality reduction algorithm to perform information gain measurement on each feature to generate a feature set with high information gain.

[0144] Sparse feature generation module: uses association rule algorithm to generate sparse feature combination sets in the feature space of feature sets with high information gain, and retains key low-frequency features and their interactions by setting frequency thresholds to obtain low-frequency feature sets.

[0145] Low-frequency feature weighting module: The low-frequency feature set is input into the gradient boosting machine for preliminary supervised training, and the initial weights are set through an adaptive weighting strategy to enhance the influence of low-frequency features in training.

[0146] Feature weight adjustment module: During supervised training, based on the gradient update trend of low-frequency features and the feature contribution distribution of Shapley values, monitor the weight changes of low-frequency features and whether their impact on the gradient boosting machine is abnormal. If abnormal, adjust the adaptive weighting strategy to correct the impact of low-frequency features.

[0147] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0148] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or may be transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium may be a solid-state hard disk.

[0149] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0150] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0151] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0152] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, and may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0153] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0154] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0155] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0156] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A data recognition method based on supervised learning, characterized in that: The steps include: S1: Collect training data from multiple data sources and perform filtering operations on the training data to remove samples that do not meet the quality threshold; The original data set containing product defect images collected from multiple sources includes high-definition images collected from three different types of cameras, defect record logs generated by automatic detection equipment, and historical defect data in the cloud storage platform. Field consistency check is performed on the collected data and the timestamp format of the log data is verified. In the data cleaning process, log records with missing fields are removed and image files with incorrect formats are deleted. The quality of the cleaned data set is evaluated, and only high-quality data samples that meet the standards are retained for model training. S2: Perform feature analysis on the filtered training data, and use the dimension reduction algorithm to perform information gain measurement on each feature to generate a feature set with high information gain; S3: Generate a sparse feature combination set in the feature space of a feature set with high information gain using an association rule algorithm, and retain key low-frequency features and their interactions by setting a frequency threshold to obtain a low-frequency feature set, specifically including: Extract each feature in the feature set with high information gain, and construct a feature space to store all possible feature combinations of the features; Use association rule algorithm to count the frequency of all feature combinations in feature space and calculate the support of each feature combination. The support is defined as the proportion of feature combinations that appear simultaneously in the training data. Generate a sparse feature combination set based on the support calculation result, and the sparse feature combination set includes all feature combinations that meet the set support range; Extract feature combinations whose support is lower than a preset frequency threshold from the sparse feature combination set, retain feature combinations containing low-frequency features, and analyze the interaction relationship between low-frequency features; The retained low-frequency features and their interaction relationships are formed into a low-frequency feature set, and the low-frequency feature set only includes low-frequency features and their interaction relationships whose support is lower than a preset frequency threshold; S4: Input the low-frequency feature set into the gradient boosting machine for preliminary supervised training, and set the initial weights through an adaptive weighting strategy to strengthen the influence of low-frequency features in training; S5: During supervised training, based on the gradient update trend of low-frequency features and the feature contribution distribution of Shapley values, monitor whether the weight changes of low-frequency features and their impact on the gradient boosting machine are abnormal. If abnormal, adjust the adaptive weighting strategy to correct the impact of low-frequency features.

2. The data recognition method based on supervised learning according to claim 1, characterized in that: Collect training data from multiple data sources and perform filtering operations on the training data to remove samples that do not meet the quality threshold, including: Obtain the original data set containing the target features, which comes from different data acquisition devices and platforms; Perform data cleaning operations on the original data set to remove data entries with missing values, data that do not meet format requirements, and data that contain obvious errors; Perform quality assessment on the original data set after data cleaning, screen the data samples based on the preset quality assessment criteria, and remove data samples that do not meet the quality assessment criteria to generate a training data set that meets the requirements.

3. The data recognition method based on supervised learning according to claim 1, characterized in that: Perform feature analysis on the filtered training data, and use the dimension reduction algorithm to perform information gain measurement on each feature to generate a feature set with high information gain, including: Construct a feature space based on the value range and distribution of each feature in the training data. The feature space contains multiple dimensions for describing the features of the training data. The dimensionality reduction algorithm is used to screen and optimize the dimensions of the feature space. The dimensionality reduction algorithm includes an analysis method based on information gain measurement. The information gain value is calculated for each dimension of the feature space. The information gain value is based on the entropy difference between the distribution of the target label and the conditional distribution of the feature value. According to a preset threshold, features with information gain values ​​higher than the threshold are retained to form a feature set with high information gain.

4. The data recognition method based on supervised learning according to claim 1, characterized in that: The low-frequency feature set is input into the gradient boosting machine for preliminary supervised training, and the initial weights are set through an adaptive weighting strategy to strengthen the influence of low-frequency features in training, including: Format the low-frequency features in the low-frequency feature set into an input form suitable for the gradient boosting machine, including normalization of numerical features and one-hot encoding of categorical features; Initialize the training parameters of the gradient boosting machine, including learning rate, maximum tree depth, subsampling ratio, and minimum number of samples for splitting nodes; In the model initialization stage, according to the weight allocation rules of low-frequency features, the initial weights are allocated to low-frequency features through an adaptive weighting strategy. The weight allocation rules include: calculating the initial weight of the low-frequency features according to their support, and the weight is proportional to the inverse of the support value; if there is a significant interaction between low-frequency features, their weights are superimposed and corrected based on the strength of the interaction; The gradient boosting machine is supervised and trained using training data containing a set of low-frequency features. The model parameters, including feature splitting points, leaf node weights, and loss function values, are updated in each iteration.

5. The data recognition method based on supervised learning according to claim 1, characterized in that: During supervised training, based on the gradient update trend of low-frequency features and the feature contribution distribution of Shapley values, monitor the weight changes of low-frequency features and whether their impact on the gradient boosting machine is abnormal. If abnormal, adjust the adaptive weighting strategy to correct the impact of low-frequency features. Specifically, In each round of supervised training, the gradient update value of the low-frequency feature is calculated, and an update trend sequence reflecting the gradient change is generated to identify whether there is a sudden increase or decrease in the gradient of the low-frequency feature; Calculate the Shapley value of low-frequency features, generate a Shapley value distribution sequence based on all supervised training rounds, and detect whether the Shapley value distribution sequence of low-frequency features has abnormal fluctuations; The low-frequency features with abnormal fluctuations or sudden increases or decreases in the Shapley value distribution sequence are marked as abnormal low-frequency features; The adaptive weighting strategy is adjusted for the detected abnormal low-frequency features, and their influence on the gradient boosting machine is corrected by modifying the weight value.

6. The data recognition method based on supervised learning according to claim 5, characterized in that: During the supervised training process, the Shapley value of the low-frequency feature is calculated, and the Shapley value distribution sequence is generated based on all supervised training rounds to detect whether the Shapley value distribution sequence of the low-frequency feature has abnormal fluctuations. Specifically: After each round of training, the Shapley value of the low-frequency feature is calculated using the following formula: ;in, The Shapley value representing the low-frequency feature, represents a set of all low-frequency features, Indicates the current low-frequency characteristics, represents a subset of low-frequency features, represents the size of the subset of low-frequency features, represents the size of the set containing all low-frequency features, Indicates the contribution of the subset of low-frequency features to the model prediction results, It represents the contribution of the subset of low-frequency features plus the low-frequency features to the model prediction results; Detect whether the Shapley value distribution has abnormal fluctuations. The determination formula for abnormal fluctuations is: ;in, Indicates the Shapley value of the current training round, represents the historical average Shapley value, Indicates the fluctuation threshold.

7. A data recognition system based on supervised learning, used to implement the data recognition method based on supervised learning according to any one of claims 1 to 6, characterized in that: It includes a data acquisition and filtering module, a feature gain screening module, a sparse feature generation module, a low-frequency feature weighting module and a feature weight adjustment module; Data collection and filtering module: collects training data from multiple data sources and performs filtering operations on the training data to remove samples that do not meet the quality threshold; Feature gain screening module: performs feature analysis on the filtered training data, and uses the dimension reduction algorithm to perform information gain measurement on each feature to generate a feature set with high information gain; Sparse feature generation module: uses association rule algorithm to generate sparse feature combination sets in the feature space of feature sets with high information gain, and retains key low-frequency features and their interactions by setting frequency thresholds to obtain low-frequency feature sets; Low-frequency feature weighting module: The low-frequency feature set is input into the gradient boosting machine for preliminary supervised training, and the initial weights are set through an adaptive weighting strategy to strengthen the influence of low-frequency features in training; Feature weight adjustment module: During supervised training, based on the gradient update trend of low-frequency features and the feature contribution distribution of Shapley values, monitor the weight changes of low-frequency features and whether their impact on the gradient boosting machine is abnormal. If abnormal, adjust the adaptive weighting strategy to correct the impact of low-frequency features.

Images