A tobacco index outlier cleaning method and apparatus

Abstract

The application provides a tobacco index outlier cleaning method and device, the tobacco index outlier cleaning method comprises the following steps: step S1: obtaining a historical tobacco data set; step S2: fitting and constructing a univariate Gaussian distribution model according to the mean and standard deviation of the training set; step S3: determining a target detection threshold based on the univariate Gaussian distribution model and the validation set, and testing the target detection threshold according to the test set; step S4: reversely solving the target detection threshold to obtain a target normal value interval corresponding to each index type, and cleaning the to-be-tested tobacco data set according to the target normal value interval to obtain a target tobacco data set. The application generates an adaptability judgment boundary according to the distribution characteristics of the data itself, which can neither cause the omission of implicit quality defects due to large batch data fluctuations, nor cause unnecessary excessive screening for stable batches with small dispersion, and is suitable for the quality control requirements of the whole tobacco industry chain in multiple scenes.

Description

Technical Field

[0001] This application belongs to the field of tobacco data processing, specifically relating to a method and apparatus for cleaning outliers in tobacco indicators. Background Technology

[0002] Tobacco leaves are the core raw material for cigarette production, and the stability of their physicochemical indicators (including nicotine, reducing sugar, total sugar, potassium, chlorine, moisture content, color, and filling value) directly determines the sensory quality and consistency of subsequent cigarette products. Throughout the entire tobacco leaf supply chain—from acquisition and grading, re-drying, storage and aging, to cigarette manufacturing and feeding—outlier detection of batch tobacco leaf indicators is essential to identify problems such as mixed varieties, drifting processing parameters, storage deterioration, and equipment malfunctions. This is a core and crucial aspect of tobacco leaf quality control.

[0003] Current common outlier detection techniques fall into two main categories: one is based on the 3σ criterion of Gaussian distribution, which assumes the data follows a normal distribution and identifies data deviating from three standard deviations of the mean as outliers; the other is based on the box plot method using the interquartile range (IQR), which calculates the difference between the upper and lower quartiles and identifies data exceeding 1.5 times the IQR interval as outliers. Both methods are widely used in standardized industrial scenarios where data distribution conforms to the normality assumption and dispersion is stable, but they exhibit clear limitations in the outlier detection scenarios for tobacco raw material indicators.

[0004] Tobacco leaf indicators are affected by multiple uncontrolled factors such as the ecology of the production area, varietal characteristics, planting year, cultivation techniques, and curing process. They naturally exhibit skewed and multimodal distribution characteristics, which do not meet the normality premise of Gaussian distribution. Directly applying the 3σ criterion will result in a large number of misjudgments. For example, normal indicators of high nicotine specialty varieties may be identified as abnormal, or low-amplitude hidden quality defects at the tail of the skewed distribution may be missed.

[0005] The dispersion of tobacco leaf indicators varies greatly depending on the grade, batch, and processing stage. For example, the dispersion of moisture content in first-cured tobacco leaves can be more than three times that of re-cured tobacco leaves. The fixed 1.5 times threshold of the IQR method cannot meet the detection needs of different scenarios. It will either miss the batch with large dispersion or over-detect the batch with small dispersion.

[0006] Currently, the industry has not yet developed a dedicated probability distribution scheme for anomaly detection that is compatible with the distribution characteristics of tobacco raw material indicators. How to obtain a scheme that is compatible with different production areas, varieties, and processing stages, while taking into account both detection accuracy and generalization, based on massive historical indicator data of multi-source tobacco raw materials, is a core technical challenge that urgently needs to be addressed in the field of tobacco quality control. Summary of the Invention

[0007] In view of this, the purpose of this application is to provide a method and apparatus for cleaning outlier values ​​in tobacco indicators to solve the above-mentioned problems.

[0008] To solve the above-mentioned technical problems, this application adopts the following technical solution: Firstly, this application provides a method for cleaning outliers in tobacco indicators. This method includes: Step S1: Obtaining a historical tobacco dataset, which includes unimodal distribution indicators from the training set, the validation set, and the test set; Step S2: Fitting and constructing a univariate Gaussian distribution model based on the mean and standard deviation of the training set; Step S3: Determining a target detection threshold based on the univariate Gaussian distribution model and the validation set, and testing the target detection threshold using the test set; Step S4: Reverse-calculating the target detection threshold to obtain the target normal value intervals corresponding to each indicator type, and cleaning the tobacco dataset to be tested based on the target normal value intervals to obtain the target tobacco dataset.

[0009] Further, step S1 includes: Step S11: Divide the original historical tobacco dataset into an original training set, an original validation set, and an original test set according to a preset ratio or quantity; Step S12: Remove abnormal data from the original training set and retain the unimodal distribution index type data to obtain the training set; Step S13: Retain the unimodal distribution index type data of the original validation set to obtain the validation set; Step S14: Retain the unimodal distribution index type data of the original test set to obtain the test set; Step S15: Determine whether the index type data in the historical tobacco dataset conforms to a Gaussian distribution: if not, transform the unimodal distribution index data according to the power transform algorithm to approximate a Gaussian distribution.

[0010] Furthermore, power transform algorithms include the Box-Cox transform algorithm and the Yeo-Johnson transform algorithm.

[0011] Furthermore, the single-peak distribution index includes appearance index, sensory index, aroma components, conventional chemical components and physical index. The appearance index includes the total appearance score, which includes oil content, maturity, softness, leaf structure, identity, color and chroma. The sensory index includes the total sensory score, which includes aroma quality, aroma quantity, off-odors, irritation and aftertaste.

[0012] Furthermore, step S2 includes: Step S21: Obtain the mean and standard deviation of each indicator type in the training set; Step S22: Fit a univariate Gaussian distribution model based on the mean and standard deviation; the formula for the univariate Gaussian distribution model is:

[0013] Where x is the sample index value, μ is the mean, and σ is the standard deviation.

[0014] Further, step S3 includes: Step S31: Input the validation set into the Gaussian distribution model and perform multi-fold cross-validation with the training set to obtain the log-likelihood value corresponding to the validation set; Step S32: Within the range of log-likelihood values, traverse and select critical scores according to a preset step size to generate a candidate threshold set; Step S33: Calculate the detection evaluation index corresponding to each candidate threshold in the candidate threshold set, and determine the target detection threshold based on the detection evaluation index; Step S34: Test the target detection threshold based on the test set to determine whether the target detection threshold meets the preset detection evaluation index: if so, then reverse solve to obtain the target normal value range corresponding to the index type.

[0015] Furthermore, step S33 includes: step S331: comparing the log-likelihood value corresponding to each data point in the validation set with each candidate threshold in the candidate threshold set, and calculating the detection evaluation index corresponding to each candidate threshold; step S332: screening each candidate threshold according to the detection evaluation index to determine the target detection threshold.

[0016] Further, step S34 includes: step S341: determine whether the data of the index type is obtained by the power transformation algorithm; if so, calculate the normal value range of the change domain based on the mean, standard deviation and target detection threshold after the power transformation algorithm; step S342: restore the normal value range of the change domain according to the inverse transformation of the power transformation algorithm to determine the target normal value range.

[0017] Furthermore, step S34 includes: step S343: determine whether the data of the indicator type conforms to a Gaussian distribution; if so, calculate the normal value range of the target based on the mean, variance and target detection threshold.

[0018] Secondly, this application provides a tobacco indicator outlier cleaning device, which includes: an acquisition module, a construction module, a determination module, and a cleaning module. The acquisition module is used to acquire historical tobacco datasets, which include unimodal distribution indicators of the training set, the validation set, and the test set. The construction module is used to fit and construct a univariate Gaussian distribution model based on the mean and standard deviation of the training set. The determination module is used to determine the target detection threshold based on the univariate Gaussian distribution model and the validation set, and to test the target detection threshold based on the test set. The cleaning module is used to reverse-engineer the target detection threshold to obtain the target normal value interval corresponding to each indicator type, and to clean the tobacco dataset to be tested based on the target normal value interval to obtain the target tobacco dataset.

[0019] As can be seen from the above technical solutions, this application addresses the shortcomings of existing technologies, such as the physicochemical indicators of tobacco leaves being easily affected by multiple factors including production area, variety, year, and processing procedures, resulting in skewed and multimodal distributions, and significant differences in the dispersion of data between different batches of tobacco leaves. Traditional methods like the 3σ criterion and the fixed-coefficient IQR box method are prone to numerous false positives and false negatives, making them unsuitable for the quality control needs of the entire tobacco leaf industry chain. This application provides a method and apparatus for cleaning outlier values ​​in tobacco indicators, which has the following technical advantages and beneficial effects: 1. This invention does not directly use empirical thresholds with a fixed multiple of standard deviation for judgment. Instead, it relies on independently partitioned validation datasets and adaptively selects target likelihood thresholds using the F2 score-optimal strategy. This dynamically balances the anomaly detection rate and false alarm rate, overcoming the rigid judgment limitations imposed by the 3σ criterion and the fixed coefficient of 1.5 times IQR. Faced with tobacco batch data exhibiting significant variations in dispersion across different processing stages such as initial curing, re-curing, storage, and tobacco processing, an adaptive judgment boundary can be generated based on the data's inherent distribution characteristics. This avoids missing latent quality defects due to large batch data fluctuations, while also preventing unnecessary over-screening of stable batches with low dispersion, thus adapting to the quality control needs of multiple scenarios across the entire tobacco industry chain.

[0020] 2. This invention constructs a single-variable Gaussian distribution model for each physicochemical indicator, such as nicotine, reducing sugar, moisture content, and filling value. It independently learns the mean and variance of each indicator under normal operating conditions to obtain the data distribution pattern of a single indicator. This can accurately locate abnormal problems of a single indicator corresponding to variety mixing, processing parameter drift, storage deterioration, and instrument detection failure. Compared with the multi-indicator joint modeling method, it has stronger traceability and makes it easier for process personnel to quickly locate the source of quality problems. At the same time, the single-variable model structure is lightweight and the calculation logic is simple. It can be embedded into production line terminals such as tobacco leaf purchasing and grading, online re-drying detection to achieve real-time data cleaning. The engineering deployment cost is low and the operating efficiency is high.

[0021] 3. This invention uses a training set specifically fitted with Gaussian distribution parameters and a validation set to independently determine the threshold, strictly distinguishing between the modeling data source and the threshold optimization data source. This avoids data leakage and overfitting problems caused by using the same dataset to train the model and set judgment rules, greatly improving the model's generalization ability on new production areas, new varieties, and new year tobacco batches.

[0022] 4. This invention reverses the optimized target likelihood threshold to an intuitive and readable normal value range for various physicochemical indicators of tobacco leaves, transforming the abstract probability likelihood judgment rule into a specific and implementable process control value range. It can complete the batch screening of test data without repeatedly calculating the probability density, and the judgment logic is highly interpretable. It is convenient for enterprises to solidify internal quality control standards for tobacco leaves and archive quality control specifications, thus reducing the usage threshold for front-line quality inspectors.

[0023] 5. The tobacco leaf index dataset cleaned by this method eliminates noisy outliers caused by equipment malfunctions, raw material contamination, and process abnormalities. This effectively ensures the reliability of the data foundation for subsequent data analysis work such as tobacco leaf formulation design, aging cycle assessment, process parameter optimization, and quality prediction modeling. It reduces the interference of abnormal data on the tobacco leaf quality analysis model from the source, helps stabilize the consistency of sensory quality and batch quality of cigarette products, and fills the gap in the tobacco leaf industry for a dedicated probabilistic anomaly detection solution adapted to the complex distribution characteristics of raw materials. Attached Figure Description

[0024] The above description of this application and the following detailed embodiments will be better understood when read in conjunction with the accompanying drawings. It should be noted that the drawings are merely examples of the claimed technical solutions.

[0025] Figure 1 This is a flowchart of the tobacco index outlier cleaning method of this application; Figure 2 This is a graph showing the change in the F2 score of the candidate threshold; Figure 3 This is a confusion matrix obtained by applying the standard detection threshold to the validation set; Figure 4 This is a schematic diagram illustrating the effect of the target detection threshold on the test set. Detailed Implementation

[0026] The detailed features and advantages of this application are described below in the specific embodiments. The content of this description is sufficient to enable any person skilled in the art to understand the technical content of this application and implement it accordingly. Based on the specification, claims and drawings disclosed in this specification, a person skilled in the art can easily understand the related objectives and advantages of this application.

[0027] The invention will now be described with reference to the accompanying drawings, in which similar reference numerals denote similar elements. While specific structures and arrangements are discussed, it should be understood that this is done merely for illustrative purposes. Those skilled in the art will recognize that other structures and arrangements can be used without departing from the spirit and scope of the invention. It will be apparent to those skilled in the art that the invention can also be used in a variety of other applications.

[0028] In this specification and claims, several terms will be used, and unless otherwise indicated, these terms will be defined to have the following meanings: The singular forms “a” and “the” include their corresponding plural forms. “At least one” means one or more, and “more” means two or more. “At least one of the following” or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be expressed as: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0029] All figures used to represent component amounts, properties (e.g., molecular weight), reaction conditions, etc., should be considered to be modified in all cases by the terms "within the unavoidable margin of error" or "about". Therefore, the numerical values ​​set forth herein are approximate and may vary depending on the desired properties sought to be obtained by the present invention. The principle of equivalents, which is applied to a minimum and not intended to limit the scope of the claims, should be applied, for example, each value should be interpreted at least according to the specified significant digits and by applying conventional rounding techniques.

[0030] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0031] In the description of this embodiment, it should be noted that the terms "upper", "lower", "inner", "bottom", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product is usually placed during use. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0032] Unless otherwise indicated, the following abbreviations have the following meanings, and any other abbreviations used herein but not defined have their generally accepted standard meanings: All other terms used herein that are specifically defined herein shall have the general meaning understood by one of ordinary skill in the art, in particular meaning that, upon reading the claims, specification and drawings of this patent, one can directly and without doubt determine how the technical solution of this patent can be implemented.

[0033] Even if there are incomplete descriptions, omissions, or ambiguities in the grammar, words, punctuation, graphics, symbols, etc. of the claims, specification, and drawings of this patent, a person skilled in the art can still arrive at the only correct understanding by reading the claims, specification, and drawings as a whole without extensive reasoning or experimentation, and effectively exclude various incorrect interpretations that are not aimed at achieving the purpose of this patent.

[0034] Those skilled in the art would first choose to read the claims, specification, and drawings of this patent to reasonably interpret the terms; secondly, they would choose to refer to the relevant definitions in other documents published by the applicant before the filing date to reasonably interpret the terms; thirdly, they would choose the references cited in this patent to reasonably interpret the terms; and finally, they would choose to combine the technical dictionaries, technical manuals, reference books, textbooks, national or industry technical standards, etc., commonly used by those skilled in the art to reasonably interpret the terms.

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0036] Please refer to Figure 1 This application provides a method for cleaning outliers in tobacco indicators. The specific steps of this method are as follows: Step S1: Obtain the historical tobacco dataset, which includes the unimodal distribution index of the training set, the unimodal distribution index of the validation set, and the unimodal distribution index of the test set.

[0037] The number of samples for each indicator in the historical tobacco dataset is no less than 50, with the training dataset accounting for 70% and the validation dataset accounting for 30%. Cross-validation is then used for training.

[0038] Step S1 includes: Step S11: Divide the original historical tobacco dataset into the original training set, the original validation set, and the original test set according to the preset ratio or quantity.

[0039] For example, the ratio of the original training set, the original validation set, and the original test set can be 6:2:2.

[0040] Using each column of indicators in the original historical tobacco dataset as a unit, clean and remove outliers, fill in the missing values ​​of each column of indicators, and label each sample indicator according to the "tobacco quality indicator standard range" (for example, the standard range of total alkaloids should be between 0.5% and 3.5% to be normal, and below 0.5% or above 3.5% to be abnormal).

[0041] It is understood that this application is based on prior experience from manual annotation. The original range of each sample indicator can also be obtained by percentile method or 3σ principle as the initial quality indicator range, and then each data is annotated according to the initial quality indicator range.

[0042] This data range is a normal range determined by a broad empirical range, but this range may lack support from real sample data, making it difficult to adapt to and fit the actual testing scenario. The final output data range of this application is based on the initial data range and, after training, yields a more accurate data range that fits the actual working conditions and can be directly applied to the detection of outliers in the physicochemical indicators of tobacco leaves.

[0043] Step S12: Remove outlier data from the original training set and retain the index type data with a unimodal distribution to obtain the training set.

[0044] The training set only needs to retain normal data to remove outlier data.

[0045] Step S13: Retain the unimodal distribution of the index type data of the original validation set to obtain the validation set.

[0046] To reduce the amount of work, outliers can be manually marked only on the validation set.

[0047] Step S14: Retain the index type data of the original test set with a unimodal distribution to obtain the test set.

[0048] Step S15: Determine whether the data of each indicator type in the historical tobacco dataset conforms to a Gaussian distribution: If not, transform the data of the unimodal distribution indicators according to the power transform algorithm to approximate the Gaussian distribution.

[0049] If the data of the indicator type itself conforms to a Gaussian distribution, no transformation is required.

[0050] Power transform algorithms can include the Box-Cox transform algorithm and the Yeo-Johnson transform algorithm.

[0051] The Box-Cox transform algorithm requires all data to be greater than 0, while the Yeo-Johnson transform algorithm has no such requirement.

[0052] This application employs the KS normality test, exemplarily setting the significance level to 0.05, to determine the data distribution characteristics of each tobacco raw material indicator: If the test result is P>0.05, the judgment index data approximately follows a Gaussian distribution and can be considered as following a Gaussian distribution; if the test result is P≤0.05, the judgment index data exhibits a skewed distribution. When the judgment index data is skewed, the power transformation algorithm is used to transform it to approximate a Gaussian distribution.

[0053] The formula for the Box-Cox transformation algorithm is shown below:

[0054] Here, λ is the adjustment parameter, and the λ that maximizes the log-likelihood can be found based on the data.

[0055] The formula for the Yeo-Johnson transformation algorithm is shown below:

[0056] Wherein, λ is the adjustment parameter, which can be determined by referring to the Box-Cox variation algorithm. This coefficient can be determined by maximizing the log-likelihood estimate.

[0057] It should be noted that the Box-Cox transform algorithm and the Yeo-Johnson transform algorithm are only applicable to data with a unimodal distribution. If used with multimodal data, the data will be distorted even after the transformation.

[0058] For example, unimodal distribution indicators include appearance indicators, sensory indicators, aroma components, conventional chemical components, and physical indicators.

[0059] The appearance indicators include the total appearance score, which includes oil content, maturity, softness, leaf structure, identity, color, and chroma.

[0060] Sensory indicators include the total sensory score, which includes aroma quality, aroma quantity, off-odors, irritation, and aftertaste.

[0061] Aroma-producing components include polyphenols, sugars, and plastid pigments.

[0062] Common chemical components include reducing sugars, total sugars, total chlorides, total alkaloids, total potassium, total nitrogen, nitrogen-base ratio, sugar-base ratio, potassium-chlorine ratio, two-sugar ratio, and two-sugar difference.

[0063] Physical properties can be found in Table 1.

[0064] Table 1: Summary of Physical Indicators

[0065]

[0066] Step S2: Based on the mean and standard deviation of the training set, fit and construct a univariate Gaussian distribution model.

[0067] Step S2 includes: Step S21: Obtain the mean and standard deviation of each indicator type in the training set.

[0068] Step S22: Fit a univariate Gaussian distribution model based on the mean and standard deviation; the formula for the univariate Gaussian distribution model is:

[0069] Where x is the sample index value, μ is the mean, and σ is the standard deviation.

[0070] Step S3: Based on the univariate Gaussian distribution model and the validation set, determine the target detection threshold, and test the target detection threshold according to the test set.

[0071] Step S3 includes: Step S31: Input the validation set into the Gaussian distribution model and perform multi-fold cross-validation with the training set to obtain the log-likelihood value corresponding to the validation set.

[0072] Preferably, a 5-fold cross-validation is used, with the number of samples in the training set and validation set allocated in a 4:1 ratio. Based on the samples in the training set and validation set, different data are alternately used as the training set and validation set. The training set has no outliers, and the proportion of outliers in each fold of the validation set should remain unchanged.

[0073] The specific partitioning rule is that the training subset corresponding to each fold retains only normal tobacco samples that meet the compliant indicators, and removes outliers within the samples. The validation subset corresponding to each fold retains both normal samples and labeled outliers, and the proportion of outliers in the validation subset of each fold is kept consistent.

[0074] It is understandable that five-fold cross-validation is used to mitigate the randomness of random partitioning of a single dataset and improve the stability of threshold screening results. The training subset removes abnormal samples to ensure that the fitting of Gaussian distribution parameters is not affected by abnormal data. Each fold validation subset maintains a consistent proportion of abnormal samples to ensure that the F2 evaluation index of each subsequent fold can be compared, avoiding the interference of sample structure differences on the threshold optimization process. Ultimately, a more adaptable index judgment interval is obtained, which improves the generalization performance of the tobacco leaf physicochemical index anomaly detection scheme.

[0075] The validation set is input into a Gaussian distribution model to obtain the probability density values ​​corresponding to each data point in the validation set. Based on the constructed Gaussian distribution model, all data points of this indicator in the validation set are substituted to calculate the probability density value of each data point. To avoid numerical instability, all probability density values ​​are converted into log-likelihood values.

[0076] Step S32: Within the range of log-likelihood values, select critical scores by iterating through the range according to a preset step size to generate a candidate threshold set.

[0077] The maximum and minimum log-likelihood values ​​corresponding to each data point in the validation set are used as the range. The system iterates through the data points according to the preset step size to obtain multiple candidate thresholds and form a candidate threshold set. The preset step size is (maximum log-likelihood value - minimum log-likelihood value) / n, where n is the number of iterations.

[0078] The search range for candidate thresholds is determined to be from the minimum to the maximum value of the log-likelihood values ​​of the validation set. Several candidate thresholds can be generated by traversing the data with equal step size, where the step size is set to (maximum log-likelihood value - minimum log-likelihood value) / n, and n is the number of iterations. This ensures that all possible threshold ranges are covered, balancing training efficiency and accuracy.

[0079] Step S33: Calculate the detection evaluation index corresponding to each candidate threshold in the candidate threshold set, and determine the target detection threshold based on the detection evaluation index.

[0080] Step S33 includes: Step S331: Compare the log-likelihood values ​​corresponding to each data point in the validation set with the candidate thresholds in the candidate threshold set, and calculate the detection evaluation index corresponding to each candidate threshold.

[0081] The evaluation metrics for testing include precision, recall, and F2 score.

[0082] The formula for the F-score is:

[0083] If β=2, then it is the F2 score, Precision is the precision rate, and Recall is the recall rate.

[0084] Furthermore, in anomaly detection, the cost of a false negative is usually far higher than that of a false positive. Therefore, the F2 score can assign a higher weight to recall than precision, which better aligns with the business logic of anomaly detection. That is, β=2.

[0085] Step S332: Based on the detection evaluation indicators, screen each candidate threshold and determine the target detection threshold.

[0086] For example, with maximizing the F2 score as the core optimization objective, the candidate threshold with the highest corresponding F2 score is selected as the optimal detection threshold for this metric. If there are multiple candidate thresholds corresponding to the same highest F2 score, the threshold that balances precision and recall can be selected as the target detection threshold.

[0087] Step S34: Test the target detection threshold based on the test set and determine whether the target detection threshold meets the preset detection evaluation index. If so, then solve the normal value range of the target corresponding to the index type.

[0088] Specifically, the test set data is subjected to power transformation, substituted into the Gaussian distribution model obtained from training, the log-likelihood of each sample is calculated, the previously obtained target detection threshold is used to distinguish between normal and abnormal, and finally the samples are compared with the real human labels, the detection indicators are statistically analyzed, and it is confirmed whether the target detection threshold is a preset detection evaluation indicator.

[0089] Step S4: Solve the target detection threshold in reverse to obtain the target normal value range corresponding to each indicator type, and clean the tobacco dataset to be tested according to the target normal value range to obtain the target tobacco dataset.

[0090] For example, the present application will be further illustrated using "total alkaloids" as a specific embodiment.

[0091] 1. Filter out values ​​less than 0 and missing values, and sort the data in ascending order using "total alkaloids" as the fitting index (sample size n=1688). The ratio of training set: validation set: test set is 6:2:2.

[0092] Normal samples: 1533, Abnormal samples: 155; Training set: 919 samples (all normal); Validation set: 384 samples (normal: 307, abnormal: 77); Test set: 385 samples (normal: 387, abnormal: 78); Validation set anomaly rate: 20.05%; Anomaly rate in the test set: 20.26%.

[0093] 2. Data transformation results: Sample size: 1688; KS test statistic: 0.0719, p-value: 0.6080; Is the original data normally distributed? No; Performing a Yeo-Johnson transform, the optimal λ = 8.0101; After the swap, Ks p-value is 0.2913; Is the transformed state normal? Yes.

[0094] The original data is subjected to an automatically optimized Yeo-Johnson transformation with λ, and the transformed data is subjected to a KS global test again to ensure that the data tends to a normal distribution.

[0095] Gaussian modeling is performed only on the training set to estimate the population mean and unbiased standard deviation. The log-likelihood (probability density) value for each sample is calculated, and -inf (which deviates from the mean by approximately 10 standard deviations) is replaced with the minimum value -1e10. The formula is as follows (μ and σ are the estimated parameters):

[0096] Search for candidate thresholds from the minimum likelihood to the maximum likelihood in the data, with a step size of 200, and use the best F2 score as the standard to find the target detection threshold.

[0097] To further determine the effectiveness of the target detection threshold, such as Figures 2-4 As shown, the object detection threshold is applied to the validation set for validation, resulting in a confusion matrix.

[0098] 3. Results of 5-fold cross-validation: Fold 1 training complete -> Precision: 0.9371, Recall: 0.9351, F2: 0.9353, Val anomaly rate: 19.48%; Fold 2 training complete -> Precision: 0.9221, Recall: 8.9221, F2: 0.9221, Val anomaly rate: 19.48%; Fold 3 training complete -> Precision: 0.9769, Recall: 8.9740, F2: 0.9740, Val anomaly rate: 20.78%; Fold 4 training complete -> Precision: 0.9028, Recall: 0.8961, F2: 0.8968, Val anomaly rate: 20.78%; Fold5 training complete -> Precision: 0.9628, Recall: 0.9605, F2: 0.9607, Val anomaly rate: 19.74%; 4. Results of running on the test set: Optimal threshold: -1.2569; Test set precision: 0.9172; Test set Recall: 0.9195; Test set F2 score: 0.9186.

[0099] 5. Data inverse solution interval: The distribution is Gaussian, with μ = 1.1511 and σ = 0.2232. The target log-likelihood threshold is -1.2569. Intermediate quantity C = -2σ 2 ·(th+0.5·1n(2πσ 2 =0.1831; The abnormal data range after the swap is: (-∞, 0.7231)U(1.5790, +∞); Yeo-Johnson inverse transform (λ=0.0101); The outlier range of the original data is: (-∞, 2.0555)U(4.7902, +∞); The original value is considered abnormal if it is <2.0555 or >4.7902.

[0100] Based on the same inventive concept, this application also provides a tobacco index outlier cleaning device, which includes: an acquisition module, a construction module, a determination module and a cleaning module.

[0101] The acquisition module is used to acquire historical tobacco datasets, which include unimodal distribution metrics for the training set, validation set, and test set.

[0102] The building block is used to fit a univariate Gaussian distribution model based on the mean and standard deviation of the training set.

[0103] The determination module is used to determine the target detection threshold based on a univariate Gaussian distribution model and a validation set, and to test the target detection threshold based on a test set.

[0104] The cleaning module is used to reverse-solve the target detection threshold to obtain the target normal value range corresponding to each indicator type, and then clean the tobacco dataset to be tested according to the target normal value range to obtain the target tobacco dataset.

[0105] Furthermore, the acquisition module is also used to divide the original historical tobacco dataset into an original training set, an original validation set, and an original test set according to a preset ratio or quantity; remove abnormal data from the original training set and retain the indicator type data with a unimodal distribution to obtain the training set; retain the indicator type data with a unimodal distribution in the original validation set to obtain the validation set; retain the indicator type data with a unimodal distribution in the original test set to obtain the test set; determine whether the indicator type data in the historical tobacco dataset conforms to a Gaussian distribution: if not, transform the data of the unimodal distribution indicators according to the power transform algorithm to approach the Gaussian distribution.

[0106] Power transformation algorithms include the Box-Cox transformation algorithm and the Yeo-Johnson transformation algorithm. Unimodal distribution indices include appearance indices, sensory indices, aroma components, conventional chemical components, and physical indices. Appearance indices include a total appearance score, which includes oil content, maturity, softness, leaf structure, identity, color, and chroma. Sensory indices include a total sensory score, which includes aroma quality, aroma quantity, off-odors, irritation, and aftertaste.

[0107] Furthermore, the module is also used to obtain the mean and standard deviation of each indicator type in the training set; based on the mean and standard deviation, a univariate Gaussian distribution model is fitted; the formula for the univariate Gaussian distribution model is:

[0108] Where x is the sample index value, μ is the mean, and σ is the standard deviation.

[0109] Furthermore, the determination module is also used to input the validation set into the Gaussian distribution model and perform multi-fold cross-validation with the training set to obtain the log-likelihood value corresponding to the validation set; within the range of the log-likelihood value, the critical score is selected by traversing according to the preset step size to generate a candidate threshold set; the detection evaluation index corresponding to each candidate threshold in the candidate threshold set is calculated, and the target detection threshold is determined according to the detection evaluation index; the target detection threshold is tested according to the test set to determine whether the target detection threshold meets the preset detection evaluation index; if so, the target normal value range corresponding to the index type is obtained by inverse solution.

[0110] Furthermore, the determination module is also used to compare the log-likelihood values ​​corresponding to each data point in the validation set with each candidate threshold in the candidate threshold set, and calculate the detection evaluation index corresponding to each candidate threshold; based on the detection evaluation index, each candidate threshold is screened to determine the target detection threshold.

[0111] Furthermore, determine whether the data of the indicator type was obtained by the power transform algorithm: if so, calculate the normal value range of the change domain based on the mean, standard deviation and target detection threshold after the power transform algorithm; restore the normal value range of the change domain according to the inverse transform algorithm to determine the target normal value range.

[0112] Furthermore, the data for determining the indicator type conforms to a Gaussian distribution: if so, the normal value range of the target is calculated by inverse solution based on the mean, variance, and target detection threshold.

[0113] It is understood that the tobacco index outlier cleaning device provided in this application corresponds to the tobacco index outlier cleaning method provided in this application. For the sake of brevity, the same or similar parts can be referred to the content of the tobacco index outlier cleaning method section, and will not be repeated here.

[0114] The modules in the aforementioned tobacco index anomaly cleaning device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the server's processor in hardware form or independent of it, or stored in the server's memory in software form, so that the processor can call and execute the corresponding operations of each module. The processor can be a central processing unit (CPU), a microprocessor, a microcontroller, etc.

[0115] The above-mentioned tobacco index outlier cleaning method and / or tobacco index outlier cleaning device can be implemented in the form of a computer-readable instruction, which can run on a computer system.

[0116] This application also provides a computer system including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described method for cleaning outlier values ​​of tobacco indicators.

[0117] The computer system can be a server. The computer system includes a processor, non-volatile storage medium, internal memory, input device, display screen, and network interface connected via a system bus. The non-volatile storage medium of the computer system can store an operating system and computer-readable instructions. When these computer-readable instructions are executed, they can cause the processor to execute a tobacco index outlier cleaning method according to various embodiments of this application. The specific implementation process of this method can be found in [reference needed]. Figure 1 The specific details will not be elaborated here.

[0118] The processor of this computer system provides computing and control capabilities, supporting the operation of the entire system. The internal memory stores computer-readable instructions, which, when executed by the processor, cause the processor to perform a method for cleaning outlier values ​​in tobacco indicators. The computer system's input devices are used for inputting various parameters, its display screen is used for display, and its network interface is used for network communication.

[0119] Based on the same inventive concept, this application provides a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps in the above-described method for cleaning outlier values ​​of tobacco indicators.

[0120] The memory in the embodiments of this application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0121] The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache.

[0122] By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), sync link DRAM (SLDRAM), and direct memory bus RAM (DR RAM).

[0123] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions or computer programs.

[0124] When computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions according to the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means.

[0125] Computer-readable storage media can be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more sets of available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. Semiconductor media can be solid-state drives (SSDs).

[0126] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0127] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0128] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0129] In the several embodiments provided in this application, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0130] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0131] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0132] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0133] In this specification, references to "an embodiment" or "a specific implementation" mean that a particular feature, structure, or characteristic described in connection with that embodiment / specific implementation is included in at least one embodiment / specific implementation of the invention. Therefore, the phrase "in one embodiment / specific implementation" appearing in various places in this specification does not necessarily refer to the same embodiment / setting, but rather to potentially different embodiments. Furthermore, specific features, structures, or characteristics may be combined in one or more embodiments / settings in any suitable manner, as will be apparent to those skilled in the art from this disclosure.

[0134] Similarly, it should be understood that in the above description of exemplary embodiments / specific implementations of the invention, various features of the invention are sometimes combined in a single embodiment / specific implementation or its figures and description, with the aim of simplifying the disclosure and aiding in the understanding of one or more of the various aspects of the invention. However, the method of description in this patent should not be construed as reflecting an intention that the claimed features of the invention are more than those expressly stated in each claim, except where explicitly stated otherwise or in obvious technical contradiction or exclusion. Rather, the inventive aspect reflected in the claims lies in not all the features of a single foregoing disclosed embodiment / specific implementation. Therefore, the claims following the detailed description are expressly incorporated herein by reference, each claim existing independently as a separate embodiment / specific implementation of the invention.

[0135] Furthermore, while some embodiments / specific implementations described herein include, but are not limited to, other features included in other embodiments / specific implementations, combinations of features from different embodiments / specific implementations are intended to be within the scope of the invention and form different embodiments / specific implementations, as will be understood by those skilled in the art. For example, in the following claims, embodiments / specific implementations of any claim can be used in any combination.

[0136] The terms and expressions used in this specification are for illustrative purposes and not for limitation. The use of these terms and expressions is not intended to exclude any equivalents of the features or portions thereof shown and described, but rather to allow for the understanding that various modifications may be made within the scope of the invention.

[0137] Therefore, it should be understood that although the invention has been specifically disclosed through preferred embodiments, exemplary embodiments and optional features, those skilled in the art may take variations or modifications of the concepts disclosed herein, and such variations and modifications are therefore considered to be within the scope of the invention as defined by the appended claims.

[0138] The specific embodiments given in this specification are examples of useful implementations of the present invention. It will be apparent to those skilled in the art that the present invention can be implemented using many variations of the devices, device components, and method steps disclosed in this specification.

[0139] The foregoing description of specific embodiments fully discloses the general features of the present invention, enabling others to easily modify and / or adapt such specific embodiments for various applications by applying knowledge within the scope of the art, without conducting excessive experimentation and without departing from the general concept of the present invention.

[0140] Therefore, based on the teachings and guidance provided herein, it is intended that such modifications and alterations be included within the meaning and scope of equivalents of the disclosed embodiments. It should be understood that the wording or terminology used herein is for descriptive purposes and is not intended to be limiting; thus, the wording or terminology in this specification will be interpreted by those skilled in the art based on the foregoing teachings and guidance.

[0141] Furthermore, the scope of the invention should not be limited to any of the exemplary embodiments described above, but only to the appended claims and their equivalents.

Claims

1. A method for cleaning outlier values ​​in tobacco indicators, characterized in that, The method for cleaning up outlier values ​​in tobacco indicators includes: Step S1: Obtain historical tobacco dataset, which includes unimodal distribution indices for the training set, validation set, and test set. Step S2: Based on the mean and standard deviation of the training set, fit and construct a univariate Gaussian distribution model; Step S3: Based on the univariate Gaussian distribution model and the validation set, determine the target detection threshold, and test the target detection threshold according to the test set; Step S4: The target detection threshold is solved in reverse to obtain the target normal value range corresponding to each indicator type, and the target tobacco dataset to be tested is cleaned according to the target normal value range to obtain the target tobacco dataset.

2. The method for cleaning outlier values ​​in tobacco indicators according to claim 1, characterized in that, Step S1 includes: Step S11: Divide the original historical tobacco dataset into the original training set, the original validation set, and the original test set according to a preset ratio or quantity; Step S12: Remove the outlier data from the original training set and retain the index type data with a unimodal distribution to obtain the training set; Step S13: Retain the index type data of the unimodal distribution of the original validation set to obtain the validation set; Step S14: Retain the index type data of the unimodal distribution of the original test set to obtain the test set; Step S15: Determine whether the data of each indicator type in the historical tobacco dataset conforms to a Gaussian distribution: If not, transform the data of the unimodal distribution indicators according to the power transform algorithm to approximate the Gaussian distribution.

3. The method for cleaning outlier values ​​in tobacco indicators according to claim 2, characterized in that, The power transform algorithms include the Box-Cox transform algorithm and the Yeo-Johnson transform algorithm.

4. The method for cleaning outlier values ​​in tobacco indicators according to claim 1, characterized in that, The unimodal distribution indicators include appearance indicators, sensory indicators, aroma components, conventional chemical components, and physical indicators. The appearance indicators include a total appearance score, which includes oil content, maturity, softness, leaf structure, identity, color, and chroma. The sensory indicators include a total sensory score, which includes aroma quality, aroma quantity, off-odors, irritation, and aftertaste.

5. The method for cleaning up outlier values ​​in tobacco indicators according to claim 1, characterized in that, Step S2 includes: Step S21: Obtain the mean and standard deviation of each indicator type in the training set; Step S22: Fit the univariate Gaussian distribution model based on the mean and the standard deviation; The formula for the univariate Gaussian distribution model is: Where x is the sample index value, μ is the mean, and σ is the standard deviation.

6. The method for cleaning up outlier values ​​in tobacco indicators according to claim 1, characterized in that, Step S3 includes: Step S31: Input the validation set into the Gaussian distribution model, and perform multi-fold cross-validation with the training set to obtain the log-likelihood value corresponding to the validation set; Step S32: Within the range of the log-likelihood value, select the critical score by traversing according to the preset step size to generate a candidate threshold set; Step S33: Calculate the detection evaluation index corresponding to each candidate threshold in the candidate threshold set, and determine the target detection threshold based on the detection evaluation index; Step S34: Test the target detection threshold according to the test set, and determine whether the target detection threshold meets the preset detection evaluation index: if so, then solve the target normal value range corresponding to the index type.

7. The method for cleaning up outlier values ​​in tobacco indicators according to claim 6, characterized in that, Step S33 includes: Step S331: Compare the log-likelihood values ​​corresponding to each data point in the validation set with each candidate threshold in the candidate threshold set, and calculate the detection evaluation index corresponding to each candidate threshold. Step S332: Based on the detection evaluation index, screen each candidate threshold and determine the target detection threshold.

8. The method for cleaning up outlier values ​​in tobacco indicators according to claim 6, characterized in that, Step S34 includes: Step S341: Determine whether the data of the index type is obtained by the power transformation algorithm: if so, calculate the normal value range of the change domain based on the mean, standard deviation and target detection threshold after the power transformation algorithm. Step S342: Based on the inverse transformation of the power transform algorithm, restore the normal value range of the change domain to determine the target normal value range.

9. The method for cleaning up outlier values ​​in tobacco indicators according to claim 6, characterized in that, Step S34 includes: Step S343: Determine if the data of the index type conforms to a Gaussian distribution: If so, calculate the normal value range of the target based on the mean, the standard deviation and the target detection threshold.

10. A device for cleaning up abnormal values ​​of tobacco indicators, characterized in that, The tobacco index outlier cleaning device includes: an acquisition module, a construction module, a determination module, and a cleaning module. The acquisition module is used to acquire historical tobacco datasets, which include unimodal distribution indicators of the training set, unimodal distribution indicators of the validation set, and unimodal distribution indicators of the test set. The building module is used to fit and construct a univariate Gaussian distribution model based on the mean and standard deviation of the training set; The determining module is used to determine the target detection threshold based on the univariate Gaussian distribution model and the validation set, and to test the target detection threshold according to the test set; The cleaning module is used to reverse-solve the target detection threshold to obtain the target normal value range corresponding to each indicator type, and to clean the tobacco dataset to be tested according to the target normal value range to obtain the target tobacco dataset.

Images