Tobacco quality judgment method and system based on numerical analysis
Through numerical analysis and machine learning, the tobacco sample data is automatically processed and the tobacco quality evaluation model is established, which solves the problems of inefficiency and subjectivity of traditional methods, and realizes rapid, accurate judgment and in-depth analysis of tobacco quality, improving quality management Foresight and initiative.
Patent Information
- Application Number
- CN202410985140.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-12-27
AI Technical Summary
Traditional tobacco quality determination methods rely on manual testing and empirical judgment, are inefficient and easily affected by subjective factors, are difficult to process large amounts of data and find potential quality problems, and cannot make quality warnings and prevention in a timely manner.
Using a numerical analysis method, we use correlation analysis method and machine learning to establish a tobacco quality evaluation model, automate the processing of data, eliminate subjective factors, and improve the accuracy and efficiency of judgment.
It realizes rapid and accurate judgment of tobacco quality, automatically process big data, eliminates subjectivity, improves the depth of data analysis and early warning and prevention capabilities, and improves the objectivity and efficiency of tobacco quality management.
Smart Images
Figure CN119204386A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of tobacco, in particular to the technical field of tobacco quality determination, and specifically to a tobacco quality determination method and system based on numerical analysis. Background Art
[0002] In the tobacco industry, from the planting and picking of raw materials to the production and sales of cigarettes, each link involves a large amount of data and information. These data not only include the physical and chemical properties of tobacco, such as length, weight, nicotine content, etc., but also include various parameters in the production process, such as temperature, humidity, pressure, etc.
[0003] However, traditional tobacco quality management methods mainly rely on manual inspection and empirical judgment, which is not only inefficient but also easily affected by subjective factors, making it difficult to ensure the accuracy and stability of quality management.
[0004] The main reason for the above problems is that the amount of data generated in the tobacco production process is huge. Therefore, it is difficult to effectively process and analyze the huge amount of data by relying solely on manual inspection and experience judgment. At the same time, traditional methods are difficult to detect potential tobacco quality problems and cannot make timely quality warnings and preventions. Summary of the invention
[0005] In order to solve the above-mentioned defects of the traditional tobacco quality determination method which mainly relies on manual detection and experience judgment, the present invention provides a new tobacco quality determination method based on numerical analysis, namely, a system.
[0006] The present invention is achieved by adopting the following technical solutions:
[0007] A tobacco quality determination method based on numerical analysis mainly includes the following steps:
[0008] 1) Collecting original sample data for evaluating tobacco quality, wherein the original sample data includes initially collected underlying tobacco sample data and real-time dynamically updated dynamic tobacco sample data;
[0009] 2) Preprocessing the collected raw sample data for evaluating tobacco quality;
[0010] 3) Analyze and process the preprocessed data: Use correlation analysis to obtain the relationship between tobacco sample data, find out the key factors affecting tobacco quality, and construct a tobacco sample data set with a small dimension but more accurate description; interpolate the constructed tobacco sample data set with a small dimension but more accurate description to further obtain compressed data;
[0011] Further, in step 3), the specific steps of the correlation analysis method include the following: 1. Using the Pearson correlation coefficient r as the analysis coefficient of the correlation analysis method, the calculation formula of the Pearson correlation coefficient r is:
[0012]
[0013] Where X and Y represent the observation values of two different sample data sets, μ X and μ Y Represent the means of two different sample data sets, n is the number of observations, σ X and σ Y Respectively represent the standard deviation of two different groups of sample data;
[0014] Extract any tobacco sample data from the original sample data after data processing, and calculate the correlation coefficient r between the tobacco sample data and other groups of tobacco sample data according to formula (1), and obtain the correlation coefficient matrix H = {r1, r2, r3, ..., r j};
[0015] II. Sort the correlation coefficient matrix H obtained in step I in descending order;
[0016] III. Setting a threshold value (the setting of the threshold value belongs to the conventional means obtainable by those skilled in the art, such as being obtained through experience or comparison of experimental data), searching the tobacco sample data set for the descending correlation coefficient matrix H, if r>w, retaining the tobacco sample data and adding the tobacco sample data to the tobacco sample data set G, if r<w, adding the tobacco sample data to the newly created tobacco sample data set K; then calculating the Pearson correlation coefficient f between the newly created tobacco sample data set K and other groups of tobacco sample data, and setting the threshold value w in the same way, if f is less than w, the sample data is removed, if f≥w, retaining the tobacco sample data and adding it to the tobacco sample data set G, thereby optimizing the tobacco sample data in the original data set, thereby constructing a new tobacco sample data set G;
[0017] IV. Dividing the tobacco sample data in the constructed tobacco sample data set G into control data and data to be processed, and compressing the control data to obtain a control dictionary;
[0018] The specific steps are as follows: first, a part of the data is designated as the control data, for example, 20% of the tobacco sample data in the tobacco sample data set G is divided into the control data, and the rest of the tobacco sample data is divided into the to-be-processed data. The purpose of compressing these data is to build a dictionary, which contains all the unique data in the control data and their corresponding compressed representations. In this way, we can use this dictionary to quickly search and compare the subsequent to-be-processed data; for example, use data structures such as hash tables, prefix trees (Trie trees) or dictionary trees (Trie Tree) to store these unique data to achieve efficient search;
[0019] V. Determine the longest data length of the reference data in the reference dictionary obtained in step IV. Because before building the adaptive sliding window, we need to know the length of the longest data in the reference dictionary. This information is crucial to determine the length of the sliding window, because the sliding window needs to be large enough to accommodate the longest reference data, but not too large to avoid adding unnecessary computational burden. By traversing all the data in the reference dictionary, comparing the length of each data, and recording the longest data length;
[0020] VI. Determine the sliding window length according to the longest data length, and construct an adaptive sliding window in the divided data to be processed based on the sliding window length. The adaptive sliding window length is greater than or equal to the longest data length to ensure that any control data can be completely contained in the sliding window. In the data to be processed, a series of sliding windows are constructed according to the sliding window length determined previously. These sliding windows will be used to traverse the data to be processed, and perform data similarity calculation in each window, for example, sliding a window of fixed length on the data to be processed, moving the position of one character or byte each time;
[0021] VII. Traversing the data to be processed according to the adaptive sliding window, and determining the similarity between the data to be processed in the adaptive sliding window and all the reference data. Specifically, when traversing the data to be processed using the adaptive sliding window, we need to determine the similarity between the data to be processed in each window and all the data in the reference dictionary;
[0022] In this specific implementation, the similarity between the data to be processed and all the control data in the adaptive sliding window is calculated using the Jaccard similarity. The Jaccard similarity calculation formula is as follows:
[0023]
[0024] Where U represents the set of data to be processed, V represents the set of control data, U∩V represents the intersection of two sets, and U∪V represents the union of two sets. The ratio between the intersection of two sets and the union of two sets ranges from 0 to 1, where 1 means that the two sets are exactly the same, and 0 means that the two sets are completely different.
[0025] VIII. Determine the interpolation probability of the data to be processed according to the determined similar data; determine the data most similar to the data to be processed from the control data based on the similarity calculated previously, set a threshold d for each similarity, and when the similarity SIM(U,V)>d, the data is similar data;
[0026] IX. Determine the interpolation probability of the data to be processed based on the similar data, including calculating the absolute value of the difference between the length of the similar data and the length of the data to be processed as the length difference; calculating the ratio of the length difference to the length of the processed data as the interpolation probability influence coefficient; calculating the normalized value of the ratio of the similarity degree and the interpolation probability influence coefficient as the interpolation probability; the corresponding calculation formula is:
[0027]
[0028] In the formula, C represents the interpolation probability, Z() represents the normalization function, and O max Indicates the similarity between similar data and the data to be processed, q d Indicates the length of the data to be processed, q q Indicates the length of similar data, |q d -q q | indicates the length difference, |q d -q q | / q d represents the interpolation probability influence coefficient;
[0029] By calculating the absolute value of the difference between the length of similar data and the length of the data to be processed as the length difference, the greater the difference between the length of similar data and the length of the data to be processed, the greater the difference between the similar data and the data to be processed. At this time, even interpolation cannot effectively reduce the compression probability. Therefore, the smaller the corresponding interpolation probability, the ratio of the length difference to the length of the processed data is used as the interpolation probability influence coefficient to determine the interpolation probability.
[0030] The normalized value of the ratio of the similarity degree and the interpolation probability influence coefficient is calculated as the interpolation probability. Through the similarity degree and the interpolation probability influence coefficient, the difference between the characters of the similar data and the data to be processed can be effectively compared, and then according to the interpolation probability, it can be accurately determined whether the data to be processed needs to be interpolated.
[0031] X. According to the interpolation probability, the data to be processed is interpolated to obtain interpolated data, and the interpolated data is compressed according to the reference dictionary to obtain compressed data;
[0032] 4) Based on the results of numerical analysis, a tobacco quality assessment model is established through machine learning. The compressed data obtained in step 3) is used to train the tobacco quality assessment model until the maximum number of iterations is reached. When predicting, the tobacco sample data to be predicted is input into the tobacco quality assessment model to obtain the tobacco quality judgment result.
[0033] Furthermore, in step 2), data preprocessing includes data cleaning, standardized mapping, and standardized data generation.
[0034] Furthermore, the data cleaning in step 2) includes deleting duplicate data records, deleting records with missing values, identifying and correcting obvious erroneous data, such as data outside the numerical range, data with incorrect format, etc., to improve the quality of the original data based on data cleaning;
[0035] Standardized mapping is to first map the data that needs to be preprocessed to the rules of the standardized data set structure, including field mapping relationships, data type conversion, and missing value processing. According to business needs and data type differences, appropriate mapping rules are formulated, and then the data that needs to be preprocessed is mapped to the standard data set structure according to the defined mapping rules to ensure that the data is correctly filled into the corresponding fields according to the rules and maintain the consistency and accuracy of the data; Among them, the field mapping relationship is defined, such as unifying the "Origin" field into "Country" or "Region"; data type conversion, such as converting text format dates into date format; missing value processing, such as filling with average values or predicting filling based on other fields;
[0036] Standardized data generation, through mapping operations, generates a standardized data set. The table contains data from different data sources, which has been uniformly mapped to a common standard structure.
[0037] A tobacco quality determination system based on numerical analysis includes a data acquisition module, which is used to automatically or manually collect original sample data for evaluating tobacco quality from various data sources, wherein the original sample data includes initially adopted underlying tobacco sample data and real-time dynamically updated dynamic tobacco sample data;
[0038] A data preprocessing module is used to preprocess the collected sample data for tobacco quality assessment; clean, standardize and format the data, and convert the fields according to the preset mapping relationship;
[0039] Data analysis module, which uses correlation analysis to obtain the relationship between tobacco sample data, finds out the key factors affecting tobacco quality, and constructs a tobacco sample data set with a small dimension but more accurate description; and further obtains compressed data after interpolation processing on the constructed tobacco sample data set with a small dimension but more accurate description;
[0040] The quality judgment module establishes a tobacco quality assessment model through machine learning based on the output results of the data analysis module, constructs the judgment logic or rules for tobacco quality, and uses the compressed data obtained by the quality analysis module to train the tobacco quality assessment model until the maximum number of iterations is reached. When predicting, the tobacco sample data to be predicted is input into the tobacco quality assessment model to obtain the tobacco quality judgment result. The quality of the tobacco sample to be tested is judged and the judgment result is output.
[0041] Furthermore, in the data analysis module, the specific steps of the correlation analysis method include the following: I. Using the Pearson correlation coefficient r as the analysis coefficient of the correlation analysis method, the calculation formula of the Pearson correlation coefficient r is:
[0042]
[0043] Where X and Y represent the observation values of two different sample data sets, μ X and μ Y Represent the means of two different sample data sets, n is the number of observations, σ X and σ Y Respectively represent the standard deviation of two different groups of sample data;
[0044] Extract any tobacco sample data from the original sample data after data processing, and calculate the correlation coefficient r between the tobacco sample data and other groups of tobacco sample data according to formula (1), and obtain the correlation coefficient matrix H = {r1, r2, r3, ..., r j};
[0045] II. Sort the correlation coefficient matrix H obtained in step I in descending order;
[0046] III. Setting a threshold value (the setting of the threshold value belongs to the conventional means obtainable by those skilled in the art, such as being obtained through experience or comparison of experimental data), searching the tobacco sample data set for the descending correlation coefficient matrix H, if r>w, retaining the tobacco sample data and adding the tobacco sample data to the tobacco sample data set G, if r<w, adding the tobacco sample data to the newly created tobacco sample data set K; then calculating the Pearson correlation coefficient f between the newly created tobacco sample data set K and other groups of tobacco sample data, and setting the threshold value w in the same way, if f is less than w, the sample data is removed, if f≥w, retaining the tobacco sample data and adding it to the tobacco sample data set G, thereby optimizing the tobacco sample data in the original data set, thereby constructing a new tobacco sample data set G;
[0047] Furthermore, the specific steps of interpolating the quality judgment module include the following: a. dividing the tobacco sample data in the tobacco sample data set G constructed in the correlation analysis method into control data and data to be processed, and compressing the control data to obtain a control dictionary;
[0048] The specific steps are as follows: first, a part of the data is designated as the control data, for example, 20% of the tobacco sample data in the tobacco sample data set G is divided into the control data, and the rest of the tobacco sample data is divided into the to-be-processed data. The purpose of compressing these data is to build a dictionary, which contains all the unique data in the control data and their corresponding compressed representations. In this way, we can use this dictionary to quickly search and compare the subsequent to-be-processed data; for example, use data structures such as hash tables, prefix trees (Trie trees) or dictionary trees (Trie Tree) to store these unique data to achieve efficient search;
[0049] b. Determine the longest data length of the reference data in the reference dictionary obtained in step a. Because before building the adaptive sliding window, we need to know the length of the longest data in the reference dictionary. This information is crucial to determine the length of the sliding window, because the sliding window needs to be large enough to accommodate the longest reference data, but not too large to avoid adding unnecessary computational burden. By traversing all the data in the reference dictionary, comparing the length of each data, and recording the longest data length;
[0050] c. Determine the sliding window length according to the longest data length, and construct an adaptive sliding window in the divided data to be processed based on the sliding window length. The adaptive sliding window length is greater than or equal to the longest data length to ensure that any control data can be completely contained in the sliding window. In the data to be processed, a series of sliding windows are constructed according to the sliding window length determined previously. These sliding windows will be used to traverse the data to be processed, and perform data similarity calculation in each window, for example, sliding a fixed-length window on the data to be processed, moving the position of one character or byte each time;
[0051] d. traversing the data to be processed according to the adaptive sliding window, and determining the similarity between the data to be processed in the adaptive sliding window and all the reference data. Specifically, when traversing the data to be processed using the adaptive sliding window, we need to determine the similarity between the data to be processed in each window and all the data in the reference dictionary;
[0052] In this specific implementation, the similarity between the data to be processed and all the control data in the adaptive sliding window is calculated using the Jaccard similarity. The Jaccard similarity calculation formula is as follows:
[0053]
[0054] Where U represents the set of data to be processed, V represents the set of control data, U∩V represents the intersection of two sets, and U∪V represents the union of two sets. The ratio between the intersection of two sets and the union of two sets ranges from 0 to 1, where 1 means that the two sets are exactly the same, and 0 means that the two sets are completely different.
[0055] e. Determine the interpolation probability of the data to be processed according to the determined similar data; determine the data most similar to the data to be processed from the control data based on the similarity calculated previously, set a threshold d for each similarity, and when the similarity SIM(U,V)>d, the data is similar data;
[0056] f. Determine the interpolation probability of the data to be processed based on the similar data, including calculating the absolute value of the difference between the length of the similar data and the length of the data to be processed as the length difference; calculating the ratio of the length difference to the length of the processed data as the interpolation probability influence coefficient; calculating the normalized value of the ratio of the similarity degree and the interpolation probability influence coefficient as the interpolation probability; the corresponding calculation formula is:
[0057]
[0058] In the formula, C represents the interpolation probability, Z() represents the normalization function, and O max Indicates the similarity between similar data and the data to be processed, q d Indicates the length of the data to be processed, qq Indicates the length of similar data, |q d -q q | indicates the length difference, |q d -q q | / q d represents the interpolation probability influence coefficient;
[0059] By calculating the absolute value of the difference between the length of similar data and the length of the data to be processed as the length difference, the greater the difference between the length of similar data and the length of the data to be processed, the greater the difference between the similar data and the data to be processed. At this time, even interpolation cannot effectively reduce the compression probability. Therefore, the smaller the corresponding interpolation probability, the ratio of the length difference to the length of the processed data is used as the interpolation probability influence coefficient to determine the interpolation probability.
[0060] The normalized value of the ratio of the similarity degree and the interpolation probability influence coefficient is calculated as the interpolation probability. Through the similarity degree and the interpolation probability influence coefficient, the difference between the characters of the similar data and the data to be processed can be effectively compared, and then according to the interpolation probability, it can be accurately determined whether the data to be processed needs to be interpolated.
[0061] g. According to the interpolation probability, the data to be processed is interpolated to obtain interpolated data, and the interpolated data is compressed according to the reference dictionary to obtain compressed data.
[0062] A terminal device comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-mentioned cigarette quality determination method.
[0063] A storage medium stores a computer program / instruction, which implements the steps of the above-mentioned tobacco quality determination method when executed by a processor.
[0064] A computer program product comprises a computer program / instruction, which implements the steps of the above-mentioned tobacco quality determination method when executed by a processor.
[0065] The beneficial effects produced by the present invention are as follows:
[0066] Automation and efficiency: This method can automatically process large amounts of data through numerical analysis and algorithm processing, significantly improving data processing efficiency. This not only reduces human intervention and errors, but also greatly shortens data processing time, making tobacco quality determination faster and more accurate.
[0067] Objectivity and accuracy: Based on objective data and algorithms, this method can eliminate the influence of subjective factors and ensure the objectivity and accuracy of the results. Through numerical correlation evaluation, the key factors affecting tobacco quality can be determined more accurately, avoiding the subjectivity and uncertainty of human judgment.
[0068] In-depth analysis and early warning prevention: This method can discover potential quality problems through in-depth mining and analysis of data. By inputting tobacco sample data into the tobacco quality judgment model, it can timely discover abnormal values or trends, provide early warning and prevent the occurrence of quality problems, and improve the foresight and initiative of tobacco quality management.
[0069] Dimensionality reduction and precise description: In the numerical analysis stage, by using the Pearson correlation coefficient to measure the correlation between different sample data, the dimension of the calculated data representation can be reduced, while more accurately representing the spatial distribution information of the data. This not only reduces the amount of calculation, but also improves the efficiency and accuracy of data analysis.
[0070] Flexibility and scalability: This method has certain flexibility and scalability. It can be customized and adjusted according to different tobacco types and needs to adapt to different application scenarios. At the same time, with the advancement of technology and the emergence of new algorithms, this method can also be further optimized and upgraded to meet higher quality requirements and analysis needs. BRIEF DESCRIPTION OF THE DRAWINGS
[0071] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0072] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0073] Figure 1 A flowchart of the tobacco quality determination method of the present invention;
[0074] Figure 2 A schematic diagram of the composition of the tobacco quality determination system of the present invention;
[0075] Figure 3 It is a schematic diagram of the composition of the terminal device described in the present invention. DETAILED DESCRIPTION
[0076] In order to more clearly understand the above-mentioned objectives, features and advantages of the present invention, the scheme of the present invention will be further described below. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.
[0077] In the description, it should be noted that the terms "first" and "second" are only used for descriptive purposes and should not be understood as indicating or implying relative importance. It should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be internal communication between two elements. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.
[0078] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present invention, rather than all of the embodiments.
[0079] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0080] like Figure 1 As shown, a tobacco quality determination method based on numerical analysis mainly includes the following steps:
[0081] 1) Collecting original sample data for evaluating tobacco quality, wherein the original sample data includes initially collected underlying tobacco sample data and real-time dynamically updated dynamic tobacco sample data;
[0082] 2) Preprocessing the collected raw sample data for tobacco quality assessment; data preprocessing includes data cleaning, standardized mapping, and standardized data generation; data cleaning includes deleting duplicate data records, deleting records with missing values, identifying and correcting obvious erroneous data, such as data outside the numerical range, data with incorrect format, etc., to improve the quality of raw data based on data cleaning;
[0083] Standardized mapping is to first map the data that needs to be preprocessed to the rules of the standardized data set structure, including field mapping relationships, data type conversion, and missing value processing. According to business needs and data type differences, appropriate mapping rules are formulated, and then the data that needs to be preprocessed is mapped to the standard data set structure according to the defined mapping rules to ensure that the data is correctly filled into the corresponding fields according to the rules and maintain the consistency and accuracy of the data; Among them, the field mapping relationship is defined, such as unifying the "Origin" field into "Country" or "Region"; data type conversion, such as converting text format dates into date format; missing value processing, such as filling with average values or predicting filling based on other fields;
[0084] Standardized data generation, through mapping operations, generates a standardized data set. The table contains data from different data sources, which have been uniformly mapped to a common standard structure;
[0085] 3) Analyze and process the preprocessed data, which mainly includes the following specific steps: I. Use the Pearson correlation coefficient r as the analysis coefficient of the correlation analysis method. The calculation formula of the Pearson correlation coefficient r is:
[0086]
[0087] Where X and Y represent the observation values of two different sample data sets, μ X and μ Y Represent the means of two different sample data sets, n is the number of observations, σ X and σ Y Respectively represent the standard deviation of two different groups of sample data;
[0088] Extract any tobacco sample data from the original sample data after data processing, and calculate the correlation coefficient r between the tobacco sample data and other groups of tobacco sample data according to formula (1), and obtain the correlation coefficient matrix H = {r1, r2, r3, ..., r j};
[0089] II. Sort the correlation coefficient matrix H obtained in step I in descending order;
[0090] III. Set the threshold w to 0.57, search the tobacco sample data set for the descending correlation coefficient matrix H, if r>w, retain the tobacco sample data and add the tobacco sample data to the tobacco sample data set G, if r<w, add the tobacco sample data to the newly created tobacco sample data set K; then calculate the Pearson correlation coefficient f between the newly created tobacco sample data set K and other groups of tobacco sample data, and similarly set the threshold w to 0.57. If f is less than w, the sample data is removed, if f≥w, keep the tobacco sample data and add it to the tobacco sample data set G, so as to optimize the tobacco sample data in the original data set, thereby constructing a new tobacco sample data set G;
[0091] IV. Dividing the tobacco sample data in the constructed tobacco sample data set G into control data and data to be processed, and compressing the control data to obtain a control dictionary;
[0092] The specific steps are as follows: first, a part of the data is designated as the control data, for example, 20% of the tobacco sample data in the tobacco sample data set G is divided into the control data, and the rest of the tobacco sample data is divided into the to-be-processed data. The purpose of compressing these data is to build a dictionary, which contains all the unique data in the control data and their corresponding compressed representations. In this way, we can use this dictionary to quickly search and compare the subsequent to-be-processed data; for example, use data structures such as hash tables, prefix trees (Trie trees) or dictionary trees (Trie Tree) to store these unique data to achieve efficient search;
[0093] V. Determine the longest data length of the reference data in the reference dictionary obtained in step IV. Because before building the adaptive sliding window, we need to know the length of the longest data in the reference dictionary. This information is crucial to determine the length of the sliding window, because the sliding window needs to be large enough to accommodate the longest reference data, but not too large to avoid adding unnecessary computational burden. By traversing all the data in the reference dictionary, comparing the length of each data, and recording the longest data length;
[0094] VI. Determine the sliding window length according to the longest data length, and construct an adaptive sliding window in the divided data to be processed based on the sliding window length. The adaptive sliding window length is greater than or equal to the longest data length to ensure that any control data can be completely contained in the sliding window. In the data to be processed, a series of sliding windows are constructed according to the sliding window length determined previously. These sliding windows will be used to traverse the data to be processed, and perform data similarity calculation in each window, for example, sliding a window of fixed length on the data to be processed, moving the position of one character or byte each time;
[0095] VII. Traversing the data to be processed according to the adaptive sliding window, and determining the similarity between the data to be processed in the adaptive sliding window and all the reference data. Specifically, when traversing the data to be processed using the adaptive sliding window, we need to determine the similarity between the data to be processed in each window and all the data in the reference dictionary;
[0096] In this specific implementation, the similarity between the data to be processed and all the control data in the adaptive sliding window is calculated using the Jaccard similarity. The Jaccard similarity calculation formula is as follows:
[0097]
[0098] Where U represents the set of data to be processed, V represents the set of control data, U∩V represents the intersection of two sets, and U∪V represents the union of two sets. The ratio between the intersection of two sets and the union of two sets ranges from 0 to 1, where 1 means that the two sets are exactly the same, and 0 means that the two sets are completely different.
[0099] VIII. Determine the interpolation probability of the data to be processed according to the determined similar data; determine the data most similar to the data to be processed from the control data based on the similarity calculated previously, set a threshold d for each similarity, and when the similarity SIM(U,V)>d, the data is similar data;
[0100] IX. Determine the interpolation probability of the data to be processed based on the similar data, including calculating the absolute value of the difference between the length of the similar data and the length of the data to be processed as the length difference; calculating the ratio of the length difference to the length of the processed data as the interpolation probability influence coefficient; calculating the normalized value of the ratio of the similarity degree and the interpolation probability influence coefficient as the interpolation probability; the corresponding calculation formula is:
[0101]
[0102] In the formula, C represents the interpolation probability, Z() represents the normalization function, and O max Indicates the similarity between similar data and the data to be processed, q d Indicates the length of the data to be processed, q q Indicates the length of similar data, |q d -q q | indicates the length difference, |q d -q q | / q d represents the interpolation probability influence coefficient;
[0103] By calculating the absolute value of the difference between the length of similar data and the length of the data to be processed as the length difference, the greater the difference between the length of similar data and the length of the data to be processed, the greater the difference between the similar data and the data to be processed. At this time, even interpolation cannot effectively reduce the compression probability. Therefore, the smaller the corresponding interpolation probability, the ratio of the length difference to the length of the processed data is used as the interpolation probability influence coefficient to determine the interpolation probability.
[0104] The normalized value of the ratio of the similarity degree and the interpolation probability influence coefficient is calculated as the interpolation probability. Through the similarity degree and the interpolation probability influence coefficient, the difference between the characters of the similar data and the data to be processed can be effectively compared, and then according to the interpolation probability, it can be accurately determined whether the data to be processed needs to be interpolated.
[0105] X. According to the interpolation probability, the data to be processed is interpolated to obtain interpolated data, and the interpolated data is compressed according to the reference dictionary to obtain compressed data;
[0106] 4) Based on the results of numerical analysis, a tobacco quality assessment model is established through machine learning. The compressed data obtained in step 3) is used to train the tobacco quality assessment model until the maximum number of iterations is reached. When predicting, the tobacco sample data to be predicted is input into the tobacco quality assessment model to obtain the tobacco quality judgment result.
[0107] like Figure 2 As shown, a tobacco quality determination system based on numerical analysis includes a data acquisition module, which is used to automatically or manually collect original sample data for evaluating tobacco quality from various data sources, wherein the original sample data includes initially adopted underlying tobacco sample data and real-time dynamically updated dynamic tobacco sample data;
[0108] A data preprocessing module is used to preprocess the collected sample data for tobacco quality assessment; clean, standardize and format the data, and convert the fields according to the preset mapping relationship;
[0109] Data analysis module, which uses correlation analysis to obtain the relationship between tobacco sample data, finds out the key factors affecting tobacco quality, and constructs a tobacco sample data set with a small dimension but more accurate description; and further obtains compressed data after interpolation processing on the constructed tobacco sample data set with a small dimension but more accurate description;
[0110] The specific steps of the correlation analysis method include the following: 1. Using the Pearson correlation coefficient r as the analysis coefficient of the correlation analysis method, the calculation formula of the Pearson correlation coefficient r is:
[0111]
[0112] Where X and Y represent the observation values of two different sample data sets, μ X and μ Y Represent the means of two different sample data sets, n is the number of observations, σ X and σ Y Respectively represent the standard deviation of two different groups of sample data;
[0113] Extract any tobacco sample data from the original sample data after data processing, and calculate the correlation coefficient r between the tobacco sample data and other groups of tobacco sample data according to formula (1), and obtain the correlation coefficient matrix H = {r1, r2, r3, ..., r j};
[0114] II. Sort the correlation coefficient matrix H obtained in step I in descending order;
[0115] III. Setting a threshold value (the setting of the threshold value belongs to the conventional means obtainable by those skilled in the art, such as being obtained through experience or comparison of experimental data), searching the tobacco sample data set for the descending correlation coefficient matrix H, if r>w, retaining the tobacco sample data and adding the tobacco sample data to the tobacco sample data set G, if r<w, adding the tobacco sample data to the newly created tobacco sample data set K; then calculating the Pearson correlation coefficient f between the newly created tobacco sample data set K and other groups of tobacco sample data, and setting the threshold value w in the same way, if f is less than w, the sample data is removed, if f≥w, retaining the tobacco sample data and adding it to the tobacco sample data set G, thereby optimizing the tobacco sample data in the original data set, thereby constructing a new tobacco sample data set G;
[0116] The specific steps of interpolating the quality judgment module include the following: a. dividing the tobacco sample data in the tobacco sample data set G constructed in the correlation analysis method into control data and data to be processed, and compressing the control data to obtain a control dictionary;
[0117] The specific steps are as follows: first, a part of the data is designated as the control data, for example, 20% of the tobacco sample data in the tobacco sample data set G is divided into the control data, and the rest of the tobacco sample data is divided into the to-be-processed data. The purpose of compressing these data is to build a dictionary, which contains all the unique data in the control data and their corresponding compressed representations. In this way, we can use this dictionary to quickly search and compare the subsequent to-be-processed data; for example, use data structures such as hash tables, prefix trees (Trie trees) or dictionary trees (Trie Tree) to store these unique data to achieve efficient search;
[0118] b. Determine the longest data length of the reference data in the reference dictionary obtained in step a. Because before building the adaptive sliding window, we need to know the length of the longest data in the reference dictionary. This information is crucial to determine the length of the sliding window, because the sliding window needs to be large enough to accommodate the longest reference data, but not too large to avoid adding unnecessary computational burden. By traversing all the data in the reference dictionary, comparing the length of each data, and recording the longest data length;
[0119] c. Determine the sliding window length according to the longest data length, and construct an adaptive sliding window in the divided data to be processed based on the sliding window length. The adaptive sliding window length is greater than or equal to the longest data length to ensure that any control data can be completely contained in the sliding window. In the data to be processed, a series of sliding windows are constructed according to the sliding window length determined previously. These sliding windows will be used to traverse the data to be processed, and perform data similarity calculation in each window, for example, sliding a fixed-length window on the data to be processed, moving the position of one character or byte each time;
[0120] d. traversing the data to be processed according to the adaptive sliding window, and determining the similarity between the data to be processed in the adaptive sliding window and all the reference data. Specifically, when traversing the data to be processed using the adaptive sliding window, we need to determine the similarity between the data to be processed in each window and all the data in the reference dictionary;
[0121] In this specific implementation, the similarity between the data to be processed and all the control data in the adaptive sliding window is calculated using the Jaccard similarity. The Jaccard similarity calculation formula is as follows:
[0122]
[0123] Where U represents the set of data to be processed, V represents the set of control data, U∩V represents the intersection of two sets, and U∪V represents the union of two sets. The ratio between the intersection of two sets and the union of two sets ranges from 0 to 1, where 1 means that the two sets are exactly the same, and 0 means that the two sets are completely different.
[0124] e. Determine the interpolation probability of the data to be processed according to the determined similar data; determine the data most similar to the data to be processed from the control data based on the similarity calculated previously, set a threshold d for each similarity, and when the similarity SIM(U,V)>d, the data is similar data;
[0125] f. Determine the interpolation probability of the data to be processed based on the similar data, including calculating the absolute value of the difference between the length of the similar data and the length of the data to be processed as the length difference; calculating the ratio of the length difference to the length of the processed data as the interpolation probability influence coefficient; calculating the normalized value of the ratio of the similarity degree and the interpolation probability influence coefficient as the interpolation probability; the corresponding calculation formula is:
[0126]
[0127] In the formula, C represents the interpolation probability, Z() represents the normalization function, and O max Indicates the similarity between similar data and the data to be processed, q d Indicates the length of the data to be processed, q q Indicates the length of similar data, |q d -q q | indicates the length difference, |q d -q q | / q d represents the interpolation probability influence coefficient;
[0128] By calculating the absolute value of the difference between the length of similar data and the length of the data to be processed as the length difference, the greater the difference between the length of similar data and the length of the data to be processed, the greater the difference between the similar data and the data to be processed. At this time, even interpolation cannot effectively reduce the compression probability. Therefore, the smaller the corresponding interpolation probability, the ratio of the length difference to the length of the processed data is used as the interpolation probability influence coefficient to determine the interpolation probability.
[0129] The normalized value of the ratio of the similarity degree and the interpolation probability influence coefficient is calculated as the interpolation probability. Through the similarity degree and the interpolation probability influence coefficient, the difference between the characters of the similar data and the data to be processed can be effectively compared, and then according to the interpolation probability, it can be accurately determined whether the data to be processed needs to be interpolated.
[0130] g. According to the interpolation probability, the data to be processed is interpolated to obtain interpolated data, and the interpolated data is compressed according to the reference dictionary to obtain compressed data.
[0131] The quality judgment module establishes a tobacco quality assessment model through machine learning based on the output results of the data analysis module, constructs the judgment logic or rules for tobacco quality, and uses the compressed data obtained by the quality analysis module to train the tobacco quality assessment model until the maximum number of iterations is reached. When predicting, the tobacco sample data to be predicted is input into the tobacco quality assessment model to obtain the tobacco quality judgment result. The quality of the tobacco sample to be tested is judged and the judgment result is output.
[0132] like Figure 3As shown, a terminal device includes a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the above-mentioned cigarette quality determination method.
[0133] A storage medium stores a computer program / instruction, which implements the steps of the above-mentioned tobacco quality determination method when executed by a processor.
[0134] A computer program product comprises a computer program / instruction, which implements the steps of the above-mentioned tobacco quality determination method when executed by a processor.
[0135] The above is only a specific implementation of the present invention, which enables those skilled in the art to understand or implement the present invention. Although detailed descriptions are given with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments, and they should all be covered by the protection scope of the claims.
Claims
1. A tobacco quality determination method based on numerical analysis, characterized in that: The main steps are as follows: 1) Collecting original sample data for evaluating tobacco quality, wherein the original sample data includes initially collected underlying tobacco sample data and real-time dynamically updated dynamic tobacco sample data; 2) Preprocessing the collected raw sample data for evaluating tobacco quality; 3) Analyze and process the preprocessed data: Use correlation analysis to obtain the relationship between tobacco sample data, find out the key factors affecting tobacco quality, and construct a tobacco sample data set with a small dimension but more accurate description; interpolate the constructed tobacco sample data set with a small dimension but more accurate description to further obtain compressed data; 4) Based on the results of numerical analysis, a tobacco quality assessment model is established through machine learning, and the compressed data obtained in step 3) is used to train the tobacco quality assessment model until the maximum number of iterations is reached. When predicting, the tobacco sample data to be predicted is input into the tobacco quality assessment model to obtain the tobacco quality judgment result.
2. A tobacco quality determination method based on numerical analysis according to claim 1, characterized in that: In step 3), the specific steps of analyzing and processing the pre-processed data include the following:
1. Using the Pearson correlation coefficient r as the analysis coefficient of the correlation analysis method, the calculation formula of the Pearson correlation coefficient r is: Where X and Y represent the observation values of two different sample data sets, μ X and μ Y Represent the means of two different sample data sets, n is the number of observations, σ X and σ Y Respectively represent the standard deviation of two different groups of sample data; Extract any tobacco sample data from the original sample data after data processing, and calculate the correlation coefficient r between the tobacco sample data and other groups of tobacco sample data according to formula (1), and obtain the correlation coefficient matrix H = {r1, r2, r3, ..., r j }; II. Sort the correlation coefficient matrix H obtained in step I in descending order; III. Setting a threshold value (the setting of the threshold value belongs to the conventional means obtainable by those skilled in the art, such as being obtained through experience or comparison of experimental data), searching the tobacco sample data set for the descending correlation coefficient matrix H, if r>w, retaining the tobacco sample data and adding the tobacco sample data to the tobacco sample data set G, if r<w, adding the tobacco sample data to the newly created tobacco sample data set K; then calculating the Pearson correlation coefficient f between the newly created tobacco sample data set K and other groups of tobacco sample data, and setting the threshold value w in the same way, if f is less than w, the sample data is removed, if f≥w, retaining the tobacco sample data and adding it to the tobacco sample data set G, thereby optimizing the tobacco sample data in the original data set, thereby constructing a new tobacco sample data set G; IV. Dividing the tobacco sample data in the constructed tobacco sample data set G into control data and data to be processed, and compressing the control data to obtain a control dictionary; V. Determine the longest data length of the comparison data in the comparison dictionary obtained in step IV. VI. Determine the sliding window length according to the longest data length, and construct an adaptive sliding window in the divided data to be processed based on the sliding window length. The adaptive sliding window length is greater than or equal to the longest data length to ensure that any control data can be completely included in the sliding window. In the data to be processed, a series of sliding windows are constructed according to the sliding window length determined previously. These sliding windows will be used to traverse the data to be processed and perform data similarity calculation in each window. VII. traversing the data to be processed according to the adaptive sliding window, and determining the similarity between the data to be processed in the adaptive sliding window and all the control data; The similarity between the data to be processed and all the control data in the adaptive sliding window is calculated using the Jaccard similarity. The Jaccard similarity calculation formula is as follows: Where U represents the set of data to be processed, V represents the set of control data, U∩V represents the intersection of two sets, and U∪V represents the union of two sets. The ratio between the intersection of two sets and the union of two sets ranges from 0 to 1, where 1 means that the two sets are exactly the same, and 0 means that the two sets are completely different. VIII. Determine the interpolation probability of the data to be processed according to the determined similar data; determine the data most similar to the data to be processed from the control data based on the similarity calculated previously, set a threshold d for each similarity, and when the similarity SIM(U,V)>d, the data is similar data; IX. determining the interpolation probability of the data to be processed according to the similar data, including calculating the absolute value of the difference between the length of the similar data and the length of the data to be processed as the length difference; The ratio of the length difference to the length of the processed data is calculated as the interpolation probability influence coefficient; the normalized value of the ratio of the similarity degree and the interpolation probability influence coefficient is calculated as the interpolation probability; the corresponding calculation formula is: In the formula, C represents the interpolation probability, Z() represents the normalization function, and O max Indicates the similarity between similar data and the data to be processed, q d Indicates the length of the data to be processed, q q Indicates the length of similar data, |q d -q q | indicates the length difference, |q d -q q | / q d represents the interpolation probability influence coefficient; X. According to the interpolation probability, the data to be processed is interpolated to obtain interpolated data, and the interpolated data is compressed according to the reference dictionary to obtain compressed data.
3. A tobacco quality determination method based on numerical analysis according to claim 2, characterized in that: The threshold value w=0.57 described in step 3).
4. A tobacco quality determination method based on numerical analysis according to claim 3, characterized in that: In step 2), data preprocessing includes data cleaning, standardized mapping and standardized data generation; data cleaning includes deleting duplicate data records, deleting records with missing values, identifying and correcting obvious erroneous data, such as data outside the numerical range, data with incorrect format, etc., to improve the quality of original data based on data cleaning; standardized mapping is to first map the data that needs to be preprocessed to the rules of the standardized data set structure, including field mapping relationships, data type conversion, and missing value processing. According to business needs and data type differences, appropriate mapping rules are formulated, and then according to the defined mapping rules, the data that needs to be preprocessed is mapped to the standard data set structure to ensure that the data is correctly filled into the corresponding fields according to the rules and maintain the consistency and accuracy of the data; standardized data generation, through mapping operations, generates a standardized data set. The table contains data from different data sources and has been uniformly mapped to a common standard structure.
5. A tobacco quality determination system based on numerical analysis, characterized in that: It includes a data acquisition module, which is used to automatically or manually collect original sample data for evaluating tobacco quality from various data sources, wherein the original sample data includes initially adopted underlying tobacco sample data and dynamic tobacco sample data dynamically updated in real time; A data preprocessing module is used to preprocess the collected sample data for tobacco quality assessment; clean, standardize and format the data, and convert the fields according to the preset mapping relationship; Data analysis module, which uses correlation analysis to obtain the relationship between tobacco sample data, finds out the key factors affecting tobacco quality, and constructs a tobacco sample data set with a small dimension but more accurate description; and further obtains compressed data after interpolation processing on the constructed tobacco sample data set with a small dimension but more accurate description; The quality judgment module establishes a tobacco quality assessment model through machine learning based on the output results of the data analysis module, constructs the judgment logic or rules for tobacco quality, and uses the compressed data obtained by the quality analysis module to train the tobacco quality assessment model until the maximum number of iterations is reached. When predicting, the tobacco sample data to be predicted is input into the tobacco quality assessment model to obtain the tobacco quality judgment result. The quality of the tobacco sample to be tested is judged and the judgment result is output.
6. A tobacco quality determination system based on numerical analysis according to claim 5, characterized in that: In the data analysis module, the specific steps of the correlation analysis method include the following: I. Using the Pearson correlation coefficient r as the analysis coefficient of the correlation analysis method, the calculation formula of the Pearson correlation coefficient r is: Where X and Y represent the observation values of two different sample data sets, μ X and μ Y Represent the means of two different sample data sets, n is the number of observations, σ X and σ Y Respectively represent the standard deviation of two different groups of sample data; Extract any tobacco sample data from the original sample data after data processing, and calculate the correlation coefficient r between the tobacco sample data and other groups of tobacco sample data according to formula (1), and obtain the correlation coefficient matrix H = {r1, r2, r3, ..., r j }; II. Sort the correlation coefficient matrix H obtained in step I in descending order; III. Set a threshold value (the setting of the threshold value is obtainable by a technician in this field through conventional means, such as being obtained through experience or through comparison of experimental data), search the tobacco sample data set for the descending correlation coefficient matrix H, if r>w, retain the tobacco sample data and add the tobacco sample data to the tobacco sample data set G, if r<w, add the tobacco sample data to the newly created tobacco sample data set K; then calculate the Pearson correlation coefficient f between the newly created tobacco sample data set K and other groups of tobacco sample data, and set the threshold value w in the same way. If f is less than w, the sample data is removed, and if f≥w, the tobacco sample data is retained and added to the tobacco sample data set G, so as to optimize the tobacco sample data in the original data set, thereby constructing a new tobacco sample data set G.
7. A tobacco quality determination system based on numerical analysis according to claim 6, characterized in that: The specific steps of interpolating the quality judgment module include the following: a. Dividing the tobacco sample data in the tobacco sample data set G constructed in the correlation analysis method into control data and data to be processed, and obtaining a control dictionary after compressing the control data; b. Determine the longest data length of the comparison data in the comparison dictionary obtained in step a; c. Determine the sliding window length according to the longest data length, and construct an adaptive sliding window in the divided data to be processed based on the sliding window length. The adaptive sliding window length is greater than or equal to the longest data length to ensure that any control data can be completely included in the sliding window. In the data to be processed, a series of sliding windows are constructed according to the sliding window length determined previously. These sliding windows will be used to traverse the data to be processed and perform data similarity calculation in each window; d. traversing the data to be processed according to the adaptive sliding window, and determining the similarity between the data to be processed in the adaptive sliding window and all the control data; The similarity between the data to be processed and all the control data in the adaptive sliding window is calculated using the Jaccard similarity. The Jaccard similarity calculation formula is as follows: Where U represents the set of data to be processed, V represents the set of control data, U∩V represents the intersection of two sets, and U∪V represents the union of two sets. The ratio between the intersection of two sets and the union of two sets ranges from 0 to 1, where 1 means that the two sets are exactly the same, and 0 means that the two sets are completely different. e. Determine the interpolation probability of the data to be processed according to the determined similar data; determine the data most similar to the data to be processed from the control data based on the similarity calculated previously, set a threshold d for each similarity, and when the similarity SIM(U,V)>d, the data is similar data; f. determining the interpolation probability of the data to be processed according to the similar data, including calculating the absolute value of the difference between the length of the similar data and the length of the data to be processed as the length difference; The ratio of the length difference to the length of the processed data is calculated as the interpolation probability influence coefficient; the normalized value of the ratio of the similarity degree and the interpolation probability influence coefficient is calculated as the interpolation probability; the corresponding calculation formula is: In the formula, C represents the interpolation probability, Z() represents the normalization function, and O max Indicates the similarity between similar data and the data to be processed, qd indicates the length of the data to be processed, q q Indicates the length of similar data, |q d -q q | indicates the length difference, |q d -q q | / q d represents the interpolation probability influence coefficient; g. According to the interpolation probability, the data to be processed is interpolated to obtain interpolated data, and the interpolated data is compressed according to the reference dictionary to obtain compressed data.
8. A tobacco quality determination system based on numerical analysis, comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the cigarette quality determination method described in any one of claims 1 to 3.
9. A storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the tobacco quality determination method according to any one of claims 1 to 3 are implemented.
10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the tobacco quality determination method according to any one of claims 1 to 3 are implemented.