Method for analyzing availability of near infrared spectrum data of tobacco leaves

Abstract

The invention discloses a tobacco leaf near infrared spectrum data availability analysis method, and belongs to the field of tobacco leaf data availability analysis. According to the method, to-be-evaluated data are preprocessed, characteristic wave band selection is performed on the preprocessed data, abnormal wave band data are separated, spectrum matching based on multi-criterion fusion measurement is performed on each sample of the data and a reference spectrum obtained through spectrum library spectrum interpolation, and the multi-criterion fusion measurement of each sample is obtained; and obtaining an availability analysis result of the data in combination with historical batch data. According to the method, the spectrum data of the abnormal tobacco leaves can be effectively removed, the available spectrum data can be screened, the accuracy and applicability in subsequent general model training or application can be effectively improved, and the experimental analysis cost is reduced.

Description

Technical Field

[0001] The invention relates to the field of tobacco leaf data availability analysis, in particular to a method for analyzing availability of near infrared spectroscopy data oriented to a general model. Background Art

[0002] With the acceleration of the integration process of the tobacco industry and the attention of major brands to high-quality tobacco resources, the integration and efficient use of tobacco resources have become the core challenges for the development of the industry. The traditional tobacco formula research and development and quality control model are gradually unable to meet the needs of modern production. Digital detection, intelligent control and lean management technologies based on tobacco quality indicators are gradually being adopted at home and abroad to cope with the complexity and diversity of the production process.

[0003] Near infrared spectroscopy technology is widely used in laboratory tobacco leaf chemical index detection and online detection of production lines due to its accuracy, stability and reliability. This technology provides important support for the detection of tobacco leaf flavor and quality grade. However, the data quality and the scientific nature of the algorithm jointly determine the effectiveness of the spectral detection method. Abnormal spectral data will seriously affect the applicability and accuracy of the model, and thus affect the evaluation of tobacco leaf quality.

[0004] At present, practitioners have conducted a lot of research on abnormal sample screening, but there are few studies on data availability evaluation for general models. Therefore, how to combine technologies such as characteristic band selection, spectral feature extraction, and spectral similarity matching to complete the availability evaluation of spectral data has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] In order to solve the problems in the prior art, the present invention proposes a method for analyzing the availability of tobacco leaf near-infrared spectral data. Considering that abnormal tobacco leaf spectral data will seriously affect the training efficiency and accuracy of the application-side data processing model, the present invention intends to first perform availability analysis on the tobacco leaf spectral data and filter out problematic data, thereby improving the training efficiency and training accuracy of the application-side data processing model. At the same time, for the application link of the trained model, the method of the present invention can eliminate worthless data to be detected and improve the efficiency of the detection process.

[0006] The method for analyzing the availability of tobacco leaf near-infrared spectral data of the present invention comprises the following steps:

[0007] Step 1: Preprocess the near infrared spectrum data of the tobacco leaves to be evaluated;

[0008] Step 2: Select characteristic bands for the preprocessed data to select band data with high representativeness and low redundancy;

[0009] Step 3: interpolate the standard spectral data of tobacco leaves stored in the spectral library to match the dimension of the data after band selection;

[0010] Step 4: Calculate the similarity measure of the multi-criteria fusion of the data after the band selection in step 2 and the tobacco standard spectrum data after the interpolation operation in step 3, and obtain the mean and variance of the multi-criteria similarity measure;

[0011] Step 5: Based on the results obtained in step 4, perform availability analysis on tobacco leaf near-infrared spectral data and eliminate unusable data.

[0012] Compared with the prior art, the present invention has the following beneficial effects:

[0013] 1) The present invention preprocesses the near infrared spectrum data of the tobacco leaves to be evaluated to correct the errors that may occur in the spectrum acquisition process (such as mechanical errors caused by the equipment), and selects a characteristic band subset from the preprocessed data to retain the physical information in the data to the greatest extent, so as to reduce the redundancy of the data. The number of band selections is generally pre-set by the virtual dimension in the data.

[0014] 2) The present invention adopts a multi-criteria fusion similarity measurement method to evaluate the availability of data, comprehensively considering the autocorrelation coefficient, spectral angle distance, spectral information divergence and Euclidean distance between data, which can better identify abnormal samples in the data and complete the availability evaluation of spectral data.

[0015] 3) The present invention combines historical data for usability evaluation. The historical data is the tobacco leaf near-infrared spectral data that has been previously considered as usable data. The use of historical data can improve the accuracy of usability evaluation and avoid the overall data offset that may exist in a single batch of data to be evaluated. At the same time, it can also fully reuse existing data resources. The present invention can effectively eliminate abnormal tobacco leaf spectral data, complete the screening of usable spectral data, reduce or avoid unusable data, and improve the accuracy and applicability of subsequent general models in the training and application process. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of the near infrared spectroscopy data availability analysis method of the present invention;

[0017] Figure 2 The spectral curves of the three unavailable data are shown in Figure 1. The horizontal axis is the wave number (cm -1 ), the ordinate is absorbance;

[0018] Figure 3 It is the spectrum curve diagram of tobacco leaf data;

[0019] Figure 4 It is the histogram distribution of the similarity measure values ​​between the mixed tobacco samples and the non-tobacco samples. DETAILED DESCRIPTION

[0020] The present invention is further described and illustrated below in conjunction with specific embodiments. The embodiments are merely exemplary of the present disclosure and do not define the scope of limitation. The technical features of each embodiment of the present invention may be combined accordingly without conflicting with each other.

[0021] like Figure 1 As shown, the embodiment of the present invention mainly includes the following steps:

[0022] Step 1: Near infrared spectral data of tobacco leaves to be evaluated Perform correction preprocessing; N represents the number of samples in the data, and D represents the number of bands.

[0023] The purpose of preprocessing is to correct the errors that may occur in the spectrum acquisition process, including but not limited to geometric correction, atmospheric correction, radiation calibration and radiation normalization.

[0024] Step 2: Select the characteristic bands of the preprocessed data to select the band data with high representativeness and low redundancy L is the number of bands of the near-infrared spectral data after band selection; the purpose of characteristic band selection is to obtain the essential characteristic components of the data.

[0025] In some embodiments of the present invention, feature band selection refers to selecting a feature subset from the band set of the original data to retain the physical information in the image to the greatest extent possible to reduce the redundancy of the data. The number of bands selected is generally determined by the virtual dimension of the image. The physical information is selected from the generalized Rayleigh entropy of the bands, and the redundancy is selected from the covariance or cross-correlation matrix between the bands.

[0026] Step 3: interpolate the tobacco leaf standard spectral data stored in the spectral library to match the dimension of the data after band selection;

[0027] Step 4: Calculate the similarity measure of the multi-criteria fusion of the data after the band selection in step 2 and the tobacco standard spectrum data after the interpolation operation in step 3, and obtain the mean and variance of the multi-criteria similarity measure;

[0028] Step 5: Based on the results obtained in step 4, perform availability analysis on tobacco leaf near-infrared spectroscopy data and eliminate unusable data.

[0029] In a specific embodiment of the present invention, step 2 is implemented by the following method:

[0030] 2.1) Initialize the selected band set X S is empty; set the counter k=1; set the number of bands to be selected to L;

[0031] 2.2) From the candidate band set X UTraverse all unselected bands, and calculate the score of the t-th band x t as score(x t ):

[0032]

[0033] where corr(x, x t ) represents the correlation between band x and the t-th band x t ;

[0034] 2.3) Select the band with the maximum score from the current set of candidate bands X U as the target band and add it to the set of selected bands X S , and delete this band from the set of candidate bands X U ;

[0035] 2.4) If k < L at this time, then increment k by 1 and return to step 2.2); otherwise, output the set of selected bands X S , and the data after feature band selection is represented as

[0036] In a specific embodiment of the present invention, step 4 is implemented by the following method:

[0037] Input the N samples g i contained in the near-infrared spectral data to be evaluated after band selection, i = 1, 2,..., N. Each sample g i is an L-dimensional vector containing the spectral reflectance of L bands, and calculate the four similarity measures between each sample g i and the tobacco leaf standard reference spectral data c p . Among them, the four similarity measures are the correlation coefficient, spectral angle distance, spectral information divergence, and Euclidean distance;

[0038] The specific calculation methods of the four similarity measures are as follows:

[0039] ① Correlation coefficient COR(g i , c p ):

[0040]

[0041] where (g i ) l and (c p ) l are the data of g i and c p at the l-th band respectively, is the mean value of the data of the near

[0042] -infrared spectral data to be evaluated and the data in the spectral library at each band;

[0043] ② Spectral angular distance SAD (g i ,c p ):

[0044]

[0045] ③ Spectral information divergence SID (g i ,c p ):

[0046]

[0047] in represent the g after normalization i and c p spectral values ​​in band l; and

[0048] ④Euclidean distance Eud(g i ,c p ):

[0049]

[0050] Based on the four similarity metrics, a multi-criteria fusion similarity metric is obtained; further, the mean and variance of the multi-criteria similarity metric of this batch of near-infrared spectral data are obtained.

[0051] The similarity measure of multi-criteria fusion is:

[0052]

[0053] This similarity measurement criterion effectively combines multiple distance measurement methods such as spectral angle distance, spectral information divergence, Euclidean distance and correlation coefficient, which improves the robustness and accuracy of the algorithm. The mean of the multi-criteria similarity measurement is:

[0054]

[0055] In a specific embodiment of the present invention, step 5 is implemented by the following method:

[0056] The mean and variance of the multi-criteria similarity metric of the current batch data are averaged with the mean and variance of the multi-criteria similarity metric of the historical tobacco leaf near-infrared spectrum data to obtain the average mean μ p and variance mean σ p

[0057]

[0058] Among them, M is the total batches of data, among which the total number of batches of historical tobacco leaf near-infrared spectral data is M-1; M is 1, which means there is no historical data. The historical data is the tobacco leaf near-infrared spectral data that has been previously screened by humans and algorithms and is considered to be available for subsequent models. is the multi-criteria similarity mean of the i-th batch of data. The use of historical data can improve the accuracy of usability evaluation and fully reuse existing data resources. The tobacco leaf near-infrared spectral data after removing unusable samples by the method of the present invention will also recalculate the mean and variance of the multi-criteria similarity measure and store them as historical data to assist in the usability analysis of subsequent batches of data.

[0059] The present invention utilizes the 3sigma principle in the normal distribution, that is, based on the assumption of normal distribution, it is considered that most of the variation in the data is caused by random errors, and outliers beyond a certain range are considered to be gross errors. Specifically, the present invention calculates the usability of the near-infrared spectral data of the tobacco leaf to be evaluated, and the evaluation function is as follows:

[0060] Γ min =μ p -3σ p

[0061] Γ max =μ p +3σ p

[0062]

[0063] Among them, S(g i ,c p ) is the similarity measure of multi-criteria fusion of sample i, Γ represents the availability evaluation threshold of the data to be evaluated, where Γ min represents the minimum boundary threshold, Γ max represents the maximum boundary threshold; F(i) represents the availability evaluation function of the i-th sample in the data to be evaluated, which is used to evaluate whether there is abnormal data in the data to be evaluated, i = 1, 2, ..., N, where 1 indicates that the data is available and 0 indicates that it is unavailable;

[0064] When F(i) = 0, it means that the sample detection error does not meet the standard, the data to be detected is unusable data, and it is removed from the data to be evaluated;

[0065] When F(i)=1, it means that the sample is available data, and the larger F(i) is, the greater the availability of the data.

[0066] An embodiment of the present invention further provides a tobacco leaf near infrared spectroscopy data availability analysis system, which comprises:

[0067] A data preprocessing module, used for preprocessing the near infrared spectrum data of the tobacco leaves to be evaluated;

[0068] The characteristic band selection module performs characteristic band selection on the preprocessed data to select band data with high representativeness and low redundancy;

[0069] The spectrum library preprocessing module performs interpolation operations on the tobacco leaf standard spectrum data stored in the spectrum library to match the dimension of the data after band selection;

[0070] The spectrum matching module is used to calculate the similarity measure of the multi-criteria fusion between the data after band selection and the tobacco leaf standard spectrum data after interpolation operation, and obtain the mean and variance of the multi-criteria similarity measure;

[0071] The data availability evaluation module is used to analyze the availability of tobacco leaf near-infrared spectral data and eliminate unusable data.

[0072] In a specific embodiment of the present invention, the input near infrared spectrum data includes 202 samples g i ,i=1,2,...,202, each sample g i is a 1609-dimensional vector containing spectral reflectances of 1609 bands. In this example, the histogram distribution of the similarity metric values ​​of the mixed tobacco sample and the non-tobacco sample is as follows: Figure 4 As shown in Figure 1, after querying the sample label, samples with values ​​less than -2 are all non-tobacco samples. From the histogram results, it can be concluded that the similarity metric has an obvious effect in distinguishing tobacco samples from non-tobacco samples. According to the calculated F(i) value, 3 data are detected to be unavailable, and their spectral curves are as follows Figure 2 As shown in the figure, the data labels can be queried to find that the data are plastic, hemp rope and weed powder, not tobacco leaf samples. The other qualified samples in the data set are all tobacco leaf samples. The spectral curve of the tobacco leaf sample is shown in Figure 3 As shown, different colors represent different samples. Figure 2 and Figure 3 It can be seen that the spectral curve of tobacco leaves is significantly different from that of non-tobacco leaf samples, which verifies the effectiveness of the method.

[0073] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. For ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A method for analyzing the availability of tobacco leaf near infrared spectroscopy data, comprising the following steps: Step 1: Preprocess the near infrared spectrum data of the tobacco leaves to be evaluated; Step 2: Select characteristic bands for the preprocessed data to select band data with high representativeness and low redundancy; Step 3: interpolate the standard spectral data of tobacco leaves stored in the spectral library to match the dimension of the data after band selection; Step 4: Calculate the similarity measure of the multi-criteria fusion of the data after the band selection in step 2 and the tobacco standard spectrum data after the interpolation operation in step 3, and obtain the mean and variance of the multi-criteria similarity measure; Step 5: Based on the results obtained in step 4, perform availability analysis on tobacco leaf near-infrared spectral data and eliminate unusable data.

2. The method according to claim 1, characterized in that: The correction preprocessing includes one or more of geometric correction, atmospheric correction, radiation calibration or radiation normalization.

3. The method according to claim 1, characterized in that The tobacco leaf near infrared spectrum data to be evaluated in step 1 also includes multiple tobacco leaf near infrared spectrum data samples, which are expressed as N represents the number of samples in the data, and D represents the number of bands.

4. The method according to claim 1, characterized in that The characteristic band selection in step 2 specifically includes: 2.1) Initialize the selected band set X S is empty; set counter k = 1; set the number of bands to be selected to L; 2.2) From the candidate band set X U Traverse all unselected bands and calculate the tth band x t The score score(x t ): Among them, corr(x,x t ) represents the difference between band x and t band x t The size of the correlation; 2.3) Select the current candidate band set X U The band with the largest score in is added as the target band to the selected band set X S and from the candidate band set X U Delete the band in 2.4) If k < L at this time, then add 1 to k and return to step 2.2); otherwise, output the selected band set X s , the data after the feature band selection is expressed as 5. The method according to claim 4, characterized in that The step 3) is specifically to interpolate the tobacco leaf standard reference spectrum data in the spectrum library to obtain a tobacco leaf standard reference spectrum curve to match the band number L of the near-infrared spectrum data after the characteristic band selection.

6. The method according to claim 1, characterized in that The step 4 comprises: The N samples g contained in the near-infrared spectral data to be evaluated after input band selection i ,i=1,2,…,N, each sample g i For an L-dimensional vector containing the spectral reflectance of L bands, calculate each sample g i Tobacco leaf standard reference spectrum data c p The four similarity measures are correlation coefficient, spectral angle distance, spectral information divergence and Euclidean distance; Based on the four similarity metrics, a multi-criteria fusion similarity metric is obtained; further, the mean and variance of the multi-criteria similarity metric of this batch of near-infrared spectral data are obtained.

7. The method according to claim 6, characterized in that The specific calculation methods of the four similarity metrics are as follows: ①Correlation coefficient COR(g i ,c p ): Among them (g i ) l and (c p ) l g i and c p In the data of band l, is the mean value of the near-infrared spectral data to be evaluated and the data in the spectral library in each band; ② Spectral angular distance SAD (g i ,c p ): ③ Spectral information divergence SID (g i ,c p ): in represent the g after normalization i and c p spectral values ​​in band l; and ④Euclidean distance Eud(g i ,c p ):

8. The method according to claim 7, characterized in that The similarity measure obtained by multi-criteria fusion based on the four similarity measures is: The mean of the multi-criteria similarity measure is:

9. The method according to claim 1, characterized in that: The step 5 comprises: The mean and variance of the multi-criteria similarity metric of the current batch data are averaged with the mean and variance of the multi-criteria similarity metric of the historical tobacco leaf near-infrared spectrum data to obtain the average mean μ p and variance mean σ p Among them, M is the total batches of data, of which the total number of batches of historical tobacco leaf near-infrared spectroscopy data is M-1; is the mean of multi-criteria similarity of the i-th batch of data; Calculate the availability of the near infrared spectral data of the tobacco leaves to be evaluated. The evaluation function is as follows: C min =μ p -3s p C max =μ p +3s p Where Γ represents the availability assessment threshold of the data to be evaluated, where Γ min represents the minimum boundary threshold, Γ max represents the maximum boundary threshold; F(i) represents the availability evaluation function of the i-th sample in the data to be evaluated, which is used to evaluate whether there is abnormal data in the data to be evaluated, i = 1, 2, ..., N, where 1 indicates that the data is available and 0 indicates that it is unavailable; When F(i) = 0, it means that the sample detection error does not meet the standard, the data to be detected is unusable data, and it is removed from the data to be evaluated; When F(i)=1, it means that the sample is available data, and the larger F(i) is, the greater the availability of the data.

10. A tobacco leaf near infrared spectroscopy data availability analysis system for implementing the method according to any one of claims 1 to 9, characterized in that include: A data preprocessing module, used for preprocessing the near infrared spectrum data of the tobacco leaves to be evaluated; The characteristic band selection module performs characteristic band selection on the preprocessed data to select band data with high representativeness and low redundancy; The spectrum library preprocessing module performs interpolation operations on the tobacco leaf standard spectrum data stored in the spectrum library to match the dimension of the data after band selection; The spectrum matching module is used to calculate the similarity measure of the multi-criteria fusion between the data after band selection and the tobacco leaf standard spectrum data after interpolation operation, and obtain the mean and variance of the multi-criteria similarity measure; The data availability evaluation module is used to analyze the availability of tobacco leaf near-infrared spectral data and eliminate unusable data.

Images