Metrology analysis method for spectrum redundant information fusion modeling

Abstract

The invention provides a measurement analysis method for spectrum redundant information fusion modeling, and belongs to the field of spectrum rapid nondestructive analysis application, and the method comprises the steps: firstly, carrying out the regression of an original spectrum data set, obtaining a spectrum calibration prediction model, and obtaining the parameters of the spectrum calibration prediction model through the autoregression operation of training sample data; a redundant information comparison function is designed, redundant signals and partially uncertain noise signals are fused into a spectrum calibration prediction model, an effective approximate value of the redundant information comparison function is estimated, and whether the wavelength point needs to be reserved in the model or not is determined by evaluating the approximate value of the redundant information comparison function. And then calculating the influence of each wavelength on the predictive capability of the optimized spectrum calibration predictive model, and completing metering analysis. According to the method, the conventional calibration model can be corrected and optimized by fusing the characteristics of the spectrum redundant information, and the consumption of data information detection can be reduced by utilizing useless information which can be discarded originally on the premise of limited detection resources.

Description

Technical Field

[0001] The present invention relates to the application field of spectral rapid non-destructive analysis, and in particular to a metrological analysis method for spectral redundant information fusion modeling. Background Art

[0002] Spectral analysis is an efficient, fast, non-destructive and pollution-free measurement technology that has been successfully applied in agriculture, petrochemicals, food and other fields. It can quickly and efficiently determine the chemical composition and physical and chemical properties of samples, and has become a very important analytical method in corporate production and scientific research.

[0003] Since spectral data has the characteristics of diverse components, low information density, and high data dimension, the data analysis process often faces the challenges of difficult processing of high-dimensional data and inaccurate extraction of spectral response information. Too much redundant information can easily lead to high model dimensions and overfitting of the model. In addition, the spectral absorption bands of agricultural objects are mainly the absorption of spectral band information of frequency doubling and combined frequency at the molecular level. There are problems of spectral information overlap and redundant data that cannot be simply extracted and removed, and it contains other redundant information such as background noise and sample uncertainty. Therefore, the processing of redundant signals and noise signals is particularly important in improving the efficiency of model analysis and the accuracy of model prediction. The processing of redundant data generally involves screening information variables, and finding spectral wavelength variables with large amounts of information through correlation analysis of spectral data and component data, thereby removing the interference of redundant signals and noise signals.

[0004] At present, the conventional operation for processing redundant signals and noise signals in the spectral analysis process is to directly eliminate "irrelevant" variables, or reduce noise interference through spectral preprocessing methods, or use the uninformative variable removal (UVE) method to improve the signal-to-noise ratio of the data; the main purpose of these methods is to reduce the impact of redundant signals and noise signals on the model, without considering that redundant signals and some uncertain noise signals can be utilized. Summary of the invention

[0005] The purpose of the present invention is to provide a metrological analysis method for spectral redundant information fusion modeling to solve the technical problem that redundant information and uncertain noise information in the conventional spectral analysis process are unusable.

[0006] The method of the present invention realizes the fusion of redundant signals into the spectral analysis model through the innovative design of the modeling scheme, recycles the "useless" information, prevents the waste of digital resources, realizes digital energy saving, and at the same time, achieves the purpose of reducing the complexity of the model and improving the accuracy of the model in the research of spectral metrology analysis.

[0007] The plan involves the rational use of data resources in the fields of agriculture and agricultural and sideline production, as well as green, low-carbon and environmentally friendly analysis technologies. It demonstrates the feasibility of applying spectral technology for the recovery and integrated utilization of redundant information, and is innovative and practical.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0009] A metrological analysis method for spectral redundant information fusion modeling, firstly, a spectral calibration prediction model is obtained by regressing the original spectral data set, the parameters of the spectral calibration prediction model are obtained by the autoregressive operation of the training sample data, a redundant information comparison function is designed, the redundant signal and part of the uncertain noise signal are integrated into the spectral calibration prediction model, the effective approximate value of the redundant information comparison function is estimated, and by evaluating the approximate value of the redundant information comparison function, it is determined whether the wavelength point needs to be retained in the model, thereby completing the optimization of the spectral calibration prediction model, and then calculating the influence of each wavelength on the prediction ability of the optimized spectral calibration prediction model to complete the metrological analysis.

[0010] 2. According to the metrological analysis method of spectral redundant information fusion modeling according to claim 1, it is characterized by: assuming that the spectral information of a wavelength point is affected by redundant signals and noise signals, resulting in the absence of information of the corresponding wavelength point in the spectral data, or the amount of spectral information at the corresponding wavelength point is less than a set value, the spectral signal of the corresponding wavelength is re-introduced into the spectral calibration prediction model for training to obtain a recalculated result. If the change information of each wavelength point is used to retrain the model, the calculation process will become complicated and the calculation cost will become high.

[0011] 3. The quantitative analysis method of spectral redundant information fusion modeling according to claim 1 is characterized in that the process of the original spectral data set is:

[0012] The spectral data set is denoted as Z = {z i |i=1,2...n}, where the subscript i refers to the i-th sample, z t =(x t ,y t )∈X×Y, X represents p-dimensional spectral data, corresponding to p discrete spectral wavelength points, Y represents the target label value, and the Y value of the training set is known.

[0013] 4. The metrological analysis method of spectral redundant information fusion modeling according to claim 2 is characterized in that the process of the spectral calibration prediction model processing spectral information is:

[0014] Step 1: Establish a spectral calibration analysis model according to the conventional partial least squares method, and construct an empirical risk function L(Z, θ) to measure the degree of inconsistency between the predicted value of the calibration model and the target label value, where θ is the empirical risk parameter;

[0015] Step 2: Optimize the empirical risk function L(Z, θ) to obtain the minimum risk value, thereby completing the estimation of the parameter θ, that is,

[0016]

[0017] Assuming the spectral analysis calibration model: convex function and second-order differentiable, based on all sample data, the empirical risk function can be defined by the residual sum of squares between the calibration model prediction value and the actual value of the target label, that is,

[0018]

[0019] in, Represents the predicted value of the calibration model for the i-th spectral wavelength point;

[0020] Step 3: Let the set of all spectral wavelength points be w = {w j |j=1, 2...p, each wavelength point corresponds to a dimension of the spectral data. Assuming that the spectral data at the jth wavelength point incorporates redundant information, the spectral data x i After receiving the redundant information interference, the data changes to As the subscript j changes, 1≤j≤p, the redundant data is added one by one for each wavelength point, and the resulting estimate of the empirical risk parameter is The estimated value after the change is recorded as and

[0021]

[0022] The leave-one-out cross-validation method is adopted to make the model perturb each wavelength variable with redundant information.

[0023] 5. The quantitative analysis method of spectrum redundant information fusion modeling according to claim 1 is characterized in that the redundant information comparison function construction process is:

[0024] In view of the interference caused by the integration of redundant information, a redundant information comparison function is constructed to characterize the changes in the empirical risk function L(Z, θ) caused by the integration of redundant information data into the model, and the calibration model operation comparison is performed;

[0025] Assumptions With x i The data difference between is an arbitrarily small value that converges to 0, and the change in L(Z, θ) is obtained by estimating The optimized calculated value of

[0026]

[0027] in, represents the initial empirical risk function value when redundant information data is added, δ represents the weight of the L0 function, and

[0028] The construction of the redundant information comparison function is actually to use redundant information to introduce the regularization term of the L function to realize the constraints of model optimization during the parameter estimation of the empirical risk function L(Z, θ).

[0029] 6. The quantitative analysis method of spectrum redundant information fusion modeling according to claim 1 is characterized in that the solution process of the redundant information comparison function is:

[0030] In the empirical risk function L(Z, θ), Z is the experimental record data, so the minimization optimization process of the L function mainly depends on the debugging of the parameter θ. The following is the solution process of the redundant information comparison function, using L(θ) to represent L(Z, θ);

[0031] Before incorporating redundant information, the minimum value of L(θ) is determined Then for the jth wavelength variable w f (j=1, 2...p) adds redundant information, After being disturbed by redundant information data, it becomes The change is related to the weight value δ, and is a quantity less than the set value. Then the parameter change that is less than the set value is recorded as Since the data difference converges to 0, the small amount can be converted into the differential amount, that is,

[0032]

[0033] According to the first-order derivative optimization principle, the optimization equation can be derived as follows:

[0034]

[0035] Among them, the symbol Represents the first-order derivative operation of the function L with respect to θ;

[0036] Performing Taylor expansion on the equation yields,

[0037]

[0038] Among them, the symbol represents the second-order derivative operation, ignoring Δ δThe high-order small quantity of is the third-order and above derivative term. By formula (7), we can estimate Δ δ The value is,

[0039]

[0040] If and only if When L(θ) can be minimized, If the value is 0, then formula (8) can be simplified to

[0041]

[0042] Then we can get,

[0043]

[0044] It is defined as the measurement operator formed by the calibration model after fusing the redundant information data at the jth wavelength point, namely, the redundant information comparison function. As the subscript j changes 1≤j≤p, the redundant information comparison function is used to characterize the redundant information integration comparison value within the entire measurement band.

[0045] The present invention has the following beneficial effects due to the adoption of the above technical solution:

[0046] The present invention can integrate the characteristics of spectral redundant information to correct and optimize the conventional calibration model. Under the premise of limited detection resources, the "useless" information that would have been discarded can be used to reduce the consumption of data information detection, reduce detection costs and technical construction costs; moreover, based on the original fast and non-destructive technical characteristics of spectral analysis, the method of the present invention embodies the recycling and reuse of data resources and realizes digital energy saving. It is a new key technical method for the development of green and low-carbon transformation industries. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and preferred embodiments. However, it should be noted that many details listed in the specification are only for the purpose of enabling the reader to have a thorough understanding of one or more aspects of the present invention, and these aspects of the present invention can be implemented even without these specific details.

[0049] like Figure 1As shown in the figure, a metrological analysis method for spectral redundant information fusion modeling is first obtained by regressing the original spectral data set to obtain a conventional spectral calibration prediction model (referred to as calibration model), and the parameters of the model are obtained by the autoregressive operation of the training sample data. Then, the influence of each wavelength on the prediction ability of the model is calculated. It is assumed that the spectral information of this wavelength point is affected by redundant signals and noise signals, resulting in the absence of information of this wavelength point in the spectral data, or the amount of spectral information at this wavelength point is very sparse. The spectral signal of this wavelength is re-introduced into the calibration model for training to obtain the recalculated result. If the change information of each wavelength point is used to retrain the model, the calculation process will become complicated and the calculation cost will be very high. Therefore, a new function (called redundant information comparison function, i.e. RICF) is designed to integrate redundant signals and partially uncertain noise signals into the calibration model to estimate the effective approximate value of RICF. By evaluating the approximate value of RICF, it is determined whether the wavelength point needs to be retained in the model, and then the model optimization is completed.

[0050] Through the innovative design of the modeling scheme, redundant signals can be integrated into the spectral analysis model and "useless" information can be recycled, which can prevent the waste of digital resources and achieve digital energy saving. At the same time, in the research of spectral metrology and analysis, the purpose of reducing model complexity and improving model accuracy can be achieved.

[0051] 1. Data preparation

[0052] The spectral data set is denoted as Z = {z i |i=1,2...n}, where the subscript i refers to the i-th sample, z i =(x i ,y i )∈X×Y, X represents p-dimensional spectral data, corresponding to p discrete spectral wavelength points, and Y represents the target label value (the Y value of the training set is known).

[0053] 2. Model preparation

[0054] Step 1: Establish a spectral calibration analysis model according to the conventional partial least squares (PLS) method, and construct an empirical risk function L(Z, θ) to measure the degree of inconsistency between the predicted value of the calibration model and the target label value, where θ is the empirical risk parameter.

[0055] Step 2: Optimize the empirical risk function L(Z, θ) to obtain the minimum risk value, thereby completing the estimation of the parameter θ, that is,

[0056]

[0057] Assume that the spectral analysis calibration model is a strictly convex function and second-order differentiable; based on all sample data, the empirical risk function can be defined by the residual sum of squares between the calibration model prediction value and the actual value of the target label, that is,

[0058]

[0059] in, Represents the predicted value of the calibration model for the i-th spectral wavelength point.

[0060] Step 3: Let the set of all spectral wavelength points be w = {w j |j=1,2...p}, each wavelength point corresponds to a dimension of spectral data. Assuming that the spectral data at the jth wavelength point incorporates redundant information, the spectral data x i After receiving the redundant information interference, the data changes to As the subscript j changes (1≤j≤p), redundant data is added one by one for each wavelength point, and the resulting estimate of the empirical risk parameter is The estimated value after the change is recorded as and

[0061]

[0062] The leave-one-out cross-validation method is adopted to make the model perturb each wavelength variable with redundant information.

[0063] 3. Construction of RICF function

[0064] In response to the interference caused by the integration of redundant information, we construct a redundant information comparison function (i.e., RICF function) to characterize the changes in the empirical risk function L(Z,θ) caused by the integration of redundant information data into the model, and perform calibration model operation comparison.

[0065] Step 4: Assume With x i The data difference between is an arbitrarily small value that converges to 0, and the change in L(Z, θ) is obtained by estimating The optimized calculated value of

[0066]

[0067] in, represents the initial empirical risk function value when redundant information data is added, δ represents the weight of the L0 function, and

[0068] It can be seen that the construction of the RICF function is actually to use redundant information to introduce the regularization term of the L function to realize the constraints of model optimization during the parameter estimation of the empirical risk function L(Z,θ).

[0069] 4. Solution of RICF function

[0070] Reviewing the L(Z, θ) function, where Z is the experimental record data, the minimization optimization process of the L function mainly depends on the debugging of the parameter θ. For convenience, the following describes the solution process of the RICF function, using L(θ) to represent L(Z, θ).

[0071] Step 5: Before incorporating redundant information, determine by solving the minimum value of L(θ) Then for the jth wavelength variable w f (j=1, 2...p) adds redundant information, After being disturbed by redundant information data, it becomes This change is related to the weight value δ, and is a small amount, so this small parameter change is recorded as Since the data difference converges to 0, the small amount can be converted into the differential amount, that is,

[0072]

[0073] According to the first-order derivative optimization principle, the optimization equation can be derived as follows:

[0074]

[0075] Among them, the symbol Represents the first-order derivative operation of the function L with respect to θ.

[0076] Step 6: Perform Taylor expansion on the equation to obtain:

[0077]

[0078] Among them, the symbol represents the second-order derivative operation, ignoring θ δ The high-order small quantities (3rd order and above derivatives) of . By formula (7), we can estimate Δ δ The value is,

[0079]

[0080] Step 7: If and only if When L(θ) can be minimized, If the value is 0, then formula (8) can be simplified to

[0081]

[0082] In summary,

[0083]

[0084] It is defined as the measurement operator formed by the calibration model after fusing the redundant information data at the jth wavelength point, namely the RICF function. As the subscript j changes (1≤j≤p), the RICF function can be used to characterize the redundant information integration comparison value within the full measurement band.

[0085] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A quantitative analysis method for spectral redundant information fusion modeling, characterized by: Firstly, a spectral calibration prediction model is obtained by regressing the original spectral data set. The parameters of the spectral calibration prediction model are obtained through the autoregression operation of the training sample data. A redundant information comparison function is designed, and the redundant signal and some uncertain noise signals are integrated into the spectral calibration prediction model. The effective approximate value of the redundant information comparison function is estimated. By evaluating the approximate value of the redundant information comparison function, it is determined whether the wavelength point needs to be retained in the model, thereby completing the optimization of the spectral calibration prediction model. Then, the influence of each wavelength on the prediction ability of the optimized spectral calibration prediction model is calculated to complete the metrological analysis.

2. The quantitative analysis method of spectral redundant information fusion modeling according to claim 1 is characterized by: Assuming that the spectral information of a wavelength point is affected by redundant signals and noise signals, resulting in the lack of information of the corresponding wavelength point in the spectral data, or the amount of spectral information at the corresponding wavelength point is less than the set value, the spectral signal of the corresponding wavelength is re-introduced into the spectral calibration prediction model for training to obtain a recalculated result. If the change information of each wavelength point is used to retrain the model, the calculation process will become complicated and the calculation cost will become high.

3. The quantitative analysis method of spectral redundant information fusion modeling according to claim 1 is characterized by: The process for the original spectral dataset is: The spectral data set is denoted as z = {z i |i=1,2...n}, where the subscript i refers to the i-th sample, z i =(x i ,y i )∈X×Y, X represents p-dimensional spectral data, corresponding to p discrete spectral wavelength points, Y represents the target label value, and the Y value of the training set is known.

4. The quantitative analysis method of spectral redundant information fusion modeling according to claim 2 is characterized by: The process of spectral calibration prediction model processing spectral information is as follows: Step 1: Establish a spectral calibration analysis model according to the conventional partial least squares method, and construct an empirical risk function L(Z, θ) to measure the degree of inconsistency between the predicted value of the calibration model and the target label value, where θ is the empirical risk parameter; Step 2: Optimize the empirical risk function L(Z, θ) to obtain the minimum risk value, thereby completing the estimation of the parameter θ, that is, Assuming the spectral analysis calibration model: convex function and second-order differentiable, based on all sample data, the empirical risk function can be defined by the residual sum of squares between the calibration model prediction value and the actual value of the target label, that is, in, Represents the predicted value of the calibration model for the i-th spectral wavelength point; Step 3: Let the set of all spectral wavelength points be w = {w j |j=1,2...p}, each wavelength point corresponds to a dimension of spectral data. Assuming that the spectral data at the jth wavelength point incorporates redundant information, the spectral data x i After receiving the redundant information interference, the data changes to As the subscript j changes, 1≤j≤p, the redundant data is added one by one for each wavelength point, and the resulting estimate of the empirical risk parameter is The estimated value after the change is recorded as and The leave-one-out cross-validation method is adopted to make the model perturb each wavelength variable with redundant information.

5. The quantitative analysis method of spectral redundant information fusion modeling according to claim 1 is characterized by: The construction process of the redundant information comparison function is: In view of the interference caused by the integration of redundant information, a redundant information comparison function is constructed to characterize the changes in the empirical risk function L(Z, θ) caused by the integration of redundant information data into the model, and the calibration model operation comparison is performed; Assumptions With x i The data difference between is an arbitrarily small value that converges to 0, and the change in L(Z, θ) is obtained by estimating The optimized calculated value of in, represents the initial empirical risk function value when redundant information data is added, δ represents the weight of the L0 function, and The construction of the redundant information comparison function is actually to use redundant information to introduce the regularization term of the L function to realize the constraints of model optimization during the parameter estimation of the empirical risk function L(Z, θ).

6. The quantitative analysis method of spectral redundant information fusion modeling according to claim 1 is characterized by: The solution process of the redundant information comparison function is: In the empirical risk function L(Z, θ), Z is the experimental record data, so the minimization optimization process of the L function mainly depends on the debugging of the parameter θ. The following is the solution process of the redundant information comparison function, using L(θ) to represent L(Z, θ); Before incorporating redundant information, the minimum value of L(θ) is determined Then for the jth wavelength variable w j (j=1, 2...p) adds redundant information, After being disturbed by redundant information data, it becomes The change is related to the weight value δ, and is a quantity less than the set value. Then the parameter change that is less than the set value is recorded as Since the data difference converges to 0, the small amount can be converted into the differential amount, that is, According to the first-order derivative optimization principle, the optimization equation can be derived as follows: Among them, the symbol Represents the first-order derivative operation of the function L with respect to θ; Performing Taylor expansion on the equation yields, Among them, the symbol represents the second-order derivative operation, ignoring Δ δ The high-order small quantity of is the third-order and above derivative term. By formula (7), we can estimate Δ δ The value is, If and only if When L(θ) can be minimized, If the value is 0, then formula (8) can be simplified to Then we can get, It is defined as the measurement operator formed by the calibration model after fusing the redundant information data at the jth wavelength point, namely, the redundant information comparison function. As the subscript j changes 1≤j≤p, the redundant information comparison function is used to characterize the redundant information integration comparison value within the entire measurement band.

Images