Spectral data processing method, system, medium, product and equipment for integrated modeling

Abstract

The present invention provides a spectral data processing method, system, medium, product and equipment for integrated modeling, belonging to the field of data processing technology. By combining a variety of preprocessing and wavelength selection methods, data utilization is maximized at different angles, and the prediction accuracy and robustness of the model are improved; through multiple first-level PLS models and different data processing strategies, the diversity and comprehensiveness of the model are improved, and the stability and adaptability are enhanced; the model building and verification process is optimized, and through automated preprocessing and wavelength selection, the present invention significantly simplifies the modeling process, reduces the experimental trial and error time, and improves the quality and fault tolerance of data processing; the present invention significantly improves the accuracy, robustness and adaptability of the model as a whole, and simplifies the modeling process and optimization process, providing an efficient and reliable solution for the field of near-infrared spectral analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a spectral data processing method, system, medium, product and equipment for integrated modeling. Background Art

[0002] The statements in this section merely provide background art related to the present invention and do not necessarily constitute prior art.

[0003] Near Infrared Spectrum Instrument (NIR) technology is based on molecular vibration spectrum and analyzes through the interaction of light waves with substances in the near-infrared region. This technology has developed rapidly since the 1970s and has become an extremely important analytical tool in modern analytical chemistry. It is widely used in many fields due to its advantages such as non-destructiveness, no need for complex sample pretreatment, rapidity, and the ability to perform online measurements. In the actual modeling process, partial least squares regression (PLS) is one of the most commonly used techniques for spectral data analysis.

[0004] Although PLS performs well in many cases, it still faces some key limitations when processing high-dimensional and complex spectral data. Industry experts have developed a variety of solutions to improve the accuracy and robustness of the model. First of all, data preprocessing is one of the key steps, involving denoising, baseline correction, normalization, and derivatives. These methods help reduce non-systematic variation in the data and improve the quality and effect of subsequent modeling. Wavelength selection is also an important part of modeling. The main categories include filtering, wrapping, and embedded. Extracting the wavelength of optimal information from high-dimensional data can not only optimize model accuracy and stability, but also significantly reduce computational complexity.

[0005] However, in the actual PLS modeling process, there are many different preprocessing methods and wavelength selection algorithms, which often require a lot of time and resources to conduct trial and error experiments to determine the optimal data processing strategy. Moreover, traditional PLS modeling usually relies on a single data processing strategy, which limits the model's ability to adapt to different sample types and complex data structures. When the training set changes or the measurement conditions change, the original model often exhibits low robustness and cannot achieve efficient and accurate spectral data processing. Summary of the invention

[0006] In order to address the deficiencies in the prior art, the present invention provides a spectral data processing method, system, medium, product and equipment for integrated modeling, which can not only optimize the model construction and verification process, but also significantly improve the robustness and accuracy of the model, ensure the stability and high performance of the model in a changing environment, and improve the efficiency and accuracy of spectral data processing.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a spectral data processing method with integrated modeling.

[0009] A spectral data processing method based on integrated modeling includes the following processes:

[0010] Will k Preprocessing algorithms and m The wavelength selection algorithms are combined to form k × m Different data processing strategies were used to perform PLS modeling respectively. The performance of each PLS model was evaluated by leave-one-out cross-validation using the set evaluation index. i The best performing PLS model is used as the first-level learner, where k and m are all positive integers greater than or equal to 2;

[0011] The prediction results of the first-level learner are generated by the method of split cross prediction. The prediction results of each first-level learner will be used as a feature. i Features, will i feature combinations as the training set for the secondary learner, where i is a positive integer greater than or equal to 2;

[0012] use i The first-level learner predicts the test set and generates i The prediction results will be i The average of the prediction results is used as the test set for the secondary learner;

[0013] Select a variety of different regression algorithms as secondary learners, use the cross-validation method to train the secondary learners on the training set of the secondary learners, and use the test set of the secondary learners to test them. After optimizing the optimal parameters of the regression algorithms of different secondary learners, select the secondary learner with the best processing effect as the final processing model;

[0014] The final processing model is used to perform spectral data analysis on the acquired near-infrared spectral data to obtain spectral data analysis results.

[0015] As a further limitation of the first aspect of the present invention, the preprocessing algorithm includes: a smoothing filtering algorithm, a baseline correction algorithm, a scatter correction algorithm and a standardization algorithm.

[0016] As a further limitation of the first aspect of the present invention, the wavelength selection algorithm includes: a filtering algorithm, a packaging algorithm and an embedded algorithm.

[0017] As a further limitation of the first aspect of the present invention, the set evaluation indicators include: root mean square error and determination coefficient.

[0018] As a further limitation of the first aspect of the present invention, the regression algorithm includes: a linear regression algorithm, a Bayesian regression algorithm, a ridge regression algorithm and a regression algorithm based on bagging.

[0019] As a further limitation of the first aspect of the present invention, searching for optimal parameters of regression algorithms of different secondary learners includes:

[0020] The cross-validation method is used to train the secondary learner on its training set. The parameters are optimized according to the effect of cross-validation. The root mean square error and determination coefficient are used as evaluation indicators to find the optimal parameters for each secondary learner.

[0021] In a second aspect, the present invention provides a spectral data processing system with integrated modeling.

[0022] A spectral data processing system with integrated modeling, comprising:

[0023] The first-level learner generation unit is configured to: k Preprocessing algorithms and m The wavelength selection algorithms are combined to form k × m Different data processing strategies were used to perform PLS modeling respectively. The performance of each PLS model was evaluated by leave-one-out cross-validation using the set evaluation index. i The best performing PLS model is used as the first-level learner, where k and m are all positive integers greater than or equal to 2;

[0024] The second-level learner training set generation unit is configured to generate the prediction results of the first-level learner by the method of split cross prediction. The prediction results of each first-level learner will be used as a feature. i Features, will i feature combinations as the training set for the secondary learner, where i is a positive integer greater than or equal to 2;

[0025] The secondary learner test set generation unit is configured to: i The first-level learner predicts the test set and generates i The prediction results will be i The average of the prediction results is used as the test set for the secondary learner;

[0026] The final processing model generating unit is configured to: select a plurality of different regression algorithms as the secondary learners, train the secondary learners by using the cross-validation method on the training set of the secondary learners, and test the secondary learners by using the test set of the secondary learners, and after optimizing the optimal parameters of the regression algorithms of different secondary learners, select the secondary learner with the best processing effect as the final processing model;

[0027] The spectral data analysis unit is configured to: perform spectral data analysis on the acquired near-infrared spectral data using the final processing model to obtain spectral data analysis results.

[0028] In a third aspect, the present invention provides a computer device, comprising: a processor and a computer-readable storage medium;

[0029] a processor adapted to execute a computer program;

[0030] A computer-readable storage medium having a computer program stored therein, wherein when the computer program is executed by the processor, the spectral data processing method of integrated modeling as described in the first aspect of the present invention is implemented.

[0031] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program is suitable for being loaded by a processor and executing the spectral data processing method of integrated modeling as described in the first aspect of the present invention.

[0032] In a fifth aspect, the present invention provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the spectral data processing method of integrated modeling as described in the first aspect of the present invention.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. The present invention innovatively proposes a spectral data processing method of integrated modeling, which can not only optimize the model construction and verification process, but also significantly improve the robustness and accuracy of the model, ensure the stability and high performance of the model in a changing environment, and improve the efficiency and accuracy of spectral data processing.

[0035] 2. The present invention significantly improves the accuracy, robustness and adaptability of the model through diversified data processing strategies and integrated modeling methods, while simplifying the modeling process and optimization process, providing an efficient and reliable solution for the field of near-infrared spectroscopy analysis.

[0036] 3. The present invention improves the accuracy and robustness of the model by combining a variety of preprocessing and wavelength selection methods to maximize data utilization at different angles, thereby improving the model prediction accuracy and robustness.

[0037] 4. The present invention improves the diversity and comprehensiveness of the model and enhances its stability and adaptability through multiple primary PLS models and different data processing strategies.

[0038] 5. The present invention optimizes the model building and verification process, significantly simplifies the modeling process through automated preprocessing and wavelength selection, reduces experimental trial and error time, and improves the quality and fault tolerance of data processing.

[0039] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0041] Figure 1 A schematic diagram of the modeling process of the first-level learner provided in Example 1 of the present invention;

[0042] Figure 2 A flow chart for constructing new training set features and new test set features provided in Example 1 of the present invention;

[0043] Figure 3 A schematic diagram of the modeling process of the secondary learner provided in Example 1 of the present invention;

[0044] Figure 4 A schematic diagram of a spectral data processing system for integrated modeling provided in Example 2 of the present invention;

[0045] Figure 5 A schematic diagram of a computer device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0047] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0048] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0049] Embodiment 1:

[0050] This implementation proposes a spectral data processing method for integrated modeling, in which the multi-data processing strategy integrated modeling process includes:

[0051] S1: Get labels and data;

[0052] S2: outlier screening;

[0053] S3: Divide the training set and test set;

[0054] S4: Construction of PLS ​​models (first-level learner) based on different data processing strategies;

[0055] S5: Construction of new training sets and new test sets in the Stacking process;

[0056] S6: Construction of the secondary learner in the Stacking process.

[0057] In step S1 of this implementation, specifically, it includes:

[0058] Spectral data acquisition: Use a near-infrared spectrometer to measure the sample and obtain spectral data;

[0059] Label acquisition: Label data refers to the actual component content of the sample, which is usually obtained through standard analytical methods (such as high performance liquid chromatography). These labels will serve as the target variables of the model;

[0060] Data integration: Integrate spectral data and label data to ensure data integrity and accuracy and form an initial data set.

[0061] In step S2 of this implementation, specifically, it includes:

[0062] Outlier detection: Use statistical methods (such as Z-Score, box plots, etc.) or model-based methods (such as principal component analysis) to identify outliers in spectral data;

[0063] Outlier elimination: eliminate abnormal samples according to the set standards to ensure the representativeness and reliability of the remaining data;

[0064] Data verification: Check the data after removing outliers to confirm the data quality and consistency so as to proceed to the next step of processing.

[0065] In step S3 of this implementation, specifically, it includes:

[0066] Use a random partitioning method or a sample distance-based partitioning method (such as the Kennard-Stone method) to divide the dataset into a training set (80%) and a test set (20%).

[0067] In step S4 of this implementation, specifically, a PLS model (first-level learner) is constructed based on different data processing strategies, such as Figure 1 As shown, including:

[0068] S4.1: Build a preprocessing algorithm library.

[0069] Preprocessing is a crucial step in spectral analysis. It improves data quality by removing noise, correcting baselines, and standardizing data. The following are commonly used preprocessing methods (k types can be selected):

[0070] Smoothing filtering: such as moving average (MA), median filter (MF), Savitzky-Golay filter (SG), wavelet transform;

[0071] Baseline correction: such as first-order derivative, second-order derivative, multivariate scatter correction (MSC);

[0072] Scatter correction: such as extended multiplicative scatter correction (eMSC), standard normal transformation (SNV);

[0073] Normalization: such as automatic scaling (AU), minimum-maximum normalization (MMN).

[0074] S4.2: Build a wavelength selection algorithm library.

[0075] Wavelength selection is an important step in optimizing model performance. By selecting wavelengths that contribute significantly to the analysis, the data dimension and computational complexity can be reduced. The following are commonly used wavelength selection methods (m types can be selected):

[0076] Filtering methods: such as variance analysis, correlation coefficient method, mutual information method;

[0077] Encapsulated methods: such as recursive feature elimination (RFE), stepwise regression, and greedy algorithms;

[0078] Embedded methods: such as LASSO regression, ridge regression, decision tree wavelength selection.

[0079] S4.3: Combined preprocessing and wavelength selection algorithm.

[0080] The k preprocessing algorithms and the m wavelength selection algorithms are combined to form k×m (ie, n data processing strategies) different data processing strategies (including data processing strategy 1, data processing strategy 2, data processing strategy 3, data processing strategy 4...data processing strategy n).

[0081] S4.4: PLS model (first-level learner) construction.

[0082] For the k×m data processing strategies constructed, PLS modeling was performed respectively, and the root mean square error (RMSE) and determination coefficient (R²) were used as evaluation indicators. The leave-one-out cross-validation method was used to evaluate the model performance. The first i PLS models with the best performance were selected as the first-level learners, and diversified prediction results were output, including: PLS_model_1, PLS_model_2, PLS_model_3, PLS_model_4···PLS_model_i.

[0083] In step S5 of this implementation, specifically, the construction of new training set features and new test set features in the stacking process includes:

[0084] S5.1: Construction of new training set features.

[0085] Generate the prediction results of the first-level learner through the fold-by-fold cross-prediction method (such as 5-fold or 10-fold). The specific steps are as follows:

[0086] Divide the training set into K subsets: Use K-fold cross validation method.

[0087] Fold-by-fold validation: Select one subset as the validation set and the remaining subsets as the training set, train each first-level learner (i PLS models), predict the validation set, and save the prediction results.

[0088] Specifically, Figure 2 As shown in the figure, the fifth subset is selected as the validation set for the first time, and the first to fourth subsets are selected as training sets (i.e., the first four trains) for training to obtain the prediction result predict1; according to the above rules, the fourth subset is selected as the validation set for the second time, the third subset is selected as the validation set for the third time, the second subset is selected as the validation set for the fourth time, and the first subset is selected as the validation set for the fifth time, and finally the prediction results predict1, predict2, predict3, predict4 and predict5 are obtained respectively.

[0089] Merge prediction results: Merge the prediction results of all folds to form a complete training set (i.e., new_train_feature) as the training data for the second-level learner; specifically, the prediction result of each first-level learner (PLS model) will be used as a feature, with a total of i features (corresponding to i PLS models).

[0090] S5.2: Construction of new test set features.

[0091] The first-level learner (PLS model) predicts the test set and generates multiple prediction results (predict_1, predict_2, predict_3, predict_4, and predict_5). The average of these prediction results is used as the test set of the second-level learner (i.e., new_test_feature).

[0092] In step S6 of this implementation, specifically, the construction of the secondary learner in the stacking process includes:

[0093] The secondary learner is an integrated model used to combine the prediction results of the primary learner to improve the performance of the final prediction. The present invention selects a variety of different regression methods as the secondary learner to enhance the integration effect.

[0094] S6.1: Second level learner regression algorithm, e.g. Figure 3 As shown, including:

[0095] Linear regression: Find the optimal weights by linearly combining the predictions of the first-level learners.

[0096] Bayesian regression: Using Bayesian inference method, considering the uncertainty of prediction results, and improving the robustness of the model;

[0097] Ridge regression: Add L2 regularization term on the basis of linear regression to prevent overfitting and improve the generalization ability of the model;

[0098] Bagging-based regression methods: such as bagging regression tree, which generates multiple regression models by resampling the training set multiple times and averages their prediction results to reduce the variance of the model; or bagging_linear algorithm and bagging_bayes algorithm.

[0099] S6.2: The training process of the secondary learner includes:

[0100] Training: On the new training set generated in the previous step (including New_features_train_1, New_features_train_2, ... New_features_train_i), use the cross-validation method to train the secondary learner;

[0101] Parameter tuning: Use the generated new test sets (including New_features_test_1, New_features_test_2, ... New_features_test_i) to perform parameter tuning according to the results of cross-validation (such as using the Optuna method or grid search), and use the root mean square error (RMSE) and determination coefficient (R²) as evaluation indicators to find the optimal parameters;

[0102] Model selection: After optimizing the optimal parameters of the regression algorithms of different secondary learners, the secondary learner with the best prediction effect (for example, including Results_1, Results_2, Results_3 and Results_4) is compared and selected as the final model.

[0103] In this implementation, the partial least squares (PLS) integrated modeling method based on different data processing strategies provides technical improvement and efficiency optimization for near-infrared spectroscopy (NIR) analysis. The final processing model is used to perform spectral data analysis on the acquired near-infrared spectral data to obtain spectral data analysis results, such as identifying a substance and determining its chemical composition and relative content based on its spectrum.

[0104] Embodiment 2:

[0105] like Figure 4 As shown, this implementation provides a spectral data processing system with integrated modeling, including:

[0106] The first-level learner generation unit is configured to: combine k preprocessing algorithms and m The wavelength selection algorithms are combined to form k × m Different data processing strategies were used to perform PLS modeling respectively. The performance of each PLS model was evaluated by leave-one-out cross-validation using the set evaluation index. i The best performing PLS model is used as the first-level learner, where k and m are all positive integers greater than or equal to 2;

[0107] The second-level learner training set generation unit is configured to generate the prediction results of the first-level learner by the method of split cross prediction. The prediction results of each first-level learner will be used as a feature. i Features, willi feature combinations as the training set for the secondary learner, where i is a positive integer greater than or equal to 2;

[0108] The secondary learner test set generation unit is configured to: i The first-level learner predicts the test set and generates i The prediction results will be i The average of the prediction results is used as the test set for the secondary learner;

[0109] The final processing model generating unit is configured to: select a plurality of different regression algorithms as the secondary learners, train the secondary learners by using the cross-validation method on the training set of the secondary learners, and test the secondary learners by using the test set of the secondary learners, and after optimizing the optimal parameters of the regression algorithms of different secondary learners, select the secondary learner with the best processing effect as the final processing model;

[0110] The spectral data analysis unit is configured to: perform spectral data analysis on the acquired near-infrared spectral data using the final processing model to obtain spectral data analysis results.

[0111] The specific working details of each unit are described in Example 1 and will not be repeated here.

[0112] It is understandable that the above-mentioned units can be separately or all combined into one or several other units to constitute, or some of the units can be further divided into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of a unit can also be realized by multiple units, or the function of multiple units can be realized by one unit. In other embodiments of the present application, the spectral data processing system of the integrated modeling can also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0113] According to another embodiment of the present application, the system described in this embodiment can be constructed, and the spectral data processing method of integrated modeling of the embodiment of the present application can be implemented by running a computer program (including program code) capable of executing the steps involved in the corresponding method described in Example 1 on a general computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0114] Embodiment 3:

[0115] like Figure 5 As shown, this implementation provides an electronic device, which includes a processor 1001, a communication interface 1002, and a computer-readable storage medium 1003. The processor 1001, the communication interface 1002, and the computer-readable storage medium 1003 may be connected via a bus or other means.

[0116] Among them, the communication interface 1002 is used to receive and send data, the computer-readable storage medium 1003 can be stored in the memory of the electronic device, the computer-readable storage medium 1003 is used to store a computer program, the computer program includes program instructions, and the processor 1001 is used to execute the program instructions stored in the computer-readable storage medium 1003.

[0117] The processor 1001 (or CPU (Central Processing Unit)) is the computing core and control core of the electronic device, which is suitable for implementing one or more instructions, and specifically suitable for loading and executing one or more instructions to implement corresponding method processes or corresponding functions.

[0118] The processor 1001 is configured to execute the following process:

[0119] k preprocessing algorithms and m The wavelength selection algorithms are combined to form k × m Different data processing strategies were used to perform PLS modeling respectively. The performance of each PLS model was evaluated by leave-one-out cross-validation using the set evaluation index. i The best performing PLS model is used as the first-level learner, where k and m are all positive integers greater than or equal to 2;

[0120] The prediction results of the first-level learner are generated by the method of split cross prediction. The prediction results of each first-level learner will be used as a feature. i Features, will i feature combinations as the training set for the secondary learner, where i is a positive integer greater than or equal to 2;

[0121] use i The first-level learner predicts the test set and generates i The prediction results will be i The average of the prediction results is used as the test set for the secondary learner;

[0122] Select a variety of different regression algorithms as secondary learners, use the cross-validation method to train the secondary learners on the training set of the secondary learners, and use the test set of the secondary learners to test them. After optimizing the optimal parameters of the regression algorithms of different secondary learners, select the secondary learner with the best processing effect as the final processing model;

[0123] The final processing model is used to perform spectral data analysis on the acquired near-infrared spectral data to obtain spectral data analysis results.

[0124] The specific working details of each unit are described in Example 1 and will not be repeated here.

[0125] Embodiment 4:

[0126] This implementation provides a computer-readable storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the electronic device and, of course, extended storage media supported by the electronic device. The computer-readable storage medium provides a storage space that stores the processing system of the electronic device.

[0127] In addition, the storage space also stores one or more instructions suitable for being loaded and executed by the processor, and these instructions may be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it may also be at least one computer-readable storage medium located away from the aforementioned processor.

[0128] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes the one or more instructions stored in the computer-readable storage medium to implement the following process:

[0129] k preprocessing algorithms and m The wavelength selection algorithms are combined to form k × m Different data processing strategies were used to perform PLS modeling respectively. The performance of each PLS model was evaluated by leave-one-out cross-validation using the set evaluation index. i The best performing PLS model is used as the first-level learner, where k and m are all positive integers greater than or equal to 2;

[0130] The prediction results of the first-level learner are generated by the method of split cross prediction. The prediction results of each first-level learner will be used as a feature. i Features, will i feature combinations as the training set for the secondary learner, where i is a positive integer greater than or equal to 2;

[0131] use i The first-level learner predicts the test set and generates i The prediction results will be i The average of the prediction results is used as the test set for the secondary learner;

[0132] Select a variety of different regression algorithms as secondary learners, use the cross-validation method to train the secondary learners on the training set of the secondary learners, and use the test set of the secondary learners to test them. After optimizing the optimal parameters of the regression algorithms of different secondary learners, select the secondary learner with the best processing effect as the final processing model;

[0133] The final processing model is used to perform spectral data analysis on the acquired near-infrared spectral data to obtain spectral data analysis results.

[0134] The specific working details of each unit are described in Example 1 and will not be repeated here.

[0135] Embodiment 5:

[0136] The present implementation provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the following process:

[0137] k preprocessing algorithms and m The wavelength selection algorithms are combined to form k × m Different data processing strategies were used to perform PLS modeling respectively. The performance of each PLS model was evaluated by leave-one-out cross-validation using the set evaluation index. i The best performing PLS model is used as the first-level learner, where k and m are all positive integers greater than or equal to 2;

[0138] The prediction results of the first-level learner are generated by the method of split cross prediction. The prediction results of each first-level learner will be used as a feature. i Features, will i feature combinations as the training set for the secondary learner, where i is a positive integer greater than or equal to 2;

[0139] use i The first-level learner predicts the test set and generates i The prediction results will be i The average of the prediction results is used as the test set for the secondary learner;

[0140] Select a variety of different regression algorithms as secondary learners, use the cross-validation method to train the secondary learners on the training set of the secondary learners, and use the test set of the secondary learners to test them. After optimizing the optimal parameters of the regression algorithms of different secondary learners, select the secondary learner with the best processing effect as the final processing model;

[0141] The final processing model is used to perform spectral data analysis on the acquired near-infrared spectral data to obtain spectral data analysis results.

[0142] The specific working details of each unit are described in Example 1 and will not be repeated here.

[0143] A person skilled in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0144] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server, a data center, etc. that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0145] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A spectral data processing method for integrated modeling, characterized in that: The process includes: Get tags and data, including: Spectral data acquisition: Use a near-infrared spectrometer to measure the sample and obtain spectral data; Label acquisition: Label data refers to the actual component content of the sample, which is usually obtained through standard analytical methods. The label will be used as the target variable of the model; Data integration: Integrate spectral data and label data to ensure data integrity and accuracy and form an initial data set; Outlier screening, including: Outlier detection: using statistical or model-based methods to identify outliers in spectral data; Outlier elimination: eliminate abnormal samples according to the set standards to ensure the representativeness and reliability of the remaining data; Data verification: Check the data after removing outliers to confirm the data quality and consistency so as to proceed to the next step of processing; Use random partitioning or sample distance-based partitioning to divide the data set into training and test sets. k kinds of preprocessing algorithms and m The wavelength selection algorithms are combined to form k × m Different data processing strategies were used to perform PLS modeling respectively. The root mean square error and determination coefficient were used as evaluation indicators. The leave-one-out cross-validation method was used to evaluate the performance of each PLS model. i The best performing PLS model is used as the first-level learner to output a variety of prediction results, among which, k and m are all positive integers greater than or equal to 2; The prediction results of the first-level learner are generated by the method of split cross prediction. The prediction results of each first-level learner will be used as a feature. i Features, will i feature combinations as the training set for the secondary learner, where i is a positive integer greater than or equal to 2; The prediction results of the first-level learner are generated by the method of cross-prediction analysis, and the specific steps are as follows: Divide the training set into K subsets: Use K-fold cross validation method; Fold-by-fold validation: Select one subset as the validation set and the remaining subsets as the training set. i Train a first-level learner, make predictions on the validation set, and save the prediction results; The first time, the fifth subset is selected as the validation set, and the first to fourth subsets are selected as the training sets. Training is performed to obtain the prediction result predict1; the second time, the fourth subset is selected as the validation set, the third time, the fourth time, the second subset is selected as the validation set, and the fifth time, the first subset is selected as the validation set. Finally, the prediction results predict1, predict2, predict3, predict4 and predict5 are obtained respectively; use i The first-level learner predicts the test set and generates i The prediction results will be i The prediction results are averaged as the test set for the second-level learner. The first-level learner predicts the test set and generates multiple prediction results predict_1, predict_2, predict_3, predict_4 and predict_5. The average of these prediction results is used as the test set for the second-level learner. Select a variety of different regression algorithms as secondary learners, use the cross-validation method to train the secondary learners on the training set of the secondary learners, and use the test set of the secondary learners to test them. After optimizing the optimal parameters of the regression algorithms of different secondary learners, select the secondary learner with the best processing effect as the final processing model; Optimize the optimal parameters of the regression algorithms of different secondary learners, including: The cross-validation method is used to train the secondary learner on the training set of the secondary learner, and the parameters are tuned according to the effect of cross-validation. The root mean square error and determination coefficient are used as evaluation indicators to find the optimal parameters for each secondary learner; the final processing model is used to analyze the acquired near-infrared spectral data to obtain the spectral data analysis results; The preprocessing algorithm includes: a smoothing filter algorithm, a baseline correction algorithm, a scatter correction algorithm and a standardization algorithm; The wavelength selection algorithm includes: a filtering algorithm, a packaging algorithm and an embedded algorithm; The regression algorithm of the secondary learner includes: a linear regression algorithm, a Bayesian regression algorithm, a ridge regression algorithm and a regression algorithm based on bagging.

2. A spectral data processing system with integrated modeling, characterized in that: The spectral data processing system using the integrated modeling of claim 1 comprises: The first-level learner generation unit is configured to: combine k preprocessing algorithms and m The wavelength selection algorithms are combined to form k × m Different data processing strategies were used to perform PLS modeling respectively. The performance of each PLS model was evaluated by leave-one-out cross-validation using the set evaluation index. i The best performing PLS model is used as the first-level learner, where k and m are all positive integers greater than or equal to 2; The second-level learner training set generation unit is configured to generate the prediction results of the first-level learner by the method of split cross prediction. The prediction results of each first-level learner will be used as a feature. i Features, will i feature combinations as the training set for the secondary learner, where i is a positive integer greater than or equal to 2; The prediction results of the first-level learner are generated by the method of cross-prediction analysis, and the specific steps are as follows: Divide the training set into K subsets: Use K-fold cross validation method; Fold-by-fold validation: Select one subset as the validation set and the remaining subsets as the training set. i Train a first-level learner, make predictions on the validation set, and save the prediction results; The secondary learner test set generation unit is configured to: i The first-level learner predicts the test set and generates i The prediction results will be i The average of the prediction results is used as the test set for the secondary learner; The final processing model generating unit is configured to: select a plurality of different regression algorithms as the secondary learners, train the secondary learners by using the cross-validation method on the training set of the secondary learners, and test the secondary learners by using the test set of the secondary learners, and after optimizing the optimal parameters of the regression algorithms of different secondary learners, select the secondary learner with the best processing effect as the final processing model; Optimize the optimal parameters of the regression algorithms of different secondary learners, including: The cross-validation method is used to train the secondary learner on the training set of the secondary learner, and the parameters are optimized according to the effect of cross-validation. The root mean square error and determination coefficient are used as evaluation indicators to find the optimal parameters for each secondary learner. The spectral data analysis unit is configured to: perform spectral data analysis on the acquired near-infrared spectral data using the final processing model to obtain spectral data analysis results.

3. A computer device, characterized in that: include: A processor and a computer readable storage medium; a processor adapted to execute a computer program; A computer-readable storage medium having a computer program stored therein, wherein when the computer program is executed by the processor, the spectral data processing method for integrated modeling as claimed in claim 1 is implemented.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the spectral data processing method of integrated modeling as claimed in claim 1.

5. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the spectral data processing method of integrated modeling as claimed in claim 1 is implemented.

Images