A Near-Infrared Spectral Modeling Method and System Based on Random Mask Autoencoder

Abstract

This invention discloses a near-infrared spectral modeling method and system based on a random masked autoencoder. The method includes: acquiring the original near-infrared spectrum and component content of a sample; preprocessing the original near-infrared spectrum to obtain a masked near-infrared spectrum; training a constructed autoencoder model using the masked near-infrared spectrum as input to obtain a trained encoder; inputting the masked near-infrared spectrum into the trained encoder to obtain near-infrared spectral abstract features; training a constructed near-infrared spectral qualitative / quantitative analysis model using the near-infrared spectral abstract features as input; and inputting the near-infrared spectrum of the sample to be tested into the abstract features obtained from the trained encoder into the trained near-infrared spectral qualitative / quantitative analysis model to obtain predicted values ​​of the component content of the sample. This invention analyzes component content based on near-infrared spectroscopy to obtain the relationship between the abstract features extracted from the original spectrum and the component content.

Description

Technical Field

[0001] This invention relates to the field of near-infrared spectroscopy analysis technology, and in particular to a near-infrared spectral modeling method and system based on a random mask autoencoder. Background Technology

[0002] In recent years, near-infrared spectroscopy has been widely used in various industries, such as manufacturing, agriculture, forestry, marine science, and biopharmaceuticals, primarily for the qualitative and quantitative analysis of components and their concentrations. Compared to traditional laboratory methods, near-infrared spectroscopy offers advantages such as being non-destructive, rapid, and accurate.

[0003] Due to the high dimensionality (the number of wavelengths is much greater than the number of samples) and multicollinearity of near-infrared spectral data, wavelength screening is usually required before using traditional machine learning models to identify characteristic spectra. With the development of artificial intelligence, building models using artificial neural networks has become a research hotspot. However, the features extracted by artificial neural networks from the raw spectra are abstract and cannot be explained in terms of their physical meaning, resulting in a situation where "we know what, but not why." Therefore, there is an urgent need for a modeling method that can accurately interpret the abstract features extracted (i.e., white-boxing the black-box model). Summary of the Invention

[0004] The purpose of this invention is to provide a near-infrared spectral modeling method and system based on a random mask autoencoder, which can analyze the component content based on the near-infrared spectrum and obtain the relationship between the abstract features extracted from the original spectrum and the component content.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] A near-infrared spectral modeling method based on a random mask autoencoder, the method comprising:

[0007] Obtain the raw near-infrared spectrum and component content of the sample;

[0008] The original near-infrared spectrum is augmented to obtain an amplified near-infrared spectrum;

[0009] The amplified near-infrared spectrum is randomly masked to obtain a masked near-infrared spectrum;

[0010] Construct an autoencoder model; the autoencoder model includes an encoder and a decoder;

[0011] Using the masked near-infrared spectrum as input and the original near-infrared spectrum as output, the autoencoder model is trained to obtain a trained autoencoder model; the trained autoencoder model includes a trained encoder and a trained decoder.

[0012] The masked near-infrared spectrum is input into the trained encoder to obtain near-infrared spectral abstract features;

[0013] Construct a qualitative / quantitative analysis model for near-infrared spectroscopy;

[0014] Using the abstract features of the near-infrared spectrum as input and the component content corresponding to the original near-infrared spectrum as output, the near-infrared spectral qualitative / quantitative analysis model is trained to obtain a trained near-infrared spectral qualitative / quantitative analysis model.

[0015] The near-infrared spectral abstract features of the sample to be tested are input into the trained near-infrared spectral qualitative / quantitative analysis model to obtain the predicted values ​​of the component content of the sample to be tested; the near-infrared spectral abstract features of the sample to be tested are obtained by inputting the near-infrared spectrum of the sample to be tested into the trained encoder.

[0016] Optionally, the step of performing data enhancement on the original near-infrared spectrum to obtain an amplified near-infrared spectrum specifically includes:

[0017] The original near-infrared spectrum is added to a normally distributed noise with a mean of 0 to obtain a jittery near-infrared spectrum;

[0018] The original near-infrared spectrum is multiplied by a normally distributed noise with a mean of 1 to obtain a scaled near-infrared spectrum;

[0019] Multiplying the original near-infrared spectrum by the amplitude distortion function yields the amplitude-distorted near-infrared spectrum;

[0020] Multiplying the original near-infrared spectrum by the time-distortion function yields the time-distorted near-infrared spectrum;

[0021] Multiplying the original near-infrared spectrum by the rotation function yields the rotated near-infrared spectrum;

[0022] After the original near-infrared spectrum is uniformly divided into multiple sub-intervals, the spectrum is rearranged to obtain the replaced near-infrared spectrum;

[0023] By setting any one of the multiple sub-interval spectra to 0, a clipped near-infrared spectrum is obtained.

[0024] The amplified near-infrared spectrum is obtained by sequentially splicing the jittered near-infrared spectrum, the scaled near-infrared spectrum, the amplitude-distorted near-infrared spectrum, the time-distorted near-infrared spectrum, the rotated near-infrared spectrum, the displacement near-infrared spectrum, and the cropped near-infrared spectrum.

[0025] Optionally, the step of randomly masking the amplified near-infrared spectrum to obtain a masked near-infrared spectrum specifically includes:

[0026] The amplified near-infrared spectrum is divided into M sub-intervals; M is a positive integer greater than 0;

[0027] Randomly generate M binary sequences;

[0028] The spectrum of the sub-interval corresponding to "1" in the binary sequence is retained, and the spectrum of the sub-interval corresponding to "0" in the binary sequence is removed to obtain the masked near-infrared spectrum.

[0029] Optionally, the step of training the autoencoder model using the masked near-infrared spectrum as input and the original near-infrared spectrum as output to obtain a trained autoencoder model specifically includes:

[0030] Set the number of iterations;

[0031] Using the masked near-infrared spectrum as input and the original near-infrared spectrum as output, the autoencoder model is trained using an error backpropagation algorithm;

[0032] The training of the autoencoder model stops after reaching the specified number of iterations, resulting in a trained autoencoder model.

[0033] Optionally, the near-infrared spectroscopy qualitative / quantitative analysis model is a machine learning model or a deep learning model.

[0034] A near-infrared spectral modeling system based on a random mask autoencoder, the system comprising:

[0035] The acquisition module is used to acquire the raw near-infrared spectrum and component content of the sample;

[0036] The data enhancement module is used to enhance the original near-infrared spectrum to obtain an amplified near-infrared spectrum;

[0037] The masking module is used to randomly mask the amplified near-infrared spectrum to obtain the masked near-infrared spectrum;

[0038] The first building module is used to build an autoencoder model; the autoencoder model includes an encoder and a decoder.

[0039] The first training module is used to train the autoencoder model with the masked near-infrared spectrum as input and the original near-infrared spectrum as output to obtain a trained autoencoder model; the trained autoencoder model includes a trained encoder and a trained decoder.

[0040] The abstract feature acquisition module is used to input the masked near-infrared spectrum into the trained encoder to obtain near-infrared spectral abstract features.

[0041] The second building module is used to construct near-infrared spectroscopy qualitative / quantitative analysis models;

[0042] The second training module is used to train the near-infrared spectral qualitative / quantitative analysis model by taking the abstract features of the near-infrared spectrum as input and the component content corresponding to the original near-infrared spectrum as output, so as to obtain the trained near-infrared spectral qualitative / quantitative analysis model.

[0043] The prediction module is used to input the abstract features of the near-infrared spectrum of the sample to be tested into the trained near-infrared spectral qualitative / quantitative analysis model to obtain the predicted values ​​of the component content of the sample to be tested; the abstract features of the near-infrared spectrum of the sample to be tested are obtained by inputting the near-infrared spectrum of the sample to be tested into the trained encoder.

[0044] Optionally, the data enhancement module includes:

[0045] The jitter submodule is used to add the original near-infrared spectrum to a normally distributed noise with a mean of 0 to obtain a jittered near-infrared spectrum;

[0046] The scaling submodule is used to multiply the original near-infrared spectrum with a normal distribution noise with a mean of 1 to obtain a scaled near-infrared spectrum;

[0047] The amplitude distortion submodule is used to multiply the original near-infrared spectrum by the amplitude distortion function to obtain the amplitude-distorted near-infrared spectrum;

[0048] The time-warping submodule is used to multiply the original near-infrared spectrum by the time-warping function to obtain the time-warped near-infrared spectrum;

[0049] The rotation submodule is used to multiply the original near-infrared spectrum by a rotation function to obtain a rotated near-infrared spectrum;

[0050] The substitution submodule is used to rearrange the multiple sub-interval spectra obtained by uniformly dividing the original near-infrared spectrum to obtain the substitution near-infrared spectrum;

[0051] The cropping submodule is used to set any one of the multiple sub-interval spectra to 0 to obtain a cropped near-infrared spectrum.

[0052] The splicing submodule is used to sequentially splice the jittered near-infrared spectrum, the scaled near-infrared spectrum, the amplitude-distorted near-infrared spectrum, the time-distorted near-infrared spectrum, the rotated near-infrared spectrum, the displacement near-infrared spectrum, and the cropped near-infrared spectrum to obtain the amplified near-infrared spectrum.

[0053] Optionally, the masking module includes:

[0054] The sub-interval division module is used to divide the amplified near-infrared spectrum into M sub-intervals; M is a positive integer greater than 0;

[0055] The binary sequence generation submodule is used to randomly generate M binary sequences;

[0056] The masking submodule is used to retain the spectrum of the sub-interval corresponding to "1" in the binary sequence and remove the spectrum of the sub-interval corresponding to "0" in the binary sequence to obtain the masked near-infrared spectrum.

[0057] Optionally, the first training module includes:

[0058] The settings submodule is used to set the number of iterations;

[0059] The training submodule is used to train the autoencoder model by taking the masked near-infrared spectrum as input and the original near-infrared spectrum as output, and using the error backpropagation algorithm.

[0060] The completion submodule is used to stop training the autoencoder model after reaching the number of iterations, thus obtaining the trained autoencoder model.

[0061] Optionally, the near-infrared spectroscopy qualitative / quantitative analysis model is a machine learning model or a deep learning model.

[0062] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0063] This invention provides a near-infrared spectral modeling method based on a random masked autoencoder, comprising: acquiring the original near-infrared spectrum and component content of a sample; performing data augmentation on the original near-infrared spectrum to obtain an amplified near-infrared spectrum; randomly masking the amplified near-infrared spectrum to obtain a masked near-infrared spectrum; constructing an autoencoder model; the autoencoder model including an encoder and a decoder; training the autoencoder model with the masked near-infrared spectrum as input and the original near-infrared spectrum as output to obtain a trained autoencoder model; the trained autoencoder model including a trained encoder and a trained decoder; and applying the masked near-infrared spectrum to the sample. Infrared spectra are input into a trained encoder to obtain near-infrared spectral abstract features; a near-infrared spectral qualitative / quantitative analysis model is constructed; using the near-infrared spectral abstract features as input and the component content corresponding to the original near-infrared spectrum as output, the near-infrared spectral qualitative / quantitative analysis model is trained to obtain a trained near-infrared spectral qualitative / quantitative analysis model; the near-infrared spectral abstract features of the sample to be tested are input into the trained near-infrared spectral qualitative / quantitative analysis model to obtain the predicted value of the component content of the sample to be tested; the near-infrared spectral abstract features of the sample to be tested are obtained by inputting the near-infrared spectrum of the sample to the trained encoder. This invention establishes a mapping relationship between the abstract features extracted by the autoencoder and the component content characteristic wavelength spectrum through a random masking strategy, enabling the analysis of component content based on near-infrared spectra and obtaining the relationship between the abstract features extracted from the original spectrum and the component content. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 A flowchart of the near-infrared spectral modeling method based on a random mask autoencoder provided by the present invention;

[0066] Figure 2 A system block diagram of the near-infrared spectral modeling system based on a random mask autoencoder provided by the present invention;

[0067] Figure 3 The images show the near-infrared spectra of 60 gasoline samples from this embodiment of the invention.

[0068] Figure 4 This is a schematic diagram of dataset partitioning in an embodiment of the present invention;

[0069] Figure 5 This is a schematic diagram of the autoencoder model in an embodiment of the present invention;

[0070] Figure 6 This is a schematic diagram of the prediction results in an embodiment of the present invention.

[0071] Symbol explanation:

[0072] Acquisition Module-1, Data Augmentation Module-2, Masking Module-3, First Construction Module-4, First Training Module-5, Abstract Feature Acquisition Module-6, Second Construction Module-7, Second Training Module-8, Prediction Module-9. Detailed Implementation

[0073] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0074] The purpose of this invention is to provide a near-infrared spectral modeling method and system based on a random mask autoencoder, which can analyze the component content based on the near-infrared spectrum and obtain the relationship between the abstract features extracted from the original spectrum and the component content.

[0075] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0076] like Figure 1 As shown, the near-infrared spectral modeling method based on a random mask autoencoder provided by this invention includes:

[0077] Step S1: Obtain the raw near-infrared spectrum and component content of the sample.

[0078] Specifically, the original near-infrared spectrum of the sample was acquired using a near-infrared spectrometer; the component content was obtained by professionals using traditional laboratory analysis methods.

[0079] Furthermore, such as Figure 2 As shown, 60 gasoline samples were scanned using near-infrared spectroscopy with a wavelength range of 900–1700 nm and a wavelength interval of 2 nm to obtain their corresponding raw near-infrared spectra, denoted as X. The matrix dimension of X is 60*401, and each row of X represents the spectrum of a gasoline sample. The octane number of each sample was determined using traditional laboratory methods, denoted as Y. The matrix dimension of Y is 60*1, and each row of Y represents the octane number of a gasoline sample.

[0080] In addition, such as Figure 3As shown, the original near-infrared spectrum and component content of the sample are randomly split into training and validation sets, respectively; that is, the original near-infrared spectrum X is split into Xtraining and Xvalidation sets. train and X test ; where X train The matrix dimension is 50*401, X test The matrix dimension is 10*401; the component Y to be analyzed is split into Y0 and Y1. train and Y test ; where Y train The matrix dimension is 50*1, Y test The matrix dimension is 10*1; the number of wavelength points contained in the original near-infrared spectrum is denoted as L; further, L = 401.

[0081] Step S2: Perform data augmentation on the original near-infrared spectrum to obtain the amplified near-infrared spectrum; the data augmentation methods include dithering, scaling, amplitude distortion, time distortion, rotation, displacement and cropping.

[0082] S2 specifically includes:

[0083] Step S21: Add the original near-infrared spectrum to the normal distribution noise with a mean of 0 to obtain the jittered near-infrared spectrum.

[0084] Specifically, the formula for jitter enhancement data is: Where f(σ) is a randomly generated normally distributed noise, and this normally distributed noise is related to X. train The lengths are equal and the mean is 0 with a standard deviation of σ; in this embodiment, σ = 0.05.

[0085] Step S22: Multiply the original near-infrared spectrum with a normally distributed noise with a mean of 1 to obtain the scaled near-infrared spectrum.

[0086] Specifically, the formula for scaling and enhancing data is: Where g(δ) is a randomly generated normally distributed noise, and this normally distributed noise is related to X. train The lengths are equal and the mean is 1 with a standard deviation of δ; in this embodiment, δ = 0.05.

[0087] Step S23: Multiply the original near-infrared spectrum by the amplitude distortion function to obtain the amplitude-distorted near-infrared spectrum.

[0088] Specifically, the formula for amplitude distortion enhancement data is: Where h(γ, k) is a randomly generated normally distributed noise of length k with a mean of 1 and a standard deviation of γ; in this embodiment, γ = 0.05, k = 4; MagW(·) is the amplitude distortion function, which is generated from k random noise points using cubic spline interpolation and is related to X. trainRandom noise of equal length.

[0089] Step S24: Multiply the original near-infrared spectrum by the time-distortion function to obtain the time-distorted near-infrared spectrum.

[0090] Specifically, the formula for time-warped data enhancement is: Where h(γ, k) is a randomly generated normal distribution noise of length k with a mean of 1 and a standard deviation of γ, and TimeW(·) is a time warp function; in this embodiment, γ = 0.05 and k = 4.

[0091] The specific calculation method for TimeW(·) is as follows:

[0092]

[0093] Interp(.) is a piecewise linear interpolation function; Cumsum(·) is a summation function.

[0094] Step S25: Multiply the original near-infrared spectrum by the rotation function to obtain the rotated near-infrared spectrum.

[0095] Specifically, the formula for rotational enhancement data is: Where Rot(·) is the rotation function, α is a random number uniformly distributed in the range [-1, 1], representing the randomly generated rotation axis; θ is a random number uniformly distributed in the range [-π, π], representing the randomly generated rotation angle.

[0096] Step S26: After rearranging the multiple sub-interval spectra obtained by uniformly dividing the original near-infrared spectrum, the replaced near-infrared spectrum is obtained.

[0097] Specifically, the formula for permutation-enhanced data is: in, It is X train The sequence is divided into N uniform sub-intervals; k(1), k(2), ..., k(N) are sequences obtained by randomly shuffling the ordered sequence 1, 2, ..., N. Furthermore, N = 5.

[0098] Step S27: Set any one of the sub-interval spectra in the multiple sub-interval spectra to 0 to obtain the clipped near-infrared spectrum;

[0099] Specifically, the formula for cropping and enhancing data is: Where l is a randomly generated positive integer from the sequential sequence 1, 2, ..., N.

[0100] Step S28: Sequentially splice the jittered near-infrared spectrum, scaled near-infrared spectrum, amplitude-distorted near-infrared spectrum, time-distorted near-infrared spectrum, rotated near-infrared spectrum, replaced near-infrared spectrum, and cropped near-infrared spectrum to obtain the amplified near-infrared spectrum.

[0101] Specifically, regarding X train Data augmentation is performed on the raw near-infrared spectra (per row) of each sample in the X sample, i.e., X is augmented separately. train Seven data augmentation techniques were applied, including jitter, scaling, amplitude distortion, time distortion, rotation, displacement, and cropping, to obtain the amplified near-infrared spectrum. in, The matrix dimension is 350*401.

[0102] Step S3: Randomly mask the amplified near-infrared spectrum to obtain the masked near-infrared spectrum.

[0103] S3 specifically includes:

[0104] Step S31: Divide the amplified near-infrared spectrum into M sub-intervals; M is a positive integer greater than 0.

[0105] Specifically, the entire spectral range of the amplified near-infrared spectrum is divided into several sub-intervals; the division method includes uniform division or non-uniform division.

[0106] Furthermore, the entire spectral range is uniformly divided into 10 sub-intervals, each containing 40 wavelength points.

[0107] Step S32: Randomly generate M binary sequences.

[0108] Specifically, a binary sequence of the same length as the number of sub-intervals is randomly generated, and the elements in the sequence can only take the values ​​"1" and "0".

[0109] Furthermore, a random binary sequence of length 10 is generated, meaning that the elements in the sequence only contain the values ​​"1" and "0". Without loss of generality, a randomly generated binary sequence would be: [0, 1, 1, 0, 1, 1, 1, 0, 1, 0].

[0110] Step S33: Retain the spectrum of the sub-interval corresponding to "1" in the binary sequence and remove the spectrum of the sub-interval corresponding to "0" in the binary sequence to obtain the masked near-infrared spectrum.

[0111] Specifically, based on the positions of "1" and "0" in the binary sequence, the corresponding sub-interval spectra in the amplified near-infrared spectrum are retained and removed respectively to obtain the masked near-infrared spectrum.

[0112] Furthermore, the near-infrared spectrum of the mask is Since there are 6 "1" elements in the binary sequence, therefore The matrix dimension is 350*240.

[0113] Step S4: Construct an autoencoder model; the autoencoder model includes an encoder and a decoder.

[0114] Specifically, the autoencoder model consists of two parts: an encoder and a decoder. The topology of the encoder and decoder can be either a fully connected neural network or a convolutional neural network. The number of neurons in the input layer of the encoder is equal to the length of the masked near-infrared spectrum. The number of neurons in the output layer of the decoder is equal to the length of the original near-infrared spectrum. The output layer of the encoder is connected to the input layer of the decoder, meaning they contain the same number of neurons.

[0115] Furthermore, the topology of the autoencoder model M is constructed; such as... Figure 4 As shown, the autoencoder model M consists of two cascaded parts: encoder E and decoder D. Both encoder E and decoder D employ fully connected neural network topologies. The number of neurons in the input layer of encoder E is equal to the length of the masked near-infrared spectrum, i.e., 240; the number of neurons in the output layer of decoder D is equal to the length of the original near-infrared spectrum, i.e., 401. The output layer of encoder E is connected to the input layer of decoder D, containing the same number of neurons; furthermore, the total number of neurons is set to 10.

[0116] Step S5: Using the masked near-infrared spectrum as input and the original near-infrared spectrum as output, train the autoencoder model to obtain a trained autoencoder model; the trained autoencoder model includes a trained encoder and a trained decoder.

[0117] S5 specifically includes:

[0118] Step S51: Set the number of iterations; specifically, set the number of iterations to 100.

[0119] Step S52: Using the masked near-infrared spectrum as input and the original near-infrared spectrum as output, train the autoencoder model using the backpropagation algorithm; wherein, the backpropagation algorithm is SGD (stochastic gradient descent) or ADAM algorithm.

[0120] Specifically, will and X train These are used as the input and output of the autoencoder model M, respectively, and the model is trained using stochastic gradient descent.

[0121] Step S53: Stop training the autoencoder model after reaching the required number of iterations, thus obtaining the trained autoencoder model. Specifically, after training, a trained autoencoder model is obtained.

[0122] Step S6: Input the masked near-infrared spectrum into the trained encoder to obtain the near-infrared spectral abstract features.

[0123] Specifically, the masked near-infrared spectrum is input into the trained autoencoder model, and the output of each neuron is calculated at the output end of the encoder to extract the abstract features; the dimension of the abstract features is equal to the number of neurons in the encoder output layer.

[0124] Furthermore, Input into the trained autoencoder model The output of each neuron is calculated at the output of encoder E, and the near-infrared spectral abstract features are extracted. in, The matrix dimension is 350*10.

[0125] Step S7: Construct a near-infrared spectroscopy qualitative / quantitative analysis model.

[0126] Specifically, the qualitative / quantitative analysis model for near-infrared spectroscopy is a machine learning model or a deep learning model. The machine learning model includes backpropagation (BP) neural networks, support vector machines, decision trees, or random forests; the deep learning model is a convolutional neural network.

[0127] Furthermore, S is constructed using the BP neural network method, with a topology of 10-5-1, meaning that the input layer has 10 neurons, the hidden layer has 5 neurons, and the output layer has 1 neuron.

[0128] Step S8: Using the abstract features of near-infrared spectroscopy as input and the component content corresponding to the original near-infrared spectrum as output, train the near-infrared spectroscopy qualitative / quantitative analysis model to obtain the trained near-infrared spectroscopy qualitative / quantitative analysis model.

[0129] Specifically, will and Y train The inputs are respectively fed into the input and output terminals of S, and the model is trained using stochastic gradient descent to obtain a trained near-infrared spectroscopy qualitative / quantitative analysis model.

[0130] Furthermore, the verification set X test After the samples undergo steps S1-S6, the validation set X is obtained. test Abstract features of near-infrared spectroscopy Will Input into the trained near-infrared spectroscopy qualitative / quantitative analysis model The trained near-infrared spectroscopy qualitative / quantitative analysis model The trained near-infrared spectroscopy qualitative / quantitative analysis model was validated and obtained. The output results; the trained near-infrared spectroscopy qualitative / quantitative analysis model. The output result and the true value Y test For example Figure 5 As shown; coefficient of determination R 2 The result is 0.9272, indicating that the constructed model has good generalization performance.

[0131] Step S9: Input the abstract features of the near-infrared spectrum of the sample to be tested into the trained near-infrared spectral qualitative / quantitative analysis model to obtain the predicted values ​​of the component content of the sample to be tested; the abstract features of the near-infrared spectrum of the sample to be tested are obtained by inputting the near-infrared spectrum of the sample to be tested into the trained encoder.

[0132] Specifically, the near-infrared spectrum of the sample to be tested is input into the trained encoder to obtain the abstract features of the near-infrared spectrum of the sample to be tested; the abstract features of the near-infrared spectrum of the sample to be tested are input into the trained near-infrared spectral qualitative / quantitative analysis model to obtain the predicted values ​​of the component content of the sample to be tested.

[0133] like Figure 6 The near-infrared spectral modeling system based on a random mask autoencoder shown includes:

[0134] Acquisition module 1 is used to acquire the raw near-infrared spectrum and component content of the sample.

[0135] Data augmentation module 2 is used to augment the original near-infrared spectrum to obtain an amplified near-infrared spectrum.

[0136] Masking module 3 is used to randomly mask the amplified near-infrared spectrum to obtain a masked near-infrared spectrum.

[0137] The first building module 4 is used to build the autoencoder model; the autoencoder model includes an encoder and a decoder.

[0138] The first training module 5 is used to train the autoencoder model with the masked near-infrared spectrum as input and the original near-infrared spectrum as output, so as to obtain the trained autoencoder model; the trained autoencoder model includes a trained encoder and a trained decoder.

[0139] Abstract feature acquisition module 6 is used to input the masked near-infrared spectrum into the trained encoder to obtain near-infrared spectral abstract features.

[0140] The second building module 7 is used to build a near-infrared spectroscopy qualitative / quantitative analysis model; specifically, the near-infrared spectroscopy qualitative / quantitative analysis model is a machine learning model or a deep learning model.

[0141] The second training module 8 is used to train the near-infrared spectroscopy qualitative / quantitative analysis model by taking the abstract features of near-infrared spectroscopy as input and the component content corresponding to the original near-infrared spectrum as output, so as to obtain the trained near-infrared spectroscopy qualitative / quantitative analysis model.

[0142] Prediction module 9 is used to input the abstract features of the near-infrared spectrum of the sample to be tested into the trained near-infrared spectral qualitative / quantitative analysis model to obtain the predicted values ​​of the component content of the sample to be tested; the abstract features of the near-infrared spectrum of the sample to be tested are obtained by inputting the near-infrared spectrum of the sample to be tested into the trained encoder.

[0143] Data enhancement module 2 includes:

[0144] The jitter submodule is used to add the original near-infrared spectrum to a normally distributed noise with a mean of 0 to obtain a jittered near-infrared spectrum.

[0145] The scaling submodule is used to multiply the original near-infrared spectrum with a normally distributed noise with a mean of 1 to obtain a scaled near-infrared spectrum.

[0146] The amplitude distortion submodule is used to multiply the original near-infrared spectrum by the amplitude distortion function to obtain the amplitude-distorted near-infrared spectrum.

[0147] The time-warping submodule is used to multiply the original near-infrared spectrum by the time-warping function to obtain the time-warped near-infrared spectrum.

[0148] The rotation submodule is used to multiply the original near-infrared spectrum by a rotation function to obtain a rotated near-infrared spectrum.

[0149] The substitution submodule is used to rearrange the spectra of multiple sub-intervals obtained by uniformly dividing the original near-infrared spectrum to obtain the substituted near-infrared spectrum.

[0150] The cropping submodule is used to set any one of the multiple sub-interval spectra to 0 to obtain the cropped near-infrared spectrum.

[0151] The stitching submodule is used to sequentially stitch together jittered near-infrared spectra, scaled near-infrared spectra, amplitude-distorted near-infrared spectra, time-distorted near-infrared spectra, rotated near-infrared spectra, replaced near-infrared spectra, and cropped near-infrared spectra to obtain amplified near-infrared spectra.

[0152] Mask module 3 includes:

[0153] The sub-interval division module is used to divide the amplified near-infrared spectrum into M sub-intervals; M is a positive integer greater than 0.

[0154] The binary sequence generation submodule is used to randomly generate M binary sequences.

[0155] The masking submodule is used to retain the spectrum of the sub-interval corresponding to "1" in the binary sequence and remove the spectrum of the sub-interval corresponding to "0" in the binary sequence to obtain the masked near-infrared spectrum.

[0156] The first training module 5 includes:

[0157] The settings submodule is used to set the number of iterations.

[0158] The training submodule is used to train the autoencoder model by taking the masked near-infrared spectrum as input and the original near-infrared spectrum as output, and employing the error backpropagation algorithm.

[0159] The completion submodule is used to stop training the autoencoder model after reaching the required number of iterations, thus obtaining the trained autoencoder model.

[0160] This invention generates binary sequences through a masking strategy, corresponding to various wavelengths across the entire band. Each binary sequence represents a possible permutation and combination of wavelengths. Based on this permutation and combination, abstract features are extracted and a model is established. This allows for the construction of a relationship between the permutation and combination of wavelengths and a qualitative / quantitative analysis model (such as predicting gasoline component content). Furthermore, the performance of different masks (different binary sequences, different wavelength permutations and combinations, different models) can be compared to determine which mask (binary sequence, wavelength permutation and combination) is superior. By establishing a mapping relationship between the abstract features extracted by the autoencoder and the characteristic wavelength spectra of component content through a random masking strategy, the invention enables the analysis of component content based on near-infrared spectroscopy and provides the relationship between the abstract features extracted from the original spectrum and the component content.

[0161] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0162] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A near-infrared spectral modeling method based on a random mask autoencoder, characterized in that, The method includes: Obtain the raw near-infrared spectrum and component content of the sample; The original near-infrared spectrum is augmented to obtain an amplified near-infrared spectrum; The amplified near-infrared spectrum is randomly masked to obtain a masked near-infrared spectrum, specifically including: The amplified near-infrared spectrum is divided into M sub-intervals; M is a positive integer greater than 0; Randomly generate M binary sequences; The spectrum of the sub-interval corresponding to "1" in the binary sequence is retained, and the spectrum of the sub-interval corresponding to "0" in the binary sequence is removed to obtain the masked near-infrared spectrum; Construct an autoencoder model; the autoencoder model includes an encoder and a decoder; Using the masked near-infrared spectrum as input and the original near-infrared spectrum as output, the autoencoder model is trained to obtain a trained autoencoder model, specifically including: Set the number of iterations; Using the masked near-infrared spectrum as input and the original near-infrared spectrum as output, the autoencoder model is trained using an error backpropagation algorithm; wherein the error backpropagation algorithm is either SGD or ADAM algorithm. The training of the autoencoder model stops after reaching the number of iterations, and the trained autoencoder model is obtained. The trained autoencoder model includes a trained encoder and a trained decoder; The masked near-infrared spectrum is input into the trained encoder to obtain near-infrared spectral abstract features; Construct a qualitative / quantitative analysis model for near-infrared spectroscopy; Using the abstract features of the near-infrared spectrum as input and the component content corresponding to the original near-infrared spectrum as output, the near-infrared spectral qualitative / quantitative analysis model is trained to obtain a trained near-infrared spectral qualitative / quantitative analysis model. The near-infrared spectral abstract features of the sample to be tested are input into the trained near-infrared spectral qualitative / quantitative analysis model to obtain the predicted values ​​of the component content of the sample to be tested; the near-infrared spectral abstract features of the sample to be tested are obtained by inputting the near-infrared spectrum of the sample to be tested into the trained encoder.

2. The near-infrared spectral modeling method based on a random mask autoencoder according to claim 1, characterized in that, The process of data enhancement of the original near-infrared spectrum to obtain the amplified near-infrared spectrum specifically includes: The original near-infrared spectrum is added to a normally distributed noise with a mean of 0 to obtain a jittery near-infrared spectrum; The original near-infrared spectrum is multiplied by a normally distributed noise with a mean of 1 to obtain a scaled near-infrared spectrum; Multiplying the original near-infrared spectrum by the amplitude distortion function yields the amplitude-distorted near-infrared spectrum; Multiplying the original near-infrared spectrum by the time-distortion function yields the time-distorted near-infrared spectrum; Multiplying the original near-infrared spectrum by the rotation function yields the rotated near-infrared spectrum; After the original near-infrared spectrum is uniformly divided into multiple sub-intervals, the spectrum is rearranged to obtain the replaced near-infrared spectrum; By setting any one of the multiple sub-interval spectra to 0, a clipped near-infrared spectrum is obtained. The amplified near-infrared spectrum is obtained by sequentially splicing the jittered near-infrared spectrum, the scaled near-infrared spectrum, the amplitude-distorted near-infrared spectrum, the time-distorted near-infrared spectrum, the rotated near-infrared spectrum, the displacement near-infrared spectrum, and the cropped near-infrared spectrum.

3. The near-infrared spectral modeling method based on a random mask autoencoder according to claim 1, characterized in that, The near-infrared spectroscopy qualitative / quantitative analysis model is a machine learning model or a deep learning model.

4. A near-infrared spectral modeling system based on a random mask autoencoder, characterized in that, The system includes: The acquisition module is used to acquire the raw near-infrared spectrum and component content of the sample; The data enhancement module is used to enhance the original near-infrared spectrum to obtain an amplified near-infrared spectrum; A masking module is used to randomly mask the amplified near-infrared spectrum to obtain a masked near-infrared spectrum; the masking module includes: The sub-interval division module is used to divide the amplified near-infrared spectrum into M sub-intervals; M is a positive integer greater than 0; The binary sequence generation submodule is used to randomly generate M binary sequences; The masking submodule is used to retain the spectrum of the sub-interval corresponding to "1" in the binary sequence and remove the spectrum of the sub-interval corresponding to "0" in the binary sequence to obtain the masked near-infrared spectrum. The first building module is used to build an autoencoder model; the autoencoder model includes an encoder and a decoder. The first training module is used to train the autoencoder model with the masked near-infrared spectrum as input and the original near-infrared spectrum as output to obtain a trained autoencoder model; the trained autoencoder model includes a trained encoder and a trained decoder. The first training module includes: The settings submodule is used to set the number of iterations; The training submodule is used to train the autoencoder model by taking the masked near-infrared spectrum as input and the original near-infrared spectrum as output, and employing an error backpropagation algorithm; wherein the error backpropagation algorithm is either SGD or ADAM algorithm. The completion submodule is used to stop training the autoencoder model after the number of iterations has been reached, thus obtaining the trained autoencoder model. The abstract feature acquisition module is used to input the masked near-infrared spectrum into the trained encoder to obtain near-infrared spectral abstract features. The second building module is used to construct near-infrared spectroscopy qualitative / quantitative analysis models; The second training module is used to train the near-infrared spectral qualitative / quantitative analysis model by taking the abstract features of the near-infrared spectrum as input and the component content corresponding to the original near-infrared spectrum as output, so as to obtain the trained near-infrared spectral qualitative / quantitative analysis model. The prediction module is used to input the abstract features of the near-infrared spectrum of the sample to be tested into the trained near-infrared spectral qualitative / quantitative analysis model to obtain the predicted values ​​of the component content of the sample to be tested; the abstract features of the near-infrared spectrum of the sample to be tested are obtained by inputting the near-infrared spectrum of the sample to be tested into the trained encoder.

5. The near-infrared spectral modeling system based on a random mask autoencoder according to claim 4, characterized in that, The data enhancement module includes: The jitter submodule is used to add the original near-infrared spectrum to a normally distributed noise with a mean of 0 to obtain a jittered near-infrared spectrum; The scaling submodule is used to multiply the original near-infrared spectrum with a normal distribution noise with a mean of 1 to obtain a scaled near-infrared spectrum; The amplitude distortion submodule is used to multiply the original near-infrared spectrum by the amplitude distortion function to obtain the amplitude-distorted near-infrared spectrum; The time-warping submodule is used to multiply the original near-infrared spectrum by the time-warping function to obtain the time-warped near-infrared spectrum; The rotation submodule is used to multiply the original near-infrared spectrum by a rotation function to obtain a rotated near-infrared spectrum; The substitution submodule is used to rearrange the multiple sub-interval spectra obtained by uniformly dividing the original near-infrared spectrum to obtain the substitution near-infrared spectrum; The cropping submodule is used to set any one of the multiple sub-interval spectra to 0 to obtain a cropped near-infrared spectrum. The splicing submodule is used to sequentially splice the jittered near-infrared spectrum, the scaled near-infrared spectrum, the amplitude-distorted near-infrared spectrum, the time-distorted near-infrared spectrum, the rotated near-infrared spectrum, the displacement near-infrared spectrum, and the cropped near-infrared spectrum to obtain the amplified near-infrared spectrum.

6. The near-infrared spectral modeling system based on a random mask autoencoder according to claim 4, characterized in that, The near-infrared spectroscopy qualitative / quantitative analysis model is a machine learning model or a deep learning model.

Images