Training method and related equipment for spectral recognition model

Abstract

The present invention relates to a training method for a spectral recognition model and related equipment, comprising the following steps: acquiring spectral data and cleaning it to obtain cleaned spectral data; sparsely encoding the cleaned spectral data to obtain sparsely represented spectral features; using the sparsely represented spectral features to train a preset convolutional neural network to obtain an initial spectral recognition model; if target spectral data is different from the data type in a database, transfer learning is performed on the initial model based on the target spectral data to obtain a transferred spectral recognition model; cross-validating the transferred spectral recognition model to obtain a verification value; if the verification value is not within a preset range, tuning a hyperparameter through a Bayesian optimization algorithm until the verification value is within the preset range, and finally obtaining a target spectral recognition model; the technical problem that traditional grid search and random search methods are inefficient in high-dimensional space and difficult to find the optimal hyperparameter combination is solved.

Description

Technical Field

[0001] The present invention relates to the field of spectroscopy, and in particular to a training method for a spectral recognition model and related equipment. Background Art

[0002] Spectral recognition technology has important applications in many fields, including environmental monitoring, medicine, materials science, and astronomy. However, traditional spectral recognition models often rely on manual feature extraction and empirical formulas, which is not only time-consuming and labor-intensive, but also difficult to cope with complex and changeable spectral data. Therefore, spectral recognition models based on deep learning have gradually become a research hotspot. Deep learning models, especially convolutional neural networks (CNNs), have achieved remarkable success in fields such as image and speech recognition. Applying them to spectral recognition can improve the accuracy and robustness of the model.

[0003] However, the application of deep learning models in spectral recognition still faces multiple challenges. First, spectral data is usually high-dimensional and sparse. Directly using it for model training may lead to high computational complexity, long training time, and overfitting problems. Second, there are significant differences between different types of spectral data. Directly using a model may not be able to effectively generalize to different types of spectral data, resulting in unstable model performance. In addition, the selection of model hyperparameters has a significant impact on the final performance, but traditional grid search and random search methods are inefficient in high-dimensional space, and it is difficult to find the optimal hyperparameter combination.

[0004] Therefore, it is particularly important to propose an effective spectral recognition model training method to address the above problems. By cleaning and sparsely encoding spectral data, the dimension and sparsity of the data can be reduced, and the training efficiency and effect of the model can be improved. Using transfer learning technology, the pre-trained model can be applied to different types of spectral data, thereby improving the generalization ability and stability of the model. Finally, by tuning the hyperparameters of the model through the Bayesian optimization algorithm, the optimal hyperparameter combination can be found in a relatively short time, further improving the performance of the model. Summary of the invention

[0005] The main purpose of the present invention is to provide a training method and related equipment for a spectral recognition model, which solves the technical problem that traditional grid search and random search methods are inefficient in high-dimensional space and difficult to find the optimal hyperparameter combination.

[0006] To achieve the above object, the present invention provides a method for training a spectrum recognition model, comprising the following steps: Acquiring spectral data in a spectral database, and cleaning the spectral data to obtain cleaned spectral data; Performing sparse coding on the cleaned spectral data to obtain sparse representation spectral features; The preset convolutional neural network is trained by using the sparse representation spectral features to obtain an initial spectral recognition model; When receiving target spectrum data, if the target spectrum data is of a different data type from the spectrum data in the spectrum database, performing transfer learning training on the initial spectrum recognition model based on the target spectrum data to obtain a transfer spectrum recognition model; Cross-validating the migration spectrum recognition model to obtain a validation value corresponding to the migration spectrum recognition model; wherein the validation value is used to evaluate the performance of the migration spectrum recognition model; If the verification value is not within the preset range, the hyperparameters of the migration spectrum recognition model are tuned by a preset Bayesian optimization algorithm until the verification value of the migration spectrum recognition model is within the preset range to obtain a target spectrum recognition model.

[0007] Further, the spectral data is cleaned to obtain cleaned spectral data, including: Performing adaptive median filtering and wavelet threshold denoising on the spectral data to obtain denoised spectral data; Performing outlier detection and elimination on the denoised spectral data to obtain outlier-eliminated spectral data; Using a cubic spline interpolation method, nonlinear interpolation and spectrum correction are performed on the outlier-removed spectrum data to obtain corrected spectrum data; Performing standard normal transformation and multivariate scattering correction on the corrected spectral data to obtain preprocessed spectral data; Performing continuous wavelet transform and singular value decomposition on the preprocessed spectral data to obtain decomposed spectral data; The decomposed spectral data is cleaned to obtain cleaned spectral data.

[0008] Furthermore, the cleaned spectral data is sparsely encoded to obtain sparse representation spectral features, including: Embed the cleaned spectral data into a high-dimensional feature space to obtain a high-dimensional feature representation; Using a sparse autoencoder to sparsely encode the high-dimensional feature representation to obtain an initial sparse representation; Applying a sparse coding network to optimize the initial sparse representation to obtain an optimized sparse representation; Regularizing the optimized sparse representation to obtain a regularized sparse representation; Optimizing and solving the regularized sparse representation to obtain a global optimal sparse representation; Multi-scale integration and feature extraction are performed on the global optimal sparse representation to obtain sparse representation spectral features.

[0009] Furthermore, the preset convolutional neural network is trained by the sparse representation spectral feature to obtain an initial spectral recognition model, including: Performing data enhancement processing on the sparsely represented spectral features to obtain enhanced spectral feature data; wherein the data enhancement processing includes random noise addition, spectrum shift, spectrum stretching and compression transformation; Inputting the enhanced spectral feature data into a preset convolutional neural network, wherein the convolutional neural network includes a plurality of convolutional layers, a pooling layer and a fully connected layer; Performing multi-scale feature extraction on the enhanced spectral feature data through the convolution layers of different sizes and numbers to obtain a multi-scale convolution feature map; Performing a maximum pooling or an average pooling operation on the multi-scale convolutional feature map through the pooling layer to obtain a reduced-dimensional feature representation; Performing full connection processing on the dimension reduction feature representation through the full connection layer to obtain fully connected spectral feature data; The fully connected spectral feature data of the output of the fully connected layer is normalized using a softmax function to obtain spectral prediction values ​​of each category; wherein the spectra of each category include organic spectra, inorganic spectra, metal spectra, non-metal spectra, gas spectra and liquid spectra; Obtaining preset actual labels corresponding to the spectra of each category; wherein the actual labels include target values ​​corresponding to the spectra of each category; calculating a loss value between the target value and the predicted value of the spectrum of each category; Calculating the gradient of each network parameter in the convolutional neural network based on the loss value pair through a preset back propagation algorithm; Using a preset Adam optimizer, the weight parameters of the convolutional neural network are updated based on the gradients of the network parameters; The convolutional neural network is updated based on the updated weight parameters of the convolutional neural network to obtain an initial spectrum recognition model.

[0010] Furthermore, the initial spectrum recognition model is subjected to transfer learning training based on the target spectrum data to obtain a transfer spectrum recognition model, including: Selectively freezing the pre-trained weights of the initial spectral recognition model to obtain a partially frozen weight model; wherein the selective freezing includes freezing the weight parameters of all layers except the last layer of the initial spectral recognition model to retain the basic features learned by the initial spectral recognition model, and the last layer is a fully connected layer; Obtaining a target fully connected layer, and replacing the last fully connected layer in the partially frozen weight model with the target fully connected layer to obtain a model adapted to the new task; Using a mini-batch gradient descent method, fine-tuning the target fully connected layer newly added to the model adapted to the new task based on the target spectral data to obtain a fine-tuned target fully connected layer; Adaptively adjusting the learning rate of the fine-tuning target fully connected layer by using a cosine annealing learning rate strategy to obtain an adjusted target fully connected layer; Gradually unfreeze some convolutional layers, and jointly optimize the unfrozen parts of the convolutional layers and the adjusted target fully connected layers through an adaptive moment estimation optimizer to obtain a jointly optimized layer; The joint optimization layer is subjected to integrated parameter learning and parameter fusion processing to obtain a migration spectrum recognition model.

[0011] Further, the target spectrum data includes a specified training data set, and the cross-validation of the migration spectrum recognition model to obtain a validation value corresponding to the migration spectrum recognition model includes: Step S61, dividing the designated training data set of the migration spectrum recognition model into k mutually exclusive subsets; Step S62, dividing the subset to obtain a validation set and k-1 training sets; wherein the validation set is used to evaluate the performance of the migration spectrum recognition model, and the training set is used to train the migration spectrum recognition model; Step S63, fine-tuning the migration spectrum recognition model using a training set to obtain a fine-tuned migration spectrum recognition model; Step S64, using the verification set to evaluate the fine-tuned migration spectrum recognition model to obtain a corresponding evaluation value; Step S65, obtaining a target evaluation value corresponding to the migration spectrum recognition model, and obtaining a first verification value based on the target evaluation value and the evaluation value; Step S66, repeat the above steps S62 to S65 k times, each time selecting a different subset as the verification set, to obtain k verification values; In step S67, an average value of k verification values ​​is calculated, and the average value is used as the verification value corresponding to the migration spectrum recognition model.

[0012] Further, the hyperparameters of the migration spectrum recognition model are tuned by a preset Bayesian optimization algorithm until the verification value of the migration spectrum recognition model is within a preset range to obtain a target spectrum recognition model, including: The hyperparameter space of the migration spectrum recognition model is modeled by Gaussian process regression to obtain the hyperparameter prior distribution; Performing Monte Carlo sampling on the hyperparameter prior distribution to obtain a set of candidate hyperparameters; Performing parameter evaluation on the candidate hyperparameter set to obtain an optimal hyperparameter combination; Based on the optimal hyperparameter combination, reconfiguring the parameters of the migration spectrum recognition model to obtain an optimized migration spectrum recognition model; Refining the hyperparameters of the optimized migration spectrum recognition model by using a tree-shaped Parzen estimator to obtain a fine-tuned hyperparameter set; The migration spectrum recognition model is iteratively tuned based on the fine-tuning hyperparameter set by using a preset Bayesian optimization algorithm until the verification value of the migration spectrum recognition model is within a preset range, so as to obtain a target spectrum recognition model.

[0013] The present invention also provides a training device for a spectrum recognition model, comprising: An acquisition module, used for acquiring spectral data in a spectral database, and cleaning the spectral data to obtain cleaned spectral data; An encoding module, used for performing sparse encoding on the cleaned spectral data to obtain sparse representation spectral features; A training module, used to perform model training on a preset convolutional neural network through the sparse representation spectral features to obtain an initial spectral recognition model; a learning module, configured to, upon receiving target spectral data, perform transfer learning training on the initial spectral recognition model based on the target spectral data to obtain a transfer spectral recognition model if the target spectral data is of a different data type from the spectral data in the spectral database; A verification module, used to cross-validate the migration spectrum recognition model to obtain a verification value corresponding to the migration spectrum recognition model; wherein the verification value is used to evaluate the performance of the migration spectrum recognition model; The tuning module is used to tune the hyperparameters of the migration spectrum recognition model by a preset Bayesian optimization algorithm if the verification value is not within a preset range, until the verification value of the migration spectrum recognition model is within a preset range, so as to obtain a target spectrum recognition model.

[0014] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0015] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0016] The training method of the spectrum recognition model provided by the present invention comprises the following steps: obtaining spectrum data in a spectrum database, and cleaning the spectrum data to obtain cleaned spectrum data; sparsely encoding the cleaned spectrum data to obtain sparsely represented spectrum features; training a preset convolutional neural network model through the sparsely represented spectrum features to obtain an initial spectrum recognition model; when receiving target spectrum data, if the target spectrum data is different from the spectrum data in the spectrum database, then performing transfer learning training on the initial spectrum recognition model based on the target spectrum data to obtain a migration spectrum recognition model; cross-validating the migration spectrum recognition model to obtain a verification value corresponding to the migration spectrum recognition model; if the verification value is not within a preset range, then tuning the hyperparameters of the migration spectrum recognition model through a preset Bayesian optimization algorithm until the verification value of the migration spectrum recognition model is within a preset range to obtain the target spectrum recognition model; through the above technical means, the traditional grid search and random search methods are inefficient in high-dimensional space and it is difficult to find the optimal hyperparameter combination technical problem. When the target spectrum data is different from the data type in the original spectrum database, the initial model is retrained using the transfer learning technology. This can effectively utilize the knowledge of existing models, improve recognition performance on new data, and reduce the time and resource consumption of retraining. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic diagram of the steps of a method for training a spectrum recognition model in one embodiment of the present invention; Figure 2 is a structural block diagram of a training device for a spectrum recognition model in one embodiment of the present invention; Figure 3 It is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0018] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0020] like Figure 1 As shown, Figure 1 Schematic diagram of the training method steps of the spectrum recognition model in one embodiment of the present invention; In one embodiment of the present invention, a method for training a spectrum recognition model is provided, comprising the following steps: Acquiring spectral data in a spectral database, and cleaning the spectral data to obtain cleaned spectral data; Step S1, sparsely encode the cleaned spectral data to obtain sparsely represented spectral features.

[0021] Specifically, the process of sparsely encoding the cleaned spectral data to obtain sparse representation of spectral features, first of all, needs to be understood that this process is a technical means adopted in a specific application scenario, such as in the fields of materials science or environmental monitoring, in order to improve data processing efficiency and model prediction accuracy. In this process, the original spectral data collected first often contains a large amount of noise and redundant information, which not only increases the computational burden, but also may interfere with subsequent data analysis. Therefore, the first step is to preprocess these original data, namely the so-called "cleaning", by removing noise, filling missing values ​​and other operations to ensure the quality of the data, and lay a good foundation for the next step of sparse coding. Next, the goal of sparse coding is to extract the most representative features from the cleaned spectral data. The core of this step is to find a method that can represent the original data with as few non-zero elements as possible, namely the so-called "sparse representation". In actual operation, this can be achieved by solving optimization problems, such as using the L1 regularization method, which tends to produce sparse solutions, that is, the weights of most features are close to zero, and only a few features have significant non-zero weights, thereby achieving data compression while retaining key information. Taking materials science as an example, when researchers need to identify the composition of a specific material from a large amount of spectral data, sparse encoding of the cleaned spectral data can effectively highlight the spectral features that are closely related to the properties of the target material, making subsequent classification or regression analysis more accurate and efficient. This technology not only improves the speed of data analysis, but also enhances the generalization ability of the model, making it perform better when faced with unseen data. In summary, sparse encoding of cleaned spectral data to obtain sparse representation of spectral features is an important means to improve the quality and efficiency of data processing, especially for scenarios where useful information needs to be extracted quickly and accurately from large amounts of complex data.

[0022] Step S2, training a preset convolutional neural network model by using the sparse representation spectral features to obtain an initial spectral recognition model.

[0023] Specifically, the process of training the preset convolutional neural network model through the sparse representation of spectral features to obtain the initial spectral recognition model is based on the previous cleaning of spectral data and sparse coding. The main purpose of this stage is to use the spectral features obtained after sparse coding to build a model that can effectively identify and classify spectral data through deep learning technology, especially convolutional neural network (CNN). Convolutional neural networks perform well in processing images, sounds, and sequence data such as spectra due to their powerful feature extraction capabilities and sensitivity to local patterns. Specifically, in the field of materials science, when you have a set of spectral feature data sets that have been processed by sparse coding, you can use it as input and feed it to a pre-designed convolutional neural network model for training. During the training process, the convolution layer automatically learns how to extract more valuable information from the input spectral features, such as absorption peaks or emission lines unique to different materials, while the pooling layer is responsible for reducing the data dimension, helping the model to better focus on important features while reducing the risk of overfitting. In addition, the fully connected layer is used to integrate all the extracted features and finally output the prediction results, such as material type or purity grade. In order to ensure the accuracy and generalization ability of the model, cross-validation and other techniques are also needed to evaluate the model performance during the training process, and adjust the network structure or parameter settings based on feedback. For example, in environmental monitoring applications, if the goal is to identify the presence or absence of specific pollutants in the air and their concentration levels or a material (wood or paper scraps, etc.) through spectral data, then the spectral features after sparse coding will be used as input data. After training with a carefully designed convolutional neural network, an initial spectral recognition model can be generated. This model can not only quickly and accurately identify the target pollutants, but also maintain a high prediction accuracy when facing new samples. In short, model training of the preset convolutional neural network through sparse representation of spectral features is one of the key steps to achieve efficient and accurate spectral data recognition. It combines the efficient feature extraction of sparse coding with the powerful learning ability of convolutional neural networks, providing strong support for solving complex data analysis problems.

[0024] Step S3: when receiving the target spectrum data, if the target spectrum data is of a different data type from the spectrum data in the spectrum database, transfer learning training is performed on the initial spectrum recognition model based on the target spectrum data to obtain a transfer spectrum recognition model.

[0025] Specifically, when receiving the target spectral data, if the target spectral data is of a different data type from the spectral data in the spectral database, the initial spectral recognition model is trained through transfer learning based on the target spectral data to obtain the transfer spectral recognition model. This process is to cope with the differences between different data types and ensure the applicability and accuracy of the model in the new environment. In practical applications, such as in the field of materials science, such a situation may be encountered: an initial spectral recognition model for a certain type of material has been obtained by cleaning the spectral data and sparse coding, but when another type of material needs to be identified, the spectral characteristics of the two types of materials are significantly different, and the direct use of the original model may result in poor recognition results. At this time, it is necessary to use the technology of transfer learning to enable the model to adapt to the new data type. Specifically, the core idea of ​​transfer learning is to use the knowledge learned in the source domain (i.e., the existing spectral data) to help the learning process of the target domain (i.e., the newly received target spectral data). First, the characteristics of the target spectral data, including but not limited to the wavelength range, resolution, etc., are analyzed, and compared with the data in the spectral database to confirm the differences between the two. Then, based on these differences, a suitable transfer learning strategy is selected. For example, a fine-tuning method can be used, that is, while keeping most of the parameters of the initial spectral recognition model unchanged, only the last layer or several layers are retrained to adapt to the characteristics of the target spectral data. This can not only make full use of the generalization ability of the existing model, but also avoid the problem of overfitting to a certain extent. Taking pollutant detection in environmental monitoring as an example, it is assumed that the initial model is trained based on spectral data in the visible light region, which is specifically used to identify certain organic pollutants in the air. However, when it is necessary to expand to the detection of heavy metal ions or non-metallic substances, since the latter usually need to be measured in the ultraviolet or infrared light region, the spectral data of the two are essentially different. In this case, through transfer learning, the last few layers of the initial model can be adjusted first, so that the model can learn the new features of the heavy metal ion spectrum. At the same time, domain adaptation technology can also be introduced to further narrow the gap between the source domain and the target domain and improve the performance of the model on new tasks. After such transfer learning training, a migration spectral recognition model specifically for the target spectral data can be obtained. The model not only inherits the advantages of the original model, but also has the ability to process new types of data, thereby greatly expanding the scope of application and flexibility of the model. In summary, by performing transfer learning training on the initial spectral recognition model based on the target spectral data, the challenges brought by data type differences can be effectively addressed, so that the model can still maintain high accuracy and high efficiency when facing different types of tasks. This method is of great significance for promoting cross-domain spectral data analysis and application.

[0026] Step S4, cross-validating the migration spectrum recognition model to obtain a validation value corresponding to the migration spectrum recognition model; wherein the validation value is used to evaluate the performance of the migration spectrum recognition model.

[0027] Specifically, the migration spectrum recognition model is cross-validated to obtain a validation value corresponding to the migration spectrum recognition model; wherein the validation value is used to evaluate the performance of the migration spectrum recognition model, which is a key step to ensure the reliability and stability of the model in practical applications. In the analysis of spectral data in fields such as environmental monitoring or materials science, although the model obtained by transfer learning training has already possessed preliminary recognition capabilities, its performance on unseen data still needs to be determined through rigorous validation. Cross-validation is a widely used model evaluation method, which divides the data set into multiple subsets, takes each subset as a test set in turn, and the remaining subsets as training sets, repeatedly trains and tests the model multiple times, and finally summarizes the results of each test to evaluate the overall performance of the model. This method can not only effectively avoid accidental deviations caused by a single data division method, but also provide more comprehensive information about the generalization ability of the model. In specific implementation, first, a validation data set containing target spectral data needs to be prepared. This data set should cover as many possible situations as possible in the target field to ensure the representativeness of the validation results. Then, this data set is randomly divided into several non-overlapping subsets according to a certain ratio (such as the common 5-fold or 10-fold cross-validation). Next, the cross-validation process begins: each time one of the subsets is selected as the test set, and the remaining subsets are combined into the training set. The training set is used to train the migration spectrum recognition model, and then predictions are made on the test set, and the prediction results are recorded. This process needs to be repeated until each subset has at least one chance to be used as a test set. Finally, based on the prediction results on all test sets, the average performance indicators of the model, such as accuracy, recall, F1 score, etc., are calculated, and these indicators together constitute the "validation value" described. Taking the detection of heavy metal ions or heavy metal ion subclasses (lead, iron, etc.) in environmental monitoring as an example, it is assumed that a spectral recognition model specifically used to identify heavy metal ion subclasses has been obtained through transfer learning training. In order to evaluate the performance of this model, a set of spectral data containing heavy metal ion subclasses of different types or / and concentrations can be collected from actual monitoring points as a validation data set. Next, a 10-fold cross-validation is performed on the data set, and 1 / 10 of the data is selected as the test set each time, and the remaining 9 / 10 is used as the training set. In this way, the model will be trained and tested on different data combinations, so as to more realistically reflect its performance in actual applications. After multiple rounds of validation, the prediction results on all test sets are summarized to calculate the model's average accuracy, recall, F1 score and other performance indicators. If these indicators meet the expected standards, it means that the model has good generalization ability and stability and can be put into use in a practical environment; otherwise, it is necessary to further optimize the model structure or adjust the training parameters until the performance requirements are met. In short, cross-validation of the migration spectrum recognition model to obtain the validation value is an important means to ensure that the model can maintain good performance when facing unknown data.Through this process, not only can the generalization ability of the model be evaluated, but potential problems can also be discovered, providing a basis for further optimization of the model. This is of great significance for promoting the application of spectral analysis technology in environmental monitoring, materials science and other fields.

[0028] Step S5, if the verification value is not within a preset range, the hyperparameters of the migration spectrum recognition model are tuned by a preset Bayesian optimization algorithm until the verification value of the migration spectrum recognition model is within a preset range to obtain a target spectrum recognition model.

[0029] Specifically, if the verification value is not within the preset range, the hyperparameters of the migration spectrum recognition model are tuned by a preset Bayesian optimization algorithm until the verification value of the migration spectrum recognition model is within the preset range to obtain the target spectrum recognition model. This process is an important part of model optimization, which aims to ensure that the model achieves optimal performance in practical applications. In spectral data analysis in fields such as environmental monitoring or material science, even if a preliminary model is obtained through transfer learning training, if its performance fails to meet the expected standards, it is necessary to further improve the performance of the model through hyperparameter tuning. Specifically, when the verification value (such as accuracy, recall rate, F1 score, etc.) obtained by cross-validation is lower than the preset standard, it indicates that the model's generalization ability on unseen data is insufficient, and the hyperparameters of the model need to be adjusted. Hyperparameters refer to those parameters that are not directly optimized by the learning algorithm during the training process, but are set in advance by the user, such as learning rate, batch size, number of network layers, etc. The selection of these parameters has an important impact on the final performance of the model. Although traditional hyperparameter tuning methods such as grid search or random search are simple and intuitive, they are inefficient in high-dimensional space and difficult to find the optimal solution. In contrast, the Bayesian optimization algorithm predicts the model performance under different hyperparameter combinations by building a probabilistic model, and on this basis selects the hyperparameters that are most likely to improve performance for trial, thereby achieving efficient optimization. In actual operation, first define an objective function that takes hyperparameters as input and outputs the performance indicators of the model on the validation set. Then, initialize a Bayesian optimizer, specify the optimized objective function, the search space of hyperparameters, and the optimization strategy. Next, start the optimization process, and the optimizer will dynamically adjust the selection of hyperparameters based on historical test results, gradually approaching the optimal solution. After each round of optimization, the model will be retrained and verified until the verification value reaches the preset range. For example, in the application of different heavy metal ions or heavy metal ion sub-category detection in environmental monitoring, if the average accuracy of the model obtained by 10-fold cross validation is only 80%, which is lower than the expected 90%, then the Bayesian optimization algorithm can be started to tune the hyperparameters. The optimizer may try to adjust hyperparameters such as learning rate, batch size, number of convolution kernels, etc. After each adjustment, the model will be retrained and the performance will be evaluated on the validation set. As the optimization proceeds, the performance of the model gradually improves. Finally, when the validation value reaches or exceeds 90%, the optimization process ends and the target spectral recognition model obtained can be put into use in the actual environment. This process can not only significantly improve the performance of the model, but also save a lot of time and computing resources. Through the Bayesian optimization algorithm, a set of optimal or near-optimal hyperparameter configurations can be found in a relatively short period of time, so that the model can maintain high accuracy and high stability when processing new spectral data. This is especially important for application scenarios with high real-time requirements such as environmental monitoring.In general, tuning the hyperparameters of the migration spectrum recognition model through the preset Bayesian optimization algorithm until the verification value is within the preset range is a key step to ensure the excellent performance of the model in practical applications, which will help promote the in-depth application and development of spectral analysis technology in various fields.

[0030] In a specific embodiment, cleaning the spectral data to obtain cleaned spectral data includes: Performing adaptive median filtering and wavelet threshold denoising on the spectral data to obtain denoised spectral data; Performing outlier detection and elimination on the denoised spectral data to obtain outlier-eliminated spectral data; Using a cubic spline interpolation method, nonlinear interpolation and spectrum correction are performed on the outlier-removed spectrum data to obtain corrected spectrum data; Performing standard normal transformation and multivariate scattering correction on the corrected spectral data to obtain preprocessed spectral data; Performing continuous wavelet transform and singular value decomposition on the preprocessed spectral data to obtain decomposed spectral data; The decomposed spectral data is cleaned to obtain cleaned spectral data.

[0031] Specifically, the process of cleaning the spectral data to obtain cleaned spectral data is a systematic, multi-step preprocessing process, which aims to eliminate noise, correct outliers, correct spectral curves, and improve the quality and availability of data through a series of mathematical transformations. This process is crucial to ensure the effectiveness of subsequent sparse coding, model training and other steps. The specific implementation method of this process will be introduced in detail below, and its significance in practical applications will be explained by taking the detection of air pollutants in environmental monitoring as an example. First, the spectral data is subjected to adaptive median filtering and wavelet threshold denoising to obtain denoised spectral data. Spectral data is often affected by various factors during the acquisition process, such as instrument noise, environmental interference, etc., resulting in different degrees of noise in the data. Adaptive median filtering is an effective denoising method, which can adaptively adjust the size of the filter window according to the change of the local signal, thereby removing impulse noise while retaining edge information. Wavelet threshold denoising is to decompose the signal into different frequency scales, remove high-frequency noise according to the set threshold, and finally reconstruct the signal. This method is particularly suitable for processing spectral data containing complex background noise. For example, in the detection of air pollutants, the original spectral data may contain more random noise due to the thermal noise of the instrument or the electromagnetic interference in the environment. Through adaptive median filtering and wavelet threshold denoising, the influence of these noises can be significantly reduced and the signal-to-noise ratio of the data can be improved. Then, the denoised spectral data is subjected to outlier detection and elimination to obtain outlier-eliminated spectral data. Outliers refer to data points that are significantly deviated from the normal range. They may be caused by equipment failure, human error, etc. The existence of outliers will cause serious interference to subsequent data analysis and modeling, so they must be eliminated. Commonly used outlier detection methods include statistical-based methods (such as Z-score, IQR), clustering-based methods (such as DBSCAN), etc. In the detection of air pollutants, if a spectral data point is significantly higher or lower than other data points and cannot be explained by reasonable physical or chemical reasons, it can be judged as an outlier. Through outlier detection and elimination, it can be ensured that there is no obvious erroneous data in the data set, laying the foundation for subsequent processing. Subsequently, the cubic spline interpolation method is used to perform nonlinear interpolation and spectral correction on the spectral data after outliers are removed to obtain corrected spectral data. When spectral data are collected under different conditions, problems such as wavelength shift and intensity change may occur, which will affect the consistency and comparability of the data. The cubic spline interpolation method is a smooth interpolation method that approximates the original data by constructing a continuous and smooth polynomial function, thereby realizing nonlinear interpolation and correction of the data. In the detection of air pollutants, if the spectral data of different samples have a slight shift in wavelength, or the intensity is inconsistent due to changes in the intensity of the light source, it can be corrected by the cubic spline interpolation method to make all data comparable within the same wavelength range.Next, the corrected spectral data is subjected to standard normal transformation and multiplicative scatter correction to obtain preprocessed spectral data. Standard Normal Variate (SNV) is a commonly used spectral preprocessing method. It eliminates the baseline drift between spectra by subtracting the mean value of the spectrum from each spectral data point and dividing it by the standard deviation so that all spectral data have the same mean value and standard deviation. Multiplicative Scatter Correction (MSC) corrects the scattering effect between spectra by dividing each spectral data point by a reference spectrum and then multiplying it by the mean value of the reference spectrum. The combination of these two methods can effectively eliminate the baseline drift and scattering effect in the spectral data and improve the consistency and reliability of the data. In the detection of air pollutants, if the spectral data of different samples have intensity differences due to the scattering effect, the SNV and MSC processing can make these data more consistent, which is convenient for subsequent analysis and modeling. Finally, the preprocessed spectral data is subjected to continuous wavelet transform and singular value decomposition to obtain decomposed spectral data. Continuous Wavelet Transform (CWT) is a multi-resolution analysis method that can decompose signals into different scales to extract features in different frequency ranges. Singular Value Decomposition (SVD) is a matrix decomposition technique that can decompose a matrix into the product of three matrices to extract the main components of the data. By combining CWT and SVD, important features in spectral data can be effectively extracted while removing redundant information. In air pollutant detection, CWT can be used to extract the characteristic wavelengths of different air pollutants, while SVD can further compress the data dimensions, retain the most important features, and provide high-quality input data for subsequent sparse coding and model training. The decomposed spectral data is cleaned to obtain cleaned spectral data. This final cleaning step is mainly to further check and correct problems that may be left over from the previous steps to ensure that the final spectral data is clean, accurate, and consistent. For example, you can check again whether there are outliers, or use visualization tools to check whether the distribution of the data is reasonable. In air pollutant detection, this step can help researchers confirm the quality of the data and ensure that subsequent analysis and modeling can be based on reliable data. In summary, the process of cleaning the spectral data to obtain cleaned spectral data is a comprehensive preprocessing process involving multiple steps.Through a series of methods such as adaptive median filtering and wavelet threshold denoising, outlier detection and removal, nonlinear interpolation and spectral correction of cubic spline interpolation, standard normal transformation and multivariate scattering correction, continuous wavelet transform and singular value decomposition, the quality of spectral data can be significantly improved, providing a solid foundation for subsequent sparse coding, model training and other steps. In the application of air pollutant detection in environmental monitoring, this process can not only improve the accuracy and reliability of detection, but also provide strong data support for environmental management and pollution control.

[0032] In a specific embodiment, sparse encoding is performed on the cleaned spectral data to obtain sparse representation spectral features, including: Embed the cleaned spectral data into a high-dimensional feature space to obtain a high-dimensional feature representation; Using a sparse autoencoder to sparsely encode the high-dimensional feature representation to obtain an initial sparse representation; Applying a sparse coding network to optimize the initial sparse representation to obtain an optimized sparse representation; Regularizing the optimized sparse representation to obtain a regularized sparse representation; Optimizing and solving the regularized sparse representation to obtain a global optimal sparse representation; Multi-scale integration and feature extraction are performed on the global optimal sparse representation to obtain sparse representation spectral features.

[0033] Specifically, the process of sparsely encoding the cleaned spectral data to obtain sparsely represented spectral features is a multi-level, multi-step feature extraction and optimization process, which aims to extract the most representative and discriminative features from high-dimensional and complex spectral data. This process is crucial to improving the efficiency and accuracy of subsequent model training. The following will introduce the specific implementation method of this process in detail, and take the detection of air pollutants in environmental monitoring as an example to illustrate its significance in practical applications. First, the cleaned spectral data is embedded in a high-dimensional feature space to obtain a high-dimensional feature representation. Spectral data is essentially a series of correspondences between wavelengths and intensities, usually expressed as a one-dimensional vector. However, in order to better capture the complex structure and potential associations in the data, these one-dimensional vectors need to be mapped to a higher-dimensional feature space. This step can be achieved by a variety of methods, such as nonlinear mapping using kernel functions (such as RBF kernels), or converting data into high-dimensional vectors through an embedding layer in deep learning. In air pollutant detection, the original spectral data may contain hundreds of wavelength points. By embedding it into a high-dimensional feature space, the interactions and complex relationships between different wavelength points can be captured, providing rich information for subsequent sparse coding. Next, a sparse autoencoder is used to sparsely encode the high-dimensional feature representation to obtain an initial sparse representation. A sparse autoencoder is a special neural network structure whose goal is to minimize the number of non-zero elements in the coding layer while maintaining the main features of the input data. In this way, the most critical features in the data can be extracted while removing redundant information. In air pollutant detection, a sparse autoencoder can learn which wavelength points or bands are most important for identifying specific types of air pollutants, thereby generating a sparse feature representation. This representation not only reduces the dimension of the data, but also retains the most important information. Subsequently, a sparse coding network is applied to optimize the initial sparse representation to obtain an optimized sparse representation. A sparse coding network is a neural network structure specifically used for sparse coding, which further optimizes the feature representation through multiple layers of nonlinear transformations and sparse constraints. In this process, the network will continuously adjust the weights and biases to make the output feature representation more sparse and more representative. In air pollutant detection, the noise and redundant information in the initial sparse representation can be further removed through the optimization processing of the sparse coding network, thereby improving the purity and discrimination of the features. Next, the optimized sparse representation is subjected to regularization constraints to obtain a regularized sparse representation. Regularization is a technique to prevent overfitting. By adding regularization terms to the loss function, the complexity of the model is limited to make it more generalized. In sparse coding, commonly used regularization methods include L1 regularization and L2 regularization. L1 regularization encourages the generation of sparse feature representations by penalizing the absolute value of the weight; L2 regularization makes the weights smoother by penalizing the square of the weight.In air pollutant detection, regularization constraints can ensure that the sparse representation is not only sparse, but also has good generalization ability and will not overfit the training data. Then, the regularized sparse representation is optimized and solved to obtain the global optimal sparse representation. Optimization solution is to find the model parameters that minimize the loss function through numerical optimization methods such as gradient descent, Adam, etc. In sparse coding, the goal of optimization solution is to find a feature representation that can not only maintain the main features of the input data but also meet the sparsity and regularization constraints. In air pollutant detection, optimization solution can ensure that the generated sparse representation is not only sparse and pure, but also maintains good performance on different types of spectral data. Finally, multi-scale integration and feature extraction are performed on the global optimal sparse representation to obtain sparse representation spectral features. Multi-scale integration refers to extracting features at different scales and fusing these features to capture multi-level information in the data. Feature extraction is to extract the most representative features from the optimized sparse representation through a specific algorithm or model. In air pollutant detection, through multi-scale integration and feature extraction, features in different wavelength ranges can be captured, thereby more comprehensively characterizing different types of air pollutants. For example, some air pollutants may have obvious absorption peaks in the ultraviolet region, while others have characteristic peaks in the infrared region. Through multi-scale integration, these features can be captured at the same time, improving the accuracy and robustness of recognition. In summary, the process of sparsely encoding the cleaned spectral data to obtain sparsely represented spectral features is a complex process involving multiple steps. By embedding spectral data into a high-dimensional feature space, using sparse autoencoders for sparse encoding, applying sparse coding networks for optimization processing, regularization constraints, optimization solutions, and multi-scale integration and feature extraction, the most representative and discriminative features can be effectively extracted from high-dimensional and complex spectral data. This process not only improves the efficiency of data processing, but also provides high-quality input data for subsequent model training and application. In the application of air pollutant detection in environmental monitoring, this process can significantly improve the accuracy and reliability of detection, and provide strong data support for environmental management and pollution control.

[0034] In a specific embodiment, a preset convolutional neural network is trained by using the sparse representation spectral features to obtain an initial spectral recognition model, including: Performing data enhancement processing on the sparsely represented spectral features to obtain enhanced spectral feature data; wherein the data enhancement processing includes random noise addition, spectrum shift, spectrum stretching and compression transformation; Inputting the enhanced spectral feature data into a preset convolutional neural network, wherein the convolutional neural network includes a plurality of convolutional layers, a pooling layer and a fully connected layer; Performing multi-scale feature extraction on the enhanced spectral feature data through the convolution layers of different sizes and numbers to obtain a multi-scale convolution feature map; Performing a maximum pooling or an average pooling operation on the multi-scale convolutional feature map through the pooling layer to obtain a reduced-dimensional feature representation; Performing full connection processing on the dimension reduction feature representation through the full connection layer to obtain fully connected spectral feature data; The fully connected spectral feature data of the output of the fully connected layer is normalized using a softmax function to obtain spectral prediction values ​​of each category; wherein the spectra of each category include organic spectra, inorganic spectra, metal spectra, non-metal spectra, gas spectra and liquid spectra; Obtaining preset actual labels corresponding to the spectra of each category; wherein the actual labels include target values ​​corresponding to the spectra of each category; calculating a loss value between the target value and the predicted value of the spectrum of each category; Calculating the gradient of each network parameter in the convolutional neural network based on the loss value pair through a preset back propagation algorithm; Using a preset Adam optimizer, the weight parameters of the convolutional neural network are updated based on the gradients of the network parameters; The convolutional neural network is updated based on the updated weight parameters of the convolutional neural network to obtain an initial spectrum recognition model.

[0035] Specifically, the process of training the preset convolutional neural network model through the sparsely represented spectral features to obtain the initial spectral recognition model is a complex and sophisticated machine learning process, which aims to extract the most discriminative features from the sparsely encoded spectral features and perform efficient classification through the convolutional neural network (CNN). This process is crucial to improving the recognition accuracy and generalization ability of spectral data. The specific implementation method of this process will be introduced in detail below, and its significance in practical applications will be explained by taking the detection of heavy metal ions in environmental monitoring as an example. First, the sparsely represented spectral features are subjected to data enhancement processing to obtain enhanced spectral feature data. Data enhancement is a technology that improves the generalization ability of the model by increasing the diversity and quantity of training data. In spectral data, common data enhancement methods include random noise addition, spectrum shift, spectrum stretching and compression transformation, etc. Random noise addition can add a certain amount of random noise to the spectral data to simulate the noise interference that may exist in the actual acquisition process; spectrum shift can move the entire spectrum along the wavelength axis by a certain distance to simulate the spectral changes under different conditions; spectrum stretching and compression transformation can change the wavelength interval of the spectrum to simulate the spectral differences under different instruments or environmental conditions. In the detection of heavy metal ions, more training samples can be generated through these data enhancement methods, making the model more robust when facing actual data. Then, the enhanced spectral feature data is input into the preset convolutional neural network. Convolutional neural network is a deep learning model specially used to process grid-structured data (such as images, spectra, etc.). It gradually extracts high-level features of data through multiple convolutional layers, pooling layers and fully connected layers. In the detection of heavy metal ions, the preset convolutional neural network can include multiple convolutional layers, each of which slides on the input data through the convolution kernel to extract local features; the pooling layer reduces the size of the feature map and reduces the computational complexity through the maximum pooling or average pooling operation; the fully connected layer integrates all the extracted features together and outputs the final classification result. The enhanced spectral feature data is subjected to multi-scale feature extraction through the convolutional layers of different sizes and numbers to obtain a multi-scale convolutional feature map. Multi-scale feature extraction refers to extracting features of different scales from input data through convolution kernels of different sizes. In heavy metal ion detection, different types of heavy metal ions may have characteristic peaks in different wavelength ranges. Through multi-scale feature extraction, these features of different scales can be captured to improve the recognition ability of the model. The multi-scale convolution feature map is subjected to maximum pooling or average pooling operation through the pooling layer to obtain a reduced-dimensional feature representation. The pooling operation slides a small window on the feature map and takes the maximum value or average value in the window, thereby reducing the size of the feature map and reducing the computational complexity while retaining the most important features. In heavy metal ion detection, the pooling operation can effectively reduce the dimension of the feature map and improve the computational efficiency of the model.The fully connected layer performs full connection processing on the dimension reduction feature representation to obtain fully connected spectral feature data. The fully connected layer integrates all the extracted features together and generates a high-dimensional feature vector through the fully connected operation. In heavy metal ion detection, the fully connected layer can fuse the features extracted by different convolutional layers to generate a comprehensive feature representation, which provides a basis for the final classification. The fully connected spectral feature data output by the fully connected layer is normalized using the softmax function to obtain the spectral prediction values ​​of each category. The softmax function can convert the output of the fully connected layer into a probability distribution, and the prediction value of each category represents the probability of the category in all categories. In heavy metal ion detection, the probability distribution of different types of heavy metal ions in the sample can be obtained through the softmax function, thereby achieving classification. The actual labels corresponding to the spectra of each category set in advance are obtained. The actual label is the real category of each training sample, which is used to supervise the training of the model. In heavy metal ion detection, the actual label can be different types of heavy metal ions, such as lead, mercury, cadmium, etc. The loss value between the target value and the predicted value of the spectrum of each category is calculated. The loss value is used to measure the difference between the predicted value of the model and the actual label. Common loss functions include cross entropy loss, mean square error loss, etc. In heavy metal ion detection, the performance of the model during the training process can be evaluated by calculating the loss value. Through the preset back propagation algorithm, the gradient of each network parameter in the convolutional neural network is calculated based on the loss value pair. The back propagation algorithm is a method for calculating gradients through the chain rule, which is used to update the parameters in the network. In heavy metal ion detection, the gradient of each network parameter can be calculated through the back propagation algorithm to provide a basis for parameter updating. Through the preset Adam optimizer, the weight parameters of the convolutional neural network are updated based on the gradients of the network parameters. The Adam optimizer is an optimization algorithm for adaptive learning rate, which accelerates the convergence of the model by maintaining the independent learning rate of each parameter. In heavy metal ion detection, the Adam optimizer can efficiently update the network parameters and improve the training speed and performance of the model. The convolutional neural network is updated based on the weight parameters of the updated convolutional neural network to obtain an initial spectral recognition model. In each iteration, the performance of the model is gradually optimized by updating the network parameters. In heavy metal ion detection, through multiple iterative training, an initial spectral recognition model that can accurately identify different types of heavy metal ions can be finally obtained. In summary, the process of training the preset convolutional neural network model through the sparse representation spectral features to obtain the initial spectral recognition model is a complex process involving multiple steps such as data enhancement, multi-scale feature extraction, pooling operation, full connection processing, loss calculation, back propagation and parameter update. This process can not only extract the most discriminative features from high-dimensional and complex spectral data, but also achieve efficient classification through convolutional neural networks.In the application of heavy metal ion detection in environmental monitoring, this process can significantly improve the accuracy and reliability of detection, and provide strong data support for environmental management and pollution control.

[0036] In a specific embodiment, the initial spectrum recognition model is subjected to transfer learning training based on the target spectrum data to obtain a transfer spectrum recognition model, including: Selectively freezing the pre-trained weights of the initial spectral recognition model to obtain a partially frozen weight model; wherein the selective freezing includes freezing the weight parameters of all layers except the last layer of the initial spectral recognition model to retain the basic features learned by the initial spectral recognition model, and the last layer is a fully connected layer; Obtaining a target fully connected layer, and replacing the last fully connected layer in the partially frozen weight model with the target fully connected layer to obtain a model adapted to the new task; Using a mini-batch gradient descent method, fine-tuning the target fully connected layer newly added to the model adapted to the new task based on the target spectral data to obtain a fine-tuned target fully connected layer; Adaptively adjusting the learning rate of the fine-tuning target fully connected layer by using a cosine annealing learning rate strategy to obtain an adjusted target fully connected layer; Gradually unfreeze some convolutional layers, and jointly optimize the unfrozen parts of the convolutional layers and the adjusted target fully connected layers through an adaptive moment estimation optimizer to obtain a jointly optimized layer; The joint optimization layer is subjected to integrated parameter learning and parameter fusion processing to obtain a migration spectrum recognition model.

[0037] Specifically, the process of performing transfer learning training on the initial spectrum recognition model based on the target spectrum data to obtain the transfer spectrum recognition model is a complex process of gradually optimizing and adapting to new tasks. This process aims to utilize the basic features learned by the initial spectrum recognition model on the existing data, and through targeted adjustment and optimization, enable it to adapt to the new target spectrum data, thereby improving the performance of the model in the new task. The specific implementation method of this process will be described in detail below, and its significance in practical applications will be explained by taking the detection of heavy metal ions in environmental monitoring as an example. First, the pre-trained weights of the initial spectrum recognition model are selectively frozen to obtain a partially frozen weight model. Selective freezing refers to freezing only some layers of the model, rather than all layers, in transfer learning. Specifically, the weight parameters of all layers except the last layer of the initial spectrum recognition model are frozen. The purpose of this approach is to retain the basic features learned by the initial spectrum recognition model on the existing data, and these basic features are usually universal to different tasks. For example, in heavy metal ion detection, the initial spectrum recognition model may have been trained on organic spectrum data or air pollutants. Through selective freezing, these basic features can be retained so that they can be further optimized on new heavy metal ion spectrum data. Next, a target fully connected layer is obtained, and the target fully connected layer replaces the last fully connected layer in the partially frozen weight model to obtain a model adapted to the new task. The fully connected layer is the last layer of the convolutional neural network, which is responsible for integrating the extracted features into the final classification result. In transfer learning, it is usually necessary to redesign the last fully connected layer according to the requirements of the new task. For example, in heavy metal ion detection, if the goal is to identify new types of heavy metal ions, a new fully connected layer can be designed so that its output dimension matches the number of categories of the new task. By replacing the fully connected layer, the model can better adapt to the classification requirements of the new task. Then, the target fully connected layer newly added to the model adapted to the new task is fine-tuned based on the target spectral data using the mini-batch gradient descent method to obtain a fine-tuned target fully connected layer. The mini-batch gradient descent method is a commonly used optimization algorithm that gradually optimizes the model by calculating the gradient on each mini-batch data and updating the model parameters. In the fine-tuning training process, the newly added target fully connected layer is mainly focused on because this part of the layer needs to learn the specific features of the new task. Through the mini-batch gradient descent method, the weights of the target fully connected layer can be quickly adjusted to better adapt to the new task. Next, the cosine annealing learning rate strategy is used to adaptively adjust the learning rate of the fine-tuned target fully connected layer to obtain the adjusted target fully connected layer. The cosine annealing learning rate strategy is a method for dynamically adjusting the learning rate. By periodically changing the learning rate, it helps the model converge faster during training and avoid falling into local minima.In heavy metal ion detection, the weight of the target fully connected layer can be optimized more effectively and the performance of the model can be improved by using the cosine annealing learning rate strategy. Subsequently, some convolutional layers are gradually unfrozen, and the unfrozen parts of the convolutional layers and the adjusted target fully connected layers are jointly optimized by the adaptive moment estimation optimizer to obtain the joint optimization layer. In transfer learning, in addition to fine-tuning the last layer, some convolutional layers can also be gradually unfrozen so that these layers can also adapt to new tasks. Adaptive moment estimation optimizer (Adam) is a commonly used optimization algorithm that accelerates the convergence of the model by adaptively adjusting the learning rate of each parameter. In heavy metal ion detection, the performance of the model in the new task can be further improved by gradually unfreezing some convolutional layers and using the Adam optimizer for joint optimization. Finally, the joint optimization layer is subjected to integrated parameter learning and parameter fusion processing to obtain a migration spectrum recognition model. Integrated parameter learning refers to further improving the performance of the model by combining multiple optimization methods or multiple models. Parameter fusion processing is to fuse the parameters or prediction results of multiple models in some way to generate the final model. In heavy metal ion detection, by integrating parameter learning and parameter fusion processing, the advantages of different optimization methods can be fully utilized to generate a more robust and accurate migration spectrum recognition model. In summary, the process of performing transfer learning training on the initial spectrum recognition model based on the target spectrum data to obtain the migration spectrum recognition model is a complex process of gradual optimization and adaptation to new tasks. Through selective freezing processing, replacement of fully connected layers, fine-tuning training, adaptive learning rate adjustment, gradual unfreezing of convolutional layers and joint optimization, as well as integrated parameter learning and parameter fusion processing, the initial spectrum recognition model can be effectively adapted to the new target spectrum data, improving the performance of the model in new tasks. In the application of heavy metal ion detection in environmental monitoring, this process can significantly improve the accuracy and robustness of detection, and provide strong data support for environmental management and pollution control.

[0038] In a specific embodiment, the target spectrum data includes a specified training data set, and the cross-validation of the migration spectrum recognition model to obtain a validation value corresponding to the migration spectrum recognition model includes: Step S61, dividing the designated training data set of the migration spectrum recognition model into k mutually exclusive subsets; Step S62, dividing the subset to obtain a validation set and k-1 training sets; wherein the validation set is used to evaluate the performance of the migration spectrum recognition model, and the training set is used to train the migration spectrum recognition model; Step S63, fine-tuning the migration spectrum recognition model using a training set to obtain a fine-tuned migration spectrum recognition model; Step S64, using the verification set to evaluate the fine-tuned migration spectrum recognition model to obtain a corresponding evaluation value; Step S65, obtaining a target evaluation value corresponding to the migration spectrum recognition model, and obtaining a first verification value based on the target evaluation value and the evaluation value; Step S66, repeat the above steps S62 to S65 k times, each time selecting a different subset as the verification set, to obtain k verification values; In step S67, an average value of k verification values ​​is calculated, and the average value is used as the verification value corresponding to the migration spectrum recognition model.

[0039] Specifically, the following is a detailed explanation of this process, combined with the application scenario of heavy metal ion detection in environmental monitoring.

[0040] The target spectral data includes a designated training data set, and the process of cross-validating the migration spectrum recognition model to obtain the verification value corresponding to the migration spectrum recognition model is a systematic, multi-step evaluation process, which aims to ensure the generalization ability and stability of the model on unseen data, especially the performance in more subtle recognition tasks. The specific implementation method of this process will be described in detail below, and its significance in practical applications will be explained by taking the detection of heavy metal ions in environmental monitoring as an example. First, in step S61, the designated training data set of the migration spectrum recognition model is divided into k mutually exclusive subsets. The "designated training data set" here is a subset of the "target spectral data" for more detailed verification of the performance of the model. For example, in heavy metal ion detection, the target spectral data may contain spectral data of multiple heavy metal ions, and the designated training data set may be spectral data of a specific heavy metal ion (such as lead). Dividing this designated training data set into k mutually exclusive subsets can ensure that the data in each subset is independent, thereby reducing the randomness of data division. For example, if the specified training data set contains 1000 spectral samples of lead, k=10 can be selected to divide the data set into 10 subsets, each containing 100 samples. Then, in step S62, the subsets are divided to obtain a validation set and k-1 training sets. In each cross-validation process, one subset is selected as the validation set, and the remaining k-1 subsets are used as training sets. The validation set is used to evaluate the performance of the model, while the training set is used to train the model. For example, in the first cross-validation, the first subset can be selected as the validation set, and the remaining 9 subsets can be selected as training sets. In this way, it can be ensured that each subset has the opportunity to be used as a validation set, thereby more comprehensively evaluating the performance of the model. Then, in step S63, the migration spectrum recognition model is fine-tuned by the training set to obtain a fine-tuned migration spectrum recognition model. In each cross-validation, the migration spectrum recognition model is fine-tuned using the training set to adapt to the current training data. Fine-tuning can be performed by mini-batch gradient descent or other optimization algorithms, with the focus on adjusting the last layer of the model (usually the fully connected layer) to better adapt it to the current training data. For example, in the detection of heavy metal ions, fine-tuning the model through the training set can optimize the performance of the model on the current data, especially the recognition ability of specific heavy metal ions (such as lead). Next, in step S64, the fine-tuned migration spectrum recognition model is evaluated using the validation set to obtain a corresponding evaluation value. The evaluation value can be a performance indicator such as accuracy, recall, F1 score, etc. By evaluating the performance of the model through the validation set, we can understand the performance of the model on unseen data, especially in subtle recognition tasks. For example, in the detection of heavy metal ions, the accuracy of the model is evaluated through the validation set, and the recognition ability of the model for specific heavy metal ions (such as lead) on new data can be judged.Subsequently, step S65 obtains the target evaluation value corresponding to the migration spectrum recognition model, and calculates based on the target evaluation value and the evaluation value to obtain a first verification value. The target evaluation value is a pre-set benchmark value used to evaluate whether the performance of the model meets expectations. By comparing the evaluation value on the validation set with the target evaluation value, the first verification value can be calculated. For example, if the target evaluation value is 90% accuracy, and the evaluation value on the validation set is 88%, the first verification value can be 88%. Then, step S66 repeats the above steps S62-S65 k times, each time selecting a different subset as the validation set, and obtaining k verification values. By repeating the above steps k times, each time selecting a different subset as the validation set, k verification values ​​can be obtained. These k verification values ​​reflect the performance of the model under different data partitions, especially in subtle recognition tasks. For example, in a cross-validation of k=10, 10 verification values ​​can be obtained through 10 verifications, which can help evaluate the stability and consistency of the model on different data. Finally, step S67 calculates the average value of k validation values, and uses the average value as the validation value corresponding to the migration spectrum recognition model. By calculating the average value of k validation values, a comprehensive performance index can be obtained to evaluate the overall performance of the model, especially in subtle recognition tasks. For example, if the 10 validation values ​​are 88%, 89%, 90%, 91%, 90%, 89%, 88%, 90%, 91%, and 92%, respectively, the average value is 90%, and this 90% is the validation value corresponding to the migration spectrum recognition model. In summary, the process of cross-validating the migration spectrum recognition model to obtain the validation value corresponding to the migration spectrum recognition model is a systematic, multi-step evaluation process. By dividing the specified training data set into k mutually exclusive subsets, dividing the subsets, fine-tuning the model through the training set, evaluating the model using the validation set, calculating the validation value, repeating the validation process, and calculating the average validation value, the performance of the model on unseen data can be comprehensively evaluated, especially in more subtle recognition tasks. In the application of heavy metal ion detection in environmental monitoring, this process can ensure the reliability and stability of the model in practical applications, and provide strong data support for environmental management and pollution control.

[0041] In a specific embodiment, the hyperparameters of the migration spectrum recognition model are tuned by a preset Bayesian optimization algorithm until the verification value of the migration spectrum recognition model is within a preset range to obtain a target spectrum recognition model, including: The hyperparameter space of the migration spectrum recognition model is modeled by Gaussian process regression to obtain the hyperparameter prior distribution; Performing Monte Carlo sampling on the hyperparameter prior distribution to obtain a set of candidate hyperparameters; Performing parameter evaluation on the candidate hyperparameter set to obtain an optimal hyperparameter combination; Based on the optimal hyperparameter combination, reconfiguring the parameters of the migration spectrum recognition model to obtain an optimized migration spectrum recognition model; Refining the hyperparameters of the optimized migration spectrum recognition model by using a tree-shaped Parzen estimator to obtain a fine-tuned hyperparameter set; The migration spectrum recognition model is iteratively tuned based on the fine-tuning hyperparameter set by using a preset Bayesian optimization algorithm until the verification value of the migration spectrum recognition model is within a preset range, so as to obtain a target spectrum recognition model.

[0042] Specifically, the process of tuning the hyperparameters of the migration spectrum recognition model by a preset Bayesian optimization algorithm until the verification value of the migration spectrum recognition model is within a preset range to obtain the target spectrum recognition model is a systematic and iterative optimization process, which aims to find the optimal hyperparameter combination by intelligently searching the hyperparameter space, thereby significantly improving the performance of the model. The specific implementation method of this process will be introduced in detail below, and its significance in practical applications will be explained by taking the detection of specific heavy metal ions (lead or iron) in environmental monitoring as an example. First, the hyperparameter space of the migration spectrum recognition model is modeled by Gaussian process regression to obtain a hyperparameter prior distribution. Gaussian process regression is a non-parametric regression method that can model the hyperparameter space and generate a probability distribution, namely, a hyperparameter prior distribution. This distribution describes the possibility of different combinations of hyperparameters in model performance. In the detection of specific heavy metal ions, hyperparameters may include learning rate, batch size, number of convolution kernels, etc. Through Gaussian process regression, the relationship between these hyperparameters can be established, providing a basis for subsequent optimization. Next, Monte Carlo sampling is performed on the hyperparameter prior distribution to obtain a candidate hyperparameter set. Monte Carlo sampling is a random sampling method that generates a candidate hyperparameter set by randomly extracting samples from the hyperparameter prior distribution. Each combination in these candidate hyperparameter sets represents a possible hyperparameter configuration. In the detection of specific heavy metal ions, multiple different hyperparameter combinations can be generated through Monte Carlo sampling, providing diverse choices for subsequent evaluation. Then, parameter evaluation is performed on the candidate hyperparameter set to obtain the optimal hyperparameter combination. Parameter evaluation refers to applying each candidate hyperparameter combination to the migration spectrum recognition model, evaluating the performance of the model through the validation set, and thus selecting the optimal hyperparameter combination. The evaluation indicators can be accuracy, recall rate, F1 score, etc. In the detection of specific heavy metal ions, a hyperparameter combination that optimizes the model performance can be found by applying each candidate hyperparameter combination to the model and evaluating its performance on the validation set. Next, based on the optimal hyperparameter combination, the migration spectrum recognition model is reconfigured to obtain an optimized migration spectrum recognition model. Re-parameter configuration refers to applying the optimal hyperparameter combination to the model, retraining and validating the model to ensure the performance of the model under the optimal hyperparameter configuration. In the detection of specific heavy metal ions, by reconfiguring the hyperparameters of the model, the performance of the model on the validation set can be significantly improved, providing a basis for further optimization. Subsequently, the hyperparameters of the optimized migration spectrum recognition model are refined by the tree-shaped Parzen estimator to obtain a fine-tuned hyperparameter set. The tree-shaped Parzen estimator is an optimization method based on probability density estimation. By constructing a probability model, it gradually approaches the optimal hyperparameter combination.In the detection of specific heavy metal ions, the optimal hyperparameter combination can be further refined through the tree-shaped Parzen estimator to generate a more accurate set of fine-tuned hyperparameters. Finally, the migration spectrum recognition model is iteratively tuned based on the fine-tuned hyperparameter set through the preset Bayesian optimization algorithm until the verification value of the migration spectrum recognition model is within the preset range to obtain the target spectrum recognition model. The Bayesian optimization algorithm is an iterative optimization method that gradually approaches the optimal solution by continuously updating the probability distribution of hyperparameters. In each iteration, the algorithm selects the hyperparameter combination that is most likely to improve the performance of the model for evaluation, and updates the probability distribution of the hyperparameters based on the evaluation results. This process continues until the verification value of the model reaches the preset range. In the detection of specific heavy metal ions, the Bayesian optimization algorithm can be used to gradually optimize the hyperparameters so that the performance of the model on the verification set meets the expected standards, thereby obtaining the final target spectrum recognition model. In summary, the process of tuning the hyperparameters of the migration spectrum recognition model by a preset Bayesian optimization algorithm until the verification value of the migration spectrum recognition model is within a preset range to obtain the target spectrum recognition model is a systematic and iterative optimization process. Through Gaussian process regression modeling, Monte Carlo sampling, parameter evaluation, re-parameter configuration, hyperparameter refinement and Bayesian optimization, the optimal hyperparameter combination can be effectively found to significantly improve the performance of the model. In the application of specific heavy metal ion detection in environmental monitoring, this process can ensure the high accuracy and high stability of the model in practical applications, and provide strong data support for environmental management and pollution control.

[0043] The above describes the training method of the spectrum recognition model in the embodiment of the present invention. The following describes the training device of the spectrum recognition model in the embodiment of the present invention. Figure 2 , an embodiment of a training device for a spectrum recognition model in an embodiment of the present invention includes: An acquisition module 21 is used to acquire spectral data in a spectral database and clean the spectral data to obtain cleaned spectral data; An encoding module 22, used for performing sparse encoding on the cleaned spectral data to obtain sparse representation spectral features; A training module 23 is used to perform model training on a preset convolutional neural network through the sparse representation spectral features to obtain an initial spectral recognition model; The learning module 24 is configured to, upon receiving target spectrum data, perform transfer learning training on the initial spectrum recognition model based on the target spectrum data to obtain a transfer spectrum recognition model if the target spectrum data is of a different data type from the spectrum data in the spectrum database; A verification module 25 is used to cross-validate the migration spectrum recognition model to obtain a verification value corresponding to the migration spectrum recognition model; wherein the verification value is used to evaluate the performance of the migration spectrum recognition model; The tuning module 26 is used to tune the hyperparameters of the migration spectrum recognition model by a preset Bayesian optimization algorithm if the verification value is not within a preset range, until the verification value of the migration spectrum recognition model is within a preset range, so as to obtain a target spectrum recognition model.

[0044] In this embodiment, for the specific implementation of each unit in the above system embodiment, please refer to the above method embodiment, which will not be repeated here.

[0045] Reference Figure 3 The present invention also provides a computer device in an embodiment, wherein the internal structure of the computer device can be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0046] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0047] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0048] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0049] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0050] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for training a spectral recognition model, characterized in that: The following steps are involved: Acquiring spectral data in a spectral database, and cleaning the spectral data to obtain cleaned spectral data; Performing sparse coding on the cleaned spectral data to obtain sparse representation spectral features; The preset convolutional neural network is trained by using the sparse representation spectral features to obtain an initial spectral recognition model; When receiving target spectrum data, if the target spectrum data is of a different data type from the spectrum data in the spectrum database, performing transfer learning training on the initial spectrum recognition model based on the target spectrum data to obtain a transfer spectrum recognition model; Cross-validating the migration spectrum recognition model to obtain a validation value corresponding to the migration spectrum recognition model; wherein the validation value is used to evaluate the performance of the migration spectrum recognition model; If the verification value is not within the preset range, the hyperparameters of the migration spectrum recognition model are tuned by a preset Bayesian optimization algorithm until the verification value of the migration spectrum recognition model is within the preset range to obtain a target spectrum recognition model.

2. The training method of the spectrum recognition model according to claim 1, characterized in that: Cleaning the spectral data to obtain cleaned spectral data includes: Performing adaptive median filtering and wavelet threshold denoising on the spectral data to obtain denoised spectral data; Performing outlier detection and elimination on the denoised spectral data to obtain outlier-eliminated spectral data; Using a cubic spline interpolation method, nonlinear interpolation and spectrum correction are performed on the outlier-removed spectrum data to obtain corrected spectrum data; Performing standard normal transformation and multivariate scattering correction on the corrected spectral data to obtain preprocessed spectral data; Performing continuous wavelet transform and singular value decomposition on the preprocessed spectral data to obtain decomposed spectral data; The decomposed spectral data is cleaned to obtain cleaned spectral data.

3. The training method of the spectrum recognition model according to claim 1, characterized in that: The cleaned spectral data is sparsely encoded to obtain sparse representation spectral features, including: Embed the cleaned spectral data into a high-dimensional feature space to obtain a high-dimensional feature representation; Using a sparse autoencoder to sparsely encode the high-dimensional feature representation to obtain an initial sparse representation; Applying a sparse coding network to optimize the initial sparse representation to obtain an optimized sparse representation; Regularizing the optimized sparse representation to obtain a regularized sparse representation; Optimizing and solving the regularized sparse representation to obtain a global optimal sparse representation; Multi-scale integration and feature extraction are performed on the global optimal sparse representation to obtain sparse representation spectral features.

4. The method for training a spectrum recognition model according to claim 1, characterized in that: The preset convolutional neural network is trained by the sparse representation spectral feature to obtain an initial spectral recognition model, including: Performing data enhancement processing on the sparsely represented spectral features to obtain enhanced spectral feature data; wherein the data enhancement processing includes random noise addition, spectrum shift, spectrum stretching and compression transformation; Inputting the enhanced spectral feature data into a preset convolutional neural network, wherein the convolutional neural network includes a plurality of convolutional layers, a pooling layer and a fully connected layer; Performing multi-scale feature extraction on the enhanced spectral feature data through the convolution layers of different sizes and numbers to obtain a multi-scale convolution feature map; Performing a maximum pooling or an average pooling operation on the multi-scale convolutional feature map through the pooling layer to obtain a reduced-dimensional feature representation; Performing full connection processing on the dimension reduction feature representation through the full connection layer to obtain fully connected spectral feature data; The fully connected spectral feature data of the output of the fully connected layer is normalized using a softmax function to obtain spectral prediction values ​​of each category; wherein the spectra of each category include organic spectra, inorganic spectra, metal spectra, non-metal spectra, gas spectra and liquid spectra; Obtaining preset actual labels corresponding to the spectra of each category; wherein the actual labels include target values ​​corresponding to the spectra of each category; calculating a loss value between the target value and the predicted value of the spectrum of each category; Calculating the gradient of each network parameter in the convolutional neural network based on the loss value pair through a preset back propagation algorithm; Using a preset Adam optimizer, the weight parameters of the convolutional neural network are updated based on the gradients of the network parameters; The convolutional neural network is updated based on the updated weight parameters of the convolutional neural network to obtain an initial spectrum recognition model.

5. The method for training a spectrum recognition model according to claim 1, characterized in that: The performing transfer learning training on the initial spectrum recognition model based on the target spectrum data to obtain a transfer spectrum recognition model includes: The pre-trained weights of the initial spectral recognition model are selectively frozen to obtain a partially frozen weight model; wherein the selective freezing process includes freezing the weight parameters of all layers except the last layer of the initial spectral recognition model to retain the basic features learned by the initial spectral recognition model, and the last layer is a fully connected layer; Obtaining a target fully connected layer, and replacing the last fully connected layer in the partially frozen weight model with the target fully connected layer to obtain a model adapted to the new task; Using a mini-batch gradient descent method, fine-tuning the target fully connected layer newly added to the model adapted to the new task based on the target spectral data to obtain a fine-tuned target fully connected layer; Adaptively adjusting the learning rate of the fine-tuning target fully connected layer by using a cosine annealing learning rate strategy to obtain an adjusted target fully connected layer; Gradually unfreeze some convolutional layers, and jointly optimize the unfrozen parts of the convolutional layers and the adjusted target fully connected layers through an adaptive moment estimation optimizer to obtain a jointly optimized layer; The joint optimization layer is subjected to integrated parameter learning and parameter fusion processing to obtain a migration spectrum recognition model.

6. The method for training a spectrum recognition model according to claim 1, characterized in that: The target spectrum data includes a specified training data set, and the cross-validation of the migration spectrum recognition model to obtain a validation value corresponding to the migration spectrum recognition model includes: Step S61, dividing the designated training data set of the migration spectrum recognition model into k mutually exclusive subsets; Step S62, dividing the subset to obtain a validation set and k-1 training sets; wherein the validation set is used to evaluate the performance of the migration spectrum recognition model, and the training set is used to train the migration spectrum recognition model; Step S63, fine-tuning the migration spectrum recognition model using a training set to obtain a fine-tuned migration spectrum recognition model; Step S64, using the verification set to evaluate the fine-tuned migration spectrum recognition model to obtain a corresponding evaluation value; Step S65, obtaining a target evaluation value corresponding to the migration spectrum recognition model, and obtaining a first verification value based on the target evaluation value and the evaluation value; Step S66, repeat the above steps S62 to S65 k times, each time selecting a different subset as the verification set, to obtain k verification values; In step S67, an average value of k verification values ​​is calculated, and the average value is used as the verification value corresponding to the migration spectrum recognition model.

7. The method for training a spectrum recognition model according to claim 1, characterized in that: The hyperparameters of the migration spectrum recognition model are tuned by a preset Bayesian optimization algorithm until the verification value of the migration spectrum recognition model is within a preset range to obtain a target spectrum recognition model, including: The hyperparameter space of the migration spectrum recognition model is modeled by Gaussian process regression to obtain the hyperparameter prior distribution. Performing Monte Carlo sampling on the hyperparameter prior distribution to obtain a set of candidate hyperparameters; Performing parameter evaluation on the candidate hyperparameter set to obtain an optimal hyperparameter combination; Based on the optimal hyperparameter combination, reconfiguring the parameters of the migration spectrum recognition model to obtain an optimized migration spectrum recognition model; Refining the hyperparameters of the optimized migration spectrum recognition model by using a tree-shaped Parzen estimator to obtain a fine-tuned hyperparameter set; The migration spectrum recognition model is iteratively tuned based on the fine-tuning hyperparameter set by using a preset Bayesian optimization algorithm until the verification value of the migration spectrum recognition model is within a preset range, so as to obtain a target spectrum recognition model.

8. A training device for a spectral recognition model, characterized in that: include: An acquisition module, used for acquiring spectral data in a spectral database, and cleaning the spectral data to obtain cleaned spectral data; An encoding module, used for performing sparse encoding on the cleaned spectral data to obtain sparse representation spectral features; A training module, used to perform model training on a preset convolutional neural network through the sparse representation spectral features to obtain an initial spectral recognition model; A learning module, configured to, upon receiving target spectral data, perform transfer learning training on the initial spectral recognition model based on the target spectral data to obtain a transfer spectral recognition model if the target spectral data is of a different data type from the spectral data in the spectral database; A verification module, used to cross-validate the migration spectrum recognition model to obtain a verification value corresponding to the migration spectrum recognition model; wherein the verification value is used to evaluate the performance of the migration spectrum recognition model; The tuning module is used to tune the hyperparameters of the migration spectrum recognition model by a preset Bayesian optimization algorithm if the verification value is not within a preset range, until the verification value of the migration spectrum recognition model is within a preset range, so as to obtain a target spectrum recognition model.

9. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images